Python Cookbook 2Nd Edition Jun 1002005 [Electronic resources] نسخه متنی

Recipe 9.12. Capturing the Output and Error Streams from a Unix Shell Command

Credit: Brent Burley, Bradey Honsinger, Tobias Polzin,
Jonathan Cano, Padraig Brady

Problem

You
need to run an external process in a Unix-like environment and
capture both the output and error streams from that external
process.

Solution

The popen2 module lets you capture both streams,
but you also need help from module fcntl, to make
the streams nonblocking and thus avoid deadlocks, and from module
select, to orchestrate the action:

import os, popen2, fcntl, select
def makeNonBlocking(fd):
fl = fcntl.fcntl(fd, os.F_GETFL)
try:
fcntl.fcntl(fd, os.F_SETFL, fl | os.O_NDELAY)
except AttributeError:
fcntl.fcntl(fd, os.F_SETFL, fl | os.FNDELAY)
def getCommandOutput(command):
child = popen2.Popen3(command, 1)
 # Capture stdout and stderr from command
child.tochild.close( ) # don't need to write to child's stdin
outfile = child.fromchild
outfd = outfile.fileno( )
errfile = child.childerr
errfd = errfile.fileno( )
makeNonBlocking(outfd) # Don't deadlock! Make fd's nonblocking.
makeNonBlocking(errfd)
outdata, errdata = [  ], [  ]
outeof = erreof = False
while True:
to_check = [outfd]*(not outeof) + [errfd]*(not erreof)
ready = select.select(to_check, [  ], [  ]) # Wait for input
if outfd in ready[0]:
outchunk = outfile.read( )
if outchunk == '':
outeof = True
else:
outdata.append(outchunk)
if errfd in ready[0]:
errchunk = errfile.read( )
if errchunk == '':
erreof = True
else:
errdata.append(errchunk)
if outeof and erreof:
break
select.select([  ],[  ],[  ],.1) 
# Allow a little time for buffers to fill
err = child.wait( )
if err != 0:
raise RuntimeError, '%r failed with exit code %d\n%s' % (
command, err, ''.join(errdata))
return ''.join(outdata)
def getCommandOutput2(command):
child = os.popen(command)
data = child.read( )
err = child.close( )
if err:
raise RuntimeError, '%r failed with exit code %d' % (command, err)

Discussion

This recipe shows how to execute a Unix shell command and capture the
output and error streams in Python. By contrast,
os.system sends both streams directly to the
terminal. The function getCommandOutput presented in
this recipe executes a command and returns the
command's output. If the command fails,
getCommandOutput raises an exception, using the text
captured from the command's
stderr as part of the exception's
arguments.

Most of the complexity of this code is due to the difficulty of
capturing both the output and error streams of the child process
independently and at the same time. Normal (blocking) read calls may
deadlock if the child is trying to write to one stream, and the
parent is waiting for data on the other stream; so, the streams must
be set to nonblocking, and select must be used to
wait for data on either of the streams.

Note that the second select call is included just
to add a 0.1-second sleep after each read. Counter intuitively, this
allows the code to run much faster, since it gives the child time to
put more data in the buffer. Without it, the parent may try to read
only a few bytes at a time, which can be very expensive. Calling
time.sleep(0.1) should be exactly equivalent, but
since I was already, necessarily, calling
select.select elsewhere in the
recipe's code, I decided not to also import module
time needlessly.

If you want to capture only the output and don't
mind the error stream going to the terminal, you can use the much
simpler code presented in getCommandOutput2. If you
want to suppress the error stream altogether, that's
easy, toojust append 2>/dev/null to the
command. For example:

listing = getCommandOutput2('ls -1 2>/dev/null')

Another possibility is given by the os.popen4
function, which combines the output and error streams of the child
process. However, in that case the streams are combined in a
potentially messy way, depending on how they are buffered in the
child process, so this recipe can help.

In Python 2.4, you can use class Popen, instead of
popen2.Popen3, from the new standard library
module subprocess. However, the issues highlighted
in this recipe (namely, the need to use modules
fcntl and select to make files
nonblocking and coordinate the loop that interacts with the child
process) aren't really affected by whether you use
popen2 or subprocess.

This recipe does, as advertised, require a
rather Unix-like underlying platform. Cygwin, which does a generally
great job of emulating Unix on top of Windows, is not sufficient; for
example, it offers no way to set files to nonblocking mode, nor to
select on general files. (Under Windows, you are
allowed to select only on sockets, not on other
files.) If you must run on such problematic, non-Unix platforms, you
may prefer a very different approach, based on using temporary files:

import os, tempfile
def getCommandOutput(command):
outfile = tempfile.mktemp( )
errfile = tempfile.mktemp( )
cmd = "( %s ) > %s 2> %s" % (command, outfile, errfile)
err = os.system(cmd) >> 8
try:
if err != 0:
raise RuntimeError, '%r failed with exit code %d\n%s' % (
command, err, file(errfile).read( ))
return file(outfile).read( )
finally:
os.remove(outfile)
os.remove(errfile)

See Also

Documentation of the standard library
modules os, popen2,
fcntl, select, and
tempfile in the Library
Reference and Python in a
Nutshell; (Python 2.4 only) module
subprocess in the Library
Reference.