Python Cookbook 2Nd Edition Jun 1002005 [Electronic resources] نسخه متنی

Recipe 8.2. Measuring Memory Usage on Linux

Credit: Jean Brouwers

Problem

You need to monitor how much memory your
Python application, running under Linux, is currently using. However,
the standard library module resource does not work
correctly on Linux.

Solution

We can code our own resource measurements based on
Linux's /proc
pseudo-filesystem:

import os
_proc_status = '/proc/%d/status' % os.getpid( )
_scale = {'kB': 1024.0, 'mB': 1024.0*1024.0,
'KB': 1024.0, 'MB': 1024.0*1024.0}
def _VmB(VmKey):
''' given a VmKey string, returns a number of bytes. '''
# get pseudo file  /proc/<pid>/status
try:
t = open(_proc_status)
v = t.read( )
t.close( )
except IOError:
return 0.0  # non-Linux?
# get VmKey line e.g. 'VmRSS:  9999  kB\n ...'
i = v.index(VmKey)
v = v[i:].split(None, 3)  # split on runs of whitespace
if len(v) < 3:
return 0.0  # invalid format?
# convert Vm value to bytes
return float(v[1]) * _scale[v[2]]
def memory(since=0.0):
''' Return virtual memory usage in bytes. '''
return _VmB('VmSize:') - since
def resident(since=0.0):
''' Return resident memory usage in bytes. '''
return _VmB('VmRSS:') - since
def stacksize(since=0.0):
''' Return stack size in bytes. '''
return _VmB('VmStk:') - since

Discussion

Each of the functions in this recipe takes an optional argument
since because the typical usage of these functions
is to find out how much more memory (virtual,
resident, or stack) has been used due to a certain section of code.
Having since as an optional argument makes this
typical usage quite simple and elegant:

    m0 = memory( )
section of code you're monitoring
m1 = memory(m0)
print 'The monitored section consumed', m1, 'bytes of virtual memory'.

Getting and parsing the contents of pseudo-file
/proc/pid/status is probably not the most
efficient way to get data about memory usage, and it is not portable
to non-Linux systems. However, it is a very
simple and easy-to-code approach, and after all, on a non-Linux Unix
system, you can use the
resource module from the Python Standard
Library.

In fact, you can use resource
on Linux, but the various fields relevant to memory consumption, such
as ru_maxrss, all have a constant value of 0, just
like the various memory-consumption fields in the output of the
time shell command under Linux. The root cause of
this situation is a limitation in the Linux implementation of the
geTRusage system call, documented in man
getrusage.

See Also

Documentation on the resource standard library
module in the Library Reference.