Optimizing Linux® Performance [Electronic resources] : A Hands-On Guide to Linux® Performance Tools نسخه متنی

3.2. Linux Performance Tools: CPU and Memory

Here begins our discussion of performance tools that enable you to extract the memory performance information described previously.

3.2.1. vmstat (Virtual Memory Statistics) II

As you have seen before, vmstat can provide information about many different performance aspects of a systemalthough its primary purpose, as shown next, is to provide information about virtual memory system performance. In addition to the CPU performance statistics described in the previous chapter, it can also tell you the following:

How much swap is being used

How the physical memory is being used

How much memory is free

As you can see, vmstat provides (via the statistics it displays) a wealth of information about the health and performance of the system in a single line of text.

3.2.1.1 System-Wide Memory-Related Options

In addition to the CPU statistics vmstat can provide, you can invoke vmstat with the following command-line options when investigating memory statistics:

vmstat [-a] [-s] [-m]

As before, you can run vmstat in two modes: sample mode and average mode. The added command-line options enable you to get the performance statistics about how the Linux kernel is using memory. Table 3-1 describes the options that vmstat accepts.

Table 3-1. vmstat Command-Line Options

Option

Explanation

Table 3-2 provides a list of the memory statistics that vmstat can provide. As with the CPU statistics, when run in normal mode, the first line that vmstat provides is the average values for all the rate statistics (so and si) and the instantaneous value for all the numeric statistics (swpd, free, buff, cache, active, and inactive).

Table 3-2. Memory-Specific vmstat Output Statistics

Column

Explanation

vmstat provides a good overview of the current state of the virtual memory system for a given machine. Although it does not provide a complete and detailed list of every Linux memory performance statistic available, it does provide a compact output that can indicate how the system memory is being used overall.

3.2.1.2 Example Usage

In Listing 3.2, as you saw in the previous chapter, if vmstat is invoked without any command-line options, it displays average values for performance statistics since system boot (si and so), and it shows the instantaneous values for the rest of them (swpd, free, buff, and cache). In this case, we can see that the system has about 500MB of memory that has been swapped to disk. ~14MB of the system memory is free. ~4MB is used for buffers that contain data that has yet to be flushed to disk. ~627MB is used for the disk cache that contains data that has been read off the disk in the past.

Listing 3.2.

bash-2.05b$ vmstat
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r  b   swpd   free   buff  cache   si   so     bi   bo   in    cs us sy id wa
0  0 511012  14840   4412 642072   33   31    204  247 1110  1548  8  5 73 14

In Listing 3.3, we ask vmstat to display information about the number of active and inactive pages. The amount of inactive pages indicates how much of the memory could be swapped to disk and how much is currently being used. In this case, we can see that 1310MB of memory is active, and only 78MB is considered inactive. This machine has a large amount of memory, and much of it is being actively used.

Listing 3.3.

bash-2.05b$ vmstat -a
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r  b   swpd   free  inact active   si   so    bi    bo   in    cs us sy id wa
2  1 514004   5640 79816 1341208   33   31   204   247 1111  1548  8  5 73 14

Next, in Listing 3.4, we look at a different system, one that is actively swapping data in and out of memory. The si column indicates that swap data has been read in at a rate of 480KB, 832KB, 764KB, 344KB, and 512KB during each of those sample periods. The so column indicates that memory data has been written to swap at a rate of 9KB, 0KB, 916KB, 0KB, 1068KB, 444KB, 792KB, during each of the samples. These results could indicate that the system does not have enough memory to handle all the running processes. A simultaneously high swap-in and swap-out rate can occur when a process's memory is being saved to make way for an application that had been previously swapped to disk. This can be disastrous if two running programs both need more memory than the system can provide. For example, if two processes are using a large amount of memory, and both are trying to run simultaneously, each can cause the other's memory to be written to swap. When one of the programs needs a piece of memory, it kicks out one that the other applications needs. When the other application starts to run, it kicks out a piece of memory that the original program was using, and waits until its memory is loaded from the swap. This can cause both applications to grind to a halt while they wait for their memory to be retrieved from swap before they can continue execution. As soon as one makes a little bit of progress, it swaps out memory that the other process was using and causes the other program to slow down. This is called thrashing. When this happens, the system spends most of its time reading memory to and from the swap, and system performance dramatically slows down.

In this particular case, the swapping eventually stopped, so most likely the memory that was swapped to disk was not immediately needed by the original process. This means the swap usage was effective, the contents of the memory that was not being used was written to disk, and the memory was then given to the process that needed it.

Listing 3.4.

[ezolt@localhost book]$ vmstat 1 100
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
2  1 131560   2320   8640  53036    1    9   107    69 1137   426 10  7 74  9
0  1 131560   2244   8640  53076  480    0   716     0 1048   207  6  1  0 93
1  2 132476   3424   8592  53272  832  916  1356   916 1259   692 11  4  0 85
1  0 132476   2400   8600  53280  764    0  1040    40 1288   762 14  5  0 81
0  1 133544   2656   8624  53392  344 1068  1096  1068 1217   436  8  3  5 84
0  1 133988   2300   8620  54288  512  444  1796   444 1090   230  5  1  2 92
0  0 134780   3148   8612  53688    0  792     0   792 1040   166  5  1 92  2
0  0 134780   3148   8612  53688    0    0     0     0 1050   158  4  1 95  0
0  0 134780   3148   8612  53688    0    0     0     0 1148   451  7  2 91  0
0  0 134780   3148   8620  53680    0    0     0    12 1196   477  8  2 78 12
....

As shown in Listing 3.5, as you saw in the previous chapter, vmstat can show a vast array of different system statistics. Now as we look at it, we can see some of the same statistics that were present in some of the other output modes, such as active, inactive, buffer, cache, and used swap. However, it also has a few new statistics, such as total memory, which indicates that this system has a total of 1516MB of memory, and total swap, which indicates that this system has a total of 2048MB of swap. It can be helpful to know the system totals when trying to figure out what percentage of the swap and memory is currently being used. Another interesting statistic is the pages paged in, which indicates the total number of pages that were read from the disk. This statistic includes the pages that are read starting an application and those that the application itself may be using.

Listing 3.5.

bash-2.05b$ vmstat -s
1552528  total memory
1546692  used memory
1410448  active memory
11100  inactive memory
5836  free memory
2676  buffer memory
645864  swap cache
2097096  total swap
526280  used swap
1570816  free swap
20293225 non-nice user cpu ticks
18284715 nice user cpu ticks
17687435 system cpu ticks
357314699 idle cpu ticks
67673539 IO-wait cpu ticks
352225 IRQ cpu ticks
4872449 softirq cpu ticks
495248623 pages paged in
600129070 pages paged out
19877382 pages swapped in
18874460 pages swapped out
2702803833 interrupts
3763550322 CPU context switches
1094067854 boot time
20158151 forks

Finally, in Listing 3.6, we see that vmstat can provide information about how the Linux kernel allocates its memory. As previously described, the Linux kernel has a series of "slabs" to hold its dynamic data structures. vmstat displays each of the slabs (Cache), shows how many of the elements are being used (Num), shows how many are allocated (Total), shows the size of each element (Size), and shows the amount of memory in pages (Pages) that the total slab is using. This can be helpful when tracking down exactly how the kernel is using its memory.

Listing 3.6.

bash-2.05b$ vmstat -m
Cache                          Num   Total   Size   Pages
udf_inode_cache                  0       0    416       9
fib6_nodes                       7     113     32     113
ip6_dst_cache                    9      17    224      17
ndisc_cache                      1      24    160      24
raw6_sock                        0       0    672       6
udp6_sock                        0       0    640       6
tcp6_sock                      404     441   1120       7
ip_fib_hash                     39     202     16     202
ext3_inode_cache              1714    3632    512       8
...

vmstat provides an easy way to extract large amounts of information about the Linux memory subsystem. When combined with the other information provided on the default output screen, it provides a fair picture of the health and resource usage of the system.

3.2.2. top (2.x and 3.x)

As discussed in the previous chapter, top presents both system-wide and process-specific performance statistics. By default, top presents a list, in decreasing order, of the top CPU-consuming processes, but it can also be adjusted to sort by the total memory usage, enabling you to track down which process is using the most memory.

3.2.2.1 Memory Performance-Related Options

top does not have any special command-line options that manipulate its display of memory statistics. It is invoked with the following command line:

top

However, once running, top enables you to select whether system-wide memory information displays, and whether processes are sorted by memory usage. Sorting by memory consumption proves particularly useful to determine which process is consuming the most memory. Table 3-3 describes the different memory-related output toggles.

Table 3-3. top Runtime Toggles

Option

Explanation

Table 3-4 describes the memory performance statistics that top can provide for both the entire system and individual processes. top has two different versions, 2.x and 3.x, which have slightly different names for output statistics. Table 3-4 describes the names for both versions.

Table 3-4. top Memory Performance Statistics

Option

Explanation

top provides a large amount of memory information about the different running processes. As discussed in later chapters, you can use this information to determine exactly how an application allocates and uses memory.

3.2.2.2 Example Usage

Listing 3.7 is similar to the example run of top shown in the previous chapter. However, in this example, notice that in the buffers, we have a total amount of 1,024MB of physical memory, of which ~84MB is free.

Listing 3.7.

[ezolt@wintermute doc]$ top
top - 15:47:03 up 24 days,  4:32, 15 users,  load average: 1.17, 1.19, 1.17
Tasks: 151 total,   1 running, 138 sleeping,  11 stopped,   1 zombie
Cpu(s):  1.2% us,  0.7% sy,  0.0% ni, 93.0% id,  4.9% wa,  0.0% hi,  0.1% si
Mem:   1034320k total,   948336k used,    85984k free,    32840k buffers
Swap:  2040244k total,   276796k used,  1763448k free,   460864k cached
PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
26364 root      16   0  405m  71m 321m S  4.0  7.1 462:25.50 X
17345 ezolt     16   0  176m  73m  98m S  2.6  7.3   1:17.48 soffice.bin
18316 ezolt     16   0  2756 1096 1784 R  0.7  0.1   0:05.54 top
26429 ezolt     16   0 65588  52m  12m S  0.3  5.2  16:16.77 metacity
26510 ezolt     16   0 19728 5660  16m S  0.3  0.5  27:57.87 clock-applet
26737 ezolt     16   0 70224  35m  20m S  0.3  3.5   8:32.28 gnome-terminal
1 root     16   0  2396  448 1316 S  0.0  0.0   0:01.72 init
2 root     RT   0     0    0    0 S  0.0  0.0   0:00.88 migration/0
3 root     34  19     0    0    0 S  0.0  0.0   0:00.01 ksoftirqd/0
4 root     RT   0     0    0    0 S  0.0  0.0   0:00.35 migration/1
5 root     34  19     0    0    0 S  0.0  0.0   0:00.01 ksoftirqd/1
6 root     RT   0     0    0    0 S  0.0  0.0   0:34.20 migration/2
7 root     34  19     0    0    0 S  0.0  0.0   0:00.01 ksoftirqd/2

Again, top can be customized to display only what you are interested in observing. Listing 3.8 shows a highly configured output screen that shows only memory performance statistics.

Listing 3.8.

Mem:   1034320k total,   948336k used,    85984k free,    33024k buffers
Swap:  2040244k total,   276796k used,  1763448k free,   460680k cached
VIRT  RES  SHR %MEM SWAP CODE DATA nFLT nDRT COMMAND
405m  71m 321m  7.1 333m 1696 403m 4328    0 X
70224  35m  20m  3.5  33m  280  68m 3898    0 gnome-terminal
2756 1104 1784  0.1 1652   52 2704    0    0 top
19728 5660  16m  0.5  13m   44  19m   17    0 clock-applet
2396  448 1316  0.0 1948   36 2360   16    0 init
0    0    0  0.0    0    0    0    0    0 migration/0
0    0    0  0.0    0    0    0    0    0 ksoftirqd/0
0    0    0  0.0    0    0    0    0    0 migration/1
0    0    0  0.0    0    0    0    0    0 ksoftirqd/1
0    0    0  0.0    0    0    0    0    0 migration/2
0    0    0  0.0    0    0    0    0    0 ksoftirqd/2
0    0    0  0.0    0    0    0    0    0 migration/3

top provides a real-time update of memory statistics and a display of which processes are using which types of memory. This information becomes useful as we investigate application memory usage.

3.2.3. procinfo II

As we've seen before, procinfo provides a view of the system-wide performance characteristics. In addition to the statistics described in the previous chapter, procinfo provides a few memory statistics, similar to vmstat and top, that indicate how the current memory is being used.

3.2.3.1 Memory Performance-Related Options

procinfo does not have any options that change the output of the memory statistics displayed and, as a result, is invoked with the following command:

procinfo

procinfo displays the basic memory system memory statistics, similar to top and vmstat. These are shown in Table 3-5.

Table 3-5. procinfo CPU Statistics

Option

Explanation

Much like vmstat or top, procinfo is a low-overhead command that is good to leave running in a console or window on the screen. It gives a good indication of a system's health and performance.

3.2.3.2 Example Usage

Listing 3.9 is a typical output for procinfo. As you can see, it reports summary information about how the system is using virtual memory. In this case, the system has a total of 312MB of memory; 301MB is in use by the kernel and applications, 11MB is used by system buffers, and 11MB is not used at all.

Listing 3.9.

[ezolt@localhost procinfo-18]$ ./procinfo
Linux 2.6.6-1.435.2.3smp (bhcompile@tweety.build.redhat.com) (gcc 3.3.3 20040412 )
#1
1CPU [localhost]
Memory:      Total        Used        Free      Shared      Buffers
Mem:        320468      308776       11692           0        11604
Swap:       655192      220696      434496
Bootup: Sun Oct 24 10:03:43 2004    Load average: 0.44 0.53 0.51 3/110 32243
user  :       0:57:58.92   9.0%  page in :        0
nice  :       0:02:51.09   0.4%  page out:        0
system:       0:20:18.43   3.2%  swap in :        0
idle  :       8:47:31.54  81.9%  swap out:        0
uptime:      10:44:01.94         context : 13368094
irq  0:  38645994 timer                 irq  7:         2
irq  1:     90516 i8042                 irq  8:         1 rtc
irq  2:         0 cascade [4]           irq  9:         2
irq  3:    742857 prism2_cs             irq 10:         2
irq  4:         6                       irq 11:    562551 uhci_hcd, yenta, yen
irq  5:         2                       irq 12:   1000803 i8042
irq  6:         8                       irq 14:    207681 ide0

procinfo provides information about system performance in a single screen of information. Although it provides a few of the important statistics for memory, vmstat or top is much better suited for investigating system-wide memory usage.

3.2.4. gnome-system-monitor (II)

gnome-system-monitor is in many ways a graphical counterpart of top. It enables you to monitor individual process and observe the load on the system based on the graphs that it displays. It also provides rudimentary graphs of CPU and memory usage.

3.2.4.1 Memory Performance-Related Options

gnome-system-monitor can be invoked from the Gnome menu. (Under Red Hat 9 and higher, this is in System Tools > System Monitor.) However, it can also be invoked using the following command:

gnome-system-monitor

gnome-system-monitor has no relevant command-line options that affect the memory performance measurements.

3.2.4.2 Example Usage

When you launch gnome-system-monitor and select the System Monitor tab, you see a window similar to Figure 3-1. This window enables you to glance at the graph and see how much physical memory and swap is currently being used, and how its usage has changed over time. In this case, we see that 969MB of a total of 1,007MB is currently being used. The memory usage has been relatively flat for a while.

The graphical view of data provided by gnome-system-monitor can make it easier and faster; however, most of the details, such as how the memory is being used, are missing.

3.2.5. free

free provides an overall view of how your system is using your memory, including the amount of free memory. Although the free command may show that a particular system does not have much free memory, this is not necessarily bad. Instead of letting memory sit unused, the Linux kernel puts it to work as a cache for disk reads and as a buffer for disk writes. This can dramatically increase system performance. Because these cache and buffers can always be discarded and the memory can be used if needed by applications, free shows you the amount of free memory plus or minus these buffers.

3.2.5.1 Memory Performance-Related Options

free can be invoked using the following command line:

free [-l] [-t] [-s delay ] [-c count ]

Table 3-6 describes the parameters that modify the types of statistics that free displays. Much like vmstat, free can periodically display updated memory statistics.

Table 3-6. free Command-Line Options

Option

Explanation

free actually displays some of the most complete memory statistics of any of the memory statistic tools. The statistics that it displays are shown in Table 3-7.

Table 3-7. free Memory Statistics

Statistic

Explanation

free provides information about the system-wide memory usage of Linux. It is a fairly complete range of memory statistics.

3.2.5.2 Example Usage

Calling free without any command options gives you an overall view of the memory subsystem.

As mentioned previously, Linux tries to use all the available memory if possible to cache data and applications. In Listing 3.10, free tells us that we are currently using 234,720 bytes of memory; however, if you ignore the buffers and cache, we are only using 122,772 bytes of memory. The opposite is true of the free column. We currently have 150,428 bytes of memory free; if you also count the buffers and cached memory (which you can, because Linux throws away those buffers if the memory is needed), however, we have 262,376 bytes of memory free.

Listing 3.10.

[ezolt@wintermute procps-3.2.0]$ free
total       used       free     shared    buffers      cached
Mem:        385148     234720     150428          0       8016      103932
-/+ buffers/cache:     122772     262376
Swap:       394080      81756     312324

Although you could just total the columns yourself, the -t flag shown in Listing 3.11 tells you the totals when adding both swap and real memory. In this case, the system had 376MB of physical memory and 384MB of swap. The total amount of memory available on the system is 376MB plus 384MB, or ~760MB. The total amount of free memory was 134MB of physical memory plus 259MB of swap, yielding a total of 393MB of free memory.

Listing 3.11.

[ezolt@wintermute procps-3.2.0]$ free -t
total       used       free     shared    buffers      cached
Mem:        385148     247088     138060          0       9052      115024
-/+ buffers/cache:     123012     262136
Swap:       394080      81756     312324
Total:      779228     328844     450384

Finally, free tells you the amount of high and low memory that the system is using. This is mainly useful on 32-bit machines (such as IA32) with 1GB or more of physical memory. (32-bit machines are the only machines that will have high memory.) Listing 3.12 shows a system with a very small amount of free memory, 6MB in total. It shows a system with 876MB of low memory and 640MB of high memory. This system also has much more cached memory than buffer memory, suggesting that it may be aggressively writing data to disk rather than leaving it in the buffer cache a long time.

Listing 3.12.

fas% free -l
total       used       free     shared    buffers      cached
Mem:       1552528    1546472       6056          0       7544      701408
Low:        897192     892800       4392
High:       655336     653672       1664
-/+ buffers/cache:     837520     715008
Swap:      2097096     566316    1530780

free gives a good idea of how the system memory is being used. It may take a little while to get used to output format, but it contains all the important memory statistics.

3.2.6. slabtop

slabtop is similar to top, but instead of displaying information about the CPU and memory usage of the processes in the system, slabtop shows in real-time how the kernel is allocating its various caches and how full they are. Internally, the kernel has a series of caches that are made up of one or more slabs. Each slab consists of a set of one or more objects. These objects can be active (or used) or inactive (unused). slabtop shows you the status of the different slabs. It shows you how full they are and how much memory they are using.

3.2.6.1 Memory Performance-Related Options

slabtop is invoked using the following command line:

slabtop [--delay n sort={a | b | c | l | v | n | o | p | s | u}

The command-line options for slabtop are described in Table 3-8.

Table 3-8. slabtop Command-Line Options

Option

Explanation

slabtop provides a glimpse into the data structures of the Linux kernel. Each of these slab types is tied closed to the Linux kernel, and a description of each of these slabs is beyond the scope of this book. If a particular slab is using a large amount of kernel memory, reading the Linux kernel source code and searching the Web are two great ways to figure out what these slabs are used for.

3.2.6.2 Example Usage

As shown in Listing 3.13, by default, slabtop fills the entire console and continually updates the statistics every three seconds. In this example, you can see that the size-64 slab has the most objects, only half of which are active.

Listing 3.13.

[ezolt@wintermute proc]$ slabtop
Active / Total Objects (% used)    : 185642 / 242415 (76.6%)
Active / Total Slabs (% used)      : 12586 / 12597 (99.9%)
Active / Total Caches (% used)     : 88 / 134 (65.7%)
Active / Total Size (% used)       : 42826.23K / 50334.67K (85.1%)
Minimum / Average / Maximum Object : 0.01K / 0.21K / 128.00K
OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
66124  34395  52%    0.06K   1084       61      4336K size-64
38700  35699  92%    0.05K    516       75      2064K buffer_head
30992  30046  96%    0.15K   1192       26      4768K dentry_cache
21910  21867  99%    0.27K   1565       14      6260K radix_tree_node
20648  20626  99%    0.50K   2581        8     10324K ext3_inode_cache
11781   7430  63%    0.03K     99      119       396K size-32
9675   8356  86%    0.09K    215       45       860K vm_area_struct
6024   2064  34%    0.62K   1004        6      4016K ntfs_big_inode_cache
4520   3633  80%    0.02K     20      226        80K anon_vma
4515   3891  86%    0.25K    301       15      1204K filp
4464   1648  36%    0.12K    144       31       576K size-128
3010   3010 100%    0.38K    301       10      1204K proc_inode_cache
2344    587  25%    0.50K    293        8      1172K size-512
2250   2204  97%    0.38K    225       10       900K inode_cache
2100    699  33%    0.25K    140       15       560K size-256
1692   1687  99%    0.62K    282        6      1128K nfs_inode_cache
1141   1141 100%    4.00K   1141        1      4564K size-4096

Because the information provided by slabtop is updated periodically, it is a great way to see how the Linux kernel's memory usage changes in response to different workloads.

3.2.7. sar (II)

All the advantages of using sar as a CPU performance tool, such as the easy recording of samples, extraction to multiple output formats, and time stamping of each sample, are still present when monitoring memory statistics. sar provides similar information to the other memory statistics tools, such as the current values for the amount of free memory, buffers, cached, and swap amount. However, it also provides the rate at which these value change and provides information about the percentage of physical memory and swap that is currently being consumed.

3.2.7.1 Memory Performance-Related Options

sar can be invoked with the following command line:

sar [-B] [-r] [-R]

By default, sar displays only CPU performance statistics; so to retrieve any of the memory subsystem statistics, you must use the options described in Table 3-9.

Table 3-9. sar Command-Line Options

Option

Explanation

sar provides a fairly complete view of the Linux memory subsystem. One advantage that sar provides over other tools is that it shows the rate of change for many of the important values in addition to the absolute values. You can use these values to see exactly how memory usage is changing over time without having to figure out the differences between values at each sample. Table 3-10 shows the memory statistics that sar provides.

Table 3-10. sar Memory Statistics

Statistic

Explanation

Although sar misses the high and low memory statistics, it provides nearly every other memory statistic. The fact that it can also record network CPU and disk I/O statistics makes it very powerful.

3.2.7.2 Example Usage

Listing 3.14 shows sar providing information about the current state of the memory subsystem. From the results, we can see that the amount of memory that the system used varies from 98.87 percent to 99.25 percent of the total memory. Over the course of the observation, the amount of free memory drops from 11MB to 7MB. The percentage of swap that is used hovers around ~11 percent. The system has ~266MB of data cache and about 12MB in buffers that can be written to disk.

Listing 3.14.

[ezolt@scrffy manuscript]$ sar -r 1 5
Linux 2.6.8-1.521smp (scrffy)   10/25/2004
09:45:30 AM kbmemfree kbmemused  %memused kbbuffers  kbcached kbswpfree kbswpused  
%swpused  kbswpcad
09:45:31 AM     11732   1022588     98.87     12636    272284   1816140    224104     10
.98     66080
09:45:32 AM     10068   1024252     99.03     12660    273300   1816140    224104     10
.98     66080
09:45:33 AM      5348   1028972     99.48     12748    275292   1816140    224104     10
.98     66080
09:45:35 AM      4932   1029388     99.52     12732    273748   1816140    224104     10
.98     66080
09:45:36 AM      6968   1027352     99.33     12724    271876   1815560    224684     11
.01     66660
Average:         7810   1026510     99.25     12700    273300   1816024    224220     10
.99     66196

Listing 3.15 shows that the system is consuming the free memory at a rate of ~82 pages per second during the first sample. Later, it frees ~16 pages, and then consumes ~20 pages. The number pages of buffers grew only once during the observation, at a rate of 2.02 pages per second. Finally, the pages of cache shrunk by 2.02, but in the end, they expanded by 64.36 pages/second.

Listing 3.15.

[ezolt@scrffy manuscript]$ sar -R 1 5
Linux 2.6.8-1.521smp (scrffy)   10/25/2004
09:57:22 AM   frmpg/s   bufpg/s    campg/s
09:57:23 AM    -81.19      0.00       0.00
09:57:24 AM      8.00      0.00       0.00
09:57:25 AM      0.00      2.02      -2.02
09:57:26 AM     15.84      0.00       0.00
09:57:27 AM    -19.80      0.00      64.36
Average:       -15.54      0.40      12.55

Listing 3.16 shows that the system wrote ~53 pages of memory to disk during the third sample. The system has a relatively high fault count, which means that memory pages are being allocated and used. Fortunately, none of these are major faults, meaning that the system does not have to go to disk to fulfill the fault.

Listing 3.16.

[ezolt@scrffy dvi]$ sar -B 1 5
Linux 2.6.8-1.521smp (scrffy)   10/25/2004
09:58:34 AM  pgpgin/s pgpgout/s   fault/s   majflt/s
09:58:35 AM      0.00      0.00   1328.28       0.00
09:58:36 AM      0.00      0.00    782.18       0.00
09:58:37 AM      0.00     53.06    678.57       0.00
09:58:38 AM      0.00      0.00    709.80       0.00
09:58:39 AM      0.00      0.00    717.17       0.00
Average:         0.00     10.42    842.48       0.00

As you can see, sar is a powerful tool that enhances the functionality of the other system memory performance tools by adding the ability to archive, time stamp, and simultaneously collect many different types of statistics.

3.2.8. /proc/meminfo

The Linux kernel provides a user-readable text file called /proc/meminfo that displays current system-wide memory performance statistics. It provides a superset of the system-wide memory statistics that can be acquired with vmstat, top, free, and procinfo; but it is a little more difficult to use. If you want periodic updates, you must write a script or some code for such. If you want to store the memory performance information or coordinate it with CPU statistics, you must create a new tool or write a script. However, /proc/meminfo provides the most complete view of system memory usage.

3.2.8.1 Memory Performance-Related Options

The information in /proc/meminfo can be retrieved with the following command line:

cat /proc/meminfo

This command displays the array of statistics described in Table 3-11.

Table 3-11. /proc/meminfo Memory Statistics (All in KB)

Statistic

Explanation

/proc/meminfo provides a wealth of information about the current state of the Linux memory subsystem.

3.8.2.2 Example Usage

Listing 3.17 shows an example output of /proc/meminfo. It shows some memory statistics that are similar to those we have seen with other tools. However, some statistics show new information. One, Dirty, shows that the system currently has 24KB of data waiting to be written to disk. Another, Commited_AS, shows that we need a little more memory (needed: 1068MB versus total: 1024MB) to avoid a potential out-of-memory situation.

Listing 3.17.

[ezolt@scrffy /tmp]$ cat /proc/meminfo
MemTotal:      1034320 kB
MemFree:         10788 kB
Buffers:         29692 kB
Cached:         359496 kB
SwapCached:     113912 kB
Active:         704928 kB
Inactive:       222764 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:      1034320 kB
LowFree:         10788 kB
SwapTotal:     2040244 kB
SwapFree:      1756832 kB
Dirty:              24 kB
Writeback:           0 kB
Mapped:         604248 kB
Slab:            51352 kB
Committed_AS:  1093856 kB
PageTables:       9560 kB
VmallocTotal:  3088376 kB
VmallocUsed:     26600 kB
VmallocChunk:  3058872 kB
HugePages_Total:     0
HugePages_Free:      0
Hugepagesize:     2048 kB

/proc/meminfo is the most complete place to gather system-wide Linux memory statistics. Because you can treat it as a text file, any custom script or program can easily extract the statistics.