Performance Tuning for Linux Servers [Electronic resources] نسخه متنی

Tuning File Systems

Now that you're familiar with the layout of the various types of file systems supported in Linux, let's examine how to tune the Ext3, ReiserFS, JFS, and XFS file systems. We'll show how to tune each of the file systems by using an external log and using some of the mount options that are available on each file system.

Tuning Options for Ext3: Using a Separate Journal Device

External logs improve file system performance because the log updates are saved to a different partition than the corresponding file system, thereby reducing the number of hard disk seeks.

To use an external journal for the Ext3 file system, first run mkfs on the journal device. The block size of the external journal must be the same block size at the Ext3 file system. The example in this section uses the /dev/hdb1 device as the external log for the Ext3 file system.

There are two steps to creating an external log. The first is to format the journal; the second is to format the partition and tell mkfs that the log will be external. In the following example, the Ext3 partition will be on device /dev/hda1, and the external log will be on device /dev/hdb1. The mkfs b option sets the block size for the file system.

# mkfs.ext3 -b 4096 -O journal_dev /dev/hdb1
# mkfs.ext3 -b 4096 -J device=/dev/hdb1 /dev/hda1

The next few examples use the tiobench program to see if an external log helps the performance of this benchmark on an Ext3 file system. The tiobench benchmark is a multithreaded I/O benchmark. It is used to measure file system performance in four basic operations: sequential read, random read, sequential write, and random write.

First, the device /dev/hda1 is formatted as ext3.

# mkfs.ext3 /dev/hda1
# mount -t ext3 /dev/hda1 /ext3
# cd /ext3
# tar zxvf tiobench-0.3.3.tar.gz
# cd tiobench-0.3.3
# make
# date && ./tiobench.pl --size 500 --numruns 5 threads32 && date

The following is sample tiobench output with the log inside the partition.[View full width]

Fri Jun 27 10:17:14 PDT 2003
Run #1: ./tiotest -t 32 -f 15 -r 125 -b 4096 -d . -T
Run #2: ./tiotest -t 32 -f 15 -r 125 -b 4096 -d . -T
Run #3: ./tiotest -t 32 -f 15 -r 125 -b 4096 -d . -T
Run #4: ./tiotest -t 32 -f 15 -r 125 -b 4096 -d . -T
Run #5: ./tiotest -t 32 -f 15 -r 125 -b 4096 -d . -T
Unit information
================
File size = megabytes
Blk Size = bytes
Rate = megabytes per second
CPU% = percentage of CPU used during the test
Latency = milliseconds
Lat% = percent of requests that took longer than
X seconds
CPU Eff = Rate divided by CPU% - throughput per cpu load
Sequential Reads
File Blk Num Avg Maximum Lat%
Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s
>10s Eff
-- - - -
-- -- - - --
2.4.20-4GB 500 4096 32 6.93 3.555% 72.342 39522.87 1.56413
0.00000 195
Random Reads
File Blk Num Avg Maximum Lat%
Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s
>10s Eff
--- - -
-- -- -
2.4.20-4GB 500 4096 32 0.89 1.011% 514.676 2536.44 0.00000
0.00000 88
Sequential Writes
File Blk Num Avg Maximum Lat%
Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s
>10s Eff
--- - -
-- -- -
2.4.20-4GB 500 4096 32 4.69 6.986% 49.635 23386.91 1.28907
0.00000 67
Random Writes
File Blk Num Avg Maximum Lat%
Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s
>10s Eff
--- - -
-- -- -
2.4.20-4GB 500 4096 32 0.53 0.434% 0.302 573.19 0.00000
0.00000 122
Fri Jun 27 10:37:14 PDT 2003
# umount /ext3

Next, to determine whether an external log increases the performance of the file system under this file system benchmark, change the configuration to have an external log. In the example, the external log is located on /dev/hdb1.[View full width]

# mkfs.ext3 -b 4096 -O journal_dev /dev/hdb1
# mkfs.ext3 -b 4096 -J device=/dev/hdb1 /dev/hda1
# mount -t ext3 /dev/hda1 /ext3
# cd /ext3
# tar zxvf tiobench-0.3.3.tar.gz
# cd tiobench-0.3.3
# make
# date && ./tiobench.pl --size 500 --numruns 5 threads32 && date
tiobench output with log external.
Fri Jun 27 11:10:17 PDT 2003
Run #1: ./tiotest -t 32 -f 15 -r 125 -b 4096 -d . -T
Run #2: ./tiotest -t 32 -f 15 -r 125 -b 4096 -d . -T
Run #3: ./tiotest -t 32 -f 15 -r 125 -b 4096 -d . -T
Run #4: ./tiotest -t 32 -f 15 -r 125 -b 4096 -d . -T
Run #5: ./tiotest -t 32 -f 15 -r 125 -b 4096 -d . -T
Unit information
================
File size = megabytes
Blk Size = bytes
Rate = megabytes per second
CPU% = percentage of CPU used during the test
Latency = milliseconds
Lat% = percent of requests that took longer than
X seconds
CPU Eff = Rate divided by CPU% - throughput per cpu load
Sequential Reads
File Blk Num Avg Maximum Lat%
Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s
>10s Eff
--- - -
-- -- -
2.4.20-4GB 500 4096 32 6.54 3.496% 83.189 40436.07 2.41211 0
.00081 187
Random Reads
File Blk Num Avg Maximum Lat%
Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s
>10s Eff
--- - -
-- -- -
2.4.20-4GB 500 4096 32 0.89 0.890% 540.200 2620.12 0.00000 0
.00000 100
Sequential Writes
File Blk Num Avg Maximum Lat%
Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s
>10s Eff
--- - -
-- -- -
2.4.20-4GB 500 4096 32 4.71 6.682% 53.069 23588.61 1.31673 0
.00000 71
Random Writes
File Blk Num Avg Maximum Lat%
Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s
>10s Eff
--- - -
-- -- -
2.4.20-4GB 500 4096 32 0.53 0.432% 0.278 404.22 0.00000 0
.00000 122
Fri Jun 27 11:30:35 PDT 2003

Because tiobench does not create a large amount of metadata activity, there is no benefit to having an external log. In terms of time, the tiobench program took an additional 18 seconds to complete when the log was on an external device.

With the dbench benchmark, which creates a very large amount of metadata activity, the results show a decrease in the amount of time needed to run the benchmark and an increase in throughput for what the benchmark measures. Therefore, determining the metadata activity for your system helps determine the type of tuning that will be most useful.

In the next few examples, dbench is run first with the log inside the partition and then with the log external to the partition.

# mkfs.ext3 /dev/hda1
# mount t ext3 /dev/hda1 /ext3
# tar zxvf dbench-1.2.tar.gz
# cd dbench
# make
# date && ./dbench 20 && date
output for dbench
Fri Jun 27 14:36:39 PDT 2003
.......................+..................+..+..
+20 clients started
Throughput 15.443 MB/sec (NB=19.3037 MB/sec  154.43 MBit/sec)
Fri Jun 27 14:39:30 PDT 2003

In the next example, the log is changed to use /dev/hdb1 as the external log device. When dbench is run again, the throughput increases from 15.443 MBps to 17.2484 MBps. The time taken to run the program was reduced from 2 minutes and 51 seconds to 2 minutes and 33 seconds.

# mkfs.ext3 b 4096 O journal_dev /dev/hdb1
# mkfs.ext3  b 4096 J device=/dev/hdb1 /dev/hda1
# tar zxvf dbench-1.2.tar.gz
# cd dbench
# make
# date && ./dbench 20  && date
output for dbench
Fri Jun 27 14:52:13 PDT 2003
.................................+...........+.+
+20 clients started
Throughput 17.2484 MB/sec (NB=21.5605 MB/sec
172.484 MBit/sec)
Fri Jun 27 14:54:46 PDT 2003
# umount /ext3

Tuning Options for ReiserFS: Go Faster with an External Log

External logs improve file system performance because the log updates are saved to a different partition than the corresponding file system, thereby reducing the number of hard disk seeks.

To create a ReiserFS file system with the log on an external device, your system must have at least two unused partitions. The test system used in the following examples has spare partitions /dev/hda1 and /dev/hdb1. In the examples, the /dev/hdb1 partition is used for the external log.

# mkreiserfs  -j /dev/hdb1 /dev/hda1

In the following example, the dbench program creates file system activity. The default mount option is used with an external log on device /dev/hdb1.

# mount t reiserfs /dev/hda1 /reiserfs
# cd /reiserfs
# tar zxvf dbench-1.2.tar.gz
# cd dbench
# make
# date && ./dbench 15 && date
output from dbench
Sat Jun 28 10:23:06 PDT 2003
.................................+...........+.+
+15 clients started
Throughput 21.7191 MB/sec (NB=27.189 MB/sec
217.191 MBit/sec)
Sat Jun 28 10:24:37 PDT 2003

The next example uses the notail mount option to increase the performance of the file system. The notail option disables the storage of small files and file tails directly into the directory tree.

# mount t reiserfs o notail /dev/hda1 /reiserfs
# cd /reiserfs
# tar zxvf dbench-1.2.tar.gz
# cd dbench
# make
# date && ./dbench 15 && date
output from dbench
Sat Jun 28 10:28:42 PDT 2003
.................................+...........+.+
+15 clients started
Throughput 25.8765 MB/sec (NB=32.3456 MB/sec
258.765 MBit/sec)
Sat Jun 28 10:29:59 PDT 2003
# cd /
# umount /reiserfs

By adding the notail mount option, the throughput of the ReiserFS file system running the dbench program with 15 clients increased from 21.7191 MBps to 25.8765 MBps. The time to run the program went from 1 minute and 31 seconds to 1 minute and 17 seconds.

Tuning Options for JFS: Go Faster with an External Log

External logs improve file system performance because the log updates are saved to a different partition than the corresponding file system, thereby reducing the number of hard disk seeks.

The following examples create a baseline for the file system by using the default option of having the log of the file system inside the volume. The test program is executed again with the log on external device /dev/hdb1.

# mkfs.jfs /dev/hda1
# mount -t jfs /dev/hda1 /jfs

The stress.sh script, which creates a high number of metadata changes, shows the benefit of using an external log.

#!/bin/sh
for count in 'seq 1 30';
do
echo Count: $count
mkdir a
for i in 'seq 1 10000';
do
echo  0123456789ABCDEFGHIJKLMNOPQRSTUVW
XYZabcdefghijklmnopqrstuvwxyz > a/$i
done
mkdir b
for j in 'seq 1 10000';
do
ln -s 'pwd'/a/$j b/$j
done
rm -fr b
rm -fr a
done
# cd /jfs
# mkdir test

If you want to try this example on your own machine, place the stress.sh script in the /jfs/test subdirectory.

# date && ./stress.sh && date
Output from stress.sh script
Sat Jun 28 10:47:27 PDT 2003
Count: 1
Count: 2
...
Count: 30
Sat Jun 28 12:48:35 PDT 2003
# umount /jfs

Tuning JFS with jfs_tune

The jfs_tune can change the location of the journal. One way to increase the file system performance is by moving the journal to an external device.

The first step is to create a journal on an external device /dev/hdb1 by using mkfs.jfs.

# mkfs.jfs -J journal_dev /dev/hdb1

The next step is to attach that external journal to the file system that is located on /dev/hda1.

# jfs_tune -J device=/dev/hdb1 /dev/hda1
# mount t jfs /dev/hda1 /jfs
# date && ./stress.sh && date
output from stress.sh script
Mon Jun 30 02:42:08 PDT 2003
Count: 1
Count: 2
...
Count: 30
Mon Jun 30 04:39:08 PDT 2003

With an external log, the test program execution time was reduced by 4 minutes and 8 seconds.

Tuning Options for XFS

The examples in this section show three ways of tuning the XFS file system for running the dbench utility. The first example uses the defaults to format an XFS partition, with the log inside the partition. The second example uses the mount options logbufsize and logbsize. The third example uses an external log and the two mount options.

Using the Defaults

In the following example, an XFS partition is formatted with the log inside the partition.

# mkfs.xfs f /dev/hda1
# mount t xfs /dev/hda1 /xfs
# cd /xfs
# tar zxvf dbench-1.2.tar.gz
# cd dbench
# make
# date && ./dbench 30 && date
Output from dbench is as follows:
Fri Jun 27 15:48:52 PDT 2003
.+++++30 clients started
Throughput 1.45512 MB/sec (NB=1.8189 MB/sec  14.5512 MBit/sec)
Fri Jun 27 16:34:13 PDT 2003

Using logbufsize and logbsize

The example in this section shows how to tune an XFS file system using the mount options logbufsize and logbsize.

The logbufsize option sets the number of log buffers held in memory. The value of logbufsize can be 2 to 8. Eight is the default for file systems created with a 64KB block size, 4 for file systems created with a 32KB block size, 3 for file systems created with a 16KB block size, and 2 for other block sizes. When logbufsize is set to the maximum, more active transactions can occur at once, and metadata changes can still be performed while the log is being synced to the disk. However, should a crash occur, a higher number of metadata changes is likely to be lost, relative to setting logbufsize to a smaller value. The logbsize size option sets the size of the log buffers held in memory.

# mkfs.xfs f /dev/hda1
# mount t xfs o logbufsize=8,logbsize=32768b/dev/hda1 /xfs
# cd dbench
# date && ./dbench 30 && date
Output from dbench
Sat Jun 28 02:01:36 PDT 2003
+.............++.++.....++.++++++.+++++++++++30 clients
started ******************************
Throughput 1.97025 MB/sec (NB=2.46281 MB/sec
19.7025 MBit/sec)
Sat Jun 28 02:35:05 PDT 2003

Placing the Log on an External Device

External logs improve file system performance because the log updates are saved to a different partition than the corresponding file system, thereby reducing the number of hard disk seeks.

The example in this section runs dbench with the same parameters as in the previous examples, but the log is placed on external device /dev/hdb1.

# mkfs.xfs l logdev=/dev/hdb1,size=32768b f /dev/hda1
# mount t xfs o logbufsize=8,logbsize=32768b,logdev=
/dev/hdb1 /dev/hda1 /xfs

The mount command can be used to check the mount options for each file system, as shown in the following example.

# mount
/dev/hdb6 on / type reiserfs (rw)
proc on /proc type proc (rw)
devpts on /dev/pts type devpts (rw,mode=0620,gid=5)
shmfs on /dev/shm type shm (rw)
usbdevfs on /proc/bus/usb type usbdevfs (rw)
/dev/hda1 on /xfs type xfs (rw,logbufs=8,logbsize=32768,logdev=/dev/hdb1)
# cd dbench
# date && ./dbench 30 && date
Output from dbench
Sat Jun 28 02:57:08 PDT 2003
.....+..........+........+.....+..+..+++....
+30 clients started
Throughput 18.9072 MB/sec (NB=23.634 MB/sec
189.072 MBit/sec)
Sat Jun 28 03:00:38 PDT 2003

When the logbufsize and logbsize mount options are added, the throughput increases from 1.45512 MBps to 1.97025 MBps. When the log is moved to an external device, the throughput increases to 18.9072 MBps. Clearly, the external log increases file system performance under a test program that has a large amount of metadata activity.