Storage Tools and Diagnostics
lmdd - File System throughput tool
The lmdd tool will measure read and write throughput to a single file. This tool is part of the LMBENCH toolset.
LMBENCH and lmdd can be found at the following location. http://www.bitmover.com/lmbench/
Measure file write performance through the file system
The following instructions will run a write test via lmdd. The of parameter defines the filename that will be written to, the bs parameter defines the size of each I/O and the move parameter defines the total number of I/Os to perform. The direct option will force direct I/O. Remove the direct option from the command line in order to do buffered I/O. The parameters below will write 1GB to a file named TESTFILE using 1MB I/Os. # cd /stornext/fs-to-test
# lmdd of=TESTFILE direct=1 bs=1m move=1024m
Measure file read performance through the file system
The following instructions will run a read test via lmdd. The of parameter defines the filename that will be read from, the bs parameter defines the size of each I/O and the move parameter defines the total number of I/Os to perform. The direct option will force direct I/O. Remove the direct option from the command line in order to do buffered I/O. The parameters below will read a 1GB to a file named TESTFILE using 1MB I/Os. Note: The following command assumes that the file named TESTFILE exists.
# cd /stornext/fs-to-test
# lmdd if=TESTFILE direct=1 bs=1m move=1024m
iozone - File System I/O Benchmark
iozone is an I/O benchmarking tool that can be used to help diagnose performance problems on both SAN and LAN clients.
The iozone tool can be found at the following location. http://www.iozone.org/
It is very important to run iozone or other I/O benchmarking tools as one of the initial step in diagnosing DLC performance problems. Initially, iozone should be run directly on the gateway to confirm that there is sufficient disk bandwidth available. If there isn't, there is little point in performing end-to-end testing with DLC until this is resolved.
Here is a basic iozone command line:
iozone –s10g –r1m –i 0 –i 1 –e –c -+n –t 1
This says:
Create a 10GB file.
Use 1MB I/Os.
Run the sequential write test (“-i 0”) followed by the sequential read test (“-i 1”)
Include fysnc(2) and close(2) in timings (-e –c).
Don’t run “re-tests” (-+n)
Run one thread (“-t 1”)
It is important that the size of the file (-s10g) be chosen so that it is larger than the RAID cache. Otherwise, the I/Os will only hit cache and the results will be misleading. This test will not do a perfect job of simulating the I/Os from DLC clients but it's a first good step at determining whether there is sufficient disk bandwidth. For a closer approximation to actual DLC I/O patterns, the -I (use direct I/O) option should be used as well as increasing the number of threads (-t).
iostat - System I/O statistics
The iostat utility is part of the standard Linux distribution. This tool can show the traffic across disk devices from a particular system. In addition, iostat can be used to look for high disk utilization. High disk utilization may indicate saturation of the disk devices in some conditions.
See the iostat man page for more details.
One common operation for iostat is the -x command which prints out extended information.
The following example prints extended information (-x), puts a timestamp in the output (-t) and will print an update every tem seconds (10).
# iostat -xt 10
Time: 05:10:39 PM
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sdad 0.00 0.00 3.60 26.70 3328.00 22873.60 864.74 11.59 39.11 26.32 79.76
Time: 05:10:49 PM
sdad 0.00 0.00 0.00 0.00 0.00 0.00 0.00 16.00 0.00 0.00 100.01
Time: 05:10:59 PM
sdad 0.00 0.00 0.00 0.00 0.00 0.00 0.00 16.53 0.00 0.00 100.01
Time: 05:11:09 PM
sdad 0.00 0.00 0.00 0.00 0.00 0.00 0.00 17.00 0.00 0.00 100.00
:
Time: 05:13:49 PM
sdad 0.00 0.00 0.00 0.00 0.00 0.00 0.00 17.00 0.00 0.00 100.00
Time: 05:13:59 PM
sdad 0.00 0.00 6.20 36.80 4492.80 25356.80 694.18 6.74 7876.77 9.60 41.30
Notice in this example there are times showing >100% utilization. This can be an indicator that these disks are saturated for some reason. There may very heavy load on these disks or there may be something failing on these disks (for example, a RAID rebuild could be occuring).
A note on the SIO performance tool...
In prior investigations we have run across the SIO performance tool, provoded by NetApp. We have found this tool to be sub-optimal, especially on Windows. Our recommendation is to not utilize SIO for performance testing purposes.
Additional Information
Quantum