How to benchmark local storage system

Several tools can be used for IO test..we recommend two utilities: dd and iozone

the simplest one: dd

single process:

write test
time dd if=/dev/zero of=/dir/ddtestfile bs=1M count=1024  
 (/dir is the directory mounted to the storage device which you want to test)

read test
time dd if=/dir/ddtestfile of=/dev/null  bs=1M 
(/dir is the directory mounted to the storage device which you want to test)

bs is the record size for every write and read..

count is the number you will write with the record size.

For Disk IO test, it is recommand that the filesize should be at least 2 times of your RAM size..the bigger the better to disable the memory cache effection.. the ram of our most nodes is 16GB, so we test the file size of 32GB..

An Example:

root@umfs10 ~# df -h
......
/dev/sdc               20T  1.2M   20T   1% /dcache1
root@umfs10 /dcache# time dd if=/dev/zero of=/dcache1/ddtestfile bs=1M count=32768  
32768+0 records in
32768+0 records out

real    1m39.403s
user    0m0.010s
sys     0m45.047s

root@umfs10 /dcache# ll -h  /dcache1/ddtestfile 
-rw-r--r--  1 root root 32G Dec 11 17:02 /dcache1/ddtestfile
root@umfs10 /dcache# time dd if=/dcache1/ddtestfile of=/dev/null  bs=1M 
32768+0 records in
32768+0 records out

real    2m45.178s
user    0m0.023s
sys     0m21.365s

write filesize=32768MB
write time=1*60+39.403=99.403s
write performance=32768MB/99.403s=329MB/s

read filesize=32768MB
read time=2*60+45.176=165.176s
read performance=32768MB/165.176=198MB/s

run mutiple processes..

we recommend you run mutiple processes as many as your processors..

1 create a bash scripts

root@umfs10 ~# more 2ddwrite.sh   
#!/bin/bash
{ time dd if=/dev/zero of=/dcache/2ddwrite1 bs=1M count=1000 2>/dev/null ;} 2>>2ddwrite_time &
{ time dd if=/dev/zero of=/dcache/2ddwrite2 bs=1M count=1000 2>/dev/null ;} 2>>2ddwrite_time &

root@umfs10 ~# more 2ddread.sh 
#!/bin/bash
{ time dd  if=/dcache/2ddwrite1 of=/dev/null bs=1M  2>/dev/null ;} 2>>2ddread_time &
{ time dd  if=/dcache/2ddwrite2 of=/dev/null bs=1M  2>/dev/null ;} 2>>2ddread_time &

note:change to file size to at least 2 times of your ram size ..

2 run the script

root@umfs10 ~#sh 2ddwrite.sh

3 the output:

root@umfs10 ~# more 2ddwrite_time 

real    0m5.891s
user    0m0.000s
sys     0m1.223s

real    0m5.910s
user    0m0.003s
sys     0m1.227s

4 calculate the performance

filesize=32768MB*2=65536M
time=0.5910s . (choose the maximum time in the output file)
total write performance=2000M/5.910s=338MB/s
each write performance =338MB/s/2=169MB/s

iozone

1 testing fixed file size with fixed record size.

/usr/bin/iozone -Rab io.result  -s 32g -r 512k  -f /dir/iotmp -i 0 -i 1  
 (/dir is the directory mounted to the storage device which you want to test)

-R the output file io.result is excel compatible
-s  set testing file size to 32GB
-r set record size to 512kB
-i 0 test write and rewrite mode
-i 1 test read and reread mode

usually we use nohup  to run this command in background..

nohup /usr/bin/iozone -Rab io.result  -s 32g -r 512k  -f /dir/iotmp -i 0 -i 1 &

2 testing a range of filesize within the range of record size.

/usr/bin/iozone -Rab io.result  -n 8g -g 32g -y 32k  -q 8192k -f /dir/iotmp -i 0 -i 1  
 (/dir is the directory mounted to the storage device which you want to test)

-n  specify the smallest file size will be tested
-g  specify the largest file size will be tested
-y specify the smallest record size will be used,
-q specify the largest record size will be used

3 testing mutiple parallel threads

/usr/bin/iozone -Rb io.result  -P  -s 32g -r 512k  -l 1 -u 4 -f /dir/iotmp1  /dir/iotmp2 /dir/iotmp3  /dir/iotmp4  -i 0 -i 1

-P bind threads to the process
-l specify the least threads numer
-u specify the most threads number
-f specify the  path for the temperary files
 

directions (FYI)

1 preparing a good testing node

make sure:

1) the system is light loaded :turn off all unnessary services..especially services comsuming lots of resources..
2) leave the testing node alone: dont run "top", or "ps " during the test process,not even hit the keyboard.

2 using dd to test your storage nodes

1) figure out the device you want to test

take umfs10.aglt2.org for example

root@umfs10 /dcache# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda7             202G  5.3G  187G   3% /
/dev/sda1             981M   60M  871M   7% /boot
none                  7.9G  4.0K  7.9G   1% /dev/shm
/dev/sda2              20G  380M   18G   3% /var
AFS                   8.6G     0  8.6G   0% /afs
/dev/sdb              233G   35G  198G  16% /dcache
/dev/sdc               20T   33G   20T   1% /dcache1
root@umfs10 /dcache# 

/dev/sdc is a raid system, which is mounted to the directory /dcache1

2) using dd test write performance

root@umfs10 /dcache# time dd if=/dev/zero of=/dcache1/ddtestfile bs=1M count=32768  
32768+0 records in
32768+0 records out

real    1m39.403s
user    0m0.010s
sys     0m45.047s

3) using dd test the read performance

root@umfs10 /dcache# ll -h  /dcache1/ddtestfile 
-rw-r--r--  1 root root 32G Dec 11 17:02 /dcache1/ddtestfile
root@umfs10 /dcache# time dd if=/dcache1/ddtestfile of=/dev/null  bs=1M 
32768+0 records in
32768+0 records out

real    2m45.178s
user    0m0.023s
sys     0m21.365s

4 ) calculate the write and read rate

write filesize=32768MB
write time=1*60+39.403=99.403s
write performance=32768MB/99.403s=329MB/s

read filesize=32768MB
read time=2*60+45.176=165.176s
read performance=32768MB/165.176=198MB/s

3 using iozone to test parallel threads.

1) figure out how many processors you have on your node

on umfs10.aglt2.org , we have 8 processors. we recommend you run at least as many threads as your processor number.

2) figure out the device you want to test.

root@umfs10 /dcache# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda7             202G  5.3G  187G   3% /
/dev/sda1             981M   60M  871M   7% /boot
none                  7.9G  4.0K  7.9G   1% /dev/shm
/dev/sda2              20G  380M   18G   3% /var
AFS                   8.6G     0  8.6G   0% /afs
/dev/sdb              233G   35G  198G  16% /dcache
/dev/sdc               20T   33G   20T   1% /dcache1
root@umfs10 /dcache# 

3) using iozone test both write and read in mutiple threads mode..

/usr/bin/iozone -Rb io.result  -P  -s 32g -r 512k  -l 1 -u 8 - f /dcache1/iotmp1  /dcache1/iotmp2 /dcache1/iotmp3  /dcache1/iotmp4 /dcache1/iotmp5 /dcache1/iotmp6 /dcache1/iotmp7 /dcache1/iotmp8  -i 0 -i 1

note: if you have more than one storage device on your node, perhaps you wish to make a load balance between them..you can achieve this by specifying differently the temp files..

eg: on umfs9.aglt2.org, we have 2 raid device, one is /dev/sdb which is mounted as /dcache , the other is /dev/sdc which is mounted as /dcache1.. we hope the IO can be ran on both devices..so..just do..

/usr/bin/iozone -Rb io.result  -P  -s 32g -r 512k  -l 1 -u 8 - f /dcache/iotmp1  /dcache1/iotmp2 /dcache/iotmp3  /dcache1/iotmp4 /dcache/iotmp5 /dcache1/iotmp6 /dcache/iotmp7 /dcache1/iotmp8  -i 0 -i 1

note: it is important to specify the temp files alternatively..make sure for different threads, there is always a IO balance..

This is an example we get on the umfs10.aglt2.org:

   1   2   3   4   5   6   7   8
write    366224   715138   688733   653704   646777   623932   619243   604735
Rewrite 333197   708475   670958   670746   653379   619305   612460   595997
Read    757487   1558510   1456531   936365   899806   851331   935963   887000
Reread    758652   1571521   1426091   1116436   1006510   862514   851553   900942

(1..8 ) is the threads number

the number in the table is the total performance for mutiple threads

test 4 IO modes: write , rewrite , read and re-read..

the unit is KB/s

-- Main.bmeekhof - 11 Dec 2007