OUSWTier2 < Admins

r11 - 06 Nov 2007 - 19:06:36 - HorstSeveriniYou are here: TWiki >

OU_SWT2

Tuning on 06 Nov 2007

Tracepath to BNL:

[hs@tier2-02 ~]$ tracepath dct00.usatlas.bnl.gov
 1:  tier2-02.ochep.ou.edu (129.15.40.251)                  0.123ms pmtu 1500
 1:  129.15.40.1 (129.15.40.1)                              0.380ms 
 2:  192.168.255.233 (192.168.255.233)                      0.984ms 
 3:  164.58.10.97 (164.58.10.97)                            2.043ms 
 4:  164.58.244.174 (164.58.244.174)                      asymm  6   7.888ms 
 5:  164.58.245.78 (164.58.245.78)                        asymm  7   8.375ms 
 6:  10ge.nlr.onenet.net (164.58.245.86)                  asymm  8  24.438ms 
 7:  chi-sl-mr1-nlr-packetnet.es.net (198.125.140.222)    asymm 11  23.876ms 
 8:  chislsdn1-chislmr1.es.net (134.55.219.25)            asymm 13  23.858ms 
 9:  chiccr1-chislsdn1.es.net (134.55.207.33)             asymm 14  24.662ms 
10:  aofacr1-chicsdn1.es.net (134.55.218.94)              asymm 15  51.476ms 
11:  bnlmr1-aoacr1.es.net (134.55.217.57)                 asymm 15 123.961ms 
12:  bnlsite-bnlmr1.es.net (198.124.216.178)              asymm 16  52.647ms 
13:  dct00.usatlas.bnl.gov (192.12.15.8)                  asymm 17  52.193ms reached
     Resume: pmtu 1500 hops 13 back 17

Config info about the 1G NIC we're using:

[hs@tier2-02 ~]$ ifconfig eth1
eth1      Link encap:Ethernet  HWaddr 00:19:B9:C9:E2:EB  
          inet addr:129.15.40.251  Bcast:129.15.41.255  Mask:255.255.254.0
          inet6 addr: fe80::219:b9ff:fec9:e2eb/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:30645900 errors:0 dropped:655 overruns:0 frame:0
          TX packets:16404443 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:39453934676 (36.7 GiB)  TX bytes:5545196388 (5.1 GiB)
          Interrupt:169 Memory:f4000000-f4011100

ethtool reports for the NIC:

[root@tier2-02 hs]# ethtool -k eth1
Offload parameters for eth1:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: on
[root@tier2-02 hs]# ethtool -i eth1
driver: bnx2
version: 1.4.43-rh
firmware-version: 2.9.1
bus-info: 0000:09:00.0
[root@tier2-02 hs]# 
[root@tier2-02 hs]# 
[root@tier2-02 hs]# 
[root@tier2-02 hs]# ethtool -g eth1
Ring parameters for eth1:
Pre-set maximums:
RX:             1020
RX Mini:        0
RX Jumbo:       0
TX:             255
Current hardware settings:
RX:             100
RX Mini:        0
RX Jumbo:       0
TX:             255
[root@tier2-02 hs]# ethtool -S eth1
NIC statistics:
     rx_bytes: 39454014362
     rx_error_bytes: 0
     tx_bytes: 5545216626
     tx_error_bytes: 0
     rx_ucast_packets: 27330073
     rx_mcast_packets: 1350615
     rx_bcast_packets: 1966272
     tx_ucast_packets: 16403891
     tx_mcast_packets: 5
     tx_bcast_packets: 702
     tx_mac_errors: 0
     tx_carrier_errors: 0
     rx_crc_errors: 0
     rx_align_errors: 0
     tx_single_collisions: 0
     tx_multi_collisions: 0
     tx_deferred: 0
     tx_excess_collisions: 0
     tx_late_collisions: 0
     tx_total_collisions: 0
     rx_fragments: 0
     rx_jabbers: 0
     rx_undersize_packets: 0
     rx_oversize_packets: 0
     rx_64_byte_packets: 2959685
     rx_65_to_127_byte_packets: 1743999
     rx_128_to_255_byte_packets: 164813
     rx_256_to_511_byte_packets: 29534
     rx_512_to_1023_byte_packets: 9327
     rx_1024_to_1522_byte_packets: 25755532
     rx_1523_to_9022_byte_packets: 0
     tx_64_byte_packets: 78544
     tx_65_to_127_byte_packets: 13243762
     tx_128_to_255_byte_packets: 28521
     tx_256_to_511_byte_packets: 7223
     tx_512_to_1023_byte_packets: 42546
     tx_1024_to_1522_byte_packets: 3004002
     tx_1523_to_9022_byte_packets: 0
     rx_xon_frames: 7965
     rx_xoff_frames: 7965
     tx_xon_frames: 0
     tx_xoff_frames: 0
     rx_mac_ctrl_frames: 0
     rx_filtered_packets: 91785
     rx_discards: 0
     rx_fw_discards: 655

First test:

[root@tier2-02 hs]# iperf -c dct00.usatlas.bnl.gov -w4M -i2 -t60
------------------------------------------------------------
Client connecting to dct00.usatlas.bnl.gov, TCP port 5001
TCP window size: 2.00 MByte (WARNING: requested 4.00 MByte)
------------------------------------------------------------
[  3] local 129.15.40.251 port 40103 connected with 192.12.15.8 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 2.0 sec  23.6 MBytes  98.9 Mbits/sec
[  3]  2.0- 4.0 sec  14.1 MBytes  59.3 Mbits/sec
[  3]  4.0- 6.0 sec  19.7 MBytes  82.7 Mbits/sec
[  3]  6.0- 8.0 sec  32.4 MBytes   136 Mbits/sec
[  3]  8.0-10.0 sec  13.0 MBytes  54.7 Mbits/sec
[  3] 10.0-12.0 sec  18.3 MBytes  76.8 Mbits/sec
[  3] 12.0-14.0 sec  19.8 MBytes  83.0 Mbits/sec
[  3] 14.0-16.0 sec  18.8 MBytes  78.7 Mbits/sec
[  3] 16.0-18.0 sec  19.6 MBytes  82.2 Mbits/sec
[  3] 18.0-20.0 sec  19.7 MBytes  82.6 Mbits/sec
[  3] 20.0-22.0 sec  19.2 MBytes  80.7 Mbits/sec
[  3] 22.0-24.0 sec  14.1 MBytes  59.1 Mbits/sec
[  3] 24.0-26.0 sec  19.9 MBytes  83.4 Mbits/sec
[  3] 26.0-28.0 sec  19.5 MBytes  82.0 Mbits/sec
[  3] 28.0-30.0 sec  20.2 MBytes  84.6 Mbits/sec
[  3] 30.0-32.0 sec  20.2 MBytes  84.8 Mbits/sec
[  3] 32.0-34.0 sec  21.5 MBytes  90.2 Mbits/sec
[  3] 34.0-36.0 sec  31.6 MBytes   133 Mbits/sec
[  3] 36.0-38.0 sec  32.3 MBytes   135 Mbits/sec
[  3] 38.0-40.0 sec  19.4 MBytes  81.6 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3] 40.0-42.0 sec  16.4 MBytes  68.7 Mbits/sec
[  3] 42.0-44.0 sec  24.3 MBytes   102 Mbits/sec
[  3] 44.0-46.0 sec  24.2 MBytes   102 Mbits/sec
[  3] 46.0-48.0 sec  24.5 MBytes   103 Mbits/sec
[  3] 48.0-50.0 sec  21.9 MBytes  92.0 Mbits/sec
[  3] 50.0-52.0 sec  23.7 MBytes  99.3 Mbits/sec
[  3] 52.0-54.0 sec  24.1 MBytes   101 Mbits/sec
[  3] 54.0-56.0 sec  24.6 MBytes   103 Mbits/sec
[  3] 56.0-58.0 sec  24.4 MBytes   102 Mbits/sec
[  3] 58.0-60.0 sec  21.1 MBytes  88.6 Mbits/sec
[  3]  0.0-60.1 sec   646 MBytes  90.2 Mbits/sec

Checking for errors:

[root@tier2-02 hs]# ethtool -S eth1
NIC statistics:
     rx_bytes: 39462941949
     rx_error_bytes: 0
     tx_bytes: 6256239231
     tx_error_bytes: 0
     rx_ucast_packets: 27449347
     rx_mcast_packets: 1353008
     rx_bcast_packets: 1969504
     tx_ucast_packets: 16872779
     tx_mcast_packets: 5
     tx_bcast_packets: 703
     tx_mac_errors: 0
     tx_carrier_errors: 0
     rx_crc_errors: 0
     rx_align_errors: 0
     tx_single_collisions: 0
     tx_multi_collisions: 0
     tx_deferred: 0
     tx_excess_collisions: 0
     tx_late_collisions: 0
     tx_total_collisions: 0
     rx_fragments: 0
     rx_jabbers: 0
     rx_undersize_packets: 0
     rx_oversize_packets: 0
     rx_64_byte_packets: 2964721
     rx_65_to_127_byte_packets: 1863523
     rx_128_to_255_byte_packets: 165070
     rx_256_to_511_byte_packets: 29586
     rx_512_to_1023_byte_packets: 9335
     rx_1024_to_1522_byte_packets: 25755554
     rx_1523_to_9022_byte_packets: 0
     tx_64_byte_packets: 78581
     tx_65_to_127_byte_packets: 13244123
     tx_128_to_255_byte_packets: 28589
     tx_256_to_511_byte_packets: 7238
     tx_512_to_1023_byte_packets: 42627
     tx_1024_to_1522_byte_packets: 3472329
     tx_1523_to_9022_byte_packets: 0
     rx_xon_frames: 7965
     rx_xoff_frames: 7965
     tx_xon_frames: 0
     tx_xoff_frames: 0
     rx_mac_ctrl_frames: 0
     rx_filtered_packets: 91807
     rx_discards: 0
     rx_fw_discards: 898

Looks like a window size problem. Taking a look at our TCP kernel parameters:

[root@tier2-02 hs]# sysctl -a | grep tcp | grep mem
net.ipv4.tcp_rmem = 4096        87380   174760
net.ipv4.tcp_wmem = 4096        16384   131072
net.ipv4.tcp_mem = 786432       1048576 1572864

Changing our window size via editing /etc/sysctl.conf (add the following lines):

net.ipv4.tcp_rmem = 4096 87380 16793600
net.ipv4.tcp_wmem = 4096 87380 16793600
# maximum receive socket buffer size, default 131071
net.core.rmem_max = 16793600
# maximum send socket buffer size, default 131071
net.core.wmem_max = 16793600

Reconfiguring with:

[root@tier2-02 hs]# sysctl -p
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 1
kernel.core_uses_pid = 1
kernel.panic = 7200
net.ipv4.tcp_rmem = 4096 87380 16793600
net.ipv4.tcp_wmem = 4096 87380 16793600
net.core.rmem_max = 16793600
net.core.wmem_max = 16793600
[root@tier2-02 hs]# sysctl -a | grep tcp | grep mem
net.ipv4.tcp_rmem = 4096        87380   16793600
net.ipv4.tcp_wmem = 4096        87380   16793600
net.ipv4.tcp_mem = 786432       1048576 1572864

Rerunning the test:

[root@tier2-02 hs]# iperf -c dct00.usatlas.bnl.gov -w4M -i2 -t60
------------------------------------------------------------
Client connecting to dct00.usatlas.bnl.gov, TCP port 5001
TCP window size: 8.00 MByte (WARNING: requested 4.00 MByte)
------------------------------------------------------------
[  3] local 129.15.40.251 port 40462 connected with 192.12.15.8 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 2.0 sec  42.6 MBytes   179 Mbits/sec
[  3]  2.0- 4.0 sec  63.5 MBytes   266 Mbits/sec
[  3]  4.0- 6.0 sec  63.9 MBytes   268 Mbits/sec
[  3]  6.0- 8.0 sec  63.7 MBytes   267 Mbits/sec
[  3]  8.0-10.0 sec  65.3 MBytes   274 Mbits/sec
[  3] 10.0-12.0 sec  64.6 MBytes   271 Mbits/sec
[  3] 12.0-14.0 sec  55.4 MBytes   232 Mbits/sec
[  3] 14.0-16.0 sec  62.4 MBytes   262 Mbits/sec
[  3] 16.0-18.0 sec  56.8 MBytes   238 Mbits/sec
[  3] 18.0-20.0 sec  56.2 MBytes   236 Mbits/sec
[  3] 20.0-22.0 sec  49.9 MBytes   209 Mbits/sec
[  3] 22.0-24.0 sec  46.1 MBytes   193 Mbits/sec
[  3] 24.0-26.0 sec  43.7 MBytes   183 Mbits/sec
[  3] 26.0-28.0 sec  46.4 MBytes   195 Mbits/sec
[  3] 28.0-30.0 sec  47.0 MBytes   197 Mbits/sec
[  3] 30.0-32.0 sec  39.8 MBytes   167 Mbits/sec
[  3] 32.0-34.0 sec  45.1 MBytes   189 Mbits/sec
[  3] 34.0-36.0 sec  42.3 MBytes   178 Mbits/sec
[  3] 36.0-38.0 sec  20.1 MBytes  84.2 Mbits/sec
[  3] 38.0-40.0 sec  7.78 MBytes  32.6 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3] 40.0-42.0 sec  9.69 MBytes  40.6 Mbits/sec
[  3] 42.0-44.0 sec  11.8 MBytes  49.6 Mbits/sec
[  3] 44.0-46.0 sec  9.97 MBytes  41.8 Mbits/sec

[root@tier2-02 hs]# iperf -c dct00.usatlas.bnl.gov -w8M -i2 -t60
------------------------------------------------------------
Client connecting to dct00.usatlas.bnl.gov, TCP port 5001
TCP window size: 16.0 MByte (WARNING: requested 8.00 MByte)
------------------------------------------------------------
[  3] local 129.15.40.251 port 40528 connected with 192.12.15.8 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 2.0 sec  35.5 MBytes   149 Mbits/sec
[  3]  2.0- 4.0 sec  35.2 MBytes   148 Mbits/sec
[  3]  4.0- 6.0 sec  40.0 MBytes   168 Mbits/sec
[  3]  6.0- 8.0 sec  36.1 MBytes   151 Mbits/sec
[  3]  8.0-10.0 sec  40.4 MBytes   170 Mbits/sec
[  3] 10.0-12.0 sec  35.6 MBytes   149 Mbits/sec
[  3] 12.0-14.0 sec  35.2 MBytes   148 Mbits/sec
[  3] 14.0-16.0 sec  40.3 MBytes   169 Mbits/sec
[  3] 16.0-18.0 sec  36.4 MBytes   153 Mbits/sec
[  3] 18.0-20.0 sec  40.6 MBytes   170 Mbits/sec
[  3] 20.0-22.0 sec  36.9 MBytes   155 Mbits/sec
[  3] 22.0-24.0 sec  47.1 MBytes   198 Mbits/sec
[  3] 24.0-26.0 sec  56.4 MBytes   237 Mbits/sec
[  3] 26.0-28.0 sec  73.3 MBytes   307 Mbits/sec
[  3] 28.0-30.0 sec  87.8 MBytes   368 Mbits/sec
[  3] 30.0-32.0 sec  69.0 MBytes   289 Mbits/sec
[  3] 32.0-34.0 sec  87.4 MBytes   367 Mbits/sec
[  3] 34.0-36.0 sec  37.9 MBytes   159 Mbits/sec
[  3] 36.0-38.0 sec  43.7 MBytes   183 Mbits/sec
[  3] 38.0-40.0 sec  43.2 MBytes   181 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3] 40.0-42.0 sec  46.9 MBytes   197 Mbits/sec
[  3] 42.0-44.0 sec  43.8 MBytes   184 Mbits/sec
[  3] 44.0-46.0 sec  36.6 MBytes   153 Mbits/sec
[  3] 46.0-48.0 sec  28.8 MBytes   121 Mbits/sec
[  3] 48.0-50.0 sec  32.1 MBytes   135 Mbits/sec
[  3] 50.0-52.0 sec  23.7 MBytes  99.3 Mbits/sec
[  3] 52.0-54.0 sec  12.1 MBytes  50.7 Mbits/sec
[  3] 54.0-56.0 sec  7.64 MBytes  32.0 Mbits/sec
[  3] 56.0-58.0 sec  11.5 MBytes  48.1 Mbits/sec
[  3] 58.0-60.0 sec  11.4 MBytes  48.0 Mbits/sec
[  3]  0.0-62.1 sec  1.18 GBytes   164 Mbits/sec

Check for errors again:

[root@tier2-02 hs]# ifconfig eth1
eth1      Link encap:Ethernet  HWaddr 00:19:B9:C9:E2:EB  
          inet addr:129.15.40.251  Bcast:129.15.41.255  Mask:255.255.254.0
          inet6 addr: fe80::219:b9ff:fec9:e2eb/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:31685274 errors:0 dropped:5761 overruns:0 frame:0
          TX packets:18893480 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:40195930847 (37.4 GiB)  TX bytes:9002285016 (8.3 GiB)
          Interrupt:169 Memory:f4000000-f4011100

Problem with this NIC? Try tier2-01, which is an identical machine. Performance is the same, though. Multiple streams do allow us to fill the pipe from both tier2-01 and tier2-02:

[root@tier2-01 ~]# ~hs/bin/iperf -c dct00.usatlas.bnl.gov -w2M -i2 -t60 -P 10 | grep SUM
[SUM]  0.0- 2.0 sec   175 MBytes   735 Mbits/sec
[SUM]  2.0- 4.0 sec   220 MBytes   925 Mbits/sec
[SUM]  4.0- 8.0 sec   153 MBytes   322 Mbits/sec
[SUM]  4.0- 8.0 sec   180 MBytes   378 Mbits/sec
[SUM]  8.0-10.0 sec   219 MBytes   920 Mbits/sec
[SUM] 10.0-12.0 sec   219 MBytes   920 Mbits/sec
[SUM] 12.0-14.0 sec   219 MBytes   917 Mbits/sec
[SUM] 14.0-16.0 sec   210 MBytes   881 Mbits/sec
[SUM] 16.0-18.0 sec   169 MBytes   709 Mbits/sec
[SUM] 18.0-20.0 sec   168 MBytes   706 Mbits/sec
[SUM] 20.0-22.0 sec   172 MBytes   719 Mbits/sec
[SUM] 22.0-24.0 sec   185 MBytes   776 Mbits/sec
[SUM] 24.0-26.0 sec   209 MBytes   876 Mbits/sec
[SUM] 26.0-28.0 sec   225 MBytes   944 Mbits/sec
[SUM] 28.0-30.0 sec   221 MBytes   927 Mbits/sec
[SUM] 30.0-32.0 sec   210 MBytes   883 Mbits/sec
[SUM] 32.0-34.0 sec   212 MBytes   890 Mbits/sec
[SUM] 34.0-36.0 sec   225 MBytes   942 Mbits/sec
[SUM] 36.0-38.0 sec   220 MBytes   925 Mbits/sec
[SUM] 38.0-40.0 sec   207 MBytes   868 Mbits/sec
[SUM] 40.0-42.0 sec   223 MBytes   937 Mbits/sec
[SUM] 42.0-44.0 sec   215 MBytes   901 Mbits/sec
[SUM] 44.0-46.0 sec   209 MBytes   875 Mbits/sec
[SUM] 46.0-48.0 sec   199 MBytes   836 Mbits/sec
[SUM] 48.0-50.0 sec   219 MBytes   919 Mbits/sec
[SUM] 50.0-52.0 sec   217 MBytes   910 Mbits/sec
[SUM] 52.0-54.0 sec   200 MBytes   840 Mbits/sec
[SUM] 54.0-56.0 sec   217 MBytes   911 Mbits/sec
[SUM] 56.0-58.0 sec   222 MBytes   930 Mbits/sec
[SUM] 58.0-60.0 sec   221 MBytes   926 Mbits/sec
[SUM]  0.0-60.5 sec  6.02 GBytes   854 Mbits/sec

-- HorstSeverini - 06 Nov 2007

About This Site

Please note that this site is a content mirror of the BNL US ATLAS TWiki. To edit the content of this page, click the Edit this page button at the top of the page and log in with your US ATLAS computing account name and password.

Attachments

Admins

USATLAS Site Admin Topics
Admin Support Home
Site Setup Help
Search

Facility Integration Program
Integration home
Proof-Xrootd

Site Admin External Links
dCache
GUMS

Copyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback