MWT2_IU < Admins

r1 - 21 Sep 2007 - 18:11:56 - JosephUrbanskiYou are here: TWiki >

MWT2_IU

Tuning on 21 Sep 2007

In this test we're going to tune from our dCache gridftp gateway that has a 10G Chelsio NIC.

Tracepath to BNL:

[root@iut2-dc3 ~]#  tracepath dct00.usatlas.bnl.gov
 1:  iut2-dc3.iu.edu (149.165.225.214)                      0.076ms pmtu 9000
 1:  no reply
 2:  no reply
 3:  no reply
 4:  no reply
 5:  no reply
 6:  no reply
 7:  no reply
 8:  no reply
 9:  no reply
10:  no reply
11:  no reply
12:  no reply
13:  no reply
14:  no reply
15:  no reply
16:  no reply
17:  no reply
18:  no reply

Config info about the 10G Chelsio(cxgb) NIC we're using:

[root@iut2-dc3 ~]# ifconfig eth3
eth3      Link encap:Ethernet  HWaddr 00:07:43:01:0B:07  
          inet addr:149.165.225.214  Bcast:149.165.225.255  Mask:255.255.255.0
          inet6 addr: fe80::207:43ff:fe01:b07/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:9000  Metric:1
          RX packets:24942277 errors:0 dropped:0 overruns:0 frame:0
          TX packets:18032344 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:2294159851 (2.1 GiB)  TX bytes:3428321056 (3.1 GiB)
          Interrupt:193 Memory:fc8ff000-fc8fffff

ethtool reports for the NIC:

[root@iut2-dc3 ~]# ethtool -k eth3
Offload parameters for eth3:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: on
[root@iut2-dc3 ~]# ethtool -i eth3
driver: cxgb
version: 2.1.4a
firmware-version: N/A
bus-info: 0000:03:03.0
[root@iut2-dc3 ~]# ethtool -g eth3
Ring parameters for eth3:
Pre-set maximums:
RX:             16384
RX Mini:        0
RX Jumbo:       16384
TX:             16384
Current hardware settings:
RX:             4096
RX Mini:        0
RX Jumbo:       512
TX:             1024

First test:

[root@iut2-dc3 ~]#  iperf -c dct00.usatlas.bnl.gov -w4M -i2 -t60
------------------------------------------------------------
Client connecting to dct00.usatlas.bnl.gov, TCP port 5001
TCP window size:   256 KByte (WARNING: requested 4.00 MByte)
------------------------------------------------------------
[  3] local 149.165.225.214 port 42583 connected with 192.12.15.8 port 5001
[  3]  0.0- 2.0 sec  10.6 MBytes  44.3 Mbits/sec
[  3]  2.0- 4.0 sec  12.0 MBytes  50.2 Mbits/sec
[  3]  4.0- 6.0 sec  11.8 MBytes  49.3 Mbits/sec
[  3]  6.0- 8.0 sec  11.5 MBytes  48.3 Mbits/sec
[  3]  8.0-10.0 sec  11.7 MBytes  49.3 Mbits/sec
[  3] 10.0-12.0 sec  11.8 MBytes  49.6 Mbits/sec
[  3] 12.0-14.0 sec  11.6 MBytes  48.5 Mbits/sec
[  3] 14.0-16.0 sec  11.7 MBytes  49.1 Mbits/sec
[  3] 16.0-18.0 sec  11.6 MBytes  48.5 Mbits/sec
[  3] 18.0-20.0 sec  11.8 MBytes  49.5 Mbits/sec
[  3] 20.0-22.0 sec  11.5 MBytes  48.3 Mbits/sec
[  3] 22.0-24.0 sec  11.6 MBytes  48.6 Mbits/sec
[  3] 24.0-26.0 sec  11.8 MBytes  49.5 Mbits/sec
[  3] 26.0-28.0 sec  11.6 MBytes  48.9 Mbits/sec
[  3] 28.0-30.0 sec  11.7 MBytes  49.2 Mbits/sec
[  3] 30.0-32.0 sec  11.6 MBytes  48.7 Mbits/sec
[  3] 32.0-34.0 sec  11.5 MBytes  48.4 Mbits/sec
[  3] 34.0-36.0 sec  11.6 MBytes  48.6 Mbits/sec
[  3] 36.0-38.0 sec  11.7 MBytes  49.1 Mbits/sec
[  3] 38.0-40.0 sec  11.7 MBytes  49.0 Mbits/sec
[  3] 40.0-42.0 sec  11.6 MBytes  48.7 Mbits/sec
[  3] 42.0-44.0 sec  11.6 MBytes  48.8 Mbits/sec
[  3] 44.0-46.0 sec  11.7 MBytes  48.9 Mbits/sec
[  3] 46.0-48.0 sec  11.8 MBytes  49.4 Mbits/sec
[  3] 48.0-50.0 sec  11.7 MBytes  49.1 Mbits/sec
[  3] 50.0-52.0 sec  11.6 MBytes  48.8 Mbits/sec
[  3] 52.0-54.0 sec  11.7 MBytes  49.0 Mbits/sec
[  3] 54.0-56.0 sec  11.7 MBytes  49.3 Mbits/sec
[  3] 56.0-58.0 sec  11.6 MBytes  48.6 Mbits/sec
[  3] 58.0-60.0 sec  11.4 MBytes  47.9 Mbits/sec
[  3]  0.0-60.0 sec    349 MBytes  48.8 Mbits/sec

Changing our window size via editing /etc/sysctl.conf:

# Tuning with Shawn McKee 2007-09-21
net.ipv4.tcp_rmem = 4096 87380 20000000
net.ipv4.tcp_wmem = 4096 87380 20000000
# maximum receive socket buffer size, default 131071
net.core.rmem_max = 20000000
# maximum send socket buffer size, default 131071
net.core.wmem_max = 20000000

...and enable those changes with sysctl -p. Rerunning the test:

[root@iut2-dc3 ~]#  iperf -c dct00.usatlas.bnl.gov -w4M -i2 -t60
------------------------------------------------------------
Client connecting to dct00.usatlas.bnl.gov, TCP port 5001
TCP window size: 8.00 MByte (WARNING: requested 4.00 MByte)
------------------------------------------------------------
[  3] local 149.165.225.214 port 42619 connected with 192.12.15.8 port 5001
[  3]  0.0- 2.0 sec    128 MBytes    539 Mbits/sec
[  3]  2.0- 4.0 sec    205 MBytes    859 Mbits/sec
[  3]  4.0- 6.0 sec  91.1 MBytes    382 Mbits/sec
[  3]  6.0- 8.0 sec  57.7 MBytes    242 Mbits/sec
[  3]  8.0-10.0 sec    189 MBytes    791 Mbits/sec
[  3] 10.0-12.0 sec    190 MBytes    797 Mbits/sec
[  3] 12.0-14.0 sec    192 MBytes    804 Mbits/sec
[  3] 14.0-16.0 sec    205 MBytes    858 Mbits/sec
[  3] 16.0-18.0 sec    224 MBytes    940 Mbits/sec
[  3] 18.0-20.0 sec  78.7 MBytes    330 Mbits/sec
[  3] 20.0-22.0 sec    127 MBytes    532 Mbits/sec
[  3] 22.0-24.0 sec    187 MBytes    784 Mbits/sec
[  3] 24.0-26.0 sec    189 MBytes    793 Mbits/sec
[  3] 26.0-28.0 sec    190 MBytes    795 Mbits/sec
[  3] 28.0-30.0 sec    216 MBytes    908 Mbits/sec
[  3] 30.0-32.0 sec    196 MBytes    823 Mbits/sec
[  3] 32.0-34.0 sec    142 MBytes    597 Mbits/sec
[  3] 34.0-36.0 sec    224 MBytes    940 Mbits/sec
[  3] 36.0-38.0 sec    174 MBytes    728 Mbits/sec
[  3] 38.0-40.0 sec    119 MBytes    499 Mbits/sec
[  3] 40.0-42.0 sec    192 MBytes    804 Mbits/sec
[  3] 42.0-44.0 sec    191 MBytes    802 Mbits/sec
[  3] 44.0-46.0 sec    192 MBytes    806 Mbits/sec
[  3] 46.0-48.0 sec    214 MBytes    899 Mbits/sec
[  3] 48.0-50.0 sec    208 MBytes    872 Mbits/sec
[  3] 50.0-52.0 sec  58.9 MBytes    247 Mbits/sec
[  3] 52.0-54.0 sec    189 MBytes    794 Mbits/sec
[  3] 54.0-56.0 sec    192 MBytes    804 Mbits/sec
[  3] 56.0-58.0 sec    192 MBytes    805 Mbits/sec
[  3] 58.0-60.0 sec    204 MBytes    857 Mbits/sec
[  3]  0.0-60.0 sec  5.04 GBytes    721 Mbits/sec

Statistics:

[root@iut2-dc3 ~]# ifconfig eth3
eth3      Link encap:Ethernet  HWaddr 00:07:43:01:0B:07  
          inet addr:149.165.225.214  Bcast:149.165.225.255  Mask:255.255.255.0
          inet6 addr: fe80::207:43ff:fe01:b07/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:9000  Metric:1
          RX packets:26120940 errors:0 dropped:0 overruns:0 frame:0
          TX packets:22047253 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:2371222724 (2.2 GiB)  TX bytes:827351383 (789.0 MiB)
          Interrupt:193 Memory:fc8ff000-fc8fffff 

[root@iut2-dc3 ~]# ethtool -S eth3 > post_eth3_stats.log
[root@iut2-dc3 ~]# diff post_eth3_stats.log initial_eth3_stats.log 
2c2
<      TxOctetsOK: 13712551747
---
>      TxOctetsOK: 7725808340
4,6c4,6
<      TxUnicastFramesOK: 21824917
<      TxMulticastFramesOK: 220906
<      TxBroadcastFramesOK: 2102
---
>      TxUnicastFramesOK: 17819160
>      TxMulticastFramesOK: 220591
>      TxBroadcastFramesOK: 2101
19c19
<      RxOctetsOK: 10961508423
---
>      RxOctetsOK: 10890419362
21,23c21,23
<      RxUnicastFramesOK: 17195270
<      RxMulticastFramesOK: 8807136
<      RxBroadcastFramesOK: 119262
---
>      RxUnicastFramesOK: 16040320
>      RxMulticastFramesOK: 8797219
>      RxBroadcastFramesOK: 119169
38c38
<      TSO: 388170
---
>      TSO: 369010
41,42c41,42
<      RxCsumGood: 17002997
<      TxCsumOffload: 8859037
---
>      RxCsumGood: 15848339
>      TxCsumOffload: 5416811
52,54c52,54
<      tx_reg_pkts: 19409027
<      tx_lso_pkts: 388170
<      tx_do_cksum: 8859037
---
>      tx_reg_pkts: 15955623
>      tx_lso_pkts: 369010
>      tx_do_cksum: 5416811

-- Main.jau - 21 Sep 2007

About This Site

Please note that this site is a content mirror of the BNL US ATLAS TWiki. To edit the content of this page, click the Edit this page button at the top of the page and log in with your US ATLAS computing account name and password.

Attachments

Admins

USATLAS Site Admin Topics
Admin Support Home
Site Setup Help
Search

Facility Integration Program
Integration home
Proof-Xrootd

Site Admin External Links
dCache
GUMS

Copyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback