Why Is Read and Write Speed Slower on Nfs Share
I've got a NetApp as my nfs server, and ii Linux servers equally the nfs clients. The problem is that the newer of the two servers has extremely differing read and write speeds whenever information technology is doing read and writes simultaneously to the nfs server. Separately, reads and writes await swell for this new server. The older server does not have this outcome.
One-time host: Carp
Sun Fire x4150 with w/ 8 cores, 32 GB RAM
SLES 9 SP4
Network driver: e1000
me@bother:~> uname -a Linux carp 2.vi.5-7.308-smp #ane SMP Mon Dec 10 11:36:forty UTC 2007 x86_64 x86_64 x86_64 GNU/Linux New host: Pepper
HP ProLiant Dl360P Gen8 due west/ 8 cores, 64 GB RAM
CentOS 6.3
Network driver: tg3
me@pepper:~> uname -a Linux pepper 2.half-dozen.32-279.el6.x86_64 #one SMP Fri Jun 22 12:19:21 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux I'll jump to some graphs illustrating the read/write tests. Heres pepper and its unbalanced read/write:
and hither is carp, lookin' good:
The tests
Hither are the read/write tests I am running. I've run these separately and they wait great on pepper, but when run together (using the &), the write functioning remains solid while the read functioning suffers profoundly. The test file is twice the size of the RAM (128 GB for pepper, and 64 GB was used for carp).
# write fourth dimension dd if=/dev/zero of=/mnt/peppershare/testfile bs=65536 count=2100000 & # read time dd if=/mnt/peppershare/testfile2 of=/dev/zilch bs=65536 & The NFS server hostname is nfsc. The Linux clients have a defended NIC on a subnet thats split up from annihilation else (i.e. different subnet than primary IP). Each Linux client mounts an nfs share from server nfsc to /mnt/hostnameshare.
nfsiostat
Heres a i-minute sample during pepper's simul r/westward exam:
me@pepper:~> nfsiostat 60 nfsc:/vol/pg003 mounted on /mnt/peppershare: op/due south rpc bklog 1742.37 0.00 read: ops/s kB/s kB/op retrans avg RTT (ms) avg exe (ms) 49.750 3196.632 64.254 0 (0.0%) 9.304 26.406 write: ops/south kB/south kB/op retrans avg RTT (ms) avg exe (ms) 1642.933 105628.395 64.293 0 (0.0%) iii.189 86559.380 I don't have nfsiostat on the former host carp even so, but working on it.
/proc/mounts
me@pepper:~> true cat /proc/mounts | grep peppershare nfsc:/vol/pg003 /mnt/peppershare nfs rw,noatime,nodiratime,vers=iii,rsize=65536,wsize=65536,namlen=255,acregmin=0,acregmax=0,acdirmin=0,acdirmax=0,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=172.x.x.10,mountvers=iii,mountport=4046,mountproto=tcp,local_lock=none,addr=172.x.x.x 0 0 me@carp:~> cat /proc/mounts | grep carpshare nfsc:/vol/pg008 /mnt/carpshare nfs rw,v3,rsize=32768,wsize=32768,acregmin=0,acregmax=0,acdirmin=0,acdirmax=0,timeo=60000,retrans=3,hard,tcp,lock,addr=nfsc 0 0 Network card settings
me@pepper:~> sudo ethtool eth3 Settings for eth3: Supported ports: [ TP ] Supported link modes: 10baseT/Half 10baseT/Total 100baseT/One-half 100baseT/Full 1000baseT/One-half 1000baseT/Total Supports machine-negotiation: Aye Advertised link modes: 10baseT/One-half 10baseT/Full 100baseT/Half 100baseT/Total 1000baseT/Half 1000baseT/Full Advertised interruption frame use: Symmetric Advertised auto-negotiation: Yes Speed: 1000Mb/s Duplex: Total Port: Twisted Pair PHYAD: 4 Transceiver: internal Auto-negotiation: on MDI-X: off Supports Wake-on: g Wake-on: g Current message level: 0x000000ff (255) Link detected: yep me@carp:~> sudo ethtool eth1 Settings for eth1: Supported ports: [ TP ] Supported link modes: 10baseT/Half 10baseT/Full 100baseT/One-half 100baseT/Full 1000baseT/Full Supports auto-negotiation: Yes Advertised link modes: 10baseT/One-half 10baseT/Full 100baseT/Half 100baseT/Total 1000baseT/Full Advertised auto-negotiation: Yes Speed: 1000Mb/s Duplex: Full Port: Twisted Pair PHYAD: 1 Transceiver: internal Automobile-negotiation: on Supports Wake-on: umbg Wake-on: one thousand Current message level: 0x00000007 (7) Link detected: yes Offload settings:
me@pepper:~> sudo ethtool -chiliad eth3 Offload parameters for eth3: rx-checksumming: on tx-checksumming: on scatter-get together: on tcp-segmentation-offload: on udp-fragmentation-offload: off generic-division-offload: on generic-receive-offload: on large-receive-offload: off me@carp:~> # sudo ethtool -k eth1 Offload parameters for eth1: rx-checksumming: on tx-checksumming: on besprinkle-assemble: on tcp partitioning offload: on Its all on a LAN with a gigabit switch at full duplex between the nfs clients and nfs server. On another note, I run into quite a bit more IO wait on the CPU for pepper than bother, as expected since I suspect its waiting on nfs operations.
I've captured packets with Wireshark/Ethereal, but I'm not strong in that area, and so not sure what to look for. I don't see a bunch of packets in Wireshark that are highlighted in red/black, so thats about all I looked for :). This poor nfs performance has manifested in our Postgres environments.
Any farther thoughts or troubleshooting tips? Allow me know if I can provide further information.
UPDATE
Per @ewwhite's comment, I tried ii different tuned-adm profiles, but no alter.
To the right of my cherry mark are 2 more tests. The showtime hill is with the throughput-performance and the second is with enterprise-storage.
nfsiostat 60 of enterprise-storage profile
nfsc:/vol/pg003 mounted on /mnt/peppershare: op/due south rpc bklog 1758.65 0.00 read: ops/s kB/s kB/op retrans avg RTT (ms) avg exe (ms) 51.750 3325.140 64.254 0 (0.0%) eight.645 24.816 write: ops/s kB/s kB/op retrans avg RTT (ms) avg exe (ms) 1655.183 106416.517 64.293 0 (0.0%) iii.141 159500.441 Update 2
sysctl -a for pepper
Source: https://serverfault.com/questions/474571/nfs-client-has-unbalanced-read-and-write-speeds
0 Response to "Why Is Read and Write Speed Slower on Nfs Share"
Post a Comment