NFS error while running tcpdump

1,922 views
Skip to first unread message

Nishith Vyas

unread,
Feb 5, 2010, 8:09:49 AM2/5/10
to vglug
Hello,

I have configured NFS Server 1 IBM Web Server. The IBM Web Server can upload all data onto NFS Server. Now, today i was having slow response on IBM Web Server & by measuring the NFS, i found below error while running "tcpdump" command.

tcpdump -n -i eth1 | grep 2049

18:36:37.237451 IP 10.100.19.241.2049 > 10.100.19.88.1758143293: reply ok 1448 read [|nfs]
18:36:37.237476 IP 10.100.19.241.2049 > 10.100.19.88.539981409: reply ERR 1448
18:36:37.237481 IP 10.100.19.241.2049 > 10.100.19.88.796287348: reply ERR 1448
18:36:37.237488 IP 10.100.19.241.2049 > 10.100.19.88.1986098295: reply ERR 1448
18:36:37.237566 IP 10.100.19.241.2049 > 10.100.19.88.539762736: reply ERR 1448
18:36:37.237595 IP 10.100.19.241.2049 > 10.100.19.88.740307248: reply ERR 1448
18:36:37.237599 IP 10.100.19.241.2049 > 10.100.19.88.808333616: reply ERR 208
18:36:37.237792 IP 10.100.19.241.2049 > 10.100.19.88.1758143294: reply ok 1448 read [|nfs]
18:36:37.237810 IP 10.100.19.241.2049 > 10.100.19.88.1700950889: reply ERR 1448
18:36:37.237827 IP 10.100.19.241.2049 > 10.100.19.88.539762735: reply ERR 1332
18:36:37.237866 IP 10.100.19.241.2049 > 10.100.19.88.825830968: reply ERR 1448
18:36:37.237947 IP 10.100.19.241.2049 > 10.100.19.88.808333625: reply ERR 1200
18:36:37.237967 IP 10.100.19.241.2049 > 10.100.19.88.775435808: reply ERR 1448
18:36:37.237973 IP 10.100.19.241.2049 > 10.100.19.88.959330358: reply ERR 1448
18:36:37.237980 IP 10.100.19.241.2049 > 10.100.19.88.908078112: reply ERR 1448
18:36:37.237987 IP 10.100.19.241.2049 > 10.100.19.88.540030496: reply ERR 1448
18:36:37.238041 IP 10.100.19.241.2049 > 10.100.19.88.825897271: reply ERR 1448
18:36:37.238048 IP 10.100.19.241.2049 > 10.100.19.88.540094778: reply ERR 1448
18:36:37.238055 IP 10.100.19.241.2049 > 10.100.19.88.825237553: reply ERR 1448
18:36:37.238109 IP 10.100.19.241.2049 > 10.100.19.88.808529952: reply ERR 1448
18:36:37.238120 IP 10.100.19.88.1758143297 > 10.100.19.241.2049: 116 read [|nfs]
18:36:37.238129 IP 10.100.19.241.2049 > 10.100.19.88.842019120: reply ERR 704
18:36:37.238142 IP 10.100.19.241.2049 > 10.100.19.88.740306806: reply ERR 1448
18:36:37.238158 IP 10.100.19.241.2049 > 10.100.19.88.892805164: reply ERR 1200
18:36:37.238175 IP 10.100.19.241.2049 > 10.100.19.88.1758143288: reply ok 148 setattr [|nfs]
18:36:37.238256 IP 10.100.19.241.2049 > 10.100.19.88.1758143296: reply ok 1448
18:36:37.238263 IP 10.100.19.241.2049 > 10.100.19.88.1869440302: reply ERR 1448
16133 packets captured
23339 packets received by filter
7100 packets dropped by kernel

10.100.18.241 is the IP of NFS Server & 10.100.19.88 IP belongs to IBM Web Server.

Kindly Help




--
Regards,
Nishith N. Vyas

Nirmal Pathak

unread,
Feb 5, 2010, 4:51:02 PM2/5/10
to vg...@googlegroups.com
On 5 February 2010 18:39, Nishith Vyas <nvb...@gmail.com> wrote:
> Hello,
>
> I have configured NFS Server 1 IBM Web Server. The IBM Web Server can upload
> all data onto NFS Server. Now, today i was having slow response on IBM Web
> Server & by measuring the NFS, i found below error while running "tcpdump"
> command.
>
> tcpdump -n -i eth1 | grep 2049
>
> 18:36:37.237451 IP 10.100.19.241.2049 > 10.100.19.88.1758143293: reply ok
> 1448 read [|nfs]
> 18:36:37.237476 IP 10.100.19.241.2049 > 10.100.19.88.539981409: reply ERR
> 1448
> 18:36:37.237481 IP 10.100.19.241.2049 > 10.100.19.88.796287348: reply ERR
> 1448
> 18:36:37.237488 IP 10.100.19.241.2049 > 10.100.19.88.1986098295: reply ERR

> 10.100.18.241 is the IP of NFS Server & 10.100.19.88 IP belongs to IBM Web
> Server.
>

The error "reply ERR 1448" implies that the server's RPC layer is
denying these packets. The 1448 is the packet or perhaps the RPC
section length.

Most likely, 1448 or other numbers means the RPC section length.

This can occur due to bad NIC/module on the system. Try to update the
system to latest kernel available for the distribution or the NIC
itself requires change.

> Kindly Help
>
> Regards,
> Nishith N. Vyas

Have FuN!
--
Nirmal D Pathak.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
What I want is freedom, not life, nor pleasure, nor good!
FOSS gives that.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Nishith Vyas

unread,
Feb 11, 2010, 5:45:36 AM2/11/10
to vglug
I have changed the PC itself & put new IBM Lenovo PC as NFS Server. I did the same configuration of NFS as my previous NFS has.

The scenario is given below.

I have 3 Live Servers & 1 NFS Server. I have shared "/data" partition from NFS, which is mounted onto all 3 Live Servers.

Now, when i mount single server on NFS, then i don't get any error while running "tcpdump -n -i eth1 | grep 2049" command.
But, when i mount 2nd Live Server, immediately the error comes in the picture. I don't know what is the reason behind that.

I am using CentOS 5.4 as NFS Server with Lenovo Desktop. The model number is "M-9439-BW3"  The NIC is integrated into the motherboard.

kindly let me know what type of modifications i need to do & how... Due to this NFS problem, my Live web server performance is becoming poor & website is taking more time to open from outside. (i.e. internet side)

Waiting.
Nishith Vyas
Go Green: Print this mail only if u need it.
Call : +91 9879597301
skype id : nishith16
Ahmedabad-GUJ-IND


Jigar Raval

unread,
Feb 11, 2010, 10:55:54 PM2/11/10
to vg...@googlegroups.com

If possible try by removing sync if in your export on server and also add insecure in export file on server

/data 10.100.19.88 (rw,insecure)


Also, check

AIX clients will default to mounting version 3 NFS over TCP. If your Linux server does not support this, then you may need to specify vers=2 and/or proto=udp in your mount options.

Using netmasks in /etc/exports seems to sometimes cause clients to lose mounts when another client is reset. This can be fixed by listing out hosts explicitly.



Regards
Jigar

--- On Thu, 2/11/10, Nishith Vyas <nvb...@gmail.com> wrote:

> --
>
> Please read http://www.catb.org/~esr/faqs/smart-questions.html
> before posting.
>
> You received this message because you are subscribed to the
> "Vibrant GNU/Linux User Group".
>
> To stop receiving emails from this group, mail to
> VGLUG-un...@googlegroups.com
>
> To post to this group, send email to
> VG...@googlegroups.com
>
> For more options, visit this group at http://groups.google.com/group/VGLUG



Nishith Vyas

unread,
Feb 12, 2010, 12:59:09 AM2/12/10
to vg...@googlegroups.com
This is also not working.

Nishith Vyas

unread,
Feb 12, 2010, 1:19:27 AM2/12/10
to vg...@googlegroups.com
Hello Again,

tcpdump -vvv -n -i eth0 | grep 2049
is giving some detailed output,but overall result is same. Still the error is coming. Find below error.

11:48:09.780239 IP (tos 0x0, ttl  64, id 39559, offset 0, flags [DF], proto: TCP (6), length: 1500) 10.100.19.240.2049 > 10.100.19.86.1812320166: reply ok 1448 read [|nfs]                                                                    
11:48:09.780358 IP (tos 0x0, ttl  64, id 39560, offset 0, flags [DF], proto: TCP (6), length: 1500) 10.100.19.240.2049 > 10.100.19.86.1330861088: reply ERR 1448                                                                               
11:48:09.780374 IP (tos 0x0, ttl  60, id 63191, offset 0, flags [DF], proto: TCP (6), length: 168) 10.100.19.86.1812320168 > 10.100.19.240.2049: 116 read [|nfs]                                                                               
11:48:09.780479 IP (tos 0x0, ttl  64, id 39561, offset 0, flags [DF], proto: TCP (6), length: 1384) 10.100.19.240.2049 > 10.100.19.86.926560820: reply ERR 1332                                                                                
11:48:09.780601 IP (tos 0x0, ttl  64, id 39562, offset 0, flags [DF], proto: TCP (6), length: 1500) 10.100.19.240.2049 > 10.100.19.86.1812320167: reply ok 1448 read [|nfs]                                                                    
11:48:09.780693 IP (tos 0x0, ttl  60, id 63193, offset 0, flags [DF], proto: TCP (6), length: 168) 10.100.19.86.1812320169 > 10.100.19.240.2049: 116 read [|nfs]
11:48:09.780714 IP (tos 0x0, ttl  64, id 39563, offset 0, flags [DF], proto: TCP (6), length: 1500) 10.100.19.240.2049 > 10.100.19.86.908078112: reply ERR 1448
11:48:09.780842 IP (tos 0x0, ttl  64, id 39564, offset 0, flags [DF], proto: TCP (6), length: 1384) 10.100.19.240.2049 > 10.100.19.86.808333625: reply ERR 1332
11:48:09.780952 IP (tos 0x0, ttl  64, id 39565, offset 0, flags [DF], proto: TCP (6), length: 1500) 10.100.19.240.2049 > 10.100.19.86.1812320168: reply ok 1448 read [|nfs]
11:48:09.781063 IP (tos 0x0, ttl  60, id 63195, offset 0, flags [DF], proto: TCP (6), length: 168) 10.100.19.86.1812320170 > 10.100.19.240.2049: 116 read [|nfs]
11:48:09.781097 IP (tos 0x0, ttl  64, id 39566, offset 0, flags [DF], proto: TCP (6), length: 1500) 10.100.19.240.2049 > 10.100.19.86.1835347816: reply ERR 1448
11:48:09.781201 IP (tos 0x0, ttl  64, id 39567, offset 0, flags [DF], proto: TCP (6), length: 1384) 10.100.19.240.2049 > 10.100.19.86.1936290917: reply ERR 1332
11:48:09.781315 IP (tos 0x0, ttl  64, id 39568, offset 0, flags [DF], proto: TCP (6), length: 1500) 10.100.19.240.2049 > 10.100.19.86.1812320169: reply ok 1448 read [|nfs]
11:48:09.781434 IP (tos 0x0, ttl  60, id 63197, offset 0, flags [DF], proto: TCP (6), length: 168) 10.100.19.86.1812320171 > 10.100.19.240.2049: 116 read [|nfs]
11:48:09.781450 IP (tos 0x0, ttl  64, id 39569, offset 0, flags [DF], proto: TCP (6), length: 1500) 10.100.19.240.2049 > 10.100.19.86.1697605743: reply ERR 1448
11:48:09.781586 IP (tos 0x0, ttl  64, id 39570, offset 0, flags [DF], proto: TCP (6), length: 1384) 10.100.19.240.2049 > 10.100.19.86.1697605743: reply ERR 1332
11:48:09.781672 IP (tos 0x0, ttl  64, id 39571, offset 0, flags [DF], proto: TCP (6), length: 1500) 10.100.19.240.2049 > 10.100.19.86.1812320170: reply ok 1448 read [|nfs]
11:48:09.781798 IP (tos 0x0, ttl  60, id 63199, offset 0, flags [DF], proto: TCP (6), length: 168) 10.100.19.86.1812320172 > 10.100.19.240.2049: 116 read [|nfs]
11:48:09.781814 IP (tos 0x0, ttl  64, id 39572, offset 0, flags [DF], proto: TCP (6), length: 1500) 10.100.19.240.2049 > 10.100.19.86.775436320: reply ERR 1448
11:48:09.781926 IP (tos 0x0, ttl  64, id 39573, offset 0, flags [DF], proto: TCP (6), length: 1384) 10.100.19.240.2049 > 10.100.19.86.774977838: reply ERR 1332

My AIX NFS Version is
:-

"mount" Command on AIX

 /dev/hd4         /                jfs2   Nov 21 16:00 rw,log=/dev/hd8
         /dev/hd2         /usr             jfs2   Nov 21 16:00 rw,log=/dev/hd8
         /dev/hd9var      /var             jfs2   Nov 21 16:00 rw,log=/dev/hd8
         /dev/hd3         /tmp             jfs2   Nov 21 16:00 rw,log=/dev/hd8
         /dev/fwdump      /var/adm/ras/platform jfs2   Nov 21 16:01 rw,log=/dev/hd8
         /dev/hd1         /home            jfs2   Nov 21 16:01 rw,log=/dev/hd8
         /proc            /proc            procfs Nov 21 16:01 rw
         /dev/hd10opt     /opt             jfs2   Nov 21 16:01 rw,log=/dev/hd8
         /dev/backuplv    /backup          jfs2   Nov 21 16:01 rw,log=/dev/loglv00
         /dev/oraclelv    /oracle          jfs2   Nov 21 16:01 rw,log=/dev/loglv00

10.100.19.240 /upload/     /upload      nfs3   Feb 12 11:27

Also "nfsstat" command in AIX giving same output.

Version 3: (365616579 calls)
null       getattr    setattr    lookup     access     readlink   read
0 0%       6284111 1% 193382 0%  1318563 0% 730659 0%  0 0%       353731942 96%
write      create     mkdir      symlink    mknod      remove     rmdir
1511215 0% 185493 0%  26974 0%   0 0%       0 0%       102644 0%  84 0%
rename     link       readdir    readdir+   fsstat     fsinfo     pathconf
132247 0%  31 0%      26915 0%   88015 0%   201331 0%  30 0%      0 0%
commit
1082943 0%

The LINUX NFS Server is having below version of NFS.

rpm -qa | grep nfs
system-config-nfs-1.3.23-1.el5
nfs-utils-lib-1.0.8-7.2.z2
nfs-utils-1.0.9-33.el5

rpm -qa | grep portmap
portmap-4.0-65.2.2.1

Kindly let me know if anyone know the solution or provide me possible solution.





On Fri, Feb 12, 2010 at 9:25 AM, Jigar Raval <jiga...@yahoo.com> wrote:

ElectroMech :- Vibrant GNU/Linux User Group

unread,
Feb 12, 2010, 1:31:56 AM2/12/10
to VGLUG
Hi,


On Feb 12, 11:19 am, Nishith Vyas <nvb...@gmail.com> wrote:
> Hello Again,*
>
> tcpdump -vvv -n -i eth0 | grep 2049* is giving some detailed output,but


> overall result is same. Still the error is coming. Find below error.
>
> 11:48:09.780239 IP (tos 0x0, ttl  64, id 39559, offset 0, flags [DF], proto:
> TCP (6), length: 1500) 10.100.19.240.2049 > 10.100.19.86.1812320166: reply
> ok 1448 read
> [|nfs]

can you give us the output of wireshark or tethereal ?

Some time tcpdump messages are difficult to interpret.
>
> read more »

Nilesh J. Vaghela
http://www.electromech.info

Jigar Raval

unread,
Feb 12, 2010, 1:40:46 AM2/12/10
to vg...@googlegroups.com
Dear Nishith,

Have u tried what i sent?

Regards
Jigar

Nishith Vyas

unread,
Feb 12, 2010, 2:25:46 AM2/12/10
to vg...@googlegroups.com
Yes. But, no success.

Nishith Vyas

unread,
Feb 14, 2010, 7:19:28 AM2/14/10
to vg...@googlegroups.com
Dear Nilesh Bhai,

Any Success after verifying Wireshark data?

Jigar Raval

unread,
Feb 17, 2010, 1:28:10 AM2/17/10
to vg...@googlegroups.com

Hello,

Try with proto=tcp in export file on server.

Jigar

Nishith Vyas

unread,
Feb 23, 2010, 3:59:10 AM2/23/10
to vg...@googlegroups.com
Solved.

Jigar Raval

unread,
Feb 23, 2010, 9:50:38 AM2/23/10
to vg...@googlegroups.com

Dear Nishith,

How solved ? What changes you have made ? This will help a lot as document.

Regards
Jigar

Nishith Vyas

unread,
Feb 23, 2010, 1:43:46 PM2/23/10
to vg...@googlegroups.com
Dear Friend,

The NFS Problem is only due to 1 file,which contains 4.1 GB data. As you know that this file is located on my NFS Server,which has connected with Web Servers.

Now, Everytime, when internet users accesses website, it will automatically add 1 line in this file,which increases the file Memory. Actually, this file is meant for website hits,which was really a BAD IDEA,who implemented this scenario.Simply, that file choked up entire network. My website is getting approx 500 hits per minutes. So, you can imagine how big the situation was.

If you remove 100 lines of that file, the website will show less 100 hits.....

Finally, we have made php/html counter (simple adding integer, n+1 like) to replace the big 4.1 GB file.


That's It.

Also tell me howz RHCSS study & practical going?
Reply all
Reply to author
Forward
0 new messages