[Samba] How to troubleshoot SMBD high CPU?

[Jeremy T. Cherny]

We are seeing the SMBD process sitting at 99.9%

Are there any step-by-step troubleshooting guides to track the source of the problem?


-Jeremy
--
To unsubscribe from this list go to the following URL and read the
instructions: https://lists.samba.org/mailman/options/samba

[Steve Litt]

On Fri, 24 Feb 2012 13:01:00 +0000
"Jeremy T. Cherny" <Jeremy...@tobinsolutions.com> wrote:

> We are seeing the SMBD process sitting at 99.9%
>
> Are there any step-by-step troubleshooting guides to track the source
> of the problem?
>
>
> -Jeremy

Hi Jeremy,

My familiarity with Samba is from 12 years ago, and I know of no
specific troubleshooting guide for to-active smbd processes, but
perhaps I can give you some ideas for general troubleshooting.

Collect a symptom description. Does this always happen? Is there a set
of steps that will make it happen? How long does it take to re-happen
after restarting Samba? When was this symptom first noticed? What else
was going on around that time? Do you have multiple smbd processes, or
just one?

Look at the logs. Anything look odd or out of place? Look for
difference in log patterns between the times this symptom is occurring
and times this symptom isn't occurring.

Look for other occurrences in Google. The phrase "smbd cpu usage high"
produces some hits -- I didn't look at them, but doing so much cut your
troubleshooting time considerably.

If practical, you can start by disconnecting connected machines, one by
one, and see if it goes down. Perhaps one process on one computer is
doing it.

Best of luck.

SteveT

[Jeremy Allison]

On Fri, Feb 24, 2012 at 01:01:00PM +0000, Jeremy T. Cherny wrote:
> We are seeing the SMBD process sitting at 99.9%
>
> Are there any step-by-step troubleshooting guides to track the source of the problem?

1). Use strace -p <pid> to find out what system calls are being
done.
2). Attach with gdb and get a backtrace.

You need to do both in order to get a good idea of what
might be going on.

What Samba version is this ?

Jeremy.

[Jeremy T. Cherny]

Hi - I finally figured out how to the ipkg version of the tools installed and right now there are 3 smbd processes stuck at 99.9 %


Results of strace:

splice(0x1c, 0, 0x12, 0xbff4ee30, 0xbf70, 0) = 0
Repeated


Results of gdb with backtrace:

#0 0xb73edf33 in splice () from /lib/libc.so.6
#1 0x0839c0a0 in sys_recvfile ()
#2 0x08189f5b in ?? ()
#3 0x0000001c in ?? ()
#4 0x00000012 in ?? ()
#5 0x06760000 in ?? ()
#6 0x00000000 in ?? ()

Any ideas on what the problem may be?

Thanks.

-Jeremy

[Volker Lendecke]

On Fri, Mar 02, 2012 at 06:53:15PM +0000, Jeremy T. Cherny wrote:
> Hi - I finally figured out how to the ipkg version of the
> tools installed and right now there are 3 smbd processes
> stuck at 99.9 %
>
>
> Results of strace:
>
> splice(0x1c, 0, 0x12, 0xbff4ee30, 0xbf70, 0) = 0
> Repeated
>
>
> Results of gdb with backtrace:
>
> #0 0xb73edf33 in splice () from /lib/libc.so.6
> #1 0x0839c0a0 in sys_recvfile ()
> #2 0x08189f5b in ?? ()
> #3 0x0000001c in ?? ()
> #4 0x00000012 in ?? ()
> #5 0x06760000 in ?? ()
> #6 0x00000000 in ?? ()
>
> Any ideas on what the problem may be?

The most likely problem is that our recvfile code is broken.
Try disabling recvfile options.

With best regards,

Volker

--
SerNet GmbH, Bahnhofsallee 1b, 37081 Göttingen
phone: +49-551-370000-0, fax: +49-551-370000-9
AG Göttingen, HRB 2816, GF: Dr. Johannes Loxen
http://www.sernet.de, mailto:kon...@sernet.de

[Jeremy Allison]

On Fri, Mar 02, 2012 at 11:20:06AM -0800, Jeremy Allison wrote:

> On Fri, Mar 02, 2012 at 07:56:16PM +0100, Volker Lendecke wrote:
> > On Fri, Mar 02, 2012 at 06:53:15PM +0000, Jeremy T. Cherny wrote:
> > > Hi - I finally figured out how to the ipkg version of the
> > > tools installed and right now there are 3 smbd processes
> > > stuck at 99.9 %
> > >
> > >
> > > Results of strace:
> > >
> > > splice(0x1c, 0, 0x12, 0xbff4ee30, 0xbf70, 0) = 0
> > > Repeated
> > >
> > >
> > > Results of gdb with backtrace:
> > >
> > > #0 0xb73edf33 in splice () from /lib/libc.so.6
> > > #1 0x0839c0a0 in sys_recvfile ()
> > > #2 0x08189f5b in ?? ()
> > > #3 0x0000001c in ?? ()
> > > #4 0x00000012 in ?? ()
> > > #5 0x06760000 in ?? ()
> > > #6 0x00000000 in ?? ()
> > >
> > > Any ideas on what the problem may be?
> >
> > The most likely problem is that our recvfile code is broken.
> > Try disabling recvfile options.
>

> Either that or splice() is broken on his system, which I
> think is more likely :-).
>
> Anyway, disabling recvfile should fix it.

Ah, yes - remembering you telling me what the platform
is, I'm 100% sure this is a kernel problem, not a Samba
one.

Jeremy.

[Jeremy Allison]

On Fri, Mar 02, 2012 at 07:56:16PM +0100, Volker Lendecke wrote:

> On Fri, Mar 02, 2012 at 06:53:15PM +0000, Jeremy T. Cherny wrote:
> > Hi - I finally figured out how to the ipkg version of the
> > tools installed and right now there are 3 smbd processes
> > stuck at 99.9 %
> >
> >
> > Results of strace:
> >
> > splice(0x1c, 0, 0x12, 0xbff4ee30, 0xbf70, 0) = 0
> > Repeated
> >
> >
> > Results of gdb with backtrace:
> >
> > #0 0xb73edf33 in splice () from /lib/libc.so.6
> > #1 0x0839c0a0 in sys_recvfile ()
> > #2 0x08189f5b in ?? ()
> > #3 0x0000001c in ?? ()
> > #4 0x00000012 in ?? ()
> > #5 0x06760000 in ?? ()
> > #6 0x00000000 in ?? ()
> >
> > Any ideas on what the problem may be?
>
> The most likely problem is that our recvfile code is broken.
> Try disabling recvfile options.

Either that or splice() is broken on his system, which I
think is more likely :-).

Anyway, disabling recvfile should fix it.

Jeremy.

[Jeremy T. Cherny]

------------------------------------

Thank you for the advice. Is disabling recvfile something done in a config file? Would that still be a fix if it is a kernel issue?

-Jeremy