How to get better debugging for the kernel.

[Zaphod Beeblebrox]

I have just filed bug 211535. I'd like to be more helpful, but I'm having
some trouble with the level of optimization in the kernel. Much of kgdb's
output tells me that the value I'm looking for has been optimized away.

So... how do I compile a kernel or a module without optimization _and_ how
badly will that affect the runtime of a lightly loaded server?

My kernel is already compiled with -g. That tells me that I'm looking in
zap_leaf.c (part of ZFS). Can I simply compile zap_leaf.c without
optimization?
_______________________________________________
freebsd...@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hacke...@freebsd.org"

[Andriy Gapon]

On 03/08/2016 00:35, Zaphod Beeblebrox wrote:
> I have just filed bug 211535. I'd like to be more helpful, but I'm having
> some trouble with the level of optimization in the kernel. Much of kgdb's
> output tells me that the value I'm looking for has been optimized away.
>
> So... how do I compile a kernel or a module without optimization

I have this in my kernel configuration:
makeoptions CONF_CFLAGS="-O1"

Also I have to use amd64-gcc to build the kernel because of a problem
that happens when using clang to compile amd64 kernel modules
(particularly zfs) at that optimization level.
It's as simple as:
- pkg install amd64-xtoolchain-gcc
- add CROSS_TOOLCHAIN=amd64-gcc to the buildkernel invocation

> _and_ how
> badly will that affect the runtime of a lightly loaded server?

I think that nobody would be able to tell you that in advance.
But I expect that you won't see any difference.
Why don't you try and report back? :-)

> My kernel is already compiled with -g. That tells me that I'm looking in
> zap_leaf.c (part of ZFS). Can I simply compile zap_leaf.c without
> optimization?

--
Andriy Gapon

[Ryan Stone]

Are you using the kgdb from the base system or from ports(it's a part of
devel/gdb)? The kgdb in ports is significantly better. If you haven't
tried the version from ports already, definitely do that first.

[Andriy Gapon]

On 03/08/2016 20:14, Ryan Stone wrote:
> Are you using the kgdb from the base system or from ports(it's a part of
> devel/gdb)? The kgdb in ports is significantly better. If you haven't
> tried the version from ports already, definitely do that first.

kgdb 7.x from ports is certainly more powerful than the old base kgdb,
but clang with O2 optimizations seems to be too much even for it.


--
Andriy Gapon

[Edward Tomasz Napierała]

On 0802T1735, Zaphod Beeblebrox wrote:
> I have just filed bug 211535. I'd like to be more helpful, but I'm having
> some trouble with the level of optimization in the kernel. Much of kgdb's
> output tells me that the value I'm looking for has been optimized away.
>

> So... how do I compile a kernel or a module without optimization _and_ how

> badly will that affect the runtime of a lightly loaded server?
>

> My kernel is already compiled with -g. That tells me that I'm looking in
> zap_leaf.c (part of ZFS). Can I simply compile zap_leaf.c without
> optimization?

You can try putting "#pragma clang optimize off" at the top of the file.

[K. Macy]

On Wed, Aug 3, 2016 at 12:53 PM, Andriy Gapon <a...@freebsd.org> wrote:
> On 03/08/2016 20:14, Ryan Stone wrote:
>> Are you using the kgdb from the base system or from ports(it's a part of
>> devel/gdb)? The kgdb in ports is significantly better. If you haven't
>> tried the version from ports already, definitely do that first.
>
> kgdb 7.x from ports is certainly more powerful than the old base kgdb,
> but clang with O2 optimizations seems to be too much even for it.

Samy did a good presentation about this issue. I'm hoping I can get
him to put his slides on line. Evidently clang is much more simplistic
about how it treats callee saved registers. In essence clang will
always err on the side of saying "optimized out" even when it has
sufficient state to know otherwise. Gcc, on the other hand will
sometimes incorrectly infer that a value is valid when it is in fact
not.

I have been building some kernels with clang with dwarf4 enabled (and
thus needed to use kgdb 7.x from ports). Contrary to what I have heard
from some others I have found it to have virtually no added benefit.

There is evidently some mechanism for appending an epilogue to
functions. This would allow us to conditionally force clang to treat
all function argument as being live to the end of the function.

-M

[Zaphod Beeblebrox]

Thanks for the tips. So far, (and this is in diagnosing bug 211535), I've
managed to compile the kernel with -O0. Someone on #FreeBSD told me to do
it with "CFLAGS=-g -O0 make" ... which seems to have worked. At the very
least, I have managed to find a frame on the stack that told me where in
the filesystem was triggering the problem.

Turns out that icinga2 (a monitoring package) manages to create a directory
of 530k small files. It renames() two files each minute into the
directory. At 530k files, this panic's the kernel every 12h or so. At
lower levels of files, it panic's the kernel less.

I have not setup a test system at home to try to induce this ... and if
shutting off icinga2 fixes this (or shutting off the statistics module),
then I'm loathe to keep the production server as a guinea pig, but I do
have dumps for people to look at if they're interested.