Thread debugging tools for Linux

[Federico Lucifredi]

Hello Group Members
I got asked an interesting new question at LinuxWorld in San
Francisco this week: profiling/debugging tools for Linux threads under
(*brace*) pthreads.

Any pointers or experiences to share?

Best -Federico

_________________________________________
-- "'Problem' is a bleak word for challenge" - Richard Fish
(Federico L. Lucifredi) - http://www.lucifredi.com

[Roman Shaposhnick]

On Fri, 2007-08-10 at 22:02 +0000, Federico Lucifredi wrote:
> Hello Group Members
> I got asked an interesting new question at LinuxWorld in San
> Francisco this week: profiling/debugging tools for Linux threads under
> (*brace*) pthreads.
>
> Any pointers or experiences to share?

Hm. I guess I don't really follow the question.

Thanks,
Roman.

[Federico Lucifredi]

heh - I know, we are all spoiled today ;-)

The attendee wanted a tool like the Netbeans thread debugger for
deadlocks, only under pthreads...

Best -F

[Roman Shaposhnik]

On Sat, 2007-08-11 at 00:18 +0000, Federico Lucifredi wrote:
> heh - I know, we are all spoiled today ;-)
>
> The attendee wanted a tool like the Netbeans thread debugger for
> deadlocks, only under pthreads...

Sun Studio 12 has this thing called Thread Analyzer. It runs
on both Solaris and Linux and can be of a help sometimes. Intel
of course has Thread Checker which is an option as well. I would
love to hear about any kind of open source solution in this
area, but currently I don't know of any.

Thanks,
Roman.

[Mike Anderson]

There are a couple of possibilities.  One, you can use a tool like LTTng (Linux Trace Toolkit - Next Generation).  This will help trace
the system calls and the interaction of threads with each other and with the kernel. 

In addition, GDB has been thread aware since 5.2.  Using the "info threads" GDB command will show you the threads that are
active for the process and then you can switch to particular threads using the "th <id>" command and set breakpoints as needed. 

Since Linux uses a 1:1 threading model, each thread is independently schedulable.  Finding the parent PID will allow you to CD
to the /proc/<PID>/tasks directory.  In that directory, there will be a collection of PIDs that correspond to each of the PIDs of the
threads.  Inside each of those directories are the details for the threads (memory maps, open fds, etc.) themselves.

However, if you're running on a multi-core, then threads can migrate from one processor to another.  This means that if you
run with alternate priorities, then even though a thread is a lower priority, it could still potentially run on the alternate
processor core.  Here, LTTng would at least be able to tell you which core any potential thread is running on.

Since I haven't seen the tools that run under Java for tracing threads, I can't give you a comparison of the Java tools
with the tools I've just described.

HTH,

Mike

-- 

---------------------------------------------------------------
Mike Anderson                    703.585.9384
CTO/Chief Scientist              mi...@ThePTRGroup.com
www.ThePTRGroup.com              Embedded, Real-Time Solutions
---------------------------------------------------------------
"Software development is like making a baby.
You can't make a baby in one month by impregnating nine women.
Some things just take time."

[Vasily Levchenko]

As well as Solaris, glibc has libthread_db which widely used for debuging in gdb, but performance guys
 for some reason don't use this functionality which seems to be very useful for Thread Checking.

[Federico Mena Quintero]

On Fri, 2007-08-10 at 22:02 +0000, Federico Lucifredi wrote:

> I got asked an interesting new question at LinuxWorld in San
> Francisco this week: profiling/debugging tools for Linux threads under
> (*brace*) pthreads.

I'm pretty sure valgrind can help you with that.

Federico

[Federico Lucifredi]

Hello Mike!

Mike Anderson wrote:
> However, if you're running on a multi-core, then threads can migrate
> from one processor to another. This means that if you
> run with alternate priorities, then even though a thread is a lower
> priority, it could still potentially run on the alternate
> processor core. Here, LTTng would at least be able to tell you which
> core any potential thread is running on.

Conceivably, you could use CPU affinity to bind the threads to specific
cores, as you would for processes, I am correct?

Best -F

[Roman Shaposhnik]

On Fri, 2007-08-24 at 13:22 -0400, Federico Lucifredi wrote:
> Hello Mike!
>
> Mike Anderson wrote:
> > However, if you're running on a multi-core, then threads can migrate
> > from one processor to another. This means that if you
> > run with alternate priorities, then even though a thread is a lower
> > priority, it could still potentially run on the alternate
> > processor core. Here, LTTng would at least be able to tell you which
> > core any potential thread is running on.
>
> Conceivably, you could use CPU affinity to bind the threads to specific
> cores, as you would for processes, I am correct?

That should be possible, indeed. Although the API can really use
some polishing (that's one of the areas where we had to suffer when
we ported an OpenMP runtime library from Solaris to Linux).

Thanks,
Roman.

[Mike Anderson]

Greetings!

Yes, you should be able to use the get/set affinity calls to nail down particular threads to given processors.  I have to do this
anytime I try to do benchmarking for the embedded space while developing on my core duo.  You can also run the application
using "taskset".  I haven't tried to see if the threads inherit the affinity of the parent process though.  That would be an interesting
thing to try now that I'm back from vacation.

Regards,