Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

[ftrace] possible to implement user-space tracers?

1,444 views
Skip to first unread message

Kun Huang

unread,
Aug 11, 2015, 10:00:08 AM8/11/15
to
Hi ftrace developers

I'm developing a huge python based programs and facing performance
issue everyday. I like the ftrace system and hope there could be a
tracer to trace&report my python codes. Is it possible or is it worthy
to do this?

---

Kun
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majo...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Scotty Bauer

unread,
Aug 11, 2015, 3:20:11 PM8/11/15
to
It is possible to trace from userland, Android does it.

Essentially you need to write your data into
/sys/kernel/debug/tracing/trace_marker

then read it out of /sys/kernel/debug/tracing/trace


If you care how the implementation works you can read it in
/kernel/tracing/trace.c
(http://lxr.free-electrons.com/source/kernel/trace/trace.c) search for
tracing_mark_fops and tracing_fops.


In Android here are the relevant files you will need to see how it's
used in userland:
Atrace (for reading data out + setting things up):
https://android.googlesource.com/platform/frameworks/native/+/master/cmds/atrace/atrace.cpp
https://android.googlesource.com/platform/external/chromium-trace/+/master
(host side setup of the device)

trace-dev (for writing data in):
https://android.googlesource.com/platform/system/core/+/master/libcutils/trace-dev.c

Once we have all the data we usually run it through trace-viewer,
(https://github.com/catapult-project/catapult)

Cheers.
> [6]
>> Please read the FAQ at http://www.tux.org/lkml/ [7]
>
> Links:
> ------
> [1] http://lxr.free-electrons.com/source/kernel/trace/trace.c
> [2]
> https://android.googlesource.com/platform/frameworks/native/+/master/cmds/atrace/atrace.cpp
> [3]
> https://android.googlesource.com/platform/external/chromium-trace/+/master
> [4]
> https://android.googlesource.com/platform/system/core/+/master/libcutils/trace-dev.c
> [5] https://github.com/catapult-project/catapult
> [6] http://vger.kernel.org/majordomo-info.html
> [7] http://www.tux.org/lkml/

Kun Huang

unread,
Aug 11, 2015, 10:30:06 PM8/11/15
to
Thank you for your help Scotty :) I'm reading it.

Kun Huang

unread,
Aug 11, 2015, 11:50:06 PM8/11/15
to
Hi Scotty

I have read your links. I found the read from 'trace' and the write to
'trace_marker' are normal read&write which works like proc system.

I have questions that what's more difference between ftrace and proc?
I just know I could read data as pipeline from ftrace.

Scotty

unread,
Aug 12, 2015, 12:00:08 AM8/12/15
to
Hi Kun,



On 08/11/2015 08:42 PM, Kun Huang wrote:
> Hi Scotty
>
> I have read your links. I found the read from 'trace' and the write to
> 'trace_marker' are normal read&write which works like proc system.
>
> I have questions that what's more difference between ftrace and proc?
> I just know I could read data as pipeline from ftrace.

Well, in kernel land you can trace your functions, with very little over head. For example, in the binder driver (http://lxr.free-electrons.com/source/drivers/android/binder.c#L2423) there are trace events everywhere throughout the code. The thing is you need to build the traces into your code, if you're going to use it.

When you're in userland one way to trace your application flow is by making a wrapper around the debugfs entry (like android does) and manually write()'ing the data into the kernel buffer -- This is what android does.

Why ftrace is in debugfs and not procfs? I'm not sure you'll have to ask the original developers, I'm sure there is a good reason.

Perhaps I didn't understand your use case. Did you want to see what was happening in kernel land while you were running your python project? Or do you want to be able to trace your python functions like the way ftrace can trace kernel functions during execution?

Kun Huang

unread,
Aug 12, 2015, 2:40:05 AM8/12/15
to
I need trace python projects, with low over head and a little flexibility.

There are some pure python based trace tools, but most of them make
codes slower. I also tried systemtap, but it still become much slower
in complicated python codes. I'm now finding a better tool or some
inspirations.

Flexibility for me means that operators could write some scripts to
choose what to trace. Someone traces function A, and someone traces B
for other purposes. Using the android case as example, I think I could
control the format of input data to manage trace objects. However I'm
not confident with this idea.
0 new messages