debug and profile Lua apps under openresty

[Stefan Parvu]

Hi,

Im reading over http://openresty.org/#Profiling and I see Systemtap used there.
This is 100% Linux and wont help users on FreeBSD or other platforms that Linux.
We been using Debian 7 on our server platform but moved to FreeBSD 10 for ZFS
and better performance. We dont want to go back or test on a system different
than our production.

Questions:

Debugging Lua applications under NGINX:
 - is there anything else than printing or sending the errors to NGINX error log ?
 - howto guides ?


Profiling Lua applications under NGINX:
 - anything else than Systemtap ?
 - DTrace ?
 - else ?

Cheers,
Stefan

[Yichun Zhang (agentzh)]

Hello!

On Wed, Jul 2, 2014 at 2:33 AM, Stefan Parvu wrote:
> Im reading over http://openresty.org/#Profiling and I see Systemtap used
> there.
> This is 100% Linux and wont help users on FreeBSD or other platforms that
> Linux.

FreeBSD has an incomplete dtrace port. But the last time I checked its
userland tracing support is so weak that it's not really useful for
anything really interesting (to me). Things *might* have changed in
recent FreeBSD. You can check it out and try porting those tools over
to FreeBSD's dtrace.

Also, you can use FreeBSD's pmcstat tool to generate a C-land
flamegraph. Check out https://github.com/brendangregg/FlameGraph for
some hints on it.

Another option is to profile on Linux because most of the time the OS
difference does not matter at all. (According to my experiences, 99%
of time it is the application level issues that are OS neutral.) But
yeah, the real strength of dynamic tracing lies in live profiling and
other analysis in production.

> Debugging Lua applications under NGINX:
> - is there anything else than printing or sending the errors to NGINX error
> log ?
> - howto guides ?
>

It's up to you. We usually just check out the error logs though we
also use Raven to collect uncaught Lua exceptions seen online.

>
> Profiling Lua applications under NGINX:
> - anything else than Systemtap ?

Our focus has been on systemtap and gdb (see
https://github.com/openresty/nginx-gdb-utils#readme ). You're welcome
to contribute tools for other operating systems though.

The main obstacle for porting these tools to other OSes (like FreeBSD,
Mac OS X, and Solaris) is that it is pretty hard to find dynamic
tracing tools that are as powerful as systemtap. Even the dtrace on
solaris lacks a lot of important features for userland tracing (I know
the dtrace authors are still trying to improve the userland tracing
support). Not to mention the dtrace ports on Mac OS X and FreeBSD are
way behind Solaris's reference implementation. We need to a *lot* more
work for those systems to get everything we want. Yay for systemtap on
Linux :)

> - DTrace ?

See above.

> - else ?
>

The dynamic tracing era is just booming and we do not have many choices yet :)

It's worth mentioning that we've been trying hard not to plant
advanced debugging and profiling code within the openresty core in
order to prevent messing up the code base and limiting flexibility.
Also, we believe it is crucial to analyze the software stack as a
whole. Because when you want to profile nginx, for example, you may
also want to profile other things (like kernel and other service
processes nginx are talking to) at the same time.

Best regards,
-agentzh

[Stefan Parvu]

FreeBSD has an incomplete dtrace port. But the last time I checked its
userland tracing support is so weak that it's not really useful for
anything really interesting (to me). Things *might* have changed in
recent FreeBSD. You can check it out and try porting those tools over
to FreeBSD's dtrace.

The very first signs of DTrace on FreeBSD are since 2005 or 2006, if Im not wrong.
Currently there are some providers missing comparing to Solaris. This is what I
found on my FreeBSD current:

PROVIDER
callout_execute
dtmalloc
dtrace
fbt
io
ip
lockstat
mac
mac_framework
nfscl
priv
proc
profile
sched
sctp
syscall
tcp
udp
vfs
xbb

But things are improving and for example node.js has nice support for it.
https://blog.indutny.com/7.freebsd-dtrace - I havent looked over your work on
USDT probes - but I will.

DTrace is a nice and powerful language found on many systems by default.
Systemtap is not. Im not gonna start a head to head pro cons list, too busy coding
some Lua things :) but I stopped using systemtap on Linux because: system hungs, long
time to get it installed and functional and a bit laziness of doing previously DTrace.

So now we are off Linux (debian 7) to FreeBSD 10. Super performance and stability vs
Debian systems. OpenResty comes very handy and I need to find time to enhance the
troubleshooting part for it and DTrace on FreeBSD. Lets see if we could do something to add some
nice things for it except the USDT probes.

Another option is to profile on Linux because most of the time the OS
difference does not matter at all. (According to my experiences, 99%
of time it is the application level issues that are OS neutral.) But
yeah, the real strength of dynamic tracing lies in live profiling and
other analysis in production.

It depends. If you plan to observe and measure your system under a certain load then you
are in troubles. You will be troubled by the OS performance metrics between Linux vs FreeBSD
for example being different. Start with the fundamental run-queue length a very important
concept different implemented between OSes. And the list continues. It is *highly* important
to have a continuity between SUT and your PROD.

If you just simple check your application without measuring anything you are right a light
profiling might help. But still would be nice to enhance that part and get it done on FreeBSD.
Hope to get some time for this after we have a full stack running on OpenResty :) Not yet
there.


Thanks.

[Yichun Zhang (agentzh)]

Hello!

On Mon, Jul 14, 2014 at 5:40 AM, Stefan Parvu wrote:
> The very first signs of DTrace on FreeBSD are since 2005 or 2006, if Im not
> wrong.
> Currently there are some providers missing comparing to Solaris.

There were 2 major limitations in dtrace (including dtrace on Solaris)
the last time I checked:

1. Lack of automatic dwarf debuginfo inspection for userland tracing
(it seems that latest dtrace has CTF debuginfo, but you need to get
your compiler to emit CTF. See
http://dtrace.org/blogs/rm/2013/11/14/userland-ctf-in-dtrace/).

2. Lack of looping control flow in the D language. Quite some of my
existing tracing tools have walk through C-land data structures of
varying depth.

>
> But things are improving and for example node.js has nice support for it.
> https://blog.indutny.com/7.freebsd-dtrace

It patches the FreeBSD kernel and is just for USDT. I won't consider
it "nice support" according to my own definition :)

> - I havent looked over your work
> on
> USDT probes - but I will.

The C-land USDT in openresty's nginx should work with both systemtap
and dtrace on FreeBSD, Mac OS X, and Solaris (all tested on my side
for long).

But static probes defined are almost always insufficient and that's
why most of my existing tools in nginx systemtap toolkit and stap++ do
not make use of USDT at all.

>
> DTrace is a nice and powerful language found on many systems by default.
> Systemtap is not.

systemtap has been specific to Linux from day #1. But it does not mean
it does not provide a nice and powerful language :)

My focus has been on Linux because the production systems at $work
have been Linux only. In retrospect, I also feel lucky because the
recent Linux kernels and systemtap fits my advanced requirements
pretty well while I did not have much success with dtrace in other
systems. I might be biased here given that I am also a systemtap
contributor.

I'd love to port my tools over to dtrace if dtrace solve those two big
limitations mentioned earlier. They're quite a showstopper for me.

> but I stopped using systemtap on Linux because: system
> hungs,

That's interesting. We've been frequently using systemtap in our
global network for almost 2 years now and we've never seen such
horrible things :)

Did you report your problems to the systemtap team? The systemtap
developers are very responsive and helpful according to my experiences
:)

> long
> time to get it installed

No longer true for recent Linux kernels (3.5+). Right now in most
Linux distributions it is as simple as a single command installing the
prebuilt systemtap package. Even building systemtap from source is
easy (you only need one dependency, elfutils):

http://openresty.org/#BuildSystemtap

> So now we are off Linux (debian 7) to FreeBSD 10. Super performance and
> stability vs
> Debian systems. OpenResty comes very handy and I need to find time to
> enhance the
> troubleshooting part for it and DTrace on FreeBSD.

Contributions on this part is very welcome :)

> It depends. If you plan to observe and measure your system under a certain
> load then you
> are in troubles. You will be troubled by the OS performance metrics between
> Linux vs FreeBSD
> for example being different.

Sure. Ideally we always trace the same operating system. I was just
talking about a practical work-around that can help find many userland
bottlenecks. Theoretically we may always run into some stupid slow
primitives in certain specific operating system. Most of the time the
userland computation performance is more related to your CPU model and
C compiler/C runtime implementations anyway. (If you're focusing on
scheduler or syscall performance, then it is another story.)

Regards,
-agentzh