Sept 10 Meeting Notes

Skip to first unread message

Alexander Maidak

Sep 11, 2019, 9:37:42 AM9/11/19
New attendee's:

The Deere IBM account Team decided to Pay the LUG a visit, Welcome!
Kaz Koehring
David Lloyd
Doug Allen
John Henley

Next Month is TV-Time-Linux anything involving digital video content and linux:
  * Transcoding
  * Video storage
  * Video streaming
  * Digital content acquisition automation

Chris Cooper presents on Perf

There are counters and tracepoints inside the kernel which the perf tool can read/aggreaget on demand

You install perf with:
 debian: apt-get install linux-tools (a terrible name)
 redhat variants: yum install perf

 You'll want your debug symbols to make reading the perf output easier. symbol resolution with binary objects

There are ~2000 built in events
 - Kernel Tracepoints
 - kprobes
 - uprobes

You can see all the counters with "perf list"

Linux Perf Events Sources JPG:

Events are only instrumented went attached to.

What questions can you answer with perf?
 - Whats using the cpu?
 - What is causing all these context switches?
 - What is using all my disk IO?
 - Where to start optimizing (not just for Devs)?

CPU Profiling: Why not use Top?

"sudo perf record -a -g -F 99 --call-graph dwarf -- sleep 30"
 - "-a" Records all CPU events
 - "-g" record graphs (stack traces)
 - "-F" 99 samples per second
 - Use dwarf for stack traces (as most things omit frame pointers)

dwarf is a way to get stack traces without having frame pointers on
Chris shared an example where htop show 10% usage on CPU but load of 1
read the data with "perf report" - not useful output
You can visual things into a frame graph using brenden greggs tool

sudo perf record -a -g -F 99 -e context-switches -- sleep 30
 -> could visualize with a flamegraph as well
In the era of specter/meltdown context switches are more expensive

Chris drilled into a flamegraph for our education
Chris used one of Brendan's tools to export the perf data into a interactive SVG image which made it a lot easier to interpret.
Chris starts from the bottom when reading a flame graph from the bottom looks for "shelves"
A lot of this comes form Brendan Gregg has lots of cool tools
Any 4.0 > kernel works great
Overheads are getting better with back ends like EBPF
Chris recommends reading Brendan Greg's Linux Perf Stuff

Great presentation Chris!

William Bean

Sep 11, 2019, 10:06:25 AM9/11/19
Thanks for sending out notes, Alex!  And thanks for giving the presentation again, Chris; glad others were able to see this time.

Already excited for next month.


You received this message because you are subscribed to the Google Groups "QCLUG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
To view this discussion on the web visit
Reply all
Reply to author
0 new messages