What linux distribution is better tooled for learning about resource utilization and perf tuning

[Ruslan Rusu]

Hi here,

I'm a beginner in this space. As I read and learn was curious what is the most friendly linux distribution

which is a good if you want to learn about how to track and observe resource utilization. I searched this

list and did not find anything on this topic.

Googling around found a couple of commercial distributions RedHat, Suse which have developer licenses

programs. The binaries under this program are the same as what runs in enterprise license. 

As professional what would you recommend if you had to start again ?   

Appreciate it

[Greg Young]

How well do you already know Linux? Are you comfortable compiling kernels etc or are you just looking for a stock install? While this seems like a trivial detail it's not :)

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/mechanical-sympathy/8d6ed589-bfb5-4b5f-b1ad-ec6026fd6583%40googlegroups.com.

[David]

I can't speak for Java work (which this group seems largely based around), but for systems work in C there is very little to look out for:

1) a distro you can also use in production

2) a kernel release you can use in production

3) a matching build of the 'perf' binary for the kernel you can use in production

4) debug symbols for every component you might happen to touch

5) matching development headers for your kernel and every component you might happen to touch.

In order:

1) is important because there is little point in profiling if you're profiling the wrong code, and most probably reduces your choice to one of RHEL, CentOS, Debian, Ubuntu or SuSE, especially if you are working as part of an organization and someone is making this choice for you, or it has already been made long ago.

2) unless you really, really know what you are doing, and have a double digit percentage of time available to manage it, you almost certainly want to stick with the distro-provided kernel to start with. There are many reasons to consider managing your own kernel config, but a stock kernel is probably good enough, and significant flexibility (like isolcpus=) is possible already through command line parameters without ever needing a custom build.

3) Linux has a huge variety of profilers available, but it seems since the introduction of perf the desire to use any other tooling basically disappears. Perf is annoying because its version really wants to be exactly matched to the kernel version you are running, and so whoever ships that kernel must also provide perf. Any reasonable Linux distro will ship a matching perf, but many smaller distros might goof this up. At least it used to be true that Debian's perf (in the linux-tools package) was not always matched.

4) Perf is vastly more useful when it can print symbols. I only know of 2 distro families with mostly-complete to fully complete symbol repositories: RHEL/CentOS and Debian/Ubuntu. In Debian land most packages have a corresponding "-dbg" package after activating the repository, in Red Hat these are spelled "-dbginfo". Most smaller distros completely punt on providing comprehensive debug symbols, making it much harder to profile any app that inherits any libraries from the base system

5) Some fancier new tooling like bpftrace is so fresh that it requires building stuff from source against matching kernel headers, and in other cases (projects like SystemTap), may want to build kernel modules where you might need the full kernel source tree in use. I think most distros ship this kind of stuff, but it's worth ensuring you know where to find it and how it is installed

[Ruslan Rusu]

I know just enough to install and run things. But his falls short if you answer why my system is slow. It is even worse

when the os you happen to run your stuff does not have all the tooling to debug the root cause :)

Compiling kernels is not something i could reasonable say "yeah, recompiling with x and y will do it"

On Thursday, July 11, 2019 at 11:29:42 PM UTC-4, Greg Young wrote:

How well do you already know Linux? Are you comfortable compiling kernels etc or are you just looking for a stock install? While this seems like a trivial detail it's not :)

On Thu, Jul 11, 2019, 9:53 PM Ruslan Rusu <masquera...@gmail.com> wrote:

Hi here,

I'm a beginner in this space. As I read and learn was curious what is the most friendly linux distribution

which is a good if you want to learn about how to track and observe resource utilization. I searched this

list and did not find anything on this topic.

Googling around found a couple of commercial distributions RedHat, Suse which have developer licenses

programs. The binaries under this program are the same as what runs in enterprise license. 

As professional what would you recommend if you had to start again ?   

Appreciate it

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

[Ruslan Rusu]

David,

Thank you for taking the time to expand this. Your argumented perspective is something i was looking for.

The internet has a lot of info about disros but was very hard to converge on one. 

[Avi Kivity]

One can categorize distributions into three groups:

 - enterprise distributions, focusing on stability. RHEL 7/8 (and its derivatives, CentOS etc.), Ubuntu LTS. Debian stable is similar in its goals but does not have an enterprise backer.

 - bleeding edge distributions that feed the enterprise distributions: Fedora for RHEL, Ubuntu non-LTS for Ubuntu, Debian unstable.

 - everyone else

I recommend using the second type. This allows you to use the latest technologies, so you don't have to to compile things on your own expect if you really need something very recent (and you can often avoid the need to compile even then). Since production use will typically use an enterprise distribution, you will already be comfortable with the tools.

Among the bleeding edge distributions, I recommend Fedora. Subjectively I'm used to it and therefore am most comfortable there, but objectively it is Red Hat that drives much of the Linux performance work, and therefore Fedora receives more attention into making things work than its competition.

However, a lot of convergent evolution has taken place, so whatever you choose will not matter _a lot_. In the end, everyone runs the same things except for the package managers.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.

[Peter Booth]

For me it's a no-brainer. At least 90% of the time my day job is working for a

 large corp that uses RedHat. Most of the startup projects or personal projects

 I've done have been with Centos. It's great to be able to transfer the knowledge

 from one to another, and to know where to look for stuff. As someone pointed out, 

much of the Linux performance work is happening on RedHat one way or another.

On RedHat, if I come across a kernel or driver bug there's a strong possibility

 that someone else will have encountered it before me and a work around can be found online.

So for me, any other choice would be irrational.

[Francesco Nigro]

The problem with RHEL and recentish CentOS is that they use versions of glibc that implements many important feature directly in ASM (for pthread cond wait, spin lock/unlocks etc etc), while Fedora doesn't not. With perf dwarf call graph walker they won't (cannot) appear and won't be accounted correctly so they are not that friendly with such tools..