Sysconfig App Porn Android
Olegario Benford
Acknowledgment. I would like to thank sysadmins who have sent me the feedback for the version 1.0 of this article. It definitely helped to improve it. Mistakes and errors in the current version are my own.
The key problem with RHEL7 is its adoption of systemd as well as multiple and significant changes in standard utilities and daemons. This looks like Second-system effect: the tendency of small, elegant, and successful systems to be succeeded by over-engineered, bloated systems, due to inflated expectations and overconfidence. Troubleshooting skills needed are different from RHEL6. It also "obsoletes "pre-RHEL 7" books including most good books from O'Reilly and other publishers.
The distance between RHEL6 and RHEL7 is such that we can view RHEL7 as a new flavor of Linux introduced into an enterprise environment. Which increases the load of a sysadmin who previously supported RHEL5,6 and Suse Enterprise 12 30% or more. The amount of headache and hours you need to spend on the job increased, but salaries not so much. The staff of IT departments continues to shrink, despite the growth of the overcomplexity.
Passing RHCSA exam is now also more difficult than for RHEL 6 and requires at least a year of hands-on experience along with self-study even for those who were certified in RHEL6. "Red Hat engineer" certification became useless as complexity is such that no single person can master the whole system. Due to growing overcomplexity tech support for RHEL7 mutated into Red Hat Knowledgebase indexing service. Most problems now require extensive research to solve, hours of reading articles in Red Hat Knowledgebase, documentation, posts in Stack Overflow and similar sites. In this sense "Red Hat engineer" certification probably should be renamed to "Red Hat detective", or something like that.
Networking in Red Hat 7 is very brittle as soon as you go into complex staff such as bonding, special modes of running NFS and like. Usefulness of Network Manager for stationary rack-mounted servers connected by cables is open to review. It is just another layer of overcomplexity imposed on you without any real need for server environment. And it really complicates bonding of interfaces. Sometimes you can't make the configuration work without NM_CONTROLLED="no" directive in the interface file.
This behavior of Red Hat, which amounts to the intentional destruction of the existing server ecosystem by misguided "desktop Linux" enthusiasts within Red Hat developer community, and by Red Hat brass greed, requires some adaptation. Licensing structure of large enterprises regarding Red Hat should probably be changed: tech support should probably be diversified using your hardware vendors such as Dell and HP. If you do not use Oracle Linux it might make sense to add it to the mix, especially on the low end (self-support) and for servers running databases, where you can get a higher quality support from Oracle, than from Red Hat. Oracle provide CentOS style licensing for its distribution and is a natural replacement of CentOS in enterprise. Conversion script is available and works.
Tech support became increasingly self-support anyway. It became in home research over Red Hat knowledgebase and forums such as Stack Overflow. To save money it make sense to switch to wider use self- support licenses (or even CentOS if it works on your type of servers) on the low end, non-critical servers too, which allows buying Premium licenses for the critical servers for the same, or even less amount of money than the uniform licensing structure. Four socket servers now should be avoided due to higher licensing costs (and with the ability to get 32 cores or more on a two socket server they are not very critical, anyway as many application are single threaded and few can utilize more then 16 cores effectively). Docker should be more widely used instead of licensing virtual instances from Red Hat (the limitation on the number of virtual instances is a typical IBM-style way of extracting money from the customers). With the demise of CentOS, only Oracle Linux can fill the gap created by IBM for early Red Hat 8 adopters and that makes more wide adoption of Oracle Linux more attractive than before.
Imagine a language in which both grammar and vocabulary are changing each decade. Add to this that the syntax is complex, the vocabulary is huge and each verb has a couple of dozens of suffixes that often change its meaning, sometimes drastically. Like in Slavic languages;-).
This is the situation of "complexity trap" that we have in enterprise Linux. In this sense troubles with RHEL7 are just a tip of the iceberg. You can learn some subset when you closely work with the particular subsystem (package installation, networking, nfsd,httpd, sshd, Puppet/Ansible, Nagios, and so on and so forth) only to forget vital details after a couple quarters, or a year. I have a feeling that RHEL is spinning out of control at an increasingly rapid pace.
Many older sysadmins now have the sense that they no longer can understand the OS they need to manage and were taken hostages by desktop enthusiasts faction within Red Hat. They now need to work in a kind of hostile environment. Many complain that they feel overwhelmed with this avalanche of changes, and can't troubleshoot problems that they used to able to troubleshoot before. In RHEL7 it is easier for an otherwise competent sysadmin to miss a step, or forget something important in the stress and pressure of the moment even for tasks that he knows perfectly well. That means that SNAFU become more common. Overcomplexity of RHEL7, of course, increases the importance of checklists and personal knowledgebase (most often set of files, or a simple website, or a wiki). Keeping all previous images and restoration from images became a vital troubleshooting method.
But creation of your own knowledgebase and creation of images (for example viaRelax-and-Recover) before each major change requires time that is difficult to find and thus is often skipped: it is not uncommon to spent more time on creating documentation to the problem, than on its resolution. Here such sites as Softpanorama can help, but "document everything" mantra is definitely increasing the overload of sysadmin. At the same this step simply can't be skipped as the situation reached the stage of the inability to remember the set of information necessary for productive work. So keeping your own knowledgebase is a survival tactic.
Now even "the basic set" of commands and utilities in Red Hat is way too large for mere mortals, especially for sysadmins, who need to maintain one additional flavor of Linux (say, Suse Enterprise). Any uncommon task became a research: you need to consult man pages, as well as available on the topic Web pages such as documents at Red Hat portal, Stack Overflow discussion, and other relevant to the problem in hand sites. Often you can't perform again the task that you performed a quarter or two ago without consulting your notes and documentation.
Even worse is the resulting from overcomplexity "primitivization" of your style of work. Sometimes you discover an interesting and/or more productive way to perform an important task. If you do not perform this task frequently and do not write this up, it will soon be displaced in your memory with other things: you will completely forget about it and degrade to the "basic" scheme of doing things. This is very typically for any environment that is excessively complex.
That's probably why so few enterprise sysadmin those days have their personal .profile and .bashrc files and often simply use defaults. Similarlythe way Ansible/Puppet/Chef are used is often a sad joke -- they are used on so primitive level that execution of scripts via NFS would do the same thing with less fuss. The same is true for Nagios which in many cases is "primitivised" to a glorified ping. BTW Ansible, despite all hype is just a reincarnating of IBM JCL on a new, parallelized, level. As such it is OK for simple "waterfall" tasks but it doesn't work well for anything more complex. BTW Unix shell and REXX wiped out the floor with JCL and this is not accidental: they are better ways of doing the same set of tasks.
RHEL7 increased the level of complexity to such a level that it become painful to work with, and especially to troubleshoot complex problems. That's probably why the quality of Red Hat support deteriorated so much (it essentially became referencing service to Red Hat advisories) -- they are overwhelmed, and no longer can concentrate of a single ticket in this river of tickets related to RHEL7 that they receive daily. In many cases it is clear from their answers that they did not even tried to understand the problem you face and just searched the database for keywords.
The adoption of hugely complex, less secure (with a new security exploit approximately once a quarter), and unreliable systemd instead of init looks like Second-system effect: the tendency of small, elegant, and successful systems to be succeeded by over-engineered, bloated systems, due to inflated expectations and overconfidence.
At the same time other changes were also significant enough to classify RHEL 7 as a new flavor of Linux, distinct from RHEL4-RHEL6 family: along with systemd, multiple new utilities and daemons were introduced as well, networking commands and operations are different; also many methods of going common tasks changed, often dramatically. And old methods of administration of the system simply do not work. Especially in the area of troubleshooting.
It might well be the resulting complexity crosses some "red line" (as in "The straw that broke the camel's back ") and instantly started to be visible and annoying to vastly more people then before. In other words, with the addition of systemd quantity turned into quality Look, for example, at the recovery of forgotten sysadmin password in RHEL7 vs. RHEL6. The value of many high quality books about Linux was destroyed by this distribution, and there is no joy to see this vandalism. And you feel somewhat like a prisoner of somebody else decisions: you have essentially no control.
b1e95dc632