Re: Systems Programming By John J Donovan Pdf Free Download
Agathe Thies
Systems are built from hardware and software components. Systems programming is about implementing these components, their interfaces and the overall architecture. Individual components perform their prescribed functions and at the same time work together to form a stable and efficient system.
Systems programming is distinct from application programming. System programs provide services to other software. Via abstractions, they expose API to simplify the development of applications. They're often optimized to low-level machine architecture. Unlike application software, most system software are not directly used by end users.
Operating systems, device drivers, BIOS and other firmware, compilers, debuggers, web servers, database management systems, communication protocols and networking utilities are examples of system software. As part of operating systems, memory management, scheduling, event handling and many more essential functions are done by system software. Examples of application software are Microsoft Office, web browsers, games, and graphics software.
Application software generally don't directly access hardware or manage low-level resources. They do so via calls to system software. We may thus view system software as aiding application software with low-level access and management. Application developers can therefore focus on the business logic of their applications.
While application developers may not build system software, they can become better developers by knowing more about the design and implementation of system software. Specifically, by knowing and using system APIs correctly they can avoid implementing similar functions in their applications.
An operating system such as Linux is a collection of system programs. These deal with files and directories, manage processes for executable programs, enable I/O for those programs and allocate/release memory as needed. The OS manages users, groups and associated permissions. The OS prevents normal user programs from executing privileged operations. The shell is a special system program that allows users to interact with the system. If processes need to communicate or respond to external events, signals (aka software interrupts) facilitate this.
There are systems programs that transform other programs into machine-level instructions for execution: compilers, assemblers, macro processors, loaders and linkers. Their aim is to generate instructions optimized for speed or memory. Use of registers, loops, data structures, and algorithms are considered in these system programs.
Programming languages, editors and debuggers are also system programs. These are tools to write good and reliable system programs. They have to be easy to learn and productive for a developer while also being efficient and safe from a system perspective.
A system programmer is one who write system software and this task is called system programming or systems programming. But there's an older definition that's been used in the context of mainframes since the 1960s.
On a mainframe, a system programmer installs, upgrades, configures and maintains the operating system. She does capacity planning and evaluates new products. She's also skilled in optimizing the system for performance, troubleshooting problems and analyzing memory dumps.
On the other hand, a system administrator handles day-to-day operations such as adding/removing users, configuring access, installing software, and monitoring performance. She deals with applications whereas a system programmer is more well-versed with the mainframe hardware.
System programming languages are required to provide direct access to hardware including memory and I/O access. Performance, explicit memory management and fine-grained control at bit level are essential capabilities. Where they also offer high-level programming constructs, system programming languages (C, Rust, Swift, etc.) are also used for application programming.
Since such languages give access to low-level hardware functions, there's a risk of introducing bugs. Rust was created to balance aspects of both safety and control. C sacrifices safety for control while Java does the opposite.
Scripting languages such as Python, JavaScript and Lua are not for systems programming. However, the introduction of static typing (for safety) and Just-in-Time (JIT) compilation (for speed) has seen these languages being used for systems programming.
On the web, applications adopt the client-server architecture. Server-side logic may be implemented as many microservices distributed on a cloud platform. Applications make use of APIs served by different endpoints. In this context, we have systems programs that help create distributed applications for the cloud. System programs must be designed to address network topology changes, security concerns, high latency, and poor network connections.
Rob Pike commented that developers thought that Go was for systems programming. In fact, it was for writing any server-side code. Later he saw anything running on the cloud as systems software. Ousterhout commented in 1998 that on the web Java was being used for systems programming.
Bunardzic commented that "the only way to program a system is to program a network". Systems should be designed for concurrency, fault encapsulation/detection/recovery, upgrade without downtime, observability, and asynchronous communication. Developers who use system services and APIs must be free to choose their own technology stack. One service shouldn't depend on others.
Designing a good system is not a one-time task. The system should be designed for iteration and incremental improvements. System designers must be open to feedback. Historically, many Linux system programmers blamed application developers for program crashes instead of seeing these as opportunities to improve the Linux kernel or system tools.
All assumptions must be made explicit. Systems software should get the abstractions right, minimize if not avoid leaky abstractions. In other words, its users shouldn't need to know implementation details. For instance, ORM frameworks that interface between applications and databases are often leaky due to a conceptual mismatch between objects and relations.
Before implementation, it's beneficial to do system modelling, analysis and simulations. Unified Modelling Language (UML) can help. Small and simple systems are amenable to formal analysis. Anything else, we need to apply statistical analysis. More components there are, more complex becomes the system.
Till early and even mid-1960s, the first concern in designing computer systems is the hardware itself. Programming them becomes a secondary concern. Programming techniques are chaotic. Often they're not as intended by the hardware designers. Systems programming as a discipline is only starting to emerge.
Shaw considers assemblers, interpreters, compilers, and monitors as translators; that is, they translate code in one form to another. Referring to the figure, translator T translates A to B. He defines the following,
At the NATO Conference on Software Engineering, the merits of high-level languages are discussed: cost, maintainability, correctness. System programmers however object to high-level languages. They don't like anything to get in between themselves and the machine. Reconciling these two concerns is the main challenge in design a suitable systems programming language.
Unix operating system is invented at Bell Laboratories, first in assembly on PDP-7 (1969) and subsequently migrated to C on PDP-11 (1971). A decade later one of its inventors, Dennis Ritchie, comments that only the assembler is still written in assembler. Most other system programs are written in C language.
Researchers at the Carnegie-Mellon University invent a new systems programming language that they name BLISS. They describe it as a general purpose high-level language for writing large software systems for specific machines. They enumerate the requirements of any good systems programming language: space/time economy, access to hardware features, data structures, control structures, understandability, debugging, etc. The figure shows an example of how BLISS enables bit-level access. Such low-level access is typical of systems software.
Weicker proposes Dhrystone as a benchmark for systems programming. Currently we have the Whetstone benchmark that's tuned to measure floating-point arithmetic. Systems programs often use enumerations, records and pointer data types. Compared to numerical programs, systems programs have fewer loops, simpler compute statements, more conditional branching and more procedure calls. The Dhrystone benchmark accounts for these.
An IBM research report details the challenges and pitfalls in programming for a multi-CPU and multi-threaded environment. They refer to the System/370 architecture. Shared memory and data structures have to be handled carefully. Therefore, parallelism is a new concern for systems programming. However, multi-programming was considered in the design of Unix back in the early 1970s.
This decade sees the wide adoption of scripting languages such as Perl, Tcl, Ruby, PHP, Python and JavaScript. They're seen as languages that "glue together" various components whereas systems programming languages are used to create those components. In the 2010s, the boundaries blur as some scripting languages are used to build large systems software.
Wolfe proposes changes to how systems programming must be taught to students. The curriculum would include small challenges in various aspects of systems programming and not just focus on assemblers and compilers. Currently, students perceive systems programming as boring, the techniques themselves stale, and the courses disconnected from systems programming jobs out there.
Brewer et al. note that 35 years after the invention of C language, we still don't have a suitable alternative for systems programming. Shapiro voices a similar complaint in 2006. Advances in programming languages have not motivated systems programmers to adopt them.
7fc3f7cf58