Jobserver related developments in 2.6

[Ronald Jeninga]

Hi All,

a lot is happening in the new release we are building. But the changes I want to speak about now, relate to the jobservers.

First of all we eliminated the need for JNI code. This means that the actual (2.6) jobserver is implemented in plain Java.
This simplifies porting the jobserver code to other than Linux systems substantially. The downside is that we now need the jna library.
This library, which is also implemented in plain java and released under the LGPL, can be obtained from

https://maven.java.net/content/repositories/releases/net/java/dev/jna/jna/4.0.0/jna-4.0.0.jar

The github repository of jna.jar can be found at

https://github.com/twall/jna

It might or might not be common knowledge that the jobserver first spawns a jobexecutor process which then executes the command line as specified in the scheduling system.

This architecture is chosen in order to be able to shut down and restart jobservers in a running scheduling system. The information about running processes is kept in so called taskfiles. The jobexecutor writes the progress into the taskfile, which is evaluated by the jobserver and reported back to the scheduling server later on.

A small drawing might explain the previous a bit better (and I apologize for my limited artistic capabilities):

+--------------------+        +-------------+
|                    |        |             |
| Scheduling Server  +--------+  Jobserver  |
|                    |        |             |
+--------------------+        +---+-----+---+
                                  |     |
                     +------------+     |
                     |                  |
              +------+--------+      /--+---\
              |               |      \------/
              | Jobexecutor   +------+ task |
              |               |      | file |
              +------+--------+      \------/
                     |
              +------+--------+
              |               |
              | User Process  |
              |               |
              +---------------+


As long as the jobexecutor processes are kept alive (aren't killed), no information is lost, even if the jobserver and/or scheduling server process are killed with "kill -KILL".

For several reasons we did a rewrite of the jobexecutor. First of all an implementation in C which can be found in $SDMSHOME/src/jobserver/jobexecutor.c and another implementation  in Java, which is located in $SDMSHOME/src/jobserver/jexecutor.
The C implementation should be easily portable between Unix based operating systems. We still have to do the port for Microsoft Windows.
The Java implementation is meant for platforms for which no C port is available (yet), as well as a base for future native support of JBoss and comparable environments.

Since we didn't change the protocol between jobserver and scheduling server, the old 2.5.1 jobservers will still function without problems. And since we didn't change the layout of the taskfiles, the new jobserver can cooperate with the old jobexecutor too (which means that you can do an upgrade in a system with running jobs).

We are also planning to rewrite the jobserver itself. The intention is to do this both in Java and in C. The latter implementation is meant for platforms without sufficient Java support. As a preparation we are developing a C library which can be used to communicate with the scheduling server. This library isn't yet available in github, but will be uploaded as soon as we have finished our developments.

Best regards,

Ronald