Skip to first unread message
Mohamed Abdallah
Mar 16, 2016, 4:41:02 AM3/16/16
to schedulix
Hi all,
I have trouble understanding the concepts used in schedulix such as job server, resources, ..etc and the only reference I can find is the official documentation but it is not organized in a good way to introduce the concepts but rather as a complete reference.
I wonder if there is any good tutorials for schedulix that can help to understand its concepts and the relations between them in a better way.
Any help is appreciated,
Mohamed Abdallah
I have trouble understanding the concepts used in schedulix such as job server, resources, ..etc and the only reference I can find is the official documentation but it is not organized in a good way to introduce the concepts but rather as a complete reference.
I wonder if there is any good tutorials for schedulix that can help to understand its concepts and the relations between them in a better way.
Any help is appreciated,
Mohamed Abdallah
Ronald Jeninga
Mar 16, 2016, 12:50:04 PM3/16/16
to schedulix
Hi Mohamed,
if I answer questions here, most of the time I try to provide some insight in the concepts of schedulix.
We are aware that the documentation is more like a reference than like anything else. Still it is valuable. I myself use the documentation, both the syntax doc as well as the gui doc, every now and then.
Currently we are doing some youtube videos to explain some basic concepts. Due to our background, the videos are done in German first, but English versions will follow soon.
If the videos are made public, I'll definitely drop a note here in this group. Right now the videos target on BICsuite, but since they are very basic, everything applies to schedulix as well. At that level the main difference is the more blueish than greenish GUI.
If you feel like writing a concept guide, you're heartily invited :-)
(We use LaTeX for the documentation, which might be a challenge on itself).
Today we had some enlightening insights regarding the new GUI we want to build, which is why I'm in a very good mood.
I'll give you some hints, although I know that it will hardly replace a concept guide.
Exit State Definition (ESD):
A logical name for the end state of a Job. Most of the time it'll be something like SUCCESS or FAILURE, but anything you like can be defined.
ESDs don't do anything, it's just a name.
Exit State Profile (ESP):
An ESP is a list of possible states a Job can have. The ESP also defines which states are regarded final and which are regarded restartable.
It is important to note that an ESD FAILURE can be final in one ESP and restartable in another.
In fact, a restartable state indicates nothing more than that an operator decision is required (and rerun is only one of the options). Better said, it indicates that some exception occurred that is not covered by your job flow.
If, by example, an input file must be present and it isn't, this could be a reason for a restartable state.
Exit State Mapping (ESM):
An Exit State Mapping is a table that maps numerical exit codes to logical exit states (ESD).
Since different processes can have different opinions about right and wrong, an ESM can be defined at job level.
But most of the time processes have a comparable view on the world (e.g. Unix: 0 = OK, non-zero = Error), it makes sense to specify a default ESM at ESP level.
Resource State Definition (RSD):
Like an ESD just a name for a logical resource state.
Resource State Profile (RSP):
A list of states a resource can have.
Resource State Mapping (RSM):
This maps an ESD in combination with the current RSD to a new RSD.
If my job reaches state SUCCESS and my resource has state TO_LOAD, the resource state will be LOADED.
Named Resource (NR):
This is nothing more than a resource definition. It specifies the name and type, and maybe a RSP.
Because of the expected number of resources, they are stored within a tree structure.
There are 3 types of resources:
- static -- a "boolean" which represents abstract capabilities of the runtime environment. (RESOURCE.HAS_PERL could indicate that a perl interpreter is installed)
- system -- an "integer" which is mainly used for load control
- synchronizing -- a lockable counting semaphore with state model. Sounds complicated, but actually isn't. These are used for batch synchronization.
Environment (NE):
An environment is just a collection of static resource requests that is used to select the runtime environment of a job.
Normally at least a physical addressing of jobservers is defined (SERVER@LOCALHOST), but it is common to define a more abstract environment as well.
An environment could require something like: a Linux, perl and a postgres client installed. Any jobserver being able to satisfy such a request is valid as a potential executor.
In combination with a system resource to control the load, you have load balancing for free.
The use of Environments must be granted. Since the specification of an environment is mandatory in a job definition, this is a part of the security concept.
Footprint (FP):
A footprint is a set of system resource requirements. They aren't really needed, but make it easier to add standard requirements to jobs. (e.g. every job must require 1 PROCESS system resource).
Group:
A group is a set of users (can be empty). A user can be member of more than one groups.
There are two predefined groups:
PUBLIC -- every user is member of this group
ADMIN -- every user in this group has all privileges
Privileges are granted to groups.
User:
almost obvious, I think.
Web User:
This is not really part of the system, but more a GUI thing.
Within the Web Users dialogue, the connections to use are administrated.
I normally use a setup with (at least) two connections: an Admin user and a normal user.
On the main desktop one can switch easily between the connections.
Scope:
A scope is an execution environment or a set of execution environments.
A special scope, the one that actually starts jobs, is called jobserver.
The scope hierarchy is nothing more than an organized list of jobservers. But the hierarchy adds some functionality, especially regarding resources and parameters.
Starting from a leaf, a jobserver, it can "see" all resources that are defined on the path to the root of the tree. If two resources with the same underlying Named Resource are present on this path, only the first is seen.
It is common practice to instantiate the synchronizing named resources at a high level (quite often GLOBAL), but system and static resources tend to be instantiated at lower levels.
Say you have a two machines. Jobs can run on both machines. In order to synchronize the jobs, you'll need a resource on a level that is visible from both machines. Hence GLOBAL.
The workload on a machine isn't dependent on the load of the other machine. Load control will be done on machine level, or even below.
You also have two users defined, both run a jobserver. Static resources are used to distinguish between the two jobservers.
The jobservers don't do setuid() calls, which eliminates a potential security risk and simplifies system maintenance in an outsourced environment (any user can install a jobserver; no root privileges required).
Folder:
The folder concept is probably well-known.
It has turned out to be a good practice to use an extra folder for each batch. Jobs belonging to that batch are stored within the same folder.
But there's no law against developing your own ordering concept.
Job Definition (SE):
SE stands for Scheduling Entity, but most of the time we speak of Job Definitions, or, if the context is clear, simply Batches and Jobs.
A job definition of type job (you must admit: speaking of a job would have been substantially easier) is an object that, after being submitted, executes its run program.
Both batches and jobs can be a container for other batches and/or jobs.
Trigger:
Trigger within schedulix are a bit like database trigger: If something happens submit a job (vs. if some Insert/Update/Delete happens, execute procedure).
Hence the concept is clear. The details can be read in the fine manual. ;-)
Of course I omitted a zillion details, but I hope this overview gives you a rough idea.
Regards,
Ronald
if I answer questions here, most of the time I try to provide some insight in the concepts of schedulix.
We are aware that the documentation is more like a reference than like anything else. Still it is valuable. I myself use the documentation, both the syntax doc as well as the gui doc, every now and then.
Currently we are doing some youtube videos to explain some basic concepts. Due to our background, the videos are done in German first, but English versions will follow soon.
If the videos are made public, I'll definitely drop a note here in this group. Right now the videos target on BICsuite, but since they are very basic, everything applies to schedulix as well. At that level the main difference is the more blueish than greenish GUI.
If you feel like writing a concept guide, you're heartily invited :-)
(We use LaTeX for the documentation, which might be a challenge on itself).
Today we had some enlightening insights regarding the new GUI we want to build, which is why I'm in a very good mood.
I'll give you some hints, although I know that it will hardly replace a concept guide.
Exit State Definition (ESD):
A logical name for the end state of a Job. Most of the time it'll be something like SUCCESS or FAILURE, but anything you like can be defined.
ESDs don't do anything, it's just a name.
Exit State Profile (ESP):
An ESP is a list of possible states a Job can have. The ESP also defines which states are regarded final and which are regarded restartable.
It is important to note that an ESD FAILURE can be final in one ESP and restartable in another.
In fact, a restartable state indicates nothing more than that an operator decision is required (and rerun is only one of the options). Better said, it indicates that some exception occurred that is not covered by your job flow.
If, by example, an input file must be present and it isn't, this could be a reason for a restartable state.
Exit State Mapping (ESM):
An Exit State Mapping is a table that maps numerical exit codes to logical exit states (ESD).
Since different processes can have different opinions about right and wrong, an ESM can be defined at job level.
But most of the time processes have a comparable view on the world (e.g. Unix: 0 = OK, non-zero = Error), it makes sense to specify a default ESM at ESP level.
Resource State Definition (RSD):
Like an ESD just a name for a logical resource state.
Resource State Profile (RSP):
A list of states a resource can have.
Resource State Mapping (RSM):
This maps an ESD in combination with the current RSD to a new RSD.
If my job reaches state SUCCESS and my resource has state TO_LOAD, the resource state will be LOADED.
Named Resource (NR):
This is nothing more than a resource definition. It specifies the name and type, and maybe a RSP.
Because of the expected number of resources, they are stored within a tree structure.
There are 3 types of resources:
- static -- a "boolean" which represents abstract capabilities of the runtime environment. (RESOURCE.HAS_PERL could indicate that a perl interpreter is installed)
- system -- an "integer" which is mainly used for load control
- synchronizing -- a lockable counting semaphore with state model. Sounds complicated, but actually isn't. These are used for batch synchronization.
Environment (NE):
An environment is just a collection of static resource requests that is used to select the runtime environment of a job.
Normally at least a physical addressing of jobservers is defined (SERVER@LOCALHOST), but it is common to define a more abstract environment as well.
An environment could require something like: a Linux, perl and a postgres client installed. Any jobserver being able to satisfy such a request is valid as a potential executor.
In combination with a system resource to control the load, you have load balancing for free.
The use of Environments must be granted. Since the specification of an environment is mandatory in a job definition, this is a part of the security concept.
Footprint (FP):
A footprint is a set of system resource requirements. They aren't really needed, but make it easier to add standard requirements to jobs. (e.g. every job must require 1 PROCESS system resource).
Group:
A group is a set of users (can be empty). A user can be member of more than one groups.
There are two predefined groups:
PUBLIC -- every user is member of this group
ADMIN -- every user in this group has all privileges
Privileges are granted to groups.
User:
almost obvious, I think.
Web User:
This is not really part of the system, but more a GUI thing.
Within the Web Users dialogue, the connections to use are administrated.
I normally use a setup with (at least) two connections: an Admin user and a normal user.
On the main desktop one can switch easily between the connections.
Scope:
A scope is an execution environment or a set of execution environments.
A special scope, the one that actually starts jobs, is called jobserver.
The scope hierarchy is nothing more than an organized list of jobservers. But the hierarchy adds some functionality, especially regarding resources and parameters.
Starting from a leaf, a jobserver, it can "see" all resources that are defined on the path to the root of the tree. If two resources with the same underlying Named Resource are present on this path, only the first is seen.
It is common practice to instantiate the synchronizing named resources at a high level (quite often GLOBAL), but system and static resources tend to be instantiated at lower levels.
Say you have a two machines. Jobs can run on both machines. In order to synchronize the jobs, you'll need a resource on a level that is visible from both machines. Hence GLOBAL.
The workload on a machine isn't dependent on the load of the other machine. Load control will be done on machine level, or even below.
You also have two users defined, both run a jobserver. Static resources are used to distinguish between the two jobservers.
The jobservers don't do setuid() calls, which eliminates a potential security risk and simplifies system maintenance in an outsourced environment (any user can install a jobserver; no root privileges required).
Folder:
The folder concept is probably well-known.
It has turned out to be a good practice to use an extra folder for each batch. Jobs belonging to that batch are stored within the same folder.
But there's no law against developing your own ordering concept.
Job Definition (SE):
SE stands for Scheduling Entity, but most of the time we speak of Job Definitions, or, if the context is clear, simply Batches and Jobs.
A job definition of type job (you must admit: speaking of a job would have been substantially easier) is an object that, after being submitted, executes its run program.
Both batches and jobs can be a container for other batches and/or jobs.
Trigger:
Trigger within schedulix are a bit like database trigger: If something happens submit a job (vs. if some Insert/Update/Delete happens, execute procedure).
Hence the concept is clear. The details can be read in the fine manual. ;-)
Of course I omitted a zillion details, but I hope this overview gives you a rough idea.
Regards,
Ronald
Reply all
Reply to author
Forward
0 new messages