Is it possible to on-ice/deactivate a certain job in a batch?

[Minjie Gong]

Hello Community,

Is there any on-ice/deactivated status of a job in schedulix? 

A job in such status remains to be a part of a certain batch, however it won't run and just skip after that batch is kicked off. And it won't block any job that is dependent on it. I looked around, and found no such mechanism in schedulix?

Thanks,

Minjie

[Ronald Jeninga]

Hi Minjie,

what would be the purpose of a job that doesn't run and doesn't influence the rest of the batch?

You could define the run program to be "0" (zero). That will logically run the job, and it will have exit code 0 afterwards. That's a shortcut for "/bin/true" saving a lot of overhead.
The logic behind is: if the run program is numerical, the system regards it as a process that gets exit code "run program"; if you want to execute an executable called 4711, you'll have to add some non-numerical characters to the run program, like "./4711". Most real life executables will have a name (more or less) describing their purpose.

Another possibility is to let the job timeout on a resource (with timeout 0).

Regards,

Ronald

[Minjie Gong]

Hi Ronald,

Thank you for the input. 

I have experience of using some other schedule tool like UC4, autosys and etc. Jobs could be on-iced or deactivated in these tools.

For example, there is a batch that starts with several extract jobs to pull feeds from upstream system. It's possible that one upstream would stop providing feeds for a while, perhaps one week. All other feeds remain the same. For the purpose of not touching the structure of the whole batch, we could just put that certain job on-ice and the rest of batch would be functional as before the change. One week later, when the upstream is working again, we could bring the on-ice one back to normal.

Is there any way to import/export jobs or batches into/from an environment or a folder? As you know, there could be a few environment - dev, staging, qa and prd. The manual says jobs/batches could be copied/cloned from a folder and then be pasted into a new one. Yet per my experience, such manual process would be a little bit hard to be standardized. So I'm trying to find a convenient way to transfer jobs/batches among environment in a pre-defined format like XML or sdmsh scripts maybe.

Thanks,

Minjie

[Ronald Jeninga]

Hi Minjie,

there are some possibilities:

1. Use a batch as a stub
You have one or more jobs that are related and have to be skipped for a while. You simply add them as children of the batch if they have to be executed and remove them if not.
It'll be a good idea to use another batch as a container for the jobs (if there are more than one). That way you only have to add that top level batch as a child of your stub batch.

2. The timeout on a resource is still a very good practice. For each upstream you define a synchronizing resource with a state model (active/inactive). The jobs processing this upstream require that resource in state active, with timeout 0. If the upstream is inactive, they'll skip and get the timeout state you defined. Any dependent jobs require a SUCCESS state or a SKIPPED state (if you want to make the skipping visible). Now you can enable or disable any upstreams by altering the resource state.

3. A third possibility is to define a job (DO_IT) with an exit state profile RUN_X, DONT_RUN_X. The upstream processing job requires RUN_X and skips if the predecessor "returns" DONT_RUN_X. Any dependent jobs require either a successful execution of the upstream processor, or a DONT_RUN_X.
Now you define the run program of DO_IT as either 0 (RUN_X) or 1 (DONT_RUN_X). Toggling between the values enables or disables the processing of the upstream. You could even use a parameter instead of a fixed command line. That way you can define which upstreams should be processed at submit time.

4. If it is possible to determine which of the upstreams are working today, you could also submit the processing jobs dynamically instead of having them statically within your batch. This is a very common practice.
Information about which upstream has to be processed by the job(s) can be provided as parameters at submit.

Depending on the situation and the frequency of occurrence one of these options should do it. In a test situation or during development option 1 or 3 are interesting (option 1 is like an empty function, option 3 is like an #ifdef DEBUG). In a production environment I'd opt for one of the other possibilities. I'd take option 2 in a, more or less, static environment and option 4 in a highly dynamic environment.
There might be other good ideas to cover your requirement.


We built import/export functionality. This is part of BICsuite PROFESSIONAL. The output of an export is an "sdmsh-script" indeed.
Effectively, if you need functionality like import/export, privileges or auditing, to name a few, you'll have to invest some money.

Regards,

Ronald

[Dieter Stubler]

Hallo,
The parent child relationships of batches and jobs have a property 'static' which yoh can set to false.
When a parent is submitted only static children will submitted also.
Hope thaf helps.
regards
Dieter

[Minjie Gong]

Thank you Ronald for coming up w/ the ideas. I'll take a try. At the first glance, option 2 seems to be a very elegant way.

I agree w/ that "Effectively, if you need functionality like import/export, privileges or auditing, to name a few, you'll have to invest some money." xD

Btw, if we decide to pay for some premium features of schedulix, is this the right way to contact you by sending message from this url - http://www.schedulix.org/en/kontakt? And more importantly, is there a partner or distributor in CHN? 

Thanks,

Minjie

[Minjie Gong]

Thank you Dieter for the information.

I tried to set the 'static' property from true to false. It works, but only for one circumstance in which that job has to be independent and no others depends on it. Otherwise, it would prompt error when submitting the batch by saying "cannot resolve dependency". But in real life, it's not practical in my opinion since the job we want to put on-ice is never the tail of a batch.

I'm not sure the purpose of such a property. To me, it seems to be redundant to have this property making the job be omitted while it would be treated as a violation of the reference integrity. Don't know if there is anyone else has raised a request to change this behavior? Sorry to be a pain here.  

Thanks,

Minjie

[Ronald Jeninga]

Hi Minjie,

either you use the URL you mentioned, or you use the corresponding URL from our company site (http://www.independit.de/en/kontakt), or you send me an e-mail directly (firstname dot lastname at independit dot de).
It doesn't make any real difference actually. In the first two cases, Dieter will get an e-mail as well, in the last case, I'll inform Dieter about the e-mail. Whatever suits you best.

It might be interesting to have a look at the company site. schedulix is functionally identical with BICsuite BASIC. You'll find a comparison of features and the BICsuite documentation (which is very much like schedulix documentation since everything is generated from one source).

Unfortunately we don't have a partner in CHN yet, but we are definitely interested in partnering. This is also true for other countries in the world.

Let's talk :-)

Regards,

Ronald

[Dieter Stubler]

Hallo,
Part if the dependency definition details, there is a property called 'unresolved handling'. Set it to ignore you will not get an error on submit.
regards
Dieter

[Ronald Jeninga]

Hi Minjie,

you'll have to define the dependencies such that they will be ignored if unresolved. (UNRESOLVED HANDLING -> IGNORE in the dependency definition).
And you are not a pain at all. If I would regard all people that don't have a detailed knowledge of schedulix as a pain, I'd be unhappy for the rest of my life.

Regards,

Ronald

[Minjie Gong]

Thank you both.

By setting UNRESOLVED HANDLING -> IGNORE, the violation would be ignore at run-time. Yet unfortunately, the implied dependency will be broken.

A batch has 3 jobs, and they are supposed to run in a chain (JOB1 -> JOB2 -> JOB3). If unchecking static perperty of dependency "JOB2 -> JOB3" and unresolved handling to be "ignore", job1 and job3 would be kicked off at the same time after the batch is submitted. My expectation was job3 should run after job1 has finished in success.

Anyway, I'll explore the other options provided by Ronald later. I have a general concept about "named resource"/"environment"/"footprint", but it'll take some time to get familiar w/ the configuration. My feeling is schedulix could be really powerful and flexible, yet it could be complicated as well xD. The learning curve could be steep.

Thanks,

Minjie

[Dieter Stubler]

Hi,
you can add an explizit dependecy from job1 to job3.
regards
Dietee

[Ronald Jeninga]

Hi Minjie,

If you wrap the job(s) to be skipped with a batch, you can define the dependencies at the batch level. This would keep your dependency chain untouched, regardless of the wrapped child being static or dynamic.

So you'll have

JOB1 -> BATCH1 -> ... -> BATCHn -> JOB3
                 |                          |
              JOB2_1               JOB2_n

If JOB2_x is defined as a dynamic child, it won't be submitted, but the BATCHx will remain to preserve the dependency chain.

Actually I'm convinced that schedulix is a powerful tool, but you need to get familiar with the concepts. It's a bit like programming in some language. Most languages are simple, have only a few keywords and a comparatively simple syntax. It's the combination of the concepts that makes it powerful. We tried our best to build a system on a few concepts that can be arbitrarily combined.

regards,

Ronald

[Ronald Jeninga]

Hi Minjie,

now it's time for some good news. Dieter and I discussed the issue about disabling jobs and batches.
During the discussion we found out that we're talking about two kinds of situations:

1. The fact that a batch or job should be executed is defined by the state of the system and is likely to be different at every point in time
    In this case it will be better to have the batch dynamically handle the situation. This type of requirement can be built with the methods I described previously.
    (Use resources, dynamic submits, initial jobs that test if something has to be done, ...)

2. The fact that a batch or job should be skipped (disabled) is an exception.
    At the moment this requires redefining the batch definition, which is not very nice, although possible.

It is the second requirement that convinced us to implement the possibility of disabling jobs and batches.
At the moment we're working on a new release (2.6.1) that we plan to make available towards end of October. Though we have enough work, we think (but don't guarantee) that we can add the disable-feature to the 2.6.1 version. This will be a feature of BICsuite BASIC and therefor a feature of schedulix.

Don't hit me if we don't manage to implement it in time. But you can be assured that your feature request is high on the list. (That means, it will definitely be there in Q1/2015, but not necessarily in a stable release).

Regards,

Ronald

PS. as you notice, we are susceptible to new ideas. If you have more of them, tell us. If it's already there, I'll tell you :-)

[Minjie Gong]

Hey Ronald,

That's a good news indeed. Quite looking forward to it.

A tiny question here, if a new version w/ new features comes, will it be smooth to upgrade? Per my understanding, all the definitions of schedulix objects are stored in the database. Once there is no table rebuild, all it takes to upgrade is compiling the new source code and then replacing the new one. Is my understanding correct?

Btw, I'm trying to compile and run schedulix under cygwin on Windows. Although struggling, it would be very helpful to persuade my manager and his manager to make schedulix the scheduling solution for our department.

Thanks,

Minjie

[Ronald Jeninga]

Hi Minjie,

new versions of schedulix normally require a few schema changes.
For example, if you upgrade from 2.5.1 to 2.6 you'll have to add two "SALT" and "METHOD" columns (users and scopes), as well as two "STICKY_NAME" and "STICKY_PARENT" columns (resource_requirement, resource_allocation). Of course this is documented (look at the README.md file in 2.6).

Now upgrading from one version to another goes as follows:

1. shutdown server
2. if you have jobservers local to the scheduling server, shut them down as well
3. perform DB schema upgrade as documented. It is possible that the required syntax differs a little from the examples given; different RDBMS speak different dialects of SQL.
4. set link to current version
5. start server
6. start local jobservers

It makes sense to have the different versions of the software in different directories and a link to the currently used directory. It also makes sense to have the configuration of the system outside.
Like:

/home/schedulix
             |
             +-- schedulix-2.5.1
             |
             +-- schedulix-2.6
             |
             +-- schedulix -> schedulix-2.6
             |
             +-- etc
             ...

In a production environment the entire procedure only costs a few minutes (if you have prepared yourself) in most cases.
It depends a little on the changes to the DB schema; If you have to add a column to submitted_entity and perform an update of that column, it might take a little longer.
In the actual case, we're talking about a few minutes indeed.

Now to the windows part.
So far our policy is: free software for free systems. As it seems a lot of people need a windows agent.
One possibility is to use the mingw software to cross-compile on linux.

On one of my computers here i have set the environment variable

MINGWGCC=i686-pc-mingw32-gcc

In the jobserver directory I extended the Makefile with:

ifdef MINGWGCC
DISTRIB = $(JOBEXECUTOR) $(WINEXECUTOR)
else
DISTRIB = $(JOBEXECUTOR)
endif

as well as

$(WINEXECUTOR): jobexecutor.c
        $(MINGWGCC) -DWINDOWS $< -o $@

This produces executables that seem to work.
Unfortunately there a bit more to as you want to run the windows software as a service. We did a port of scrolllog and developed several batch files like the Linux wrapper scripts.
I won't promise anything here, but I'll discuss with Dieter if sharing the scrolllog code is a good idea.
I'm a bit uncertain about this myself, as I'm not really a windows lover (more like a severe toothache actually) and digging into windows systems to find some flaw or mistake in the compile procedure is not my idea of a nice day.

Regards,

Ronald