"Multiple problems encountered: can't start schedulix-server-mariadb, can't stop schedulix-zope, can't install examples, etc"

[alexand...@gmail.com]

Hi, good day.

We want to test out schedulix for our production, but we can't seem to install it properly.

I used the rpm installer, installed it on Centos7, followed the "RPM Repository Package for Centos 7 README" for installation instruction.

I have installed the following packages:

schedulix-repo

schedulix-base

schedulix-client

schedulix-server-mariadb

schedulix-zope

schedulix-doc

schedulix-examples

I've installed the needed swt.jar, jdbc driver, and I do have the jna file as well but we're having a couple of problems getting it to properly work.

After installing the packages, it started the schedulix-zope service right away. And we can't stop it using the command "service schedulix-zope stop" doing a sigkill will just temporarily kill it too.

Checking the /etc/init.d folder, the following services were created: 

schedulix-server, schedulix-server-mariadb, schedulix-client, schedulix-zope

The rest runs when started except for schedulix-server-mariadb which just gets stuck on displaying: 

"Starting schedulix Enterprise Job Scheduling server"

I tried installing the schedulix-examples package as well, and getting the following error:

"

Importing GPG key 0x360B3982:

 Userid     : "Ronald Jeninga <ronald....@independit.de>"

 Fingerprint: 4cbb 9b37 456c 7e96 e6a1 87c1 9198 dcd9 360b 3982

 From       : https://www.independit.de/repo/rpm-gpg/RPM-GPG-KEY-schedulix

Is this ok [y/N]: y

Public key for schedulix-examples-2.8-18.el7.noarch.rpm is not installed

 Failing package is: schedulix-examples-2.8-18.el7.noarch

 GPG Keys are configured as: https://www.independit.de/repo/rpm-gpg/RPM-GPG-KEY-schedulix

"

I can get to the admin console but when I go to Batches and Jobs I only have the SYSTEM folder available on the directory list on the right panel.

I would really appreciate if anyone could lead me to the right direction from here. I apologize for being so noob, and thanks in advance for any help you could extend.

Xander

[Ronald Jeninga]

Hi Xander,

a lot of questions, I'll try to cover them all.

But please don't apologise for being inexperienced. That's true for nearly everyone.

We are delighted you decided to evaluate the system.

Let me start with the rpm signing issue:

Last week I noticed that the gpg key I use for signing the packages was expired.

Unfortunately that was a few day after expiration, but since I can't travel backward in time, no way to fix that.

(I am an experience time traveller forward in time though ;-)

Anyway, I generated a new key, uploaded it and rebuilt all the packages which I uploaded as well.

I can't rule out that I made a mistake during the process.

And I didn't have the time to test the rpm installation yet. But I will do it this week.

Anyway, because I generated the certificate myself, it can't be verified by central authorities. This is why you need to confirm the validity of the key.

Basically this confirmation means that you don't believe that our web site has been hacked.

My road map foresees the use of an "official" key, but that'll still take some time.

Because the installation of the examples package failed, you don't find anything below folder SYSTEM.

Naturally the system is empty after a fresh installation.

We provide some convenience objects (Exit State Profile STANDARD, Exit State Mapping UNIX and Exit State Definitions SUCCESS, SKIPPED and FAILURE, or so), which are loaded during the installation of the server rpm.

The examples package creates a folder called EXAMPLES and a bunch of subfolders below the SYSTEM folder, as well as a few local jobservers.

OK, the examples package isn't that crucial at this point in time, so let me try get the more important problems fixed first.

1. Shut down Zope

    From your description it looks like Zope has been started as a service.

    What we do is that we start a "scrolllog" and instruct it to start Zope as a "foreground" process.

    But since scrolllog runs as a daemon, Zope will run in the background nevertheless.

    The effect is that if Zope crashes for some reason, it'll be immediately restarted by scrolllog.

    If the reason for crashing is a kind of soft error (an error that is caused by a temporary situation), this approach will improve the availability of the GUI.

    Since, for whatever reason (we'll find that out later), the init.d script doesn't work in your environment, you'll have to get rid of Zope in another way.

    If the Zope management form is reachable, the easiest way to get rid of the process is:

    - open http://localhost:8080/manage

    - click on "Control Panel"

    - click the "Shutdown" button

    The effect is that Zope will exit with exit code 0, which signals scrolllog that the termination of the process was intended.

    scrolllog won't automatically restart the Zope process and will terminate itself instead.

    Alternatively you can kill both the Zope process and its parent process.

2. The scheduling server

    The fact that the start gets stuck means that the server-start script doesn't seem to return.

    There is a kind of race condition which can cause this, but that's a very rare situation.

    (To elaborate this: if you, for instance, instruct Java to display garbage collect messages, it is possible that such a message collides with a server message that reports a successful startup.

    Since the server-start script doe a "grep" for this message, it won't find it in that case and it'll continue waiting although the server is running fine.

    My estimate is that this occurs at most once in a hundred startups. But hey, there exist people that win the jackpot at the first attempt).

    The more likely cause for getting stuck is a problem during startup.

    But in order to analyze that, I'm going to need the server's log file(s).

    If you feel uncomfortable posting them here, you can send me them per e-mail (ronald dot jeninga, then an at-sign and the company name, independit and the top level domain de).

The path to happiness we're going to follow is:

- get rid of the problematic processes (the Zope server and (maybe) the scheduling server)

- analyze the reasons for failing

- eliminate those reasons (I might have to build new rpms; open source environments are unfortunately pretty volatile, and I'm certainly not perfect either)

- get both the schedulix server and Zope running

- install the examples again (i.e. we try and see what happens, analyze any problems and repair them)

Don't worry about us. We're not perfect but eager to learn from our mistakes. And every improvement of the system (that includes the rpms) is important to us.

Best regards,

Ronald

[alexand...@gmail.com]

Hi Ronald,

I appreciate a how u responded soon and addressed each of my concerns in detail.

With installing the examples and the rpm signing issue. Does that mean I can't install the examples at the moment?

I tried reinstalling them just now, and it's still not going through with the same signing issue.

That's fine too for now as long as we can get the system to properly work.

I agree with this :D :

" The path to happiness we're going to follow is:

- get rid of the problematic processes (the Zope server and (maybe) the scheduling server)

- analyze the reasons for failing

- eliminate those reasons (I might have to build new rpms; open source environments are unfortunately pretty volatile, and I'm certainly not perfect either)

- get both the schedulix server and Zope running

- install the examples again (i.e. we try and see what happens, analyze any problems and repair them) "

Zope server and Scheduling server - are there separate servers for each of them? Is that why there is "schedulix-server" and "schedulix-server-mariadb" under /etc/init.d?

Regarding that too, running schedulix-server works fine I think. Although I'm not sure how to confirm except that it doesn't get stuck in displaying a "starting up server...." message and it says to check the log files but I found no errors there too. Am I right to assume that "schedulix-serer" is for the Zope Server and the "schedulix-server-mariadb" for the scheduling server?

I've run the "schedulix-server-mariadb" server a couple of times too but none went past that "starting up server.. " message.

Also, with this:

"We provide some convenience objects (Exit State Profile STANDARD, Exit State Mapping UNIX and Exit State Definitions SUCCESS, SKIPPED and FAILURE, or so), which are loaded during the installation of the server rpm."

When I try to create a job and click on "Exit State Profile", there's only "STANDARD" available. And when I go to "Environment", there's nothing there too. So we can't really proceed in creating a job and actually test out the system.

Are there anything else I need to look into? Should I wait for your test of the rpm package or do I do a reinstall of everything without using the rpm package?

Thanks,

Xander

[alexand...@gmail.com]

Hi Ronald,

I forgot to send the log file. I think it's okay to post it here.

This is the log entries after I run the commands "schedulix-server start" and then "schedulix-server-mariadb start". With the former getting executed successfully and the latter stuck on "starting server.. " message.

timestamps from 02/07/2019 2:54 to 02/07/2019 02:56.

Please see attachment.

Thanks,

Xander

[Ronald Jeninga]

Hi Xander,

the gpg key issue isn't really one. You can instruct yum/rpm to ignore that.

But we'll get to that later.

I had a look at the log file. Obviously to server processes are trying to start simultaneously. This won't work (without manual configuration and a second repository).

Fortunately this can be easily fixed. You kill both and then restart.

The scheduling server is started by scrolllog as well, just like the Zope server. Which means you'll have to kill both scrolllog and the Java process.

If you do a 

[ronald@ocelot ~]$ ps -fu schedulix | grep java | grep 300

schedul+  7066     1  0 Feb05 ?        00:00:29 /opt/schedulix/schedulix/bin/scrolllog /opt/schedulix/log/BICserver.out -s 50 -l 250000 -o /opt/schedulix/log/BICserver.pid -e /usr/lib/jvm/java-1.8.0-openjdk/bin/java -Xmx300m -Xms150m -XX:NewSize=32m -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=80 -XX:+DisableExplicitGC -XX:ParallelGCThreads=4 -cp /opt/schedulix/schedulix/lib/BICsuite.jar:/usr/share/java/postgresql-jdbc.jar:/usr/share/java/jna.jar:/usr/lib64/java/swt.jar:/opt/schedulix/schedulix/lib/guava.jar:/opt/schedulix/schedulix/lib/jna-platform.jar:/opt/schedulix/schedulix/lib/waffle-jna.jar de.independit.scheduler.BICServer /opt/schedulix/etc/server.conf

schedul+  7067  7066  0 Feb05 ?        00:09:45 /usr/lib/jvm/java-1.8.0-openjdk/bin/java -Xmx300m -Xms150m -XX:NewSize=32m -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=80 -XX:+DisableExplicitGC -XX:ParallelGCThreads=4 -cp /opt/schedulix/schedulix/lib/BICsuite.jar:/usr/share/java/postgresql-jdbc.jar:/usr/share/java/jna.jar:/usr/lib64/java/swt.jar:/opt/schedulix/schedulix/lib/guava.jar:/opt/schedulix/schedulix/lib/jna-platform.jar:/opt/schedulix/schedulix/lib/waffle-jna.jar de.independit.scheduler.BICServer /opt/schedulix/etc/server.conf

you'll find something like the above output, but you'll get multiple pairs.

Just kill them all. Thus with my output:

kill -KILL 7066 7067

After the kill, the server is down.

Now you can start the service again:

service schedulix-server start

Which also brings me to your question: why are there a schedulix-server and a schedulix-server-mariadb in /etc/init.d.

If you take a close look, you'll see that schedulix-server is simply a symbolic link to either schedulix-server-mariadb or schedulix-server-pg.

The idea behind it is that it simplifies administration: You don't need to know what DBMS is used by the schedulix server; service schedulix-server start/stop works in every schedulix environment.

The Zope server which only provides the Web-GUI is addressed by  the schedulix-zope script.

The schedulix server assumes it is the only process with write access to the repository.

This implies two things:

1. There can only be one server active (the highlander ;)

2. Never ever try to change anything in the repository with SQL means (unless we say so)

Since you've seen the Exit State Profile STANDARD, the server package seems to have been installed correctly.

So we don't have to repeat that.

And since you "clicked" on Exit State Profile, Zope seems to work as well.

There are no Environments yet, because nobody created one.

After the installation of the examples, you'll have a bunch of environments.

3 of them uniquely address a certain jobserver (the SERVER@LOCALHOST and friends). 

The others are used for specific examples like LOAD_BALANCE.

After tidying up you can try to install the examples.

a

yum --nogpgcheck schedulix-examples

should do it.

A final remark:

Although it doesn't make sense to start a second schedulix-server that uses the same repository on the same node, it is possible.

The provided init.d scripts don't support this though.

Starting a second schedulix-server on another node does make sense. If you configure everything correctly you'll end up with a kind of warm standby.

But let's not dream about New York's marathon before we can even walk :-)

HTH

Regards,

Ronald

[alexand...@gmail.com]

Hi Ronald,

I did as u suggested, and was able to load to schedulix-examples package successfully too. :D

Now I just have to start scheduling some jobs. I tried following the video tutorials on your site, but there's one thing I don't quite understand that can't let me really start. When I create a job, where is schedulix looking for the bash file that I want to run? Where do I put the bash file for it to locate?

Also, how do I make use of the example jobs for testing? I went over the documentations that came along with the rpm installer: online_en.pdf, syntax_en.pdf, and installation_en.pdf, but I don't seem to find any mention there about where it looks for the bash files loaded for jobs created.

Thanks for your assistance and prompt responses as always.

Xander

[Ronald Jeninga]

Hi Xander,

great! We're making significant progress :-)

The jobservers have a configuration parameter called USE_PATH. If this is set to true, the jobexecutor (which is started by the jobserver) will do a execvp() with the parsed command line it receives from the server.

IOW you specify a "run program" in your job definition. After submitting the job this run program is parsed by the scheduling server following bourne shell syntax rules (without the internal commands).

This results in an array of arguments (argv[] in C). This is passed to either execv() or execvp(), depending on the USE_PATH setting.

The bottom line is that the system behaves much like a shell, with the exception that it's not interactive.

Basically it is similar to calling ssh with your run program as a parameter.

If you want to execute some script, you'll either have to install it on the target computer, or you'll have to write it into the run program.

Since 2.8 schedulix knows "long parameters". Before 2.8 the max size of a parameter value was limited to 255 bytes.

It is possible to store your script in such a parameter, maybe even at folder level.

You then can call it like

/bin/bash -c "$MYSCRIPT"

If the script needs command line parameters, it'll require a sophisticated dealing with quoting and bash syntax.

You'll always have to keep in mind that this $MYSCRIPT is interpreted twice. Hence you'll have to do some experimenting to get it right.

An older (not published) example uses a run program

        run program = 'bash -c "sdms-set_variable -j $JOBID -h $SDMSHOST -p $SDMSPORT -k $KEY OLDPOSITION $POSITION;

LASTDIR=`sdms-get_variable -j $JOBID -h $SDMSHOST -p $SDMSPORT -k $KEY -n $POSITION`;

DIR=`expr \\$LASTDIR + 1`; DIR=`expr \\$DIR % 4`;

sdms-set_variable -j $JOBID -h $SDMSHOST -p $SDMSPORT -k $KEY MOVE_DIRECTION \\$DIR;

sdms-set_variable -j $JOBID -h $SDMSHOST -p $SDMSPORT -k $KEY $POSITION \\$DIR;

exit \\$DIR;"'

I won't explain what's happening here, but use it as an example to show that simple scripts are possible.

I can even give you a "worse" example:

        run program = 'sh -c "SDMSpopup.sh -c \'1=1:2=2:3=3:4=4:5=5:6=6:7=7:8=8:9=9\' `echo \'show job $MASTERID;\' | sdmsh | awk \'/DEFINED_RESOURCES :/ { start=1; i=0; } /TICTACTOE.PLAY/ { if(start==1) {if($6 != "\'"EMPTY"\'") {board[i] = $6;} else {board[i] = i+1}; i++;} } /TICTACTOE.TOGGLE/ { if(start==1) {player = ($6 == "\'"X"\'" ? "\'"O"\'" : "\'"X"\'")} } END {print "\'"\'"Player:"\'"\'" player; for(i=0;i<9;i+=3) { printf("\'"\'"%s|%s|%s\\n"\'"\'", board[i], board[i+1], board[i+2]); }} \'`"'

it is from an implementation of TicTacToe in schedulix. This is what I call "advanced quoting" ;-)

The example job definition can be submitted and the definitions can be studied.

Most of them simply open a small window with a list of command line parameters.

This is what I use in the latter example: The part of the command line that starts with `echo \'show job in fact builds a list of strings that is then displayed by the SDMSpopup.sh script and it shows the tictactoe board:

As you can see, X is winning :-P

The examples need X11 (which explains a bit that you need the swt.jar). If the jobservers are running as another user than the user that is signed on, you might have to execute a "xhost +".

Obviously the examples aren't intended for use on production systems, but are nice to play with on your local system.

You're welcome.

And because I'm curious: where are you located? (you don't need to answer; privacy is important to us).

Best regards,

Ronald

[alexand...@gmail.com]

Hi Brayan,

I think I've worded my question quite vaguely. Nevertheless, what you provided is something I needed to learn too. Thanks.

What I was really asking is the working directory where I need to save the scripts I'm going to use for programs I want to run. But I think I've figured it out already, hopefully, but please correct me if I'm mistaken. It's the bin directory in /opt/schedulix. What I did to find that out is I searched for the example bash file "SDMSpopup.sh" and found it there, so I am assuming that if I have the bash files I want run, I need to place them under /opt/schedulix/bin directory. And for the run tab when creating a job, I can just simply type inside the RUN PROGRAM field the name of the bash file.

Although I was able to create and submit a job (I followed the video tutorial this time and used the file "SDMSpopup.sh"), it's failing.

This is what I got from the error log:

------- [08-02-2019 11:29:54 +08] Start --------
Cannot open GUI (Exception:org.eclipse.swt.SWTError: No more handles [gtk_init_check() failed])
exit (19)
user CPU  : 0 / 120674
system CPU: 0 / 32683
RSS       : 37612
inblock   : 0
oublock   : 88
------- [08-02-2019 11:29:54 +08] End (19) --------

It must have been from the required swt file that perhaps it can't find? But it should have been able to find it. The examples won't get installed if that swt file is missing. And they have been successfully installed.

This is from my java.conf: "SWTJAR=/opt/rh/devtoolset-4/root/usr/lib/java/swt.jar"

I don't seem to know how to proceed from here again.

-Xander

[alexand...@gmail.com]

Hi Ronald,

Aplogies. I'm not sure who I was addressing as Brayan. :'D But I'm certain I'm directing my concerns to you.

The example programs seem to work. But I'm not getting that popup interface to control the running job similar to that in the tutorial videos.

And I can't figure out why when I create a job and use that SDMSpopup.sh file as in the tutorial video, it fails. I tried running the tic tac toe batch program examples and the status just stay idle. I don't get any popup either. Where am I supposed to see it working? Also the port 2506, what is that for?

I was so glad that I already got the application up and running. But really couldn't test it out yet as we wanted. We basically just wanna see how the schedulix could fit into the jobs we're currently automating in production. I believe it's a pretty nifty application to deploy those. We are already really pleased to find out that this exists, but would be really grateful if we could make use of it. The video tutorials are so straightforward and easy to follow along. Just that we're not getting the result expected from that in the video.

I really wanna learn and understand how the program is ran and managed too. I know that it's just an open source software and we can't really ask much. But would be glad if there's some sort of documentation for dummies like me. :'D

-xander 

[Ronald Jeninga]

Hi Xander,

I'll try again, because I did answer your question, I think.

Your jobservers are processes running on the target system that is intended to execute your scripts and programs.

Each jobserver is started by some user (no worries about that; it's taken care of by the init.d scripts, the schedulix-client script to be more precise).

The jobserver processes have a runtime environment.

If a job is submitted, the command line to execute is sent to the jobserver.

The jobserver now executes this command line from within its context.

This context will contain a PATH environment variable. And when starting your command line this PATH variable is evaluated.

Hence if your command line looks like "cp /tmp/source /tmp/target", the operating system will search the PATH for an executable called "cp".

This is actually located in /bin, which is usually one of the directories with a PATH.

Then the operating system will start this executable and "cp" will copy the /tmp/source file to /tmp/target.

So you need to place your bash scripts (or whatever executables) in some directory that is within your PATH.

In fact it's just like opening a remote shell and entering your command line by hand.

The swt error message you get means that

1. you have a sane swt.jar

2. but the process can't open a window on your desktop

If you're sitting in front of it, it probably means that you're either not signed in as schedulix or you don't have a GUI running (X11), or both.

To cover the first, you execute a "xhost +" from some shell.

But if there's no X11 running, you don't have a GUI and there doesn't exist such a thing as "opening a window".

HTH

Best regards,

Ronald

  

[Ronald Jeninga]

Hi Xander,

Apart from the fact that you can always try to "bribe" me, which would result in my undivided attention, you can always ask in the group here.

But you'll have to accept that we can't always answer immediately.

TicTacToe isn't the best example to start with. The first example, the singlejob, is much better.

It simply starts the SDMSpopup.sh script and after pressing OK, it will terminate.

If that doesn't work for some reason, TicTacToe definitely won't work.

If you look at the architecture, e.g. http://www.schedulix.org/en/highlights/architektur , you can see that the Jobservers/Agents don't necessarily run locally to the scheduling server.

Thus a network is required to start jobs remotely.

The scheduling server opens port 2506. Jobservers (and the Zope engine) connect to this port, authenticate themselves, and can give the scheduling server instructions / send it requests.

In case of jobservers the "GET NEXT JOB" command is pretty popular. (If there's something to do for the jobserver asking, the scheduling server will respond with a data structure that enables the jobserver to run the run program you specified).

With this infrastructure in place, it doesn't make sense to build a second infrastructure for this rare case that everything runs locally.

Oh and BTW I'm not aware I know somebody called Brayan. But I'm not that good in remembering names.

HTH

Best regards,

Ronald

[alexand...@gmail.com]

Hi Ronald,

Thanks a lot for answering my concerns. I do understand too and am not expecting for you or anyone in your team to respond to any of our inquiries soon. I'm actually quite surprised and really grateful that you seem to made time as soon as you. I appreciate it. Also, I'm just addressing my concern to you too because that's what I see on the other threads, that they address to whoever replies to them, I guess I should just address your team next time. I apologize.

I'm running schedulix on centos7 without GUI. So that's really the reason why I'm getting that error. Thanks for letting me know. I really had no idea. I tried googling it before posting that question here, but I didn't get a sensible answer.

And I did try running the singlejob, although without a GUI, but it did work. That's when I tried running the "SDMSpopup.sh" from the tutorial, and when it didn't work, I tried the Tic Tac Toe from the examples. I think it's working okay now. I just tried scheduling a sample job too. I still have more to learn to really find my ways around schedulix, but I think I've had a good start with your help and I've found some of my new concerns in the How To's section of the discussion area of this group.

Again, I thank you and appreciate your help a lot. It was a bit daunting for me when I was just trying to install schedulix, but it's looking fun to work around it now.

-Xander

[Ronald Jeninga]

Hi Xander,

don't worry too much about whom you're addressing in your posts. I read all of them, and so does Dieter. I definitely ignore the first line, unless it states "Hi Ronald" ;-)

I happen to like answering the questions and to elaborate on concepts. Sometimes that's a really welcome distraction from the work I'm doing at that time.

Like at the moment I'm chasing a bug in the 2.9 time scheduling module, which happens to be one of the most complex parts of the system.

Answering group questions is a pretty good method to empty my mind, which again creates room for new ideas.

People using schedulix is perceived as a reward by us. We've put a lot of effort into the system and it is great if people happily solve their problems with it.

It definitely motivates us to continue our path.

Anyway, "no GUI" is a very effective way to create problems with the examples.

But the example jobservers are a help to overcome the initial hurdles.

Installing jobservers isn't hard at all, but it helps newbies if there are some jobservers present after the initial installation.

schedulix is a mature scheduling system that has proved itself in large and complex heterogeneous environments.

This automatically implies a certain complexity, even if we try our best to keep simple things easy.

(And I think you must admit that everything wasn't that hard after all).

I think it would be a good idea to browse through the posts in this group.

Many of them explain the underlying concepts of the system. This, together with the YouTube videos and the documentation itself should give you a good starting point.

And although the examples don't run (because of the missing GUI), they still are valuable as they provide implementation patterns.

By studying them and trying out yourself (and maybe ask some questions here) you'll learn about the system.

And with a bit of luck you'll be able to implement a TicTacToe yourself.  

In fact schedulix is a bit like a distributed programming language. And it requires time and effort to learn a new language.

Best regards,

Ronald

[Ronald Jeninga]

Hi Xander,

just a moment ago I posted an announcement that relates to your problem while trying to install the scheduilix-examples package.

https://groups.google.com/forum/#!category-topic/schedulix/announcements/xlklrI28kLA

should point to that post (else search for "RPM installation / key change").

Best regards,

Ronald

[alexand...@gmail.com]

Hi Ronald,

Thanks for that. That should definitely help out those who are trying to do an RPM install as I did. I certainly wouldn't have been able to figure it out myself and didn't find it here when I tried to look either. A separate entry here is surely a great idea.

Regards,

Xander