d3/win background vs zs commands. Pros and cons?

[David Knight]

Hello my learned colleagues.

I'm wondering if anyone has experience/opinions on which command to use and why; between the 'background' and 'zs' forms of initiating a phantom routine.

My scenario: I want to create a phantom routine which runs constantly performing actions as and when things appear in a certain Windows folder. I'm uncertain which 'way' to proceed with this stand-alone d3 executable.

The way I see it, there are pros and cons for each method:

Background has the pros that I get a nice automatic audit of the times the routine runs; and also as screen output [presumably including unexpected output] goes to a file; making potential debugging/support easier. Background also handles scheduling of the command; although in my case I want it to run all the time, so scheduling is less important. If I understand correctly I can issue "background <command>" as well which could be used to fire-off the command via [say] the user-coldstart presumably without any scheduling?

In which case, I guess the method is analogous to firing off a phantom via "zsd <command>" also via the user-coldstart. So what are advantages of zsd?

Or, since I need the routine to run in the same account as the data, so the routine can see and update the data account files; maybe I'll run it from somewhere else? In which case, how do I stop it from running multiple times concurrently?

Does anyone have any real-world experience in d3/win with either method?

Any advice to give me? Especially best practice in terms of writing the d3 routine, especially concerning screen output, logging and error reporting?

Thank you in advance.

David

[Tony Gravagno]

As a VAR, support provider, or sys-admin resource for my D3 clients, I setup this sort of background processor to automate common tasks.

In short, my PHANTOM.RUNNER process is a single program that gets "zeed-off" from user-coldstart with "ZS". That program is now my manager for everything else.

The runner uses a table to determine when various jobs should be started. When the time is in a range of a start time through about an hour afterward, it will attempt to perform a task. The range is because sometimes things don't start exactly on the minute, and if more than an hour goes by it might be too late to start the job with time enough to complete. The point here is that I've found that it's somewhat inadequate to just set a background job to start at a specific time - there is some context associated with some jobs.

Within the runner, some quick tasks are performed in-line. For example, a check is made for new BASIC RUNTIME errors which are then emailed to me. This task runs every 5 minutes or so.

Other longer-running tasks are kicked off with ZS. But as you say, they need to run in different accounts. I have programs that purge old data from a single app file, and there are 15 accounts with that. So with the accounts in a list from a config item, the runner loops to ZS a process for each account. If there aren't enough phantom licenses the jobs are queued, but the runner returns immediately into its wait loop until something else needs to be done.

The above describes exactly what the Background process does, the runner is just more site-specific. It also makes it easy to run the same process for a bunch of accounts without having to create a new task for the Background program.

So you can use the Background program to kick off jobs. AND use a custom runner to loop/sleep, kick off odd jobs, and loop through selected accounts to "zee-off" other jobs".

As to logging, that's up to you, but now the runner is a central point for reporting what has been run and what has Not been run. You can have the runner check its own logs (as yet another job) to see if it missed doing something, and maybe send an email in that case or for other errors. And/Or, each program that gets "zeed off" with ZS can do its own logging, and the runner can run a process that checks to see if some account did not log a successful run. Background won't do that for you.

As to management of the runner itself, my runner checks its own compile stamp every few minutes. If the current stamp is different from that which was read when the program started, it will go through wrapup, zee-off another instance of itself (with the new object) and then terminate. So it's "self-updating" in that regard like many Windows programs.

I haven't built-in this secondary process yet, no dire need in my experience, but someone might wonder about what happens if the phantom terminates, falls into debug, or otherwise stops doing what it needs to do. For this, the runner can save a timestamp to indicate it's been through a loop. A second runner process can check the jobs file to see that there is another runner active. If the timestamp goes stale, terminate the first process, kick off a third process to monitor this second one, and the second one will continue on with tasks. So there's always a runner watching the current runner.

In summary ZS is just a one-shot tool to make something happen, Background just does scheduling, and I've opted to create my own background processor for more control than Background. But there's nothing wrong with running Background in that mix too for normal, simple, non-critical tasks that need to run on a schedule.

You have another option, which is to use the Windows Task Scheduler to invoke the D3TCL program (OS command for Windows and Linux), as a way to execute some function in the DBMS from outside. The Task Scheduler (and similar utilities) might offer functionality that would otherwise need to be implemented in the DBMS. For example, when a file is available in the OS file system, tell D3 to reach out to get it. This, compared to having a D3 phantom which checks for the presence of an OS file. Neither method is "better" unless there is some other detail involved - but with Task Scheduler (and the runner) you don't tie up a DBMS phantom just for this task.

Any help?

[David Knight]

Hi Tony,

Thank you for your detailed & well thought out response.

Yes, it does help [really!] but it's also like being hit with a stack of encyclopedias! A lot to take in & digest.

I've read your post twice; and ideas/concepts particular to my scenario are forming. I plan to read over a few more times.

One question I have though: if one assumes your Phantom-doo-dad is running via user-coldstart, my assumption is it is "in" dm,dm. How does one then run a routine that is "in" another account such that the routine 'sees' files locally? Or must all program and file references in OPENs etc be fully pathed?

Cheers,

David

[David Knight]

The more I think about this today, the more I realise I have some glaring holes in my "understanding" of d3....

I'm so used to an 'account-centric' view of d3, probably because of the terminal data-entry paradigm where I started my journey. So much so that my recently-retired app which  used terminal emulators made operators 'logon' to an account name which was specific per site; and then there was an logon-proc of the same name in the md of that account which fired off the program. In that case it was the SB+ user-login routine; and off we'd go.

It is therefore with some confusion that my latest offering; a designbais app which uses MVSP pipes to access that same 'account' seems to somehow bypass the login proc and executes the designbais' executable which handles the comms between their IIS based app and d3. I'm confused because I don't understand how designbais can 'logon' via MVSP and not have the logon-proc execute. But it works, therefore somehow it manages. Hence the 'hole' in my knowledge.

But I know the logon proc is still there, alive and functioning because if I logon via a Telnet connection & a suitable Terminal emulator; the 'old' SB+ system still fires up. Rather useful in my case; but I digress.

So, back to my issue.... I've been trying to find ways of being somewhere "else" on the system, say dm, perhaps during the user-coldstart; and fire off something which runs AS IF it were already logged onto another account, with access to the md entries found there for cataloguing a file opens/closes etc.

And my brain is starting to hurt...

I noticed the "Background" looks like one can stack TCL commands along with data; which in my case sounds a bit kludgy as I would have to logto an account, answer the questions which SB+ will prompt for and finally abort to real TCL; where I could issue a ZS of my routine and log off again. Like I said, all rather kludgy.

Or, I thought I could create a synonym for the data account normally logged to, which would avoid the logon-macro issue; and logto that, instead? Also kludgy.

So I'm thinking there has to be an easier/more elegant way to achieve this, surely?

Otherwise, for my routine I'm going to have to somehow work it into my application such that it fires off a zs version when the app starts, somehow checking to see if it already active?

Don't really like that idea either...

Thoughts ladies & gentlemen? Am I missing something here which I cannot see due to my "hole" of knowledge? If MVSP can execute things "IN" an account without actually logging on to it; can I do the same?

Cheers!

David

[Gerd Förthmann]

Hi David,

In D3 you can access any file in any account by simply specifying the account name before the file name separated by comma.
To access files that 'reside' in another account you just open them like this:

open 'acount-name,filename' to file else...

Also you can catalog programs or run them the same way.

With OSFI you can even access files on another server (server-name:account-name,file-name, )

I use this in all phantoms that get started by the user-coldstart.

--
You received this message because you are subscribed to
the "Pick and MultiValue Databases" group.
To post, email to: mvd...@googlegroups.com
To unsubscribe, email to: mvdbms+un...@googlegroups.com
For more options, visit http://groups.google.com/group/mvdbms

[David Knight]

Hey Mecki,

Yeah, thanks for that. I kinda knew that but never use it. See my other post about being 'account-centric'.

The issue I'd have with this is each site has a different account-name so I do not know if the open will accept a dynamic variable as part of the full-pathed filename. If it did, that would mean I'd have to read in the account name from either an environment variable I set in the OS or from some known, fixed-location file.

David

[Gerd Förthmann]

David,

Sounds like you already found possible solutions for your self-inflicted dilemma.
After all the account name is just a string and can be stored in a record and to open a file you can use a variable instead a constant.
read acc.name from control.file,'ACCOUNT' else acc.name = 'rubbish'; A = acc.name: ',Customer'; open A to filevar else...

But you probably kinda know that as well but never used it before, right?

I guess you also kinda know that you can create q-pointers to accounts in D3 even if you never use it?

Mecki

[David Knight]

Actually, Mecki - No, I don't know how or where to create a Q-Pointer to an account. If there is an entry in the SYSTEM file called "HOSPITAL"; how & where would I create a synonym called "HOSCARE"?

[Kevin Powick]

On Sunday, 15 October 2017 20:24:02 UTC-4, David Knight wrote:

Actually, Mecki - No, I don't know how or where to create a Q-Pointer to an account. If there is an entry in the SYSTEM file called "HOSPITAL"; how & where would I create a synonym called "HOSCARE"?

From the DM account, create a Q pointer in MDS named HOSCARE that "points" to HOSPITAL.  Assuming HOSPITAL is an account in the FSI, the q-pointer called HOSCARE would look like the following

001 Q

002 fsi:HOSPITAL

Issue a "password" command to create/update the password for HOSCARE.

Btw, SYSTEM is a q-pointer to MDS

--

Kevin Powick

 

[Kevin Powick]

My only comment that you may find interesting is that, for me, the background processor has been unreliable on D3 Windows. 

By "unreliable", I mean that it would inexplicably, and randomly, stop executing jobs at the intervals I had set.  Sometimes it would function correctly for weeks before quitting, and other times only a few days.

It may have improved in newer versions of D3, but my frustration with it led me to a similar approach taken by Tony G.  I.e. I rolled my own "job runner" with phantoms.

--

Kevin Powick

[Zumasys Reply]

David, yes we use the D3 background scheduler on many of our internal and Customer's systems to manage automated regular scheduled tasks such as D3 backups, etc

and of course we also use the various Z command with options to initiate phantoms immediately as and when required.

Pros and Cons? I agree with what you have said here.

(Both do allow you to set run-time user and account information so you should not have to set Q-pointers and have them run from DM) 

The background scheduler relies on it's 'bgsched' phantom process constantly running and remaining healthy.

In my experience, we have many customers that run great for months without any problems whatsoever, so this runs fine ........ until it doesn't. It gets logged off or aborts for some reason, after all, it's just another phantom pib.

Certainly after a failure, if you schedule a mixture of hourly and daily jobs then re-syncing a large number of these following a bgsched failure or system crash experience can be tiresome recalculating when you want them all to run next and resetting dates and times.

Using the scheduled approach does provide logging information via the JOBS and BACKGROUNDS.OUTPUT files so you can check the JOBS file for dates and times etc and the BACKGROUNDS.OUTPUT file will contain the complete screen output for that phantom job.

So this is a benefit of this approach as you have already said.

But if you were starting a phantom at USER-COLDSTART and wanted it to run forever then you would not need any scheduling and screen output is not necessary.

stopping multiple concurrent instances? setting locks would be the most obvious/simple or checking running jobs.

So I guess the bottom line is both work ok and horses for courses.

Not sure I have really added anything as I think you have it figured out already, but feel free to give me a call if you want to discuss in any more detail.

Allen.

[David Knight]

Ummmm "Both do allow you to set run-time user and account information......" - hit me with a stupid stick; but NO I do not know how to do this. Can't find it in the manual....

Can someone elaborate?

[Tony Gravagno]

If you just type Z at TCL and press Enter, it will prompt you for a user, account, and a stack of commands to execute. This eliminates the need to q-point everything as the job actually runs in the account of choice.

There is a bug in there, never fixed, where if the user (or account, I don't recall which) is DM, it forces entry of a null password even if there is no password. I code around this using something like:
  IF DM.BUG.IS.FIXED THEN
   ...
  END ELSE
   ...
  END


I feel a shmuck, but these days I prefer not to post code that has any value into public forums when that's how I earn a living, and this industry doesn't do FOSS well. Sorry.

T

[Tony Gravagno]

[ad]
David, you and I have exchanged notes for a few years about tech challenges like this. I've shifted my business model from hourly services to a monthly retainer model for clients who want occasional Q&A or more extensive mentoring, collaborative efforts with development, or just to provide system monitoring and maintenance (in part with background tasks) so that on-site staff can focus on other things. I welcome an arrangement like this where you can just chat me up in Skype and we can get through things like this quickly so that you can move more quickly through your efforts.
Regards,
T
[/ad]

On Saturday, October 14, 2017 at 10:23:48 PM UTC-7, David Knight wrote:

The more I think about this today, the more I realise I have some glaring holes in my "understanding" of d3....

And my brain is starting to hurt...

[Patrick Payne]

As Tony noted all the Z## commands are all really front ends to Z which will ask you all the information.  You can automate this with a program and Data statements.  Also, Z is really just a program in dm,bp, and the real function is a u10 oconv.

ed dm,bp, Z

191 item = user.id

192 item<1,2> = passwd

193 item<2> = md.id

194 item<2,2> = mdpasswd

195 item<3> = in.line

196 item<4> = opts

197 item<5> = user.line

198 item<6> = zline

199 item<9> = orig.line

200 job.num = oconv(item,'u10') ;*go create job item & queue it

[Tony Gravagno]

I wrote:

There is a bug in there, never fixed, where if the user (or account, I don't recall which) is DM, it forces entry of a null password even if there is no password.

Correction: The nuance is that if the user initiating the Z is DM, then the Z program doesn't prompt for a password.

I now see this is in the documentation for v9 and v10:

"If a user ID is specified and you are logged on as a user other than the DM user, a user password is requested."

"If the z command is used by the DM user, User and MD passwords are not prompted for."

Aside from that sentence ending with a preposition, it's incontrovertible - this is Not a bug.

I believe the assumption is that the DM user is already authenticated as the system administrator who should not need other passwords to admin the system.

There is precedent for this - The Unix root user has full access to everything, as does the named Administrator user in Windows.

Now I need to remember all of those systems where I wrapped that code in IF DM.BUG.ISNT.FIXED THEN ... :)

T