About script load balancing

[Albert Crespillo]

Hi!
First, congratulations for your impressive job! I'm testing hcron right now and I think It's what I was looking for!

I have some questions if you would be so kind :-)

1. Have you some trick or idea to balance the script dispatching using CPU usage on destination host as a reference (without using clustering)?
2. How many tasks can hcron handle per user? Because I'm thinking in setup my installation using only one user to handle all work (thousands tasks per day) and then divide by type of work using tree folders. Is it a crazy approach?
3. Is the 60 seconds sleep configurable?

It's amazing. What plans of future have for hcron? Are you evolving it?

Wow, so many questions :P Thanks!
Albert Crespillo

[John]

On 08/18/2012 01:33 PM, Albert Crespillo wrote:

Hi!
First, congratulations for your impressive job! I'm testing hcron right now and I think It's what I was looking for!

Hey, thanks.

I have some questions if you would be so kind :-)

1. Have you some trick or idea to balance the script dispatching using CPU usage on destination host as a reference (without using clustering)?

No. As far as the destination is concerned, hcron works blindly. However, it is
possible for round-robin DNS to do _some_ form of load balancing, but that
is a pretty crude approach.

At one point, though, I did fiddle around with dynamically updating a local
DNS so that it returned IP addresses in a particular order (e.g., based on
availability and load). But that is totally apart from anything to do with
hcron.

2. How many tasks can hcron handle per user? Because I'm thinking in setup my installation using only one user to handle all work (thousands tasks per day) and then divide by type of work using tree folders. Is it a crazy approach?

From the hcron project page (I just updated it):

hcron in Use

At Environment Canada, hcron is in 24/7 operational use managing over 1500 event definitions with more than 36000 events scheduled each day. (Update as of 2012-08-18: over 1900 event definitions, and more than 75000 events scheduled each day).

So 1000s per day is not much.

There is no fixed limit to the # of events a single user can have other than
the limit in the configuration file (hcron.conf) parameter max_events_per_user.

To make it work:
* don't target the same host for all tasks, it may not be able to handle it
* make sure that the ssh server is configured to allow a lot of concurrent
    connections within a short period of time (see MaxStartups parameter
    for sshd_config file)
* spread out the events/tasks across the day

3. Is the 60 seconds sleep configurable?

No. This is how the regular cron works. It is a periodic (every minute) scheduler.

It's amazing. What plans of future have for hcron? Are you evolving it?

I've got some ideas but I am working on another project at the moment.

Hope this helps,
John

[Albert Crespillo]

Ok thanks John! I'm trying IPVS to balance workload. We'll see if I can get it to work.

Albert