OT: Enterprise Schedulers

[Adrian Mowat]

Hi Everyone,

This is a wee bit off topic, but given the sorts of problems the Clojure community likes to solve and the enterprise background of a lot of people I thought this list might be a good place to start.

We are building a fairly large web-infrastructure running over a combination of internal and cloud-hosted VMs.  It's basically service-oriented, but we have a number of cases where we need to bulk load data from CSV file and largish JSON files.  At the moment, we are using cron to run the jobs, but it's not really providing the level of visibility and control we need.

I've added my problem statement below and I am thinking of a solution that is something along the lines of a cheaper and more agile-friendly version the enterprise schedulers I used to use when I was building ETL/data warehouse solutions a few years ago.  Before I dive in and start building something, does anyone know of any open-source projects and/or libraries I might want to look at?  

Also, if anyone else is interested in this space, I would love to hook up and bounce some ideas around.

Many Thanks

Adrian

The Problem

We need a reusable solution that allows us to schedule, execute and monitor batch processes accross all our applications.

It should...

• Be controlled by artifacts developers control
  • Probably be github friendly
  • Provide a direct relationship between an application and its tasks
• Support separate sandbox, staging and production environments
• Be scalable
• Be distributed - jobs for application X can run on the same host as application X or on a different host or cluster as needed
• Be secure
• Be easy to administer
  • Job progress and status is visible
  • Alert when a job fails
  • Easy to re-run a job
  • Easy to spin up new hosts and/or move all processing to a different host
  • Provide a standard way of organising assets like files and configs across all our applications
• Comply with our hybrid infrastructure (stuff runs internally and in the cloud)
  • Data can move internal -> cloud
  • Data can move cloud -> internal
  • Data can be processed entirely within a host
• Support different ways of triggering a job
  • Scheduled tasks
  • Run when file x arrives
  • Run job y after job x completes

[François Rey]

Considering your enterprise/cloud requirements, it seems that quartz (http://clojurequartz.info/) should be the closest to your needs. It's also integrated into Immutant (http://immutant.org/tutorials/jobs/).
More generally in the clojure world there is this stackoverflow question that may help:
http://stackoverflow.com/questions/21404130/periodically-calling-a-function-in-clojure
I have not used any of these, it's just that I needed to answer the same question a few days ago.

[François Rey]

Perhaps I should be more precise: quartz (http://quartz-scheduler.org/) is a java-based open source scheduler, and the link I gave earlier is to the clojure integration layer quartzite (http://clojurequartz.info/).
Immutant (http://immutant.org/tutorials/jobs/) seems to use another integration library named quartz-clj (https://github.com/mdpendergrass/quartz-clj).
I have no idea how quartzite and quartz-clj compare to each other.
Finally the enterprise/proprietary version of quartz is named Terracotta Quartz Scheduler and it includes the ability to specify where to run jobs.
Do let us know what you find out and decide.

[Toby Crawley]

fmj...@gmail.com writes:

> Perhaps I should be more precise: quartz (http://quartz-scheduler.org/)

> <http://quartz-scheduler.org/> is a java-based open source scheduler,

> and the link I gave earlier is to the clojure integration layer

> quartzite (http://clojurequartz.info/) <http://clojurequartz.info/>.
> Immutant (http://immutant.org/tutorials/jobs/)
> <http://immutant.org/tutorials/jobs/> seems to use another integration

> library named quartz-clj (https://github.com/mdpendergrass/quartz-clj)

> <https://github.com/mdpendergrass/quartz-clj>.

Actually, Immutant has its own Quartz integration, and is not based on
quartz-clj. You can, however, use the Quartzite API with the
cluster-aware Quartz scheduler that Immutant provides if you prefer the
Quartzite API over the Immutant one.

- Toby

[François Rey]

On 10/02/14 16:20, Toby Crawley wrote:

Actually, Immutant has its own Quartz integration, and is not based on quartz-clj. You can, however, use the Quartzite API with the cluster-aware Quartz scheduler that Immutant provides if you prefer the Quartzite API over the Immutant one. - Toby

Thanks for being more precise, as I said I've have not used any of these libraries, yet (project still in the starting block)...

On 10/02/14 14:30, Adrian Mowat wrote:

Also, if anyone else is interested in this space, I would love to hook up and bounce some ideas around.

A scheduling library would provide much of what's needed for managing these jobs, but that would be at a level which may not be too low for certain use cases, e.g. finer control over job distribution, job composition, exception handling, manual retry, etc. A layer above the scheduler would make sense for this.
Recently, while investigating the use of a finite state machine and thus searching for fsm libraries in the clojure world, I ended up looking at a couple fsm libraries used in pallet (http://palletops.com/):
- pallet-fsm (https://github.com/pallet/pallet-fsm)
- pallet-fsmop (https://github.com/pallet/pallet-fsmop)
They are used in the pallet api for managing cloud operations on remote nodes:
http://palletops.com/pallet/marginalia/0.8/uberdoc.html#pallet.core.primitives
http://palletops.com/pallet/marginalia/0.8/uberdoc.html#pallet.api (see converge method)
I don't know if you use pallet but this may be of interest, especially when reading the rationale:
https://github.com/pallet/pallet-fsmop/wiki/Rationale
An example of usage can also be found in this discussion:
https://groups.google.com/forum/#!topic/pallet-clj/ZcBrmUn-mAI
From what I understand pallet-fsmop is based on pallet-fsm and provides higher-level operations over sets of fsm that must have certain states for that purpose. These higher-level operations trigger the remote operation encapsulated by each fsm, adding some delay, timeouts, comprehensions, reducers, reporting, etc. So in your case one could imagine a similar library that uses a scheduling library instead of doing immediate or delayed execution.
In any case a single library won't satisfy all your requirements, so you will have to choose a scheduling library and compose with others...

[Adrian Mowat]

Hi François

Thanks for the info.  Quartz and it's Clojure DSLs seem to do some of what I need.  I had a quick scan of the docs and they don't appear to support triggers that are not time based (on arrival of a file, on completion of a job etc) - but it was only a quick scan so I wondered if you had noticed these facilitates during your investigations?

Do let us know what you find out and decide.

I definitely will!

Many Thanks

Adrian

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to a topic in the Google Groups "Clojure" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/clojure/95W4MlkFgnY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Adrian Mowat

Tweet: @mowat27

Am I being a bit short?  Here's why: http://emailcharter.org/, http://inboxzero.com/

[Adrian Mowat]

Hi François

I totally agree about the scheduling library being only part of the solution.

I'm aware of Pallet but I have never used it in anger.  The links you have provided look like an interesting angle.  I'll start working through them and see if I can figure out an architecture that meets my needs and I can share with you though this list

Many Thanks

Adrian

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to a topic in the Google Groups "Clojure" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/clojure/95W4MlkFgnY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[François Rey]

On 10/02/14 18:46, Adrian Mowat wrote:
> Thanks for the info. Quartz and it's Clojure DSLs seem to do some of
> what I need. I had a quick scan of the docs and they don't appear to
> support triggers that are not time based (on arrival of a file, on
> completion of a job etc) - but it was only a quick scan so I wondered
> if you had noticed these facilitates during your investigations?

Yeah I could not find such custom trigger in the quartz api either. It
makes sense since it's no longer about scheduling but instead reacting.
So a separate abstraction/library would be better, e.g. a file watch
library such as https://github.com/derekchiang/Clojure-Watch, that
triggers an immediate job with the scheduler.

[Alan Moore]

While not specifically a scheduling tool, Clara is a forward chaining rule engine that can be used to implement a FSM and/or controller logic. I would imagine you would want to layer this on top of a library like Pallet, Quartz or Storm. Clara's Storm support is useful for distributed computing.

See:

https://github.com/rbrush/clara-rules

https://github.com/apache/incubator-storm

I would use Clara to implement the "what" (policy, decision making) from the "how" (executing jobs.) With the latest 0.4 release, Clara rules can be created using data/EDN and as such would be readily created/modified by engineers.

Good luck!

Alan

[icamts]

Hi Adrian,

the answer is more off-topic than the question :) but have a look to Spagic (I'm a member of the developers' team), Mule ESB, Petals ESB or Talend ESB. You may already know Talend as an ETL solution. You'll find tools to define, configure and run instances of services or processes. Monitong application with re-start / re-run facilities. Connectors for services / processes integration.

In a ESB scenario developers will design simple processes with a quartz or file polling connector followed by a script / custom component designed to accomplist the batch task. 

Custom components can be written in clojure if you reify the required interface. The only cavevat is the AOT compilation.

Some other ideas are below, along your problem statement.

Cheers,

Luca

• Be controlled by artifacts developers control
  • Probably be github friendly

 

Put service deployment directory and estensions / plugins directory under version control and copy them as a part of the deployment process. An ESB can be deployed like a simple webapp.

• • Provide a direct relationship between an application and its tasks

Choose meaningful service names. Deploy a local ESB in the same AS of your application and use an in-memory invoker / connector. 

• Support separate sandbox, staging and production environments

 Use different ESB instances.

• Be scalable

Single task scalability is up to your code.

• Be distributed - jobs for application X can run on the same host as application X or on a different host or cluster as needed

Sevices / processes can be run on every ESB instance. Use in-memory invoker / connector or soap invoker / connector.

• Be secure

Use https for remote connection. Use sftp for file transfer. 

• Be easy to administer
  • Job progress and status is visible

Service progress notification is up to your code and not a monitor console feature. Process running step is available. 

• • Alert when a job fails

Use a mail connector to alert on job fails

• • Easy to re-run a job

Restart / rerun through monitoring console. 

• • Easy to spin up new hosts and/or move all processing to a different host

Deploy a new instance with the same configuration. No running service can be moved. Running processes may be moved. It depends on workflow engine implementation details.

• • Provide a standard way of organising assets like files and configs across all our applications

 

(Not sure what you mean.)

• Comply with our hybrid infrastructure (stuff runs internally and in the cloud)
  • Data can move internal -> cloud

 

Is it an ETL task?

• • Data can move cloud -> internal

Same as before 

• • Data can be processed entirely within a host

 

That's so

• Support different ways of triggering a job
  • Scheduled tasks

Use a quartz input connector 

• • Run when file x arrives

Use a file poller input connector 

• • Run job y after job x completes

 

Use an output connector to trigger the next job start or design a process with jobs in a sequence.

[Adrian Mowat]

Hi Luca

Thanks for the links!

I definitely have a lot of hammock time ahead of me :-)

Cheers

Adrian

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to a topic in the Google Groups "Clojure" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/clojure/95W4MlkFgnY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[icamts]

Hi Adrian,

I've been discovered :) I enjoyed exploring a ESB scenario. In abstract you may go along the lines of any of ESB solutions this way:

1) Any task is performed by suitably configured components

2) componets live in a container

3) components communicate through a message router

4) special components resolve technology itegration issues (connectors)

Another interesting approach is provided by Akka. Have a look to this thread

https://groups.google.com/forum/#!topic/akka-user/KFzPbaNglPQ

A clojure library providing erlang like actors is pulsar

https://github.com/puniverse/pulsar

Cheers,

Luca