Dynamic Workflows

[Matt Anderson]

Hello,

I started experimenting with Onyx this week to see if it would fit a use case we have for workflow processing.  The unique requirement we have is that non-developers should be able to configure and execute workflows.  What drew me to Onyx is the fact that since all of the configuration details are just data structures it should, in theory, be possible to simply submit these as configurations and execute a job dynamically (ie: without statically defined workflows, catalogs, lifecycles, etc in the deployed jar).

For example, let's say I have a pre-built set of tasks available to a user (functions I've already written and deployed) as well as standard and custom plugins for ingest/egress.  I then build a service that initializes a pool of peers on startup.  It then waits for inbound requests via some API or scheduled trigger that contain details about a job to run.  At that point, it simply parses the configuration into a job map (eg: places runtime configs into the catalog map, etc) and runs the job on the previously initialized peers.

Is this doable or am I misunderstanding something that would prevent this/make it less than ideal?  If I had multiple nodes of my service running could they share a peer group or would they have to have their own group?

Any feedback is most welcome and appreciated!

Thanks,

Matt

[Mike Drogalis]

Hi Matt!

Your understanding of Onyx is spot on. This is exactly what it was designed to do. The only modification I would make would be your description of deployment in the last paragraph. You can either launch a large pool of peers on your cluster and use the various schedulers to constrain resource allocation to each job, or you can start a new set of peers with their own `:onyx/id` to create strong, multi-tenant-like isolation. It sounds like the former would be more suitable to you.

Really glad you found Onyx since it appears to be a perfect match for your problem!

--
You received this message because you are subscribed to the Google Groups "Onyx" group.
To unsubscribe from this group and stop receiving emails from it, send an email to onyx-user+...@googlegroups.com.
To post to this group, send email to onyx...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/onyx-user/da256200-a407-4c92-b5cb-87ac47843b2b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Matt Anderson]

Great, thanks Mike!

One more question (for now): we will most likely have batch and long-running streaming jobs, which I know Onyx supports.  For the long running jobs do the peers stay allocated regardless of whether there are inputs?  For example, if I have a job with 5 steps including an input with the Kafka plugin, would the 5 peers always be allocated for that job or would only 1 stay allocated to listen to Kafka and the others pulled in as needed?

Really like what I've seen so far.  Thanks for your work on this!

Matt

[Mike Drogalis]

All 5 peers will remain allocated while that job is still alive.

To view this discussion on the web visit https://groups.google.com/d/msgid/onyx-user/b15d6ba3-2833-419f-aea4-6cdcffd37c55%40googlegroups.com.