Onyx input ingestion patterns

[Matan]

Hi,

Looking through this example, I am not sure why lifecycles are involved in the ingestion of input into the cluster, or more precisely whether they should always be used to ingest into a workflow. Can you kindly explain the exact roll of lifecycles as it relates to simply ingesting input into a workflow? I see the same pattern used for ingesting into the workflow in this other example as well. In both cases the toy segments (input-segments) are injected into the workflow using the same technique.

I wonder how you'd ingest from an external source, such as messages coming from a front-end system, if at all the scenario is well-suited with Onyx.

And also much related, this might shed some light for me on the role of plugins in the architecture.

Thanks in advance for answering,

Matan

[Mike Drogalis]

Lifecycles are behavioral hooks to that augment Onyx's runtime functionality. Plugins feed data into and out of the Onyx cluster. For some plugins, like core.async, we need to communicate the channel to use for input or output. Lifecycles give you a hook to pass in your channel so that Onyx can get a reference to it. Note that using core.async is only advised for development, since the runtime that gets access to the channel needs to be known - and thus only realistic for a single node cluster.

For production, use the others plugins - onyx-kafka, onyx-datomic, onyx-sql, etc. A full listing is available on the project README, among various other places.

--
You received this message because you are subscribed to the Google Groups "Onyx" group.
To unsubscribe from this group and stop receiving emails from it, send an email to onyx-user+unsubscribe@googlegroups.com.
To post to this group, send email to onyx...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/onyx-user/5310ed99-6e3c-4f7b-98b6-8b73c0b7cb46%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Matan Safriel]

Thanks Mike, I can kind of follow your description of plugins, channels and lifecycles, but I still fail to see, maybe at the conceptual level, why something like a plugin is needed, why simple API (fn's in the case of clojure) aren't enough. There seems to be an underlying complexity which I simply do not see yet, which likely explains why the architecture requires that machinery.

Maybe somebody can take it from that basic starting point...