Access to Driver via MorphlinesContext

[Eric Pheatt]

Is there any established provision within morphlines commands for accessing the driver program via the MorphlinesContext or other mechanisims for caching, connection reuse or other purposes? 

I have a simple HTTPClient based morphlines command working to send data to a kafka-rest-proxy that discovers the rest URL endpoint using HOCON SystemProperties/Environment variables resolution but I would rather pass an initialized KafkaProducer instance (https://github.com/epheatt/kafka-connect-morphlines/commit/43973d33d377ef973f2c310eba9b5bf68d9e2d56#diff-c908326dc8686289f9e8170788e40273R225) from the driver program so that I can use the native producer and manage/reuse the connection outside of the command context and only worry about the payload formulation in the command. I also would like to access the driver context for cache purposes so that when I am interacting with the confluent schema registry so that I can cache the schema/schemaId for reuse and evolution between command invocations for the duration of the driver program.

Thanks,

Eric 

[Wolfgang Hoschek]

For shared state like the one you are describing, consider passing it using MorphlineContext.getSettings() for writes and MorphlineContext.getTypedSettings() for convenient reads.

Wolfgang

--
You received this message because you are subscribed to the Google Groups "CDK Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cdk-dev+u...@cloudera.org.
For more options, visit https://groups.google.com/a/cloudera.org/d/optout.

[Eric Pheatt]

From a lifecycle perspective I can see that the kafkaProducer command is available by calling getCommandBuilder("kafkaProducer") on the MorphlineContext used for compilation but I can't tell how to determine if it was requested to be compiled into the morphline chain other that checking the recusively. Is there any example to short cut this so that I can just push the producer in as an object reference in the settings map if it is needed or am I stuck with potentially signaling within the command builder context settings that I want a native producer and if it is provided by the driver then use it when the doProcess is called, otherwise fallback to a rest producer?

Thanks,

Eric

[Wolfgang Hoschek]

I'm not quite following what role getCommandBuilder() would play in your use case.

In any case, a driver can pass arbitrary java objects to morphline commands via MorphlineContext.Builder.setSettings(). The morphline commands can retrieve these java objects via AbstractCommand.getContext().getSettings().get(). You can use this mechanism to pass whatever state you like from the outside world to morphline commands.

Wolfgang

[Eric Pheatt]

Ok thanks, that seems to be working in my happy path testing locally using docker from my latest commit https://github.com/epheatt/kafka-connect-morphlines/commit/99dff66711b4ea867bb0ed70056660ad7f0430b6, I guess I just need to sort out a unit test suite to properly cover the switch between the two types of producers when one wants to use rest and the other native. 

To unsubscribe from this group and stop receiving emails from it, send an email to cdk-dev+unsubscribe@cloudera.org.