Where can I download plugin jars from so I can do a terraform/API upload?

[Darragh]

I've gone through huge amounts of docs and stackoverflow around uploading plugins, and while I've found copious examples of how to upload the plugin jars ( stock plugins from Hub such as hydrator-plugin-python-transform and v), I cannot for the life of me understand where to download the jars from so that I can upload them to my own CDAP/DataFusion instance...

Am I missing something fundamental here? I don't want to have to build them as that defeats the purpose, and when deploying from the DataFusion Hub UI it all just magically works, but how the hell do I get hold of the actual plugin jars so I can upload as part of my CI/CD automation??

[Albert Shau]

Hi Darragh,

If you're trying to automate the Hub plugin deployment, take a look at https://cdap.atlassian.net/wiki/spaces/DOCS/pages/554401840/Hub+API, which describes how it is setup.

For example, by default a CDAP install will use a base url of 'https://hub.cdap.io/v2'. If you want to install the database-plugins plugin, that means you can pull the jar from https://hub.cdap.io/v2/packages/hydrator-plugin-database-plugin/2.9.2/database-plugins-2.9.2.jar and configuration json from https://hub.cdap.io/v2/packages/hydrator-plugin-database-plugin/2.9.2/database-plugins-2.9.2.json.

Regards,

Albert

On Mon, Oct 10, 2022 at 5:33 AM Darragh <gcp-analyt...@cygnvs.com> wrote:

I've gone through huge amounts of docs and stackoverflow around uploading plugins, and while I've found copious examples of how to upload the plugin jars ( stock plugins from Hub such as hydrator-plugin-python-transform and v), I cannot for the life of me understand where to download the jars from so that I can upload them to my own CDAP/DataFusion instance...

Am I missing something fundamental here? I don't want to have to build them as that defeats the purpose, and when deploying from the DataFusion Hub UI it all just magically works, but how the hell do I get hold of the actual plugin jars so I can upload as part of my CI/CD automation??

--
You received this message because you are subscribed to the Google Groups "CDAP User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cdap-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/dde1b555-104e-47b4-b411-7f8d8ed94772n%40googlegroups.com.

[Darragh]

Hi Albert,

Thanks for the reply! I'm trying to pull 2 plugin jars, and using the base url pattern you gave me I'm getting "the specific key does not exist"...

https://hub.cdap.io/v2/packages/plugin-kinesis-streaming-source/2.0.0/plugin-kinesis-streaming-source-2.0.0.jar 

https://hub.cdap.io/v2/packages/hydrator-plugin-python-transform/2.2.2/hydrator-plugin-python-transform-2.2.2.jar 

Both jars are listed in here:  https://hub.cdap.io/v2/packages.json  

  { "name": "hydrator-plugin-python-transform", "version": "2.2.2", "description": "A plugin that can execute user-provided python code to transform input record into zero or more records. Source code available at https://github.com/data-integrations/python-plugins.", "label": "Python Transform", "author": "Cask", "org": "Cask Data, Inc.", "cdapVersion": "[6.0.0-SNAPSHOT, 7.0.0-SNAPSHOT)", "created": 1616022252, "beta": false, "categories": [ "hydrator-plugin" ], "paid": false },

  { "name": "plugin-kinesis-streaming-source", "version": "2.0.0", "description": "Amazon Kinesis Streaming Source (Scala 2.12)", "label": "Amazon Kinesis Streaming Source", "author": "Cask", "org": "Cask Data, Inc.", "cdapVersion": "[6.4.0-SNAPSHOT,7.0.0-SNAPSHOT)", "created": 1662564116, "beta": false, "categories": [ "AWS", "hydrator-plugin" ], "paid": false } 

And using the kinesis plugin as an example, I can get access to the spec.json using this url: https://hub.cdap.io/v2/packages/plugin-kinesis-streaming-source/2.0.0/spec.json , but no sign of the jar itself...

Am I using the wrong endpoint, or missing some config when making the call?

Thanks,

Darragh

[Darragh]

Hi Albert,

Please ignore the previous, I completely typoed my own URLS :facepalm: 

I have the jars and json, and the pattern to get access to any others I need know.

Thanks for your help!!