Is Rest API availabe for creating source and sink connections?

[Dhanu Mohan]

hi, 

I am using cdap-sandbox docker, all functionalities in Web UI working perfectly. But i am trying to access cdap via Rest API.

Is rest api available for create connections for source and sink? 

I need to create source and sink connections, then i need to create pipeline using the created source and sink connections via rest api.  

[Albert Shau]

Hi,

The CDAP UI is built on top of the REST APIs, so you can do those things through the REST API. You will need to make sure your docker container is exposing port 11015, which is what the REST API binds to. You can create a pipeline using the deploy app API documented here. You can export a pipeline that you created in the UI and use that as the request body. You can find more information about the pipeline config json here.

Regards,

Albert

--
You received this message because you are subscribed to the Google Groups "CDAP User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cdap-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/611bf4ac-d36a-4751-b873-123262170d60%40googlegroups.com.

[Dhanu Mohan]

Thanks for the response Mr.Albert. 

Yeah 11015 port is exposed and i can able to view the pipeline details via Rest API. But my actual requirement is, i need to create source/sink connections with user account credentials via REST Api like "Add Connection" functionality in UI.

On Thursday, September 19, 2019 at 1:45:57 AM UTC+5:30, Albert Shau wrote:

Hi,

The CDAP UI is built on top of the REST APIs, so you can do those things through the REST API. You will need to make sure your docker container is exposing port 11015, which is what the REST API binds to. You can create a pipeline using the deploy app API documented here. You can export a pipeline that you created in the UI and use that as the request body. You can find more information about the pipeline config json here.

Regards,

Albert

On Wed, Sep 18, 2019 at 5:13 AM Dhanu Mohan <dha...@gmail.com> wrote:

hi, 

I am using cdap-sandbox docker, all functionalities in Web UI working perfectly. But i am trying to access cdap via Rest API.

Is rest api available for create connections for source and sink? 

I need to create source and sink connections, then i need to create pipeline using the created source and sink connections via rest api.  

--
You received this message because you are subscribed to the Google Groups "CDAP User" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cdap...@googlegroups.com.

[Albert Shau]

I see. Those APIs are not documented, but if you are using CDAP 6.x, the REST API is:

POST [host]:[port]/v3/namespaces/system/apps/dataprep/services/service/methods/contexts/[namespace]/connections

with body:

{

   "type": ...,

   "name": ...,

   "description": ...,

   "properties": {

     ...

   }

}

it's easiest to know what to put in the body by creating one in the UI, then fetching the list with:

GET [host]:[port]/v3/namespaces/system/apps/dataprep/services/service/methods/contexts/[namespace]/connections

If you are using a previous CDAP version, the path is [host]:[port]/v3/namespaces/[namespace]/apps/dataprep/services/service/methods/connections.

If you would like to see the actual code, you can look at https://github.com/data-integrations/wrangler/blob/release/4.1/wrangler-service/src/main/java/io/cdap/wrangler/service/connections/ConnectionHandler.java#L166.

Regards,

Albert

To unsubscribe from this group and stop receiving emails from it, send an email to cdap-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/6a25b508-ba58-42ae-85e5-72a33bdf5994%40googlegroups.com.

[Dhanu Mohan]

Thanks for your valuable reply.

It works fine, i am able to create connections via rest api now. Thank you very much.

Few more question from me, 

1. How to use Http streaming poller? I need to create pipelines using this to fetch some api endpoint and load the response to some sink(destination).

2. I have tried local file upload and created pipeline using wrangler then connected to sink, but when i deploy pipline, i am getting the error "Wrangler: nodes missing connections". how colud i solve this issue?

3. And where to find uploaded file paths?

[Albert Shau]

Hi Dhanu,

Were you running into specific issues with the http poller? It should just repeatedly hit the same endpoint every batch interval, but you will be responsible for parsing the contents and doing something meaningful in the subsequent pipeline stages.

File upload is purely for experimenting with data within the Wrangler UI. The file contents don't get stored in a place that you can reference later in a pipeline. 

Regards,

Albert

--

You received this message because you are subscribed to the Google Groups "CDAP User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cdap-user+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/7bf8ad21-ee53-4985-8212-867bd49e7da3%40googlegroups.com.

[Dhanu Mohan]

Hi Albert,

Thank you for your timely response.

I am explaining my scenario here, 

1. I create connections through rest API, then i test connection also via rest api

2. I need to create a pipeline using above created connections, for that where to find the schema from the connected data sources using Rest API? 

    because to create pipelines, we need stages inputSchema & outputSchema.

To unsubscribe from this group and stop receiving emails from it, send an email to cdap...@googlegroups.com.