Problem: Starting pipelines using CURL POST / LABEL mode ( API REST FULL)

24 views
Skip to first unread message

Fabio Baldaconi

unread,
Dec 12, 2019, 9:47:24 AM12/12/19
to sdc-user

Good Morning,
I come to the community for help in solving a drive problem via CURL.
I need to know if there is any way to do "CURL POST -X" by pointing to a "LABEL" containing several pipelines.

Example with a "1D" LABEL is executed by POST and all those with this tag will be executed.

I can not tell if there is this called via "API RET FULL" but if there is no other way to call via LABEL ???

-----------------------------

Example in the documentation for a pipelines
#curl -X POST http://<SDCEdge_hostname>:<port>/rest/v1/pipelines/start 

Example in the documentation for a specific pipeline

#curl -X POST http://<SDCEdge_hostname>:<port>/rest/v1/pipeline/<pipelineID>/start -H 'Content-Type: application/json;charset=UTF-8' --data-binary '{"<parameter_name1>":"<parameter_value1>","<parameter_name2":"<parameter_value2>"}'

Pat Patterson

unread,
Dec 12, 2019, 11:47:50 AM12/12/19
to Fabio Baldaconi, sdc-user
Hi Fabio,

There is no way to directly start a group of pipelines in SDC Edge by specifying a label. The closest thing I can think of would be to have a script retrieve the pipelines via their label - GET /rest/v1/pipelines?label=prod - then iterate through the resulting pipelines, starting each one.

Cheers,

Pat

--

Pat Patterson | Director of Evangelism | http://about.me/patpatterson


--
You received this message because you are subscribed to the Google Groups "sdc-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sdc-user+u...@streamsets.com.
To view this discussion on the web visit https://groups.google.com/a/streamsets.com/d/msgid/sdc-user/a7979b5f-76fc-4991-b36c-52b76da0c107%40streamsets.com.

Fabio Baldaconi

unread,
Dec 12, 2019, 1:37:28 PM12/12/19
to Pat Patterson, sdc-user
Hi Pat, how are you ?

Thank you so much for answering my question I am very happy, I understood the positioning but I do not have enough knowledge in creating this pyhton this way ... would have some sample code or some documentation that instructs how to make this deal? Thank you,  



 


                              

Fabio Baldaconi Roldão

fabio.baldaconi@roxpartner.com

11 2657 7575 | 11 98094-5021

Rua Arquiteto Olavo Redig de Campos, 105 – EZTOWER Torre B

       São Paulo - Brazil


Pat Patterson

unread,
Dec 12, 2019, 2:21:08 PM12/12/19
to Fabio Baldaconi, sdc-user
Hi Fabio,

Here is some sample Python. My edge pipelines are actually stored in Data Collector, so I can list and run them via the Data Collector REST API:

import requests

USERNAME='admin'
PASSWORD='admin'
SDC_URL='http://localhost:18630'
LABEL='mylabel'

pipelines = requests.get(f'{SDC_URL}/rest/v1/pipelines?label={LABEL}',
                         auth=(USERNAME, PASSWORD),
                         headers={'X-Requested-By': 'script'}).json()

for pipeline in pipelines:
    response = requests.post(f'{SDC_URL}/rest/v1/pipeline/{pipeline["pipelineId"]}/start',
                             auth=(USERNAME, PASSWORD),
                             headers={'X-Requested-By': 'script'}).json()
    print(f'{response["pipelineId"]} status is {response["status"]}')


The Edge REST API doesn't seem to recognize the label parameter, so it might be tricky to do this directly against the Edge instance.

Cheers,

Pat

--

Pat Patterson | Director of Evangelism | http://about.me/patpatterson

Pradeep Reddy Chinthala

unread,
Mar 21, 2022, 12:13:23 PMMar 21
to sdc-user, Pat Patterson, sdc-user, Fabio Baldaconi
Hi Pat, 
I was writing below python script to start my SDC pipeline through rest API call. I am getting response code 200 but my pipeline is not triggered. Am I missing anything here?

import requests
import json
import ssl


def run_job_instances():

    endpoint_url ='https://localhost:18630/rest/v1/pipeline/<my_pipeline_id>/start'
    # # Final response
    auth = ('USERNAME', 'PASSWORD')
    headers = {"content-type": "application/json"}
    response = requests.post(url=endpoint_url, auth = auth, headers=headers, json=jobdata)
    if response.status_code != 200:
        print ('Error pipeline running')
    # #      Label(login_screen, text="Error creating Job Instances").pack()
        print (response.status_code)
        print (response)
        exit(-1)
    else:
        print ('Started pipeline',response)
if __name__ == '__main__':
    try:
        _create_unverified_https_context = ssl._create_unverified_context
    except AttributeError:

    # Legacy Python that doesn't verify HTTPS certificates by default

        pass
    else:

    # Handle target environment that doesn't support HTTPS verification
        ssl._create_default_https_context = _create_unverified_https_context
    run_job_instances()

Drew Kreiger

unread,
Mar 28, 2022, 4:06:51 PMMar 28
to sdc-user, pradeeprdy...@gmail.com, sdc-user, Fabio Baldaconi
Hi Pradeep, 

I noticed that you have a Community Platform account. Please add your question to our new Community Forum as someone internally will help answer your question. The new community Forum will be the new home for StreamSetters to get their questions answered and receive helpful resources. 


Best, 
Drew Kreiger
Senior Community Manager @ StreamSets
Reply all
Reply to author
Forward
0 new messages