Problem: Starting pipelines using CURL POST / LABEL mode ( API REST FULL)

[Fabio Baldaconi]

Good Morning,

I come to the community for help in solving a drive problem via CURL.

I need to know if there is any way to do "CURL POST -X" by pointing to a "LABEL" containing several pipelines.

Example with a "1D" LABEL is executed by POST and all those with this tag will be executed.

I can not tell if there is this called via "API RET FULL" but if there is no other way to call via LABEL ???

-----------------------------

Example in the documentation for a pipelines

#curl -X POST http://<SDCEdge_hostname>:<port>/rest/v1/pipelines/start 

Example in the documentation for a specific pipeline

#curl -X POST http://<SDCEdge_hostname>:<port>/rest/v1/pipeline/<pipelineID>/start -H 'Content-Type: application/json;charset=UTF-8' --data-binary '{"<parameter_name1>":"<parameter_value1>","<parameter_name2":"<parameter_value2>"}'

[Pat Patterson]

Hi Fabio,

There is no way to directly start a group of pipelines in SDC Edge by specifying a label. The closest thing I can think of would be to have a script retrieve the pipelines via their label - GET /rest/v1/pipelines?label=prod - then iterate through the resulting pipelines, starting each one.

Cheers,

Pat

--

Pat Patterson | Director of Evangelism | http://about.me/patpatterson

--
You received this message because you are subscribed to the Google Groups "sdc-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sdc-user+u...@streamsets.com.
To view this discussion on the web visit https://groups.google.com/a/streamsets.com/d/msgid/sdc-user/a7979b5f-76fc-4991-b36c-52b76da0c107%40streamsets.com.

[Fabio Baldaconi]

Hi Pat, how are you ?

Thank you so much for answering my question I am very happy, I understood the positioning but I do not have enough knowledge in creating this pyhton this way ... would have some sample code or some documentation that instructs how to make this deal? Thank you,  

Fabio Baldaconi Roldão fabio.baldaconi@roxpartner.com 11 2657 7575 | 11 98094-5021 Rua Arquiteto Olavo Redig de Campos, 105 – EZTOWER Torre B        São Paulo - Brazil

[Pat Patterson]

Hi Fabio,

Here is some sample Python. My edge pipelines are actually stored in Data Collector, so I can list and run them via the Data Collector REST API:

import requests

USERNAME='admin'
PASSWORD='admin'
SDC_URL='http://localhost:18630'
LABEL='mylabel'

pipelines = requests.get(f'{SDC_URL}/rest/v1/pipelines?label={LABEL}',
                         auth=(USERNAME, PASSWORD),
                         headers={'X-Requested-By': 'script'}).json()

for pipeline in pipelines:
    response = requests.post(f'{SDC_URL}/rest/v1/pipeline/{pipeline["pipelineId"]}/start',
                             auth=(USERNAME, PASSWORD),
                             headers={'X-Requested-By': 'script'}).json()
    print(f'{response["pipelineId"]} status is {response["status"]}')

The Edge REST API doesn't seem to recognize the label parameter, so it might be tricky to do this directly against the Edge instance.

Cheers,

Pat

--

Pat Patterson | Director of Evangelism | http://about.me/patpatterson

[Pradeep Reddy Chinthala]

Hi Pat, 

I was writing below python script to start my SDC pipeline through rest API call. I am getting response code 200 but my pipeline is not triggered. Am I missing anything here?

import requests
import json
import ssl


def run_job_instances():

    endpoint_url ='https://localhost:18630/rest/v1/pipeline/<my_pipeline_id>/start'
    # # Final response
    auth = ('USERNAME', 'PASSWORD')
    headers = {"content-type": "application/json"}
    response = requests.post(url=endpoint_url, auth = auth, headers=headers, json=jobdata)
    if response.status_code != 200:
        print ('Error pipeline running')
    # #      Label(login_screen, text="Error creating Job Instances").pack()
        print (response.status_code)
        print (response)
        exit(-1)
    else:
        print ('Started pipeline',response)
if __name__ == '__main__':
    try:
        _create_unverified_https_context = ssl._create_unverified_context
    except AttributeError:

    # Legacy Python that doesn't verify HTTPS certificates by default

        pass
    else:

    # Handle target environment that doesn't support HTTPS verification
        ssl._create_default_https_context = _create_unverified_https_context
    run_job_instances()

[Drew Kreiger]

Hi Pradeep, 

I noticed that you have a Community Platform account. Please add your question to our new Community Forum as someone internally will help answer your question. The new community Forum will be the new home for StreamSetters to get their questions answered and receive helpful resources. 

https://community.streamsets.com

Best, 

Drew Kreiger

Senior Community Manager @ StreamSets