How to run shell script in dataproc cluster using airflow

[Raj Shekar]

Hi All,

I am trying to execute the shell script from airflow in dataproc cluster it's not executing shell script, is there any way to execute shell script from airflow?.

status = subprocess.getstatusoutput('gcloud compute ssh clustername-m  --project "my_project" --zone "us-central1-a" --command "sh /webtooltodm_start_mongodb_conn.sh"')

Can anyone help here how to solve.

Thanks,

Rajashekar.

[Doulot, Shah]

Hello Raj,

 

Have you tried the Bash Operator (BashOperator)

 

Example;

 

create_cluster = BashOperator(

    dag=dag,

    task_id='Create_cluster',

    bash_command='if [ `{g_path}gcloud dataproc clusters list --region europe-west1 --project={project} | grep {cluster} | wc -l` -eq 0 ]; then `{0}`; \

                    else echo "cluster_exists";  fi'.format(CREATE_CLUSTER , **PARAMS),

    email_on_failure=True,

    trigger_rule=TriggerRule.ALL_SUCCESS

)

 

 

 

Shah Doulot | Data Engineer |

Data Technology & Analytic - Sky Networks

 

Sky UK
* shah....@sky.uk
( +44 207 032 2523

 

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.
To post to this group, send email to cloud-compo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/679d3921-2e3c-46be-af82-dc7958ebf00d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--------------------------------------------------------------------

This email is from an external source. Please do not open attachments or click links from an unknown or suspicious origin. Phishing attempts can be reported by sending them to phis...@sky.uk as attachments. Thank you

--------------------------------------------------------------------

 

Information in this email including any attachments may be privileged, confidential and is intended exclusively for the addressee. The views expressed may not be official policy, but the personal views of the originator. If you have received it in error, please notify the sender by return e-mail and delete it from your system. You should not reproduce, distribute, store, retransmit, use or disclose its contents to anyone. Please note we reserve the right to monitor all e-mail communication through our internal and external networks. SKY and the SKY marks are trademarks of Sky Limited and Sky International AG and are used under licence.

Sky UK Limited (Registration No. 2906991), Sky-In-Home Service Limited (Registration No. 2067075), Sky Subscribers Services Limited (Registration No. 2340150) and Sky CP Limited (Registration No. 9513259) are direct or indirect subsidiaries of Sky Limited (Registration No. 2247735). All of the companies mentioned in this paragraph are incorporated in England and Wales and share the same registered office at Grant Way, Isleworth, Middlesex TW7 5QD

[Raj Shekar]

Hi Doulot,

Thanks for your reply,

I tried that same but not worked for me.

May airflow runs this command in gke-cluster that's why may be.

start_mongodb = BashOperator(
        task_id= 'start_conn_mongodb',
        bash_command='gcloud compute ssh rajashekar.sunkesula@clustername-m  --project "myproject" --zone "us-central1-a" --command "sh /webtooltodm_start_mongodb_conn.sh"',
        dag=dag
   )

Thanks,

Rajashekar.