Triggering DAG from a Python script

1,488 views
Skip to first unread message

Balázs Bartl

unread,
Jun 27, 2018, 8:00:16 AM6/27/18
to cloud-compo...@googlegroups.com
Hello all,

i would like to trigger my airflow DAG from a python script. I found a way in the documentation to triggering it from a cloud function:
For the python specific part of the job, I found help here:
With these help, I was able to write a short script, which fetches me a jwt token. If I decompile it, its payload looks like this:
{
  "sub": "108920719766590804550",
  "email_verified": true,
  "exp": 1530100963,
  "iat": 1530097363
}

The service account identified by the SOME-ID-com...@developer.gserviceaccount.com mail address has the following roles in the same project, where the composer environment resides:
  • BigQuery Data Editor
  • Composer User
  • Editor
  • Service Account Token Creator
I target the following URL with my POST request:

When I send the request to Composer with this token as an "Authorization: Bearer" header, I get an HTTP 401 response. 

Could you please help me, what could be wrong with my request?

Thank you very much.

Wilson Lian

unread,
Jun 27, 2018, 9:53:01 PM6/27/18
to Balázs Bartl, cloud-composer-discuss
It's hard to say without seeing your python script if you're missing something, but just to be sure, are you getting a Google-signed JWT (aka OpenID Connect token)?

best,
Wilson

On Wed, Jun 27, 2018 at 5:00 AM, Balázs Bartl <balazs...@doctusoft.com> wrote:
Hello all,

i would like to trigger my airflow DAG from a python script. I found a way in the documentation to triggering it from a cloud function:
For the python specific part of the job, I found help here:
With these help, I was able to write a short script, which fetches me a jwt token. If I decompile it, its payload looks like this:
{
  "sub": "108920719766590804550",
  "email_verified": true,
  "exp": 1530100963,
  "iat": 1530097363
}

The service account identified by the SOME-ID-compute@developer.gserviceaccount.com mail address has the following roles in the same project, where the composer environment resides:
  • BigQuery Data Editor
  • Composer User
  • Editor
  • Service Account Token Creator
I target the following URL with my POST request:

When I send the request to Composer with this token as an "Authorization: Bearer" header, I get an HTTP 401 response. 

Could you please help me, what could be wrong with my request?

Thank you very much.

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-discuss+unsub...@googlegroups.com.
To post to this group, send email to cloud-composer-discuss@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/CAGhLTk3SA6BVW65YQGp-NfCRXEHGDTs6bGPTimrMbrYbmbhezw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Tim Swast

unread,
Jun 28, 2018, 4:14:04 PM6/28/18
to balazs...@doctusoft.com, cloud-composer-discuss
Where is this Python script running from? Based on SOME-ID...@developer.gserviceaccount.com I assume either GCE or GKE? I believe you have found the correct sample to follow, if that's the case. It should be accessing via a service account: https://cloud.google.com/iap/docs/authentication-howto#authenticating_from_a_service_account

I believe that access to the Composer web environment is handled via the Composer User role (someone on the Composer team, please verify). You may need to add that role to the service account.

  •  Tim Swast
  •  Software Friendliness Engineer
  •  Google Cloud Developer Relations
  •  Seattle, WA, USA


--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.
To post to this group, send email to cloud-compo...@googlegroups.com.

Balázs Bartl

unread,
Jul 2, 2018, 4:03:48 AM7/2/18
to sw...@google.com, cloud-compo...@googlegroups.com
Hi,

at the moment, its running on my local workstation. As Wilson pointed out, I my issue was the wrong client id.

Thank you all.

Wilson Lian

unread,
Jul 2, 2018, 2:30:04 PM7/2/18
to Balázs Bartl, Tim Swast, cloud-composer-discuss
My reply from the off-thread discussion was:

<PASTE>
Looks like you have the client_id set to your .appspot.com URL. The client ID should be a string of the form http://[-0-9a-z]*.apps.googleusercontent.com. It appears in the URL as the client_id URL parameter on the OAuth account picker page when you first authenticate to the Airflow Web UI. If you've already clicked through the account picker page in your browser session, you can force it to show again by clearing your cookies or accessing the Airflow Web UI in incognito/private browsing mode.
</PASTE>

On Mon, Jul 2, 2018 at 1:03 AM, Balázs Bartl <balazs...@doctusoft.com> wrote:
Hi,

at the moment, its running on my local workstation. As Wilson pointed out, I my issue was the wrong client id.

Thank you all.
On Thu, Jun 28, 2018 at 10:14 PM Tim Swast <sw...@google.com> wrote:
Where is this Python script running from? Based on SOME-ID-compute@developer.gserviceaccount.com I assume either GCE or GKE? I believe you have found the correct sample to follow, if that's the case. It should be accessing via a service account: https://cloud.google.com/iap/docs/authentication-howto#authenticating_from_a_service_account

I believe that access to the Composer web environment is handled via the Composer User role (someone on the Composer team, please verify). You may need to add that role to the service account.

  •  Tim Swast
  •  Software Friendliness Engineer
  •  Google Cloud Developer Relations
  •  Seattle, WA, USA

On Wed, Jun 27, 2018 at 5:00 AM Balázs Bartl <balazs...@doctusoft.com> wrote:
Hello all,

i would like to trigger my airflow DAG from a python script. I found a way in the documentation to triggering it from a cloud function:
For the python specific part of the job, I found help here:
With these help, I was able to write a short script, which fetches me a jwt token. If I decompile it, its payload looks like this:
{
  "sub": "108920719766590804550",
  "email_verified": true,
  "exp": 1530100963,
  "iat": 1530097363
}

The service account identified by the SOME-ID-compute@developer.gserviceaccount.com mail address has the following roles in the same project, where the composer environment resides:
  • BigQuery Data Editor
  • Composer User
  • Editor
  • Service Account Token Creator
I target the following URL with my POST request:

When I send the request to Composer with this token as an "Authorization: Bearer" header, I get an HTTP 401 response. 

Could you please help me, what could be wrong with my request?

Thank you very much.

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-discuss+unsub...@googlegroups.com.
To post to this group, send email to cloud-composer-discuss@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-discuss+unsub...@googlegroups.com.
To post to this group, send email to cloud-composer-discuss@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/CAGhLTk36Q4CyDvAzAkFk2M5%2BuKJLTcuaVYSEBApTRV-phpzVdg%40mail.gmail.com.
Reply all
Reply to author
Forward
0 new messages