Airflow - Google Groups

Groups

Contact owners and managers

1–30 of 301

Airflow is now Apache Airflow (Incubating)

As part of that migration, this Google Group has become defunct and is now maintained in READ-ONLY mode. All active discussions must move to Apache mailing lists and Jira.

Please follow instructions at the link below:

https://groups.google.com/d/msg/airbnb_airflow/961jUu-yxR8/5c_WzmgEBAAJ

0 selected

Chris Riccomini, Siddharth Anand3

4/29/16

IMPORTANT: Airflow is migrating to Apache

I want to reiterate: please sign up for both commits and dev mailing lists. The commits mailing list

unread,

IMPORTANT: Airflow is migrating to Apache

I want to reiterate: please sign up for both commits and dev mailing lists. The commits mailing list

4/29/16

Jon Andrews, Maxime Beauchemin2

5/5/16

pip install airflow[postgres] throws RequirementParseError

`pip install pip --upgrade` ?

unread,

pip install airflow[postgres] throws RequirementParseError

`pip install pip --upgrade` ?

5/5/16

Jason Chen, … Maxime Beauchemin4

5/2/16

About start_date

Hi Maxime, Thanks for your reply. It was not my question. My question was "is it possible to

unread,

About start_date

Hi Maxime, Thanks for your reply. It was not my question. My question was "is it possible to

5/2/16

Siddharth Anand

4/29/16

Apache Airflow is now on Twitter

Keep abreast of Apache Airflow (Incubating) news by following our newly-minted Twitter account :

unread,

Apache Airflow is now on Twitter

Keep abreast of Apache Airflow (Incubating) news by following our newly-minted Twitter account :

4/29/16

Lance N., … Chris Riccomini6

4/29/16

Final task in a backfill is run over and over - caught in a weird failure loop

Comments are happening on the dev@ mailing list. Suggest you switch over to that.

unread,

Final task in a backfill is run over and over - caught in a weird failure loop

Comments are happening on the dev@ mailing list. Suggest you switch over to that.

4/29/16

Masih Tabrizi, Chris Riccomini3

4/29/16

Oracle Connection

Also, if you are able to connect to your Oracle DB outside of Airflow, then this: http://dba-oracle.

unread,

Oracle Connection

Also, if you are able to connect to your Oracle DB outside of Airflow, then this: http://dba-oracle.

4/29/16

Chris Riccomini2

4/28/16

Next Airflow meetup

Hey all, Just a reminder that this is happening. If you're interested in speaking, could you

unread,

Next Airflow meetup

Hey all, Just a reminder that this is happening. If you're interested in speaking, could you

4/28/16

4/28/16

Airflow fails at startup

sql_alchemy_conn_cmd = ./setup/configure.py I see a lot of processes started : Something weird going

unread,

Airflow fails at startup

sql_alchemy_conn_cmd = ./setup/configure.py I see a lot of processes started : Something weird going

4/28/16

Maxime Beauchemin, … r0ger7

4/28/16

Interesting blog post: "Airflow: Tips, Tricks, and Pitfalls"

got it!!! Thanks Maxime.

unread,

Interesting blog post: "Airflow: Tips, Tricks, and Pitfalls"

got it!!! Thanks Maxime.

4/28/16

r0ger, … Maxime Beauchemin7

4/27/16

Db url changes? how to update sql_alchemy_conn?

We use Amazon RDS (MySQL) and it works very well for us. Max

unread,

Db url changes? how to update sql_alchemy_conn?

We use Amazon RDS (MySQL) and it works very well for us. Max

4/27/16

Dennis O'Brien, Maxime Beauchemin2

4/27/16

ideas for troubleshooting deadlock problem?

It's not a database deadlock or anything, it just means some tasks failed and your DAG can't

unread,

ideas for troubleshooting deadlock problem?

It's not a database deadlock or anything, it just means some tasks failed and your DAG can't

4/27/16

David Palomino, Chris Riccomini3

4/27/16

attempting AD auth for Airflow

Thanks Chris! We dug into it a bit and found the problem was the quotes around the basedn value. I

unread,

attempting AD auth for Airflow

Thanks Chris! We dug into it a bit and found the problem was the quotes around the basedn value. I

4/27/16

Andrew O'Brien, Maxime Beauchemin4

4/26/16

Redundancy & Failover

> So if I understand correctly, I can just run multiple webservers (behind a load-balancer) and

unread,

Redundancy & Failover

> So if I understand correctly, I can just run multiple webservers (behind a load-balancer) and

4/26/16

Yuri Titov Bendana, … r0ger14

4/25/16

passing parameters to externally trigged dag

You can get the dag_run from the kwargs: def run_mytask(*args, **kwargs): dag_run = kwargs['

unread,

passing parameters to externally trigged dag

You can get the dag_run from the kwargs: def run_mytask(*args, **kwargs): dag_run = kwargs['

4/25/16

Siddharth Anand

4/22/16

Apache Airflow Talk at DataDaySeattle - July 23

Hi Folks! If you plan on being in the Seattle area on July 23, check out Data Day Seattle I'll be

unread,

Apache Airflow Talk at DataDaySeattle - July 23

Hi Folks! If you plan on being in the Seattle area on July 23, check out Data Day Seattle I'll be

4/22/16

Sergei Iakhnin, … Serega Sheypak13

4/22/16

Is it possible to send parameters to a workflow?

Thanks, but it doesn't work if you are not in UTC

unread,

Is it possible to send parameters to a workflow?

Thanks, but it doesn't work if you are not in UTC

4/22/16

Serega Sheypak, Frank Wang5

4/22/16

Pass external values to DAG

Thanks, I saw that, anyway it doesn't work if you you are not in UTC timezone :(

unread,

Pass external values to DAG

Thanks, I saw that, anyway it doesn't work if you you are not in UTC timezone :(

4/22/16

r0ger, Maxime Beauchemin6

4/22/16

backfill not behaving correctly?

works! I feel airflow has a lot of features that people can use but are not aware of. It can be a

unread,

backfill not behaving correctly?

works! I feel airflow has a lot of features that people can use but are not aware of. It can be a

4/22/16

r0ger, Maxime Beauchemin6

4/21/16

HTTP Operator always uses www.google.com?

awesome. so any env variable with "AIRFLOW_CONN_" prefix would work!!! Thanks. Sorry I dont

unread,

HTTP Operator always uses www.google.com?

awesome. so any env variable with "AIRFLOW_CONN_" prefix would work!!! Thanks. Sorry I dont

4/21/16

Serega Sheypak, Siddharth Anand3

4/21/16

trigger_dag writes creates dag_run record but doesn't start DAG. Example example_trigger_controller_dag doesn't start also

Hi, yes, you are right, it was my mistake. Backfill worked out of the box w/o scheduler and it

unread,

trigger_dag writes creates dag_run record but doesn't start DAG. Example example_trigger_controller_dag doesn't start also

Hi, yes, you are right, it was my mistake. Backfill worked out of the box w/o scheduler and it

4/21/16

r0ger, Serega Sheypak3

4/20/16

Using execution_date in python code for assigning variable

I did try that as well. I get an error: NameError: name 'execution_date' is not defined from

unread,

Using execution_date in python code for assigning variable

I did try that as well. I get an error: NameError: name 'execution_date' is not defined from

4/20/16

Yogesh Garg, Chris Riccomini2

4/19/16

Health Check of Airflow

Joy is working on a scheduler page to tell you the status of the scheduler in a bit more detail. I

unread,

Health Check of Airflow

Joy is working on a scheduler page to tell you the status of the scheduler in a bit more detail. I

4/19/16

4/19/16

Deploy airflow multi-tenant env: one scheduler multiple nodes deployment

We will have one pipeline per customer (1 DAG per customer). I am trying to think of how to build the

unread,

Deploy airflow multi-tenant env: one scheduler multiple nodes deployment

We will have one pipeline per customer (1 DAG per customer). I am trying to think of how to build the

4/19/16

4/18/16

Installation script with Ansible

Hello, I have created an Ansible role to install Airflow on Ubuntu / Debian. You can find the code

unread,

Installation script with Ansible

Hello, I have created an Ansible role to install Airflow on Ubuntu / Debian. You can find the code

4/18/16

Or Sher, Jeremiah Lowin4

4/18/16

Kafka -> S3 -> Redshift workflows

Hi Or, That shouldn't be an issue -- by default, XComs only search in the same DAG, but by

unread,

Kafka -> S3 -> Redshift workflows

Hi Or, That shouldn't be an issue -- by default, XComs only search in the same DAG, but by

4/18/16

Austin Barton, … Ludovic Claude8

4/18/16

Data/event driven DAG instantiation

Hello, I'm trying to do something similar here: I need to scan a directory containing sub-

unread,

Data/event driven DAG instantiation

Hello, I'm trying to do something similar here: I need to scan a directory containing sub-

4/18/16

4/18/16

Scheduler not scheduling DAG runs as expected

Hi, I' have a DAG that should be executed every hour and I need it to be scheduled precisely

unread,

Scheduler not scheduling DAG runs as expected

Hi, I' have a DAG that should be executed every hour and I need it to be scheduled precisely

4/18/16

Frank Wang, Or Sher3

4/18/16

how to update an existing dag without affecting re-try?

OK, tried it. It doesn't kick old schedules for the new Operators. Just the way I hoped it will

unread,

how to update an existing dag without affecting re-try?

OK, tried it. It doesn't kick old schedules for the new Operators. Just the way I hoped it will

4/18/16

Yogesh Garg, Jeremiah Lowin3

4/16/16

Why does celery workers start in Local/Sequential Executor mode ?

Yes, that makes sense. +1 for making it explicit. Thanks.

unread,

Why does celery workers start in Local/Sequential Executor mode ?

Yes, that makes sense. +1 for making it explicit. Thanks.

4/16/16

4/15/16

airflow.jobs.BackfillJob, how to pass configuration to DAG?

Hi, I'm writing automated e2e tests for Airflow flows. I'm using airflow.jobs.BackfillJob to

unread,

airflow.jobs.BackfillJob, how to pass configuration to DAG?

Hi, I'm writing automated e2e tests for Airflow flows. I'm using airflow.jobs.BackfillJob to

4/15/16

Dennis O'Brien4

4/13/16

debugging database connections

It looks like this was a bug in 1.7.0 but has already been addressed. 1.7.0: https://github.com/

unread,

debugging database connections

It looks like this was a bug in 1.7.0 but has already been addressed. 1.7.0: https://github.com/

4/13/16

Search

Clear search

Close search

Google apps

Main menu