Db url changes? how to update sql_alchemy_conn?

[r0ger]

We are using airflow with aws aurora. It has been pretty stable so far (1 week)

I was wondering if my db failsover, how do I update the connection user in airflow.cfg (sql_alchemy_conn) ? 

More generally:

1. can I update the connection url once my airflow is up and running?

2. Also, the db connection is needs credentials in clear text. Is there a way to provide credentials at run time so that the passwords are not exposed? This is an important issue for us, right now.

[r0ger]

I figured how to go about doing this. But I am having some issues.

I would really appreciate some help here to guide me through what I am doing wrong.

So the way I am getting my db connection url is:  sql_alchemy_conn_cmd = bash_command_to_run

This is my bash command is: 

sql_alchemy_conn_cmd = `python ~/setup/configure.py`

(configure.py is my python script located in the 'setup' directory under my ${AIRFLOW_HOME}  )

When I try to start airflow, this is the error I see:

Traceback (most recent call last):
  File "/usr/local/bin/airflow", line 4, in <module>
    from airflow import configuration
  File "/usr/local/lib/python2.7/dist-packages/airflow/__init__.py", line 16, in <module>
    from airflow.models import DAG
  File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 40, in <module>
    from airflow import settings, utils
  File "/usr/local/lib/python2.7/dist-packages/airflow/settings.py", line 51, in <module>
    SQL_ALCHEMY_CONN = conf.get('core', 'SQL_ALCHEMY_CONN')
  File "/usr/local/lib/python2.7/dist-packages/airflow/configuration.py", line 520, in get
    return conf.get(section, key, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/airflow/configuration.py", line 416, in get
    return run_command(command)
  File "/usr/local/lib/python2.7/dist-packages/airflow/configuration.py", line 50, in run_command
    command.split(), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
  File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

[r0ger]

nevermind:

just do this. 

sql_alchemy_conn_cmd = python ~/setup/configure.py

I know it is silly. apologies. hopefully, it helps someone.

But my first question still remains: 

[Maxime Beauchemin]

> can I update the connection url once my airflow is up and running?

Nope, or you'd have to restart the service. Why would you do that?

Max

[Lance N.]

Usually people handle this problem with DNS aliases. "airflow_db.x.com" points to your master. When you fail over (or promote), change the DNS name to point to the new one.

[r0ger]

> can I update the connection url once my airflow is up and running?

Nope, or you'd have to restart the service. Why would you do that?

-- Reason: we use aws aurora. It usually has multiple cluster-endpoints (basically mysql connection url) with one being active at a time.

This can flip any moment. 

So there should probably a wrapper to provide list of endpoints instead of one?

Something like mariadb (which is for jdbc): jdbc:mysql:replication://master1,slave1/test
I am not recommending anything. But I just wanted to put it out there.  But this is one of the issues we will eventually deal with in our company. 

Thanks Lance, I am going to look into this now.

[Maxime Beauchemin]

We use Amazon RDS (MySQL) and it works very well for us.

Max