Create connection extra parameter

[Praveenkumar Venkatesan]

Have a qn related to "extra" parameter when creating a connection.

In order to use DruidHook, I created a connection with conn id "druid_query_default" and "druid_ingest_default" where

Extra = "{\"endpoint\": \"/druid/indexer/v1/task\"}"

with the outer quotes. But I received an error saying 'unicode' object has no attribute 'get' on

endpoint = conn.extra_dejson.get('endpoint', '')

I believe when obj = json.loads(self.extra) is executed, obj is in unicode {u'endpoint': u'/druid/indexer/v1/task'}. Hence 'get' doesn't work.

Am I passing the dictionary in the wrong format?

[Maxime Beauchemin]

The extra field accepts normal json. You can copy the following string in there and it would work, no need to quote/escape:

{'endpoint': '/druid/indexer/v1/task'}

If there was a field called "name" you wouldn't quote the value either...

Max

[Praveenkumar Venkatesan]

my bad. I forgot to explain how I ended up with that escaped json. I tried the normal JSON like the one you specified but here is what I got.


[2015-09-30 22:39:41,657] {models.py:405} ERROR - Expecting property name: line 1 column 2 (char 1)
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/airflow/models.py", line 403, in extra_dejson
    obj = json.loads(self.extra)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 382, in raw_decode
    obj, end = self.scan_once(s, idx)
ValueError: Expecting property name: line 1 column 2 (char 1)
[2015-09-30 22:39:41,660] {models.py:408} ERROR - Failed parsing the json for conn_id druid_ingest_default


As per https://github.com/airbnb/airflow/blob/master/airflow/models.py#L407 ----- `json.loads({'endpoint': '/druid/indexer/v1/task'})`  errors out in the python console as well. But `json.loads('{"endpoint": "/druid/indexer/v1/task"}')` or the escaped one from the original question returns a unicode string in the dict.

[Maxime Beauchemin]

Sorry, my previous example should have been double quoted as in:

{"endpoint": "/druid/indexer/v1/task"}

Here's a demo of how it works from a short iPython session:

In [1]: import json

In [2]: s = '{"endpoint": "foo"}'

In [3]: json.loads(s)

Out[3]: {u'endpoint': u'foo'}

Of course in the form field you don't need the outer single quotes as shown in "In [2]".

[Praveenkumar Venkatesan]

thanks