Upgrade to Master 3/8

[salva....@gmail.com]

Hi Group,

I've just upgraded from the version installed by pip install panoramix to the current master.

cloned the master and executed python setup.py.

I'm running in test so just to start clean, I removed the previous config rm /tmp/panoramix.db and re-created it from scratch:

# Create an admin user
fabmanager create-admin --app panoramix
# Initialize the database
panoramix db upgrade
# Create default roles and permissions
panoramix init
# Load some data to play with
panoramix load_examples
# Start the development web server
panoramix runserver -d

Now my interface crashes right on the welcome page:

http://server:8088 redirects to http://server:8088/panoramix/featured

Where I get an attribute error:
 

2016-03-08 22:15:27,236:INFO:werkzeug: * Debugger pin code: 150-320-126
2016-03-08 22:15:39,327:INFO:werkzeug:72.27.253.99 - - [08/Mar/2016 22:15:39] "GET / HTTP/1.1" 302 -
2016-03-08 22:15:39,543:DEBUG:MARKDOWN:Successfuly imported extension module "markdown.extensions.tables".
2016-03-08 22:15:39,543:DEBUG:MARKDOWN:Successfully loaded extension "markdown.extensions.tables.TableExtension".
2016-03-08 22:15:39,546:DEBUG:MARKDOWN:Successfuly imported extension module "markdown.extensions.fenced_code".
2016-03-08 22:15:39,546:DEBUG:MARKDOWN:Successfully loaded extension "markdown.extensions.fenced_code.FencedCodeExtension".
2016-03-08 22:15:39,546:DEBUG:MARKDOWN:Successfuly imported extension module "markdown.extensions.codehilite".
2016-03-08 22:15:39,546:DEBUG:MARKDOWN:Successfully loaded extension "markdown.extensions.codehilite.CodeHiliteExtension".
2016-03-08 22:15:39,620:DEBUG:MARKDOWN:Successfuly imported extension module "markdown.extensions.tables".
2016-03-08 22:15:39,621:DEBUG:MARKDOWN:Successfully loaded extension "markdown.extensions.tables.TableExtension".
2016-03-08 22:15:39,621:DEBUG:MARKDOWN:Successfuly imported extension module "markdown.extensions.fenced_code".
2016-03-08 22:15:39,621:DEBUG:MARKDOWN:Successfully loaded extension "markdown.extensions.fenced_code.FencedCodeExtension".
2016-03-08 22:15:39,621:DEBUG:MARKDOWN:Successfuly imported extension module "markdown.extensions.codehilite".
2016-03-08 22:15:39,621:DEBUG:MARKDOWN:Successfully loaded extension "markdown.extensions.codehilite.CodeHiliteExtension".
2016-03-08 22:15:39,632:INFO:werkzeug:72.27.253.99 - - [08/Mar/2016 22:15:39] "GET /panoramix/featured HTTP/1.1" 500 -
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1836, in __call__
    return self.wsgi_app(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1820, in wsgi_app
    response = self.make_response(self.handle_exception(e))
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1403, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python2.7/dist-packages/flask_appbuilder/security/decorators.py", line 26, in wraps
    return f(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/panoramix-0.7.1-py2.7.egg/panoramix/views.py", line 735, in featured
    utils=utils)
  File "/usr/local/lib/python2.7/dist-packages/flask_appbuilder/baseviews.py", line 157, in render_template
    return render_template(template, **dict(list(kwargs.items()) + list(self.extra_args.items())))
  File "/usr/local/lib/python2.7/dist-packages/flask/templating.py", line 128, in render_template
    context, ctx.app)
  File "/usr/local/lib/python2.7/dist-packages/flask/templating.py", line 110, in _render
    rv = template.render(context)
  File "/usr/local/lib/python2.7/dist-packages/jinja2/environment.py", line 989, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/local/lib/python2.7/dist-packages/jinja2/environment.py", line 754, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python2.7/dist-packages/panoramix-0.7.1-py2.7.egg/panoramix/templates/panoramix/featured.html", line 1, in top-level template code
    {% extends "panoramix/basic.html" %}
  File "/usr/local/lib/python2.7/dist-packages/panoramix-0.7.1-py2.7.egg/panoramix/templates/panoramix/basic.html", line 30, in top-level template code
    {% block body %}
  File "/usr/local/lib/python2.7/dist-packages/panoramix-0.7.1-py2.7.egg/panoramix/templates/panoramix/featured.html", line 23, in block "body"
    <p>{{ utils.markdown(dataset.description) | safe }}</p>
  File "/usr/local/lib/python2.7/dist-packages/panoramix-0.7.1-py2.7.egg/panoramix/utils.py", line 263, in markdown
    'markdown.extensions.codehilite',])
  File "/usr/local/lib/python2.7/dist-packages/markdown/__init__.py", line 494, in markdown
    return md.convert(text)
  File "/usr/local/lib/python2.7/dist-packages/markdown/__init__.py", line 355, in convert
    if not source.strip():
AttributeError: 'NoneType' object has no attribute 'strip'

Reason for the upgrade is that I was having trouble using my data from druid and I wanted to test the latest code in case there was any enhancement in the area. If the code is still in flux and it's better to run something older, just let me know.

Thanks in advance.

Salva

[Maxime Beauchemin]

Hi,

Thanks for reporting this, I just pushed a hotfix to fix that for you:

https://github.com/mistercrunch/panoramix/commit/370c5af425b226050c9253a0aae38426fcbf1f98

Let me know if it works!

Max

[salva....@gmail.com]

Fixed!!

Now I got another one:

Reproduction:

clean /tmp/panoramix.db

git clone 

python setup.py install

fabmanager create-admin -app panoramix

panoramix db upgrade

panoramix init

panoramix load_examples

panoramix runserver -d

Login goes ok

Setup my druid cluster, note: it is a poc setup using imply, broker and coordinator are in the same node ports 8081 and 8082, urls are default. 

refresh metadata.

After refreshing metadata I get this exception:

2016-03-09 13:45:21,131:DEBUG:requests.packages.urllib3.connectionpool:"GET /druid/coordinator/v1/metadata/datasources HTTP/1.1" 200 None

2016-03-09 13:45:21,637:INFO:werkzeug:72.27.253.99 - - [09/Mar/2016 13:45:21] "GET /panoramix/refresh_datasources/ HTTP/1.1" 500 -

Traceback (most recent call last):

  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1836, in __call__

    return self.wsgi_app(environ, start_response)

  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1820, in wsgi_app

    response = self.make_response(self.handle_exception(e))

  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1403, in handle_exception

    reraise(exc_type, exc_value, tb)

  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1817, in wsgi_app

    response = self.full_dispatch_request()

  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1477, in full_dispatch_request

    rv = self.handle_user_exception(e)

  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1381, in handle_user_exception

    reraise(exc_type, exc_value, tb)

  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1475, in full_dispatch_request

    rv = self.dispatch_request()

  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1461, in dispatch_request

    return self.view_functions[rule.endpoint](**req.view_args)

  File "/usr/local/lib/python2.7/dist-packages/flask_appbuilder/security/decorators.py", line 26, in wraps

    return f(self, *args, **kwargs)

  File "/usr/local/lib/python2.7/dist-packages/panoramix-0.7.1-py2.7.egg/panoramix/views.py", line 675, in refresh_datasources

    cluster, str(e)),

  File "/usr/local/lib/python2.7/dist-packages/panoramix-0.7.1-py2.7.egg/panoramix/models.py", line 668, in __repr__

    return self.cluster_name

  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/attributes.py", line 237, in __get__

    return self.impl.get(instance_state(instance), dict_)

  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/attributes.py", line 578, in get

    value = state._load_expired(state, passive)

  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/state.py", line 474, in _load_expired

    self.manager.deferred_scalar_loader(self, toload)

  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/loading.py", line 664, in load_scalar_attributes

    only_load_props=attribute_names)

  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/loading.py", line 219, in load_on_ident

    return q.one()

  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2693, in one

    ret = list(self)

  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2736, in __iter__

    return self._execute_and_instances(context)

  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2749, in _execute_and_instances

    close_with_result=True)

  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2740, in _connection_from_session

    **kw)

  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 893, in connection

    execution_options=execution_options)

  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 898, in _connection_for_bind

    engine, execution_options)

  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 313, in _connection_for_bind

    self._assert_active()

  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 214, in _assert_active

    % self._rollback_exception

InvalidRequestError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (raised as a result of Query-invoked autoflush; consider using a session.no_autoflush block if this flush is occurring prematurely) (sqlite3.IntegrityError) NOT NULL constraint failed: columns.created_on [SQL: u'INSERT INTO columns (datasource_name, column_name, is_active, type, groupby, count_distinct, sum, max, min, filterable, description) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)'] [parameters: (u'purview', u'App_rsps_Min', 1, u'FLOAT', 0, 0, 0, 0, 0, 0, None)]

This was working previously with 0.7.0

Thnak you!!

[Maxime Beauchemin]

Hi,

If you install from github you should follow the `CONTRIBUTING.md` instructions. The important part is running `npm install` from the assets folder and `npm run prod`

I'll push a new version to Pypi soon and post an update to the group.

Max

[Salvador Ferrer Martinez]

Got it! 

Thank you! 

Sent from Nine

--
You received this message because you are subscribed to a topic in the Google Groups "airbnb_panoramix" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/airbnb_panoramix/YF28l_rRrEY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to airbnb_panoram...@googlegroups.com.
To post to this group, send email to airbnb_p...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/airbnb_panoramix/16846ecb-e074-4050-9231-d762df121aac%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Salvador Ferrer Martinez]

Which maybe explain as well my issue with the empty welcome page. Let me rework it and confirm. 

Sent from Nine

To view this discussion on the web, visit https://groups.google.com/d/msgid/airbnb_panoramix/BLU408-EAS133C92252B40B70848FF9B48FB30%40phx.gbl.

[salva....@gmail.com]

Hi Max I rebuilt my environment according to CONTRIBUTING.md but I still get the same error when adding a druid datasource.

ANy idea of what would be worth looking at?

THank you

On Wednesday, March 9, 2016 at 5:59:26 PM UTC-5, Salvador Ferrer Martinez wrote:

Which maybe explain as well my issue with the empty welcome page. Let me rework it and confirm. 

Sent from Nine

From: Salvador Ferrer Martinez <salva....@gmail.com>
Sent: Mar 9, 2016 5:57 PM
To: Maxime Beauchemin; airbnb_panoramix
Subject: Re: Upgrade to Master 3/8

Got it! 

Thank you! 

Sent from Nine

From: Maxime Beauchemin <maximebe...@gmail.com>
Sent: Mar 9, 2016 5:41 PM
To: airbnb_panoramix
Subject: Re: Upgrade to Master 3/8

Hi,

If you install from github you should follow the `CONTRIBUTING.md` instructions. The important part is running `npm install` from the assets folder and `npm run prod`

I'll push a new version to Pypi soon and post an update to the group.

Max

--
You received this message because you are subscribed to a topic in the Google Groups "airbnb_panoramix" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/airbnb_panoramix/YF28l_rRrEY/unsubscribe.

To unsubscribe from this group and all its topics, send an email to airbnb_panoramix+unsubscribe@googlegroups.com.
To post to this group, send email to airbnb_panoramix@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/airbnb_panoramix/16846ecb-e074-4050-9231-d762df121aac%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "airbnb_panoramix" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/airbnb_panoramix/YF28l_rRrEY/unsubscribe.

To unsubscribe from this group and all its topics, send an email to airbnb_panoramix+unsubscribe@googlegroups.com.
To post to this group, send email to airbnb_panoramix@googlegroups.com.

[Maxime Beauchemin]

I'm not sure what the problem is. You may want to use a mysql or postgres db and see if the problem persists...

[salva....@gmail.com]

I configured a postgres DB and the problem doesn't persist, it only occurs when refreshing the metadata in a druid cluster.

I'll continue researching and let you know if I figure out what's wrong.

[salva....@gmail.com]

Hi Max, I believe I've found the issue and it may be affecting teh just released 0.8.0. 

It took me some time following what the code was doing until the full exception message got in my brain:

Original exception was: (raised as a result of Query-invoked autoflush; consider using a session.no_autoflush block if this flush is occurring prematurely) (sqlite3.IntegrityError) NOT NULL constraint failed: columns.created_on

So SQLLite is complainign that we are creating new data in the database and leaving the column created_on empty.

I couldn't find where in the code the column was defined until I stumbled upon this commit:

https://github.com/mistercrunch/panoramix/commit/6f4397c7d7ea8889e97f2438edac3843125b9c80

In that commit DruidColumn didn't inherit AuditMixinNullable and it is in AuditMixinNullable that we update the 'created_on' field.

I reverted that commit in my local copy and the exception disappeared.