[ac-play-java] 403/forbidden from heroku-hosted connect add-on when sessions expire with Jira OnDemand?

[Michael Kitchin]

Hi there,

Our heroku-hosted connect add-on works exactly the way we want until (we think) the client session expires, then starts kicking back 403/forbidden in the iframe requests. 

We thought this might be token expiry so have set ac.token.expiry.secs in application.conf to a very large value with no improvements.

Recap:

• Set up dev environment (IntelliJ)
• Created the app with ac-play-java 0.10, Play 2.2.6, and Java 8
• Added descriptor, routes, views, controllers, etc.
• Added modules for UI links (mostly jiraProjectTabPanels)
• Added everything to GitHub and Heroku for turnkey deployment
  
• Added my addon to my dev Jira successfully
• Added it to our OnDemand instance via private Marketplace registration, etc.
• All runs _perfectly_, until some time has passed (see above)

Other details:

• Using H2 database (during evaluation) with the appropriate hibernate dialect, etc. (noting AC_HOST-related errors, below)
• Using a "hobbyist"-level dyno (no sleeping)

Here is a section of the log when a user logs back in and tries to reach some of our pages.

2015-11-30T23:15:13.218898+00:00 heroku[router]: at=info method=GET path="/planner/admin?project_key=((project-omitted))&tz=America%2FDenver&loc=en-US&user_id=((username-omitted))&user_key=((username-omitted))&xdm_e=https%3A%2F%2F((on-demand-domain-omitted)).atlassian.net&xdm_c=channel-((app-url-omitted))-a&cp=&lic=none&cv=1.1.64&jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJta2l0Y2hpbiIsInFzaCI6ImRiOWE4NDMwYWIzMjhmNGI1ZjBmYTcwMWRmNjVmMTk1ZTRjN2ZiNDczZjM0Y2Q2YzJkNjQzMjA3NzBhODU1OWYiLCJpc3MiOiJqaXJhOmM2Nzc3MjI4LTIxYmQtNDVlMy1iZGQxLWE0MGY4Zjg5ZTBhNSIsImNvbnRleHQiOnsidXNlciI6eyJ1c2VyS2V5IjoibWtpdGNoaW4iLCJ1c2VybmFtZSI6Im1raXRjaGluIiwiZGlzcGxheU5hbWUiOiJNaWNoYWVsIEouIEtpdGNoaW4ifX0sImV4cCI6MTQ0ODkyNTQ5MSwiaWF0IjoxNDQ4OTI1MzExfQ.Wyym7X5ZMRnmqRjfzdDWfJnEgBEfIMmVAmuXxJCSQs4" host=((heroku-domain-omitted)).herokuapp.com request_id=0855d525-ad86-4f65-8adf-4e8e884a68a6 fwd="50.170.226.183" dyno=web.1 connect=1ms service=33ms status=403 bytes=257

2015-11-30T23:15:13.205640+00:00 app[web.1]: [ [31merror [0m] o.h.e.j.s.SqlExceptionHelper - Table "AC_HOST" not found; SQL statement:

2015-11-30T23:15:13.681840+00:00 heroku[router]: at=info method=GET path="/planner/capacity?project_key=((project-omitted))&tz=America%2FDenver&loc=en-US&user_id=((username-omitted))&user_key=((username-omitted))&xdm_e=https%3A%2F%2F((on-demand-domain-omitted)).atlassian.net&xdm_c=channel-((app-url-omitted))-a&cp=&lic=none&cv=1.1.64&jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJta2l0Y2hpbiIsInFzaCI6ImMyZjFjZjNmOWMxYWJiMGY3YTAxYzQyMTg2YjRmNDA0MWU5NDJjYjYzNzFjMTYxMTRjOTdlNTNhZDU4YzVmY2EiLCJpc3MiOiJqaXJhOmM2Nzc3MjI4LTIxYmQtNDVlMy1iZGQxLWE0MGY4Zjg5ZTBhNSIsImNvbnRleHQiOnsidXNlciI6eyJ1c2VyS2V5IjoibWtpdGNoaW4iLCJ1c2VybmFtZSI6Im1raXRjaGluIiwiZGlzcGxheU5hbWUiOiJNaWNoYWVsIEouIEtpdGNoaW4ifX0sImV4cCI6MTQ0ODkyNTQ5MCwiaWF0IjoxNDQ4OTI1MzEwfQ.Ghpgg_WoYR_QH7qoUYHL_6ONDfrZnacSV1kMTSAFujA" host=((heroku-domain-omitted)).herokuapp.com request_id=8105d435-c702-4362-8623-1022e9ba36f9 fwd="50.170.226.183" dyno=web.1 connect=0ms service=28ms status=403 bytes=257

2015-11-30T23:15:13.667829+00:00 app[web.1]: [ [31merror [0m] o.h.e.j.s.SqlExceptionHelper - Table "AC_HOST" not found; SQL statement:

2015-11-30T23:15:15.038005+00:00 heroku[router]: at=info method=GET path="/planner/run?project_key=((project-omitted))&tz=America%2FDenver&loc=en-US&user_id=((username-omitted))&user_key=((username-omitted))&xdm_e=https%3A%2F%2F((on-demand-domain-omitted)).atlassian.net&xdm_c=channel-((app-url-omitted))-a&cp=&lic=none&cv=1.1.64&jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJta2l0Y2hpbiIsInFzaCI6IjkyZDM3ZjU5N2ZkZDMwNTA1MTkyZTU1OGFiNjQ0ZTVmZDQ1YjU1NzAwMGU2YTFjOGQxYzNlNzU3MDI1NjNmZWMiLCJpc3MiOiJqaXJhOmM2Nzc3MjI4LTIxYmQtNDVlMy1iZGQxLWE0MGY4Zjg5ZTBhNSIsImNvbnRleHQiOnsidXNlciI6eyJ1c2VyS2V5IjoibWtpdGNoaW4iLCJ1c2VybmFtZSI6Im1raXRjaGluIiwiZGlzcGxheU5hbWUiOiJNaWNoYWVsIEouIEtpdGNoaW4ifX0sImV4cCI6MTQ0ODkyNTQ4OSwiaWF0IjoxNDQ4OTI1MzA5fQ.v8SR0DhMIy5jyc7IZOt_A4Ff0OTaSHcI0Ch75eJl3Sc" host=((heroku-domain-omitted)).herokuapp.com request_id=0f3344f8-44b9-43dd-bc4b-98b2137fb758 fwd="50.170.226.183" dyno=web.1 connect=1ms service=58ms status=403 bytes=257

2015-11-30T23:15:15.020889+00:00 app[web.1]: [ [31merror [0m] o.h.e.j.s.SqlExceptionHelper - Table "AC_HOST" not found; SQL statement:

2015-11-30T23:16:01.421412+00:00 heroku[router]: at=info method=GET path="/atlassian-connect.json" host=((heroku-domain-omitted)).herokuapp.com request_id=d24bf9f6-f1c2-4420-b241-87191e6114ce fwd="54.85.255.242" dyno=web.1 connect=15ms service=76ms status=200 bytes=1450

2015-11-30T23:16:03.522417+00:00 heroku[router]: at=info method=GET path="/atlassian-connect.json" host=((heroku-domain-omitted)).herokuapp.com request_id=d59de75e-5954-4efe-a45b-96c23dceb5d4 fwd="54.85.255.242" dyno=web.1 connect=1ms service=40ms status=200 bytes=1450

Please let me know if I may provide any additional information. Thanks!

-Regards,

MjK

[Michael Kitchin]

Hi there,

Neglected to include the response body for that 403/forbidden, described above:

Access to this resource is forbidden without successful authentication. Please supply valid credentials.

Please let me know if I may provide any additional information. Thanks!

-Best wishes,

MjK

[Seb Ruiz]

Hi Michael,

H2 is an in-memory database and should not be used for production (it's fine for testing).

2015-11-30T23:15:13.667829+00:00 app[web.1]: [ [31merror [0m] o.h.e.j.s.SqlExceptionHelper - Table "AC_HOST" not found; SQL statement:

This line indicates that the database doesn't have the AC_HOST table, hence your app can't find the shared secret resulting in the 403s. I won't guess as to why this database table has been dropped, but could be a number of reasons.

When writing a multi-tenant and scalable service, it's important that your add-on is built in a stateless manner. Storing the database in memory probably makes this tricky; particularly if you are using more than one heroku dyno.

My recommendation would be to switch to psql or other suitable database.

Regards,

Seb

--
You received this message because you are subscribed to the Google Groups "Atlassian Connect Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to atlassian-connec...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Seb Ruiz

Atlassian

[Michael Kitchin]

HI there,

All good points -- we didn't want to invest the time setting up postgres until we had ironed out the add-on itself, but if H2 is potentially the stumbling block we'll give it a try.

Why would a missing AC_HOST table induce a 403, especially when the add-on works great when first deployed?

To unsubscribe from this group and stop receiving emails from it, send an email to atlassian-connect-dev+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Seb Ruiz

Atlassian

[Seb Ruiz]

The AC_HOST table probably does exist when the add-on is installed - it just disappears sometime between that and the failing call.

This could be because of a number of reasons, including:

• Add-on is restarted/redeployed -- the database is wiped
• Incoming request hits a different dyno than where the database was instantiated and the records don't exist
• Some other reason that the database disappears

The add-on returns a 403 as it can't verify that the incoming request is authorised. This authorisation is determined by verifying the incoming request against the stored shared secret which is exchanged during installation. If there is no data, then this verification will fail.

To unsubscribe from this group and stop receiving emails from it, send an email to atlassian-connec...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Seb Ruiz

Atlassian

--

You received this message because you are subscribed to the Google Groups "Atlassian Connect Dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to atlassian-connec...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Seb Ruiz

Atlassian

[Anders Holmgren]

Yeah I've seen this sort of behaviour with in memory storage. You install an addon and it saves the shared secret. Then you restart the server, losing the saved installations (shared secrets etc) so the jwt checks fail.

I implemented a file based storage for cases like this (not in AC Play though), which gives you the benefits of not having to set up a real db, but still being able to restart the server. 

[Michael Kitchin]

Hi there,

Thanks for the extra insight -- I'll be adding a postgres instance shortly.

I hope this don't sound dense, but I could use clarification future reference when dealing with Heroku:

We realize in-memory databases are transient, so we're not restarting the dyno (only one) between github pushes/builds, and we expect to start from scratch whenever we do push. Therefore:

1. Could Heroku randomly cycle hobbyist-level applications in a way we won't notice in the activity feed?

2. Could Play cycle database connections in such a way the in-memory DB gets evicted?

...I'm guessing (2), myself.

Thoughts?

To unsubscribe from this group and stop receiving emails from it, send an email to atlassian-connect-dev+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Seb Ruiz

Atlassian

--
You received this message because you are subscribed to the Google Groups "Atlassian Connect Dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to atlassian-connect-dev+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Seb Ruiz

Atlassian

--
You received this message because you are subscribed to the Google Groups "Atlassian Connect Dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to atlassian-connect-dev+unsub...@googlegroups.com.

[Anders Holmgren]

From memory heroku will happily passivate your instance if it is not being used for a while. At least if you are on a free instance. 

To unsubscribe from this group and stop receiving emails from it, send an email to atlassian-connec...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Seb Ruiz

Atlassian

--
You received this message because you are subscribed to the Google Groups "Atlassian Connect Dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to atlassian-connec...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Seb Ruiz

Atlassian

--
You received this message because you are subscribed to the Google Groups "Atlassian Connect Dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to atlassian-connec...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "Atlassian Connect Dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to atlassian-connec...@googlegroups.com.

[Michael Kitchin]

Hi there,

Thanks for getting back to me so quickly. 

No, not a free instance, so whatever was going on seems related to the choice of database.

We switched to postgres with minimal effort (new Procfile, some application.conf and persistence.xml changes). Couldn't have been easier, and the problem appears to be solved.

Thanks, all for your help!

-Best wishes,

MjK

To unsubscribe from this group and stop receiving emails from it, send an email to atlassian-connect-dev+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Seb Ruiz

Atlassian

--
You received this message because you are subscribed to the Google Groups "Atlassian Connect Dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to atlassian-connect-dev+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Seb Ruiz

Atlassian

--
You received this message because you are subscribed to the Google Groups "Atlassian Connect Dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to atlassian-connect-dev+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Atlassian Connect Dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to atlassian-connect-dev+unsub...@googlegroups.com.