Getting 404 for data-lift-gc JS on first page access after session creation

54 views
Skip to first unread message

Francois

unread,
Mar 23, 2018, 10:07:17 AM3/23/18
to lif...@googlegroups.com

Hello,

I'm not exactly sure how to start debuging that, so I would love to have any help.

So, I have an application that after session creation (user login), I get an HTTP 404 for https://myapp/lift/page/F8011079085101U1U3W.js (where 'F8011079085101U1U3W' is the value in data-lift-gc in <body data-lift-session-id="xxx" data-lift-gc="F8011079085101U1U3W").

If I reload the page, everything is working fine (and the new /lift/page/xxxx.js page is correctly served). If I logout and login again => same error.

I was working fine in a previous version of the app using the same version of liftweb (3.0.2), so it may not be directly related to lift, but I have no idea where to start looking for.

On the server, I only get, even with "trace" logs for liftweb:

[2018-03-23 14:55:53] INFO  net.liftweb.util.TimeHelpers - Service request (GET) /rudder-web/lift/page/F8011079085101U1U3W.js returned 404, took 5 Milliseconds

Any insight would be much appreciated.
-- 
Francois ARMAND - @fanf42
https://github.com/Normation/rudder
http://www.normation.com

Antonio Salazar Cardozo

unread,
Mar 23, 2018, 2:53:21 PM3/23/18
to Lift
This is strange… These pages are designed to be single-shot (i.e., they are only served once per page). How does
the user login flow work? Do you see more than one request for the same page id?
Thanks,
Antonio

Francois

unread,
Mar 26, 2018, 10:33:57 AM3/26/18
to lif...@googlegroups.com, Antonio Salazar Cardozo
The login workflow is as follow:

Other JS/CSS resources are found (we use a "cache" system that generate unique URL by version of our app to control cache invalidation on browser side), so it does not seems to be a problem with lift backend at that moment.

Interestingly, if I try to do several redicts with different sessions cleaning (I though it was linked to lift session creation, some kind of race condition), each time the JS is correctly loaded:

LiftRules.statelessDispatch.append {
  case Req("secure" :: "redirect" :: Nil, _, GetRequest) =>
    () => Full(RedirectResponse(REDIRECT_URL))
}
LiftRules.dispatch.append {
  case Req("secure" :: "destroy0" :: Nil, _, GetRequest) =>
    S.request.foreach(_.request.session.terminate)
    () => Full(RedirectResponse(REDIRECT_URL))
  case Req("secure" :: "destroy1" :: Nil, _, GetRequest) =>
    S.session.foreach( _.destroySession)
    () => Full(RedirectResponse(REDIRECT_URL))
  case Req("secure" :: "destroy2" :: Nil, _, GetRequest) =>
    S.session.foreach( _.destroySessionAndContinueInNewSession(() => S.redirectTo(REDIRECT_URL)))
    () => Full(RedirectResponse(REDIRECT_URL))
}


Any idea about how to debug that would be very welcome.
--
--
Lift, the simply functional web framework: http://liftweb.net
Code: http://github.com/lift
Discussion: http://groups.google.com/group/liftweb
Stuck? Help us help you: https://www.assembla.com/wiki/show/liftweb/Posting_example_code

---
You received this message because you are subscribed to the Google Groups "Lift" group.
To unsubscribe from this group and stop receiving emails from it, send an email to liftweb+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Francois

unread,
Mar 26, 2018, 10:50:26 AM3/26/18
to lif...@googlegroups.com, Antonio Salazar Cardozo
So, it appears that something change in Jetty, and it is the culprit (but for now, we don't know why):
  • jetty up to 8 => OK
  • jetty 9.3 => bad behavior.
So, something changed with Jetty 9. And now, we are lost :(

Antonio Salazar Cardozo

unread,
Mar 26, 2018, 12:04:46 PM3/26/18
to Lift
Whoa. That puts us firmly in bizarre-o land. It does look like Jetty 9.3 has HTTP/2 support,
wonder if that could be related… One question is, are you in the same session and hitting
the same server in both requests? The page JS is stored in local memory on the specific
server and served from there as well. It is evicted immediately once it's served.

Is there any chance you can post a simplified example project that showcases this issue?
Thanks,
Antonio
To unsubscribe from this group and stop receiving emails from it, send an email to liftweb+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Francois

unread,
Mar 26, 2018, 3:22:35 PM3/26/18
to lif...@googlegroups.com, Antonio Salazar Cardozo
We tested different jetty version with eclipse "run-jetty-run" plugin, so it was exactly the same code with different jetty server.
In all case, just after sesssion creation+redirect => JS problem. Just hitting f5 is ok. So yes, same session, same server, same everything.

Have you any idea to debug that? I have zero idea right now :/
To unsubscribe from this group and stop receiving emails from it, send an email to liftweb+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Francois

unread,
Mar 26, 2018, 6:09:02 PM3/26/18
to lif...@googlegroups.com, Antonio Salazar Cardozo
Some progress (if we can tell...): when the problem occures, messageCallback in LiftSession is empty, so that part:
def findFunc(funcName: String): Option[S.AFuncHolder] =
  Option(messageCallback.get(funcName))
(called in RenderVersion.doWith(..)) returns None. In the case where the JS is correctly loaded, the messageCallback map contains the method.

On 26/03/2018 21:22, Francois wrote:
We tested different jetty version with eclipse "run-jetty-run" plugin, so it was exactly the same code with different jetty server.
In all case, just after sesssion creation+redirect => JS problem. Just hitting f5 is ok. So yes, same session, same server, same everything.

Have you any idea to debug that? I have zero idea right now :/
lo

Antonio Salazar Cardozo

unread,
Mar 26, 2018, 6:19:49 PM3/26/18
to Lift
This really feels like you're operating in a different session. This is where the relevant state is set:


I would check to see if the session is the same in here and when the /lift/page request is resolved. If not, it's
possible you're seeing cookie issues? But I don't get why that would be Jetty-version-related, so it seems unlikely.
You do kind of have to dig into the Lift code to figure this one out… This is baked in pretty deep.
Thanks,
Antonio

Francois

unread,
Mar 26, 2018, 7:04:55 PM3/26/18
to lif...@googlegroups.com, Antonio Salazar Cardozo
Yep, there is clearly something strange, and you may be right in the number of session... I just don't know where they are :)

When logging, I have an empty lift sessions if I check S.session.
But then, I get several calls to findFunc(...) before the code in LiftSession.processTemplate actually put the page version id in nmessageCallback map (so these ones obviously lead to a None).
Then, there is the nmessageCallback map init in LiftSession.processTemplate.
And then, I get some call to findFunc, but with a different ID than the one in the map. The one in the map matches the one in the page, but it's not the one requested for. And we never get the correct ID asked for.

(after some more time, I think all the findFunc() with non matching ID are coming from liftActor from a different thread pool / thread global than the one which is doing the rendering).

So, this is starting to smell like a thread nightmare, and perhaps jetty 9 somehow changed his threading model in an interesting way. I will try to understand some more, but it does sound a little bit to hard for me.
To unsubscribe from this group and stop receiving emails from it, send an email to liftweb+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Antonio Salazar Cardozo

unread,
Mar 26, 2018, 7:18:53 PM3/26/18
to Lift
At this point the easiest thing would probably be to create a minimal example. Then one of us
can have a deeper dive when we have a moment to see if we can figure out what's up.

This generally works fine, and I've seen it working in Jetty 9.3 before as well, so I think something
odd is up, but it's hard to know what exactly it is without getting my hands on some code.
Thanks,
Antonio

Francois

unread,
Mar 26, 2018, 7:57:43 PM3/26/18
to lif...@googlegroups.com, Antonio Salazar Cardozo

OK, I have found the problem (but not the explication).

So, on our login page, a session is created by jetty (there is no reason why, but we certainly didn't look too much for it). So a sessionid is set by cookie.

Then, on login POST request, the jetty session is retrieved from the cookie. On login OK, we are reseting any session and start a new one for the user, which leads to that line of log:

[2018-03-27 01:42:06] INFO  org.eclipse.jetty.server.session - Session id node010os5oomttl5q1lgnl6uzve9z03 swapped for new id node01hovmtjtjd6nw1kf1f3qc17gs54

And then, there is the redirect, and things are going badly for the next GET (even if I have the correct session.

If I delete the session cookie on login page, the behavior is OK.

I don't know why it is so. I'm attaching the log output of jetty with and without the cookie. Not sure it helps to understand, but if it may help someone else with a strange behavior that can be sorted simply...

Thanks for the help!
To unsubscribe from this group and stop receiving emails from it, send an email to liftweb+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.
existing-session-cookie.log
clean-session-cookie.log

Francois

unread,
Mar 27, 2018, 12:23:17 PM3/27/18
to lif...@googlegroups.com, Antonio Salazar Cardozo
Some more information before I try to make a minimal reproducible example (but I fear I will have to put spring-security on it and I'm not particulary happy with the idea).

I added session creation / deletion LiftRules to try to understand what is happening, and I'm not sur it helped.

This is the log I get (I'm adding color and comments to help understand what is happening:

Case1: I delete by hand JSESSIONID and I click on log-in:

=====> CREATE Session [F545932277983AQVOWD // node01xjj1musg2ga11491c0a6fxvq819] with [http://localhost:8082/rudder-web/secure/index.html]
=====> Processing [F545932277983AQVOWD // node01xjj1musg2ga11491c0a6fxvq819] request: http://localhost:8082/rudder-web/secure/index.html
=====> Processing [F545932277983AQVOWD // node01xjj1musg2ga11491c0a6fxvq819] request: http://localhost:8082/rudder-web/lift/page/F545932277984A24UFO.js

Case 2: I keep the old JSESSIONID cookie in my browser:

[2018-03-27 18:02:08] INFO  org.eclipse.jetty.server.session - Session id node065hu4a0suzi1347zg1ouqe9z15 swapped for new id node01lk82vpb2p4qy1s9aim8ag6chv16
=====> CREATE Session [F5459322777632FV5T4 // node01lk82vpb2p4qy1s9aim8ag6chv16] with [http://localhost:8082/rudder-web/secure/index.html]
=====> DESTROY Session [F545932277740NPTY3Z // node065hu4a0suzi1347zg1ouqe9z15]
=====> Processing [F5459322777632FV5T4 // node01lk82vpb2p4qy1s9aim8ag6chv16] request: http://localhost:8082/rudder-web/secure/index.html
=====> DESTROY Session [F5459322777632FV5T4 // node01lk82vpb2p4qy1s9aim8ag6chv16]
=====> CREATE Session [F545932277795R4YW1K // node01lk82vpb2p4qy1s9aim8ag6chv16] with [http://localhost:8082/rudder-web/lift/page/F5459322777644N5JM5.js]
=====> Processing [F545932277795R4YW1K // node01lk82vpb2p4qy1s9aim8ag6chv16] request: http://localhost:8082/rudder-web/lift/page/F5459322777644N5JM5.js
=====> Processing [F545932277795R4YW1K // node01lk82vpb2p4qy1s9aim8ag6chv16] request: http://localhost:8082/rudder-web/lift/page/F5459322777644N5JM5.js


So, lift try to destroy session for node065hu4a0suzi1347zg1ouqe9z15 (which was created on the login page, even if the given login page should be stateless) and that somehow change its behavior for the next sessions handling, with the session setting the page.js being destroyed before the request on said page.js.


Francois

unread,
Mar 27, 2018, 12:23:27 PM3/27/18
to lif...@googlegroups.com, Antonio Salazar Cardozo
Some more information before I try to make a minimal reproducible example (but I fear I will have to put spring-security on it and I'm not particulary happy with the idea).

I added session creation / deletion LiftRules to try to understand what is happening, and I'm not sur it helped.

This is the log I get (I'm adding color and comments to help understand what is happening:

Case1: I delete by hand JSESSIONID and I click on log-in:

=====> CREATE Session [F545932277983AQVOWD // node01xjj1musg2ga11491c0a6fxvq819] with [http://localhost:8082/rudder-web/secure/index.html]
=====> Processing [F545932277983AQVOWD // node01xjj1musg2ga11491c0a6fxvq819] request: http://localhost:8082/rudder-web/secure/index.html
=====> Processing [F545932277983AQVOWD // node01xjj1musg2ga11491c0a6fxvq819] request: http://localhost:8082/rudder-web/lift/page/F545932277984A24UFO.js

Case 2: I keep the old JSESSIONID cookie in my browser:

[2018-03-27 18:02:08] INFO  org.eclipse.jetty.server.session - Session id node065hu4a0suzi1347zg1ouqe9z15 swapped for new id node01lk82vpb2p4qy1s9aim8ag6chv16
=====> CREATE Session [F5459322777632FV5T4 // node01lk82vpb2p4qy1s9aim8ag6chv16] with [http://localhost:8082/rudder-web/secure/index.html]
=====> DESTROY Session [F545932277740NPTY3Z // node065hu4a0suzi1347zg1ouqe9z15]
=====> Processing [F5459322777632FV5T4 // node01lk82vpb2p4qy1s9aim8ag6chv16] request: http://localhost:8082/rudder-web/secure/index.html
=====> DESTROY Session [F5459322777632FV5T4 // node01lk82vpb2p4qy1s9aim8ag6chv16]
=====> CREATE Session [F545932277795R4YW1K // node01lk82vpb2p4qy1s9aim8ag6chv16] with [http://localhost:8082/rudder-web/lift/page/F5459322777644N5JM5.js]
=====> Processing [F545932277795R4YW1K // node01lk82vpb2p4qy1s9aim8ag6chv16] request: http://localhost:8082/rudder-web/lift/page/F5459322777644N5JM5.js
=====> Processing [F545932277795R4YW1K // node01lk82vpb2p4qy1s9aim8ag6chv16] request: http://localhost:8082/rudder-web/lift/page/F5459322777644N5JM5.js


So, lift try to destroy session for node065hu4a0suzi1347zg1ouqe9z15 (which was created on the login page, even if the given login page should be stateless) and that somehow change its behavior for the next sessions handling, with the session setting the page.js being destroyed before the request on said page.js.




On 27/03/2018 01:57, Francois wrote:

Antonio Salazar Cardozo

unread,
Mar 27, 2018, 3:15:41 PM3/27/18
to Lift
This also seems like a bug, independent of the “accidentally stateful” nature of the
login page... I would still huugely appreciate a minimized version of this, if time permits.
Thanks,
Antonio

Francois

unread,
Mar 27, 2018, 8:25:07 PM3/27/18
to lif...@googlegroups.com, Antonio Salazar Cardozo
Issue opened here: https://github.com/lift/framework/issues/1947

The root cause seems to be the use of HttpServletRequest.changeSessionId, as demoed in the minimal project here: https://github.com/fanf/demo-bug-lift-session

Hope it helps
To unsubscribe from this group and stop receiving emails from it, send an email to liftweb+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Francois

unread,
Mar 27, 2018, 8:57:11 PM3/27/18
to lif...@googlegroups.com, Antonio Salazar Cardozo
For information, in the specific case of spring-security, you can workaround that problem by changing the "session-fixation-protection" to something else than "changeId" (see: https://docs.spring.io/spring-security/site/docs/current/reference/html/ns-config.html#ns-session-fixation).

Also, the problem appeared with Jetty 9 because it was the first version to implement changeSessionId which appeared in Servlet 3.1.

Cheers,

Antonio Salazar Cardozo

unread,
Mar 28, 2018, 2:43:51 PM3/28/18
to Lift
Huge thanks for all that info @fanf! I'll look into this when I get a chance, I have
some ideas on how we can tweak the stateless bit... The bug with the session
falling all over itself when the id changes will be a touch more complicated, I
suspect.
Thanks,
Antonio

Francois Armand

unread,
Mar 28, 2018, 2:58:53 PM3/28/18
to Lift
No problem, there is no urgency on my side since I found a workaround. 

Thanks for your help, without the query for a minimal example I would  not have found the root cause. 

Cheers! 

To unsubscribe from this group and stop receiving emails from it, send an email to liftweb+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.
--
--
Lift, the simply functional web framework: http://liftweb.net
Code: http://github.com/lift
Discussion: http://groups.google.com/group/liftweb
Stuck? Help us help you: https://www.assembla.com/wiki/show/liftweb/Posting_example_code

---
You received this message because you are subscribed to the Google Groups "Lift" group.
To unsubscribe from this group and stop receiving emails from it, send an email to liftweb+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.
--
--
Lift, the simply functional web framework: http://liftweb.net
Code: http://github.com/lift
Discussion: http://groups.google.com/group/liftweb
Stuck? Help us help you: https://www.assembla.com/wiki/show/liftweb/Posting_example_code

---
You received this message because you are subscribed to the Google Groups "Lift" group.
To unsubscribe from this group and stop receiving emails from it, send an email to liftweb+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.
--
--
Lift, the simply functional web framework: http://liftweb.net
Code: http://github.com/lift
Discussion: http://groups.google.com/group/liftweb
Stuck? Help us help you: https://www.assembla.com/wiki/show/liftweb/Posting_example_code

---
You received this message because you are subscribed to the Google Groups "Lift" group.
To unsubscribe from this group and stop receiving emails from it, send an email to liftweb+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.
--
--
Lift, the simply functional web framework: http://liftweb.net
Code: http://github.com/lift
Discussion: http://groups.google.com/group/liftweb
Stuck? Help us help you: https://www.assembla.com/wiki/show/liftweb/Posting_example_code

---
You received this message because you are subscribed to the Google Groups "Lift" group.
To unsubscribe from this group and stop receiving emails from it, send an email to liftweb+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
--
Lift, the simply functional web framework: http://liftweb.net
Code: http://github.com/lift
Discussion: http://groups.google.com/group/liftweb
Stuck? Help us help you: https://www.assembla.com/wiki/show/liftweb/Posting_example_code

---
You received this message because you are subscribed to the Google Groups "Lift" group.
To unsubscribe from this group and stop receiving emails from it, send an email to liftweb+u...@googlegroups.com.

Antonio Salazar Cardozo

unread,
Mar 30, 2018, 3:45:52 PM3/30/18
to Lift
Glad to help :)
Cheers! 

To unsubscribe from this group and stop receiving emails from it, send an email to liftweb+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.
--
--
Lift, the simply functional web framework: http://liftweb.net
Code: http://github.com/lift
Discussion: http://groups.google.com/group/liftweb
Stuck? Help us help you: https://www.assembla.com/wiki/show/liftweb/Posting_example_code

---
You received this message because you are subscribed to the Google Groups "Lift" group.
To unsubscribe from this group and stop receiving emails from it, send an email to liftweb+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.
--
--
Lift, the simply functional web framework: http://liftweb.net
Code: http://github.com/lift
Discussion: http://groups.google.com/group/liftweb
Stuck? Help us help you: https://www.assembla.com/wiki/show/liftweb/Posting_example_code

---
You received this message because you are subscribed to the Google Groups "Lift" group.
To unsubscribe from this group and stop receiving emails from it, send an email to liftweb+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.
--
--
Lift, the simply functional web framework: http://liftweb.net
Code: http://github.com/lift
Discussion: http://groups.google.com/group/liftweb
Stuck? Help us help you: https://www.assembla.com/wiki/show/liftweb/Posting_example_code

---
You received this message because you are subscribed to the Google Groups "Lift" group.
To unsubscribe from this group and stop receiving emails from it, send an email to liftweb+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.
--
--
Lift, the simply functional web framework: http://liftweb.net
Code: http://github.com/lift
Discussion: http://groups.google.com/group/liftweb
Stuck? Help us help you: https://www.assembla.com/wiki/show/liftweb/Posting_example_code

---
You received this message because you are subscribed to the Google Groups "Lift" group.
To unsubscribe from this group and stop receiving emails from it, send an email to liftweb+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
--
Lift, the simply functional web framework: http://liftweb.net
Code: http://github.com/lift
Discussion: http://groups.google.com/group/liftweb
Stuck? Help us help you: https://www.assembla.com/wiki/show/liftweb/Posting_example_code

---
You received this message because you are subscribed to the Google Groups "Lift" group.
To unsubscribe from this group and stop receiving emails from it, send an email to liftweb+unsubscribe@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages