Post upgrade problems with stale cached plugin web resources

186 views
Skip to first unread message

Clark Boylan

unread,
Nov 17, 2023, 3:27:11 PM11/17/23
to Repo and Gerrit Discussion
Hello,

We upgraded our Gerrit server to 3.8.2 from 3.7.5 today. Everything
seems to have gone well, except for problems loading the
codemirror-editor web UI editor for file editing. Specifically, you
can enter and exit edit mode just fine, but when you open a specific
file for editing that fails to load the editor with the file in it.
Firefox's console reports that GET
/plugins/codemirror-editor/static/codemirror-element.html returns a
404 not found response. Examining the 3.8.2 war file this html file is
not in the codemirror-element.jar, but static/codemirror_editor.js is.

At some point it did start working for some users; on a hunch I did a
hard refresh of the change page in my browser and then it began
working for me as well. The problem appears to be related to caching
stale Gerrit 3.7.5 content which isn't present under 3.8.2. I believe
that Gerrit versions its own resources which ensure the browsers
reload any cached content but maybe the plugins do not? It is also
possible that I am entirely mistaken, and we're simply not employing a
necessary cache TTL/eviction policy within our deployment.

All that to say, I'm wondering if this is expected behavior and
whether or not we or Gerrit need to do something different to make
behavior more consistent after Gerrit upgrades.

Clark

Nasser Grainawi

unread,
Nov 17, 2023, 6:53:09 PM11/17/23
to Clark Boylan, Repo and Gerrit Discussion
We saw the same problem with our upgrade to 3.5, so I think there's a real issue in there somewhere, but I didn't get to the root cause before I moved on to the next problem.

Nasser
 

Clark

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/CAAcH74ghAfF6TkBzmCBTcUTr5KzbU07PyfUby4UYv%2BWq5FbX3g%40mail.gmail.com.

David Ostrovsky

unread,
Nov 22, 2023, 4:16:22 PM11/22/23
to Repo and Gerrit Discussion
Nasser Grainawi schrieb am Samstag, 18. November 2023 um 00:53:09 UTC+1:
On Fri, Nov 17, 2023 at 12:27 PM Clark Boylan <clark....@gmail.com> wrote:
Hello,

We upgraded our Gerrit server to 3.8.2 from 3.7.5 today. Everything
seems to have gone well, except for problems loading the
codemirror-editor web UI editor for file editing. Specifically, you
can enter and exit edit mode just fine, but when you open a specific
file for editing that fails to load the editor with the file in it.
Firefox's console reports that GET
/plugins/codemirror-editor/static/codemirror-element.html returns a
404 not found response. Examining the 3.8.2 war file this html file is
not in the codemirror-element.jar, but static/codemirror_editor.js is.

At some point it did start working for some users; on a hunch I did a
hard refresh of the change page in my browser and then it began
working for me as well. The problem appears to be related to caching
stale Gerrit 3.7.5 content which isn't present under 3.8.2. I believe
that Gerrit versions its own resources which ensure the browsers
reload any cached content but maybe the plugins do not? It is also
possible that I am entirely mistaken, and we're simply not employing a
necessary cache TTL/eviction policy within our deployment.

All that to say, I'm wondering if this is expected behavior and
whether or not we or Gerrit need to do something different to make
behavior more consistent after Gerrit upgrades.

We saw the same problem with our upgrade to 3.5, so I think there's a real issue in there somewhere, but I didn't get to the root cause before I moved on to the next problem.


IIRC, we were trying to address similar issues with this Bazel feature request: [1],
and ended up doing `touch` in gerrit_plugin() macro during plugin build: [2].

Nasser Grainawi

unread,
Nov 27, 2023, 7:05:17 PM11/27/23
to David Ostrovsky, Repo and Gerrit Discussion
Ah, thanks for that pointer, that shows what I think is the problem. I don't know what the expected behavior would be, but could it cause a problem if that static .js file is *too new*? In that macro we set `TZ=UTC`, which means if I build a plugin, my .jar will have a timestamp like "Mon Nov 27 16:24:22 MST 2023", but then if I run `jar -xvf plugin.jar` it shows me a timestamp for all files like "Mon Nov 27 23:23:32 MST 2023". I think it's wrong to change the TZ to be different from the generating machine, it's that machine that generated the artifacts and therefore we do care about its TZ. If you really want to use UTC in the touch command, I think the way to do it would be to use `date -u` to get a timestamp in UTC and then use `touch -d ${utc_time}Z` to set it on each file.
 


Nasser
 

Clark

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/CAAcH74ghAfF6TkBzmCBTcUTr5KzbU07PyfUby4UYv%2BWq5FbX3g%40mail.gmail.com.

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

Kenyon Ralph

unread,
Nov 27, 2023, 7:12:23 PM11/27/23
to Repo and Gerrit Discussion
For build reproducibility, it should be using something like the timestamp of the commit that you're building.

David Ostrovsky

unread,
Nov 28, 2023, 1:54:01 PM11/28/23
to Repo and Gerrit Discussion
For one, I used a common idiom from our build toolchain (that wasn't added by me):

  $ cd gerrit
  $ git grep "TZ=UTC" | wc -l
  4

For another do we really care about a couple of hours of differences?

The issue could be that 3.8.4 is built/released say 2024-01-01, while 3.9.0 was
built/released say 2023-11-20. Then a gerrit site that had 3.8.4 installed, would
upgrade to newer version 3.9.0, but built earlier, with older artifact timestamps.

Nasser Grainawi

unread,
Nov 29, 2023, 11:09:26 AM11/29/23
to David Ostrovsky, Repo and Gerrit Discussion
Yes, but those were all originally added with `touch -t`, so it's a fixed time and that makes sense to also hardcode the timezone of that time.
 

For another do we really care about a couple of hours of differences?

Only if it causes a problem. That's why I asked if having too new of artifacts could be an issue. I don't know how a future date would be handled by the web server / browser.
 

The issue could be that 3.8.4 is built/released say 2024-01-01, while 3.9.0 was
built/released say 2023-11-20. Then a gerrit site that had 3.8.4 installed, would
upgrade to newer version 3.9.0, but built earlier, with older artifact timestamps.

I'm pretty confident that wasn't the problem for us and it's not the case for Clark with 3.7.5 -> 3.8.2. 3.8.2 is almost 24 hours newer.

$ jar -tvf WEB-INF/plugins/codemirror-editor.jar-3.7.5 | grep static/code*
2221764 Wed Sep 13 22:19:10 MDT 2023 static/codemirror-element.html
   3039 Wed Sep 13 22:19:10 MDT 2023 static/codemirror_editor.js

$ jar -tvf WEB-INF/plugins/codemirror-editor.jar-3.8.2 | grep static/code*
  20043 Thu Sep 14 19:59:34 MDT 2023 static/codemirror_editor.js
1400564 Thu Sep 14 19:59:34 MDT 2023 static/codemirror_element.js

Any other ideas?

Clark Boylan

unread,
Nov 29, 2023, 11:46:44 AM11/29/23
to Nasser Grainawi, David Ostrovsky, Repo and Gerrit Discussion
This might actually have been the problem for us because we build our
images and all associated plugins from scratch locally. I'm not
positive we built our 3.7.5 image after 3.8.2, but it is possible
since I think we refreshed both the 3.7 and 3.8 image at the "same"
time which means it is a race within our CI system to see which
actually builds first. Going forward we can be more strict about
building newer releases after older ones and see if that helps.
Reply all
Reply to author
Forward
0 new messages