Juris-M build: Travis CI timeout

27 views
Skip to first unread message

Frank Bennett

unread,
Aug 29, 2017, 9:10:50 AM8/29/17
to zotero-dev
Travis CI is working nicely against Juris-M, but I've hit a nasty timeout barrier, and I could use some advice.

It's not complicated. Juris-M installs a couple of large data sets, one for languages, and another for world jurisdictions. Like translators and styles, installing them is time-consuming. So when "Zotero.skipBundledFiles" is set, I limit both to the data needed for testing.

The problem is that other tests are painfully slow to run, and Travis CI forces a timeout about 50 minutes into the run, after 1,000+ fixtures have cleared, but before completion.

To speed things up, I would like to set the minimal language and jurisdiction data sets as the default for all of the test fixtures. I can set up "skipLanguageLoad" and "skipJurisdictionLoad" toggles simlar to "skipBundledFiles", but I don't know where or how I can set those toggles in the test suite for ordinary client reconstruction (as opposed to resetDB() invocations on specific test fixtures).

Any advice greatly appreciated.

Frank

Frank Bennett

unread,
Aug 29, 2017, 5:27:18 PM8/29/17
to zoter...@googlegroups.com
The timeout issue seems to have deeper roots.

After tweaking the source to use minimal language/jurisdiction
datasets in all cases, I get the same timeout failure, at 49min 58sec.

Would I be right to assume that the Zotero jobs on Travis CI have a
similar run time, but use travis-ci.com, where the time limit does not
apply?

Frank
> --
> You received this message because you are subscribed to the Google Groups
> "zotero-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to zotero-dev+...@googlegroups.com.
> To post to this group, send email to zoter...@googlegroups.com.
> Visit this group at https://groups.google.com/group/zotero-dev.
> For more options, visit https://groups.google.com/d/optout.

Frank Bennett

unread,
Aug 29, 2017, 5:39:38 PM8/29/17
to zotero-dev
Answering my own question here, um, no.

A section of the test suite that takes 3 minutes to run locally against Juris-M clears in less that 1 minute for Zotero. So there is a bottleneck in my own code somewhere. I can't think of a reason why the JM extensions should slow things down that much.

Frank

Frank Bennett

unread,
Aug 30, 2017, 7:43:12 AM8/30/17
to zotero-dev
Solved it. The bottleneck was in the resetDB invocations. Due to a bug in my own code, the entire IETF language subtag registry was being install at each iteration. Builds are now going through clean. Yay!

I have another question on how the actual zip build is supposed to happen. I'll open a separate post for that.

Frank

Emiliano Heyns

unread,
Aug 30, 2017, 10:52:02 AM8/30/17
to zotero-dev
Can you point me to the code where you do the resetdb? I never managed to get it right myself.

Frank Bennett

unread,
Aug 30, 2017, 5:14:46 PM8/30/17
to zotero-dev
I didn't change much in the end, but here is the logic of the fix. The resetDB() invocations themselves are just the ones in the existing fixtures Juris-M inherits from Zotero, e.g.:


The reset calls Zotero.reinit():


 Zotero.reinit() calls Zotero.init():


After hopping through some internal functions, Zotero.init() ultimately calls Zotero.updateSchema():


My draggy operations were in the schema update, and they were running in full every time the DB was reconstructed. Masking them or subbing in skeleton data when the skipBundledFiles option was turned on was the fix. I did that in three places (but looking at the code now, I see that it should only happen in two -- the _populateJurisdictions function should just be smartened up, rather than masked at execution):


In addition, I needed to extend the timeout applied to callbacks. That wasn't relevant to the resetDB issue, but Juris-M takes more time to complete some operations, and that was producing erratic failures in Travis. The main change for that was here (increasing 10000 Zotero timeout to 20000):


With the adjustments, the Juris-M tests complete in about 28 minutes of Travis run-time. The ceiling of free accounts is around 50 minutes, so this will be kind of pushing it was the Zotero test suite grows over time, but for the present all is well.

I'm not sure that answers your question -- as I say, the invocations of resetDB() in individual fixtures are just the ones inherited from Zotero. I just needed to avoid some slowdown in code of my own that it was calling.

Frank

Emiliano Heyns

unread,
Aug 30, 2017, 5:29:17 PM8/30/17
to zotero-dev
On Wednesday, August 30, 2017 at 11:14:46 PM UTC+2, Frank Bennett wrote:
I didn't change much in the end, but here is the logic of the fix. The resetDB() invocations themselves are just the ones in the existing fixtures Juris-M inherits from Zotero, e.g.:


The reset calls Zotero.reinit():


 Zotero.reinit() calls Zotero.init():


After hopping through some internal functions, Zotero.init() ultimately calls Zotero.updateSchema():


My draggy operations were in the schema update, and they were running in full every time the DB was reconstructed. Masking them or subbing in skeleton data when the skipBundledFiles option was turned on was the fix. I did that in three places (but looking at the code now, I see that it should only happen in two -- the _populateJurisdictions function should just be smartened up, rather than masked at execution):


In addition, I needed to extend the timeout applied to callbacks. That wasn't relevant to the resetDB issue, but Juris-M takes more time to complete some operations, and that was producing erratic failures in Travis. The main change for that was here (increasing 10000 Zotero timeout to 20000):


With the adjustments, the Juris-M tests complete in about 28 minutes of Travis run-time. The ceiling of free accounts is around 50 minutes, so this will be kind of pushing it was the Zotero test suite grows over time, but for the present all is well.


Super thanks, I'll give those a shot again. Having to wait for schemaUpdatePromise might be an issue for me (it's taken a fair amount of time in my tests), but it bears looking into again, as a manual scrub of the DB as I'm doing now is also a bit slow.

As to the run time, are you making use of parallel builds (https://docs.travis-ci.com/user/speeding-up-the-build/)? I split my test cases up in 2 halves (running on 4 containers, two halves for Zotero and two halves for Juris-M). Running stuff only after all chunks succeed is a bit of a pain on Travis unless you're on the beta (https://github.com/travis-ci/travis-ci/issues/929); it's the main reason why I moved to CircleCI before that beta was available, but it can be done. I haven't been on Travis for a while, but I think you get 4 containers for free for open source; it doesn't scale linearly exactly, but a 10 minute build should be doable this way with careful distribution of test cases.

Frank Bennett

unread,
Aug 30, 2017, 5:56:40 PM8/30/17
to zotero-dev
I didn't know about CircleCI. That's good to know, and thanks!

The Zotero tests are run as two concurrent instances of the full suite, one against FF 54.0, and another against 52.0.3. If Travis allows four streams, that should open some more margin against the future. Again very good to know.

The time taken for builds isn't a problem for me -- if anything, it forces me to think more deliberately about changes, so I'm all smiles for now. Travis is pushing the zipped client bundle to S3 now, as per design. I have a little work ahead today on ./build.sh script in my fork of zotero-standalone-build, then some changes to the Juris-M website, and we should be rolling here.

Frank

Emiliano Heyns

unread,
Aug 30, 2017, 6:14:49 PM8/30/17
to zotero-dev
On Wednesday, August 30, 2017 at 11:56:40 PM UTC+2, Frank Bennett wrote:
I didn't know about CircleCI. That's good to know, and thanks!

The Zotero tests are run as two concurrent instances of the full suite, one against FF 54.0, and another against 52.0.3. If Travis allows four streams, that should open some more margin against the future. Again very good to know.


I think they do, and even if you're not on the beta, there are workarounds to make sure the publishing stage runs only once after all test stages have succeeded. It's just that Circle has that built-in. I run my tests on whatever is the latest formal Zotero release, not a FF build; the build process picks up te download and installs it automatically.
 
The time taken for builds isn't a problem for me -- if anything, it forces me to think more deliberately about changes, so I'm all smiles for now.

I think it's no secret my process is mostly "move fast and break things"; I got serious about my test suite because of the inherent limitations of that process. Plus, TBH, my main system is a fairly anemic macbook air, and if I run my tests on it that means I'm getting an extended coffee break *again*, and I don't need more coffee.  So I will occasionally just pitch something towards circle to run it. Perhaps I should take the test time to exercise or meditate instead.
 
Travis is pushing the zipped client bundle to S3 now, as per design. I have a little work ahead today on ./build.sh script in my fork of zotero-standalone-build, then some changes to the Juris-M website, and we should be rolling here.

I push mine to github releases; it requires a stupid workaround for update.rdf (GH says "good idea, we may get to that", that was 2014 I think), but it's good enough. I have it set up so it will announce betas/test builds on the relevant GH issue, and does a full release including update rdf et all to a proper GH release if I push a tag to master. All only if the tests succeed, of course.

DevOps FTW.
Reply all
Reply to author
Forward
0 new messages