My JBake site build is "quite" slow

[Nicolas Delsaux]

Hello again

So, I use JBake to create a life stream including everything (as much as
it is possible) that I wrote in the past 15/20 years.
(http://nicolas.delsaux.free.fr/lifestream)

My last build took 2H30 hours (well, 17 hours since it was interrupted
by computer being put to sleep for the night).

Why is it so long ? I had some problems creating a good post template
(but it was solved in previous message). But what is really annoying
currently is the tag pages generations. Indeed, as I have ~9700 posts,
there are ~3000 associated tags.

My tag template is the following
https://github.com/Riduidel/lifestream/blob/master/my-lifestream/src/main/jbake/templates/tags.ftl

This template is quite similar to default one, with the exception of
linked tags being aggregated from posts one.

What I have seen is that there are long generation pauses (at least 1
minute) roughly each 50 tag generation. And this is the reason
generation is so long.

I'm gonna investigate that in the coming days ...

[Nicolas Delsaux]

So a first sampling using VisualVM give me the following result (after 1 hour running)

I find specially weird the time spent in Unsafe#park() and that synthetic accessor.

I will take a look while application hangs in debug mode ...

[Nicolas Delsaux]

Just a quick followup

I paused my application in debug while it was "hanged" and discovered that the hang was due to very slow call to ?seq_contains (on this line https://github.com/Riduidel/lifestream/blob/master/my-lifestream/src/main/jbake/templates/tags.ftl#L36).

I discovered it was slow because freemarker, when concatenating sequences, was in fact doing some kind of facade sequence : the returned sequence contains a left and right segment, each one being potentially a sequence itself.

As a consequence, when rendering big tag pages (like this one http://nicolas.delsaux.free.fr/lifestream/tags/web.html), there is a very deep stack of sequence objects that has to be navigated in each post rendering ... Yup, it's more than a O(n²). Removing the three lines reduced build time to 17 minutes, so I think I'm gonna remove that big list of tags at the end of the tag page ...

As a consequence, I would suggest that, for big JBake projects, don't do crazy things in templates 😉

--
You received this message because you are subscribed to the Google Groups "JBake Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jbake-user+...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/jbake-user/8b344faa-132a-3df4-674d-454c35226115%40gmx.fr.

[Jonathan Bullock]

Hi Nicolas,

Thanks for raising this, investigating it and sharing your findings with the community. If my memory serves me correctly then your lifestream project is probably the biggest JBake implementation in 

terms of source files and I'm sure there are plenty of areas for improvement.

Have you thought about trying a different template engine for your lifestream project?

Jon

To view this discussion on the web, visit https://groups.google.com/d/msgid/jbake-user/20750c7a-6e11-82d3-f089-b907e0c80000%40gmx.fr.

[Nicolas Delsaux]

Would be nice to have some statistics somewhere, maybe through a custom template.

Anyway, Windows tell me that the generated JBake site, it is

* 223 Mo (no images are included)

* 461 folders

* 12076 files

It generates in roughly 20 minutes.

I have not thought about switching to groovy templates, even if I know quite well Groovy, because I find the documentation of Freemarker incredibly good.

And I think that, at this size, I would be very happy to have some kind of parallel processing of templates. Because I see pages appearing at a 1 page/second rate, and I think it could be waaayyy faster if parallel'd

Anyway, it works really well and renders nicely (once again, take a look at http://nicolas.delsaux.free.fr/lifestream, which run on free.fr homepage server - arguably as slow as a Raspberry 1, it runs very well, renders nicely even with quite big HTML pages). And that's thanks to JBake. So thank you all !

[Frank Becker]

Hi Nicolas,

thanks for sharing your observations and mitigations.

I started implementing parallelization of crawling, rendering and asset processing.

https://github.com/ancho/jbake/commits/experiment/cuncurrent-render-and-crawl

It's not stable and properly tested, yet. Just a quick spike.

If you find some time, I would be interested how this works out for you. But with more than 20 minutes build time, you may run into the 10 minutes timeout

I hardcoded for the executerservice after shutdown. (Renderer and Crawler haver there own threadpool)

I will invest more time in this. Needs more testing. There maybe the one ore other unsynchronized concurrent access to a few datastructures to tackle down.

To unsubscribe from this group and stop receiving emails from it, send an email to jbake...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/jbake-user/8b344faa-132a-3df4-674d-454c35226115%40gmx.fr.

--
You received this message because you are subscribed to the Google Groups "JBake Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jbake...@googlegroups.com.

[Nicolas Delsaux]

Hello from the grave :-D

Considering my project exists since more than 8 years, and I only start now to consider automated deployment, I'm coming again at JBake to see what could possibily be optimized.

Because I plan to have it running on my Raspberry Pi 3, I need to seriously optimize things to have things running smoothly. And consider the generating process is killed each time by Raspbian.

So, I'm currently considering other course of things.

First, is it possible using JBake to have pages generated only when their source content change ?

Second, do you know how to have archives generated by date, instead of a paginated list of all pages

And finally, specifically for Frank, have you had any progress into your parallelization of JBake processing ? This would make things faster.

Oh, and to have some progress, I know have

* 269 Mo

* 1303 folders (tags make that folder number grow fast)

* 14417 files !

To unsubscribe from this group and stop receiving emails from it, send an email to jbake-user+...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/jbake-user/de85b827-8049-42f0-b47f-1d4bbb636579%40googlegroups.com.

[flug...@gmail.com]

Hi Nicolas,

> First, is it possible using JBake to have pages generated only when their source content change?

Yes. See https://jbake.org/docs/2.6.5/#persistent_content_store

> Second, do you know how to have archives generated by date, instead of a paginated list of all pages

I don't get that. published_posts should already be sorted by date. Can you elaborate on that?

> And finally, specifically for Frank, have you had any progress into your parallelization of JBake processing ? This would make things faster.

Unfortunately no. I didn't go further than that but i am still interested if it works out for you.
I can bundle a snapshot binary distribution release for you if you don't want do build it for yourself.

As I said before. The branch i mentioned is highly experimental and contains changes which are still open as pr since a few years now.