parallel rendering fixes

36 views
Skip to first unread message

Matthew Butterick

unread,
Jun 13, 2019, 9:22:30 PM6/13/19
to Pollen
I recommend that anyone who enjoys `raco pollen render -p` update to the newest Pollen. In the last week I've discovered (and hopefully corrected) a couple bugs that were causing it to silently ignore files during a parallel render.


Matthew Butterick

unread,
Jun 13, 2019, 10:33:51 PM6/13/19
to Pollen
Eh, don't bother. I just found another bug in it.

Matthew Butterick

unread,
Jun 13, 2019, 11:01:15 PM6/13/19
to Pollen
I'm afraid the parallel-processing problem is going to require more scrutiny about how it can be done safely. For the time being I've disabled parallel processing (you can still ask for `raco pollen render -p`, but it won't be parallel).

Joel Dueck

unread,
Jun 15, 2019, 4:29:51 PM6/15/19
to Pollen
I haven’t had any trouble and it’s been saving me a lot of time, so I’ll hold off updating for now. Thank you for tackling this!

Matthew Butterick

unread,
Jun 26, 2019, 2:29:31 PM6/26/19
to Pollen
Armed with more knowledge, I have reinstated parallel rendering. `raco pollen setup` is also faster.

Joel Dueck

unread,
Jun 27, 2019, 8:44:41 AM6/27/19
to Pollen
Awesome, thank you!

Shrutarshi Basu

unread,
Jun 27, 2019, 4:23:37 PM6/27/19
to Joel Dueck, Pollen

This is great. I'm wondering how this should affect Makefile-driven setups? Currently I have a project with a bunch of directories, and a Makefile that runs `raco pollen render` for all the directories, and then publish at the toplevel directory. I've been running it `make -j 4` (since that's how many cores I have), but maybe there's a better way to do it? Should I just use `raco pollen render --recursive` at the toplevel and let pollen figure everything out?

Here's a gist of my current Makefile, if it helps: https://gist.github.com/basus/e524fed3a305fe4235ba03eab68c634d

--
Shrutarshi Basu
The ByteBaker -- computer science is not about computers

Matthew Butterick

unread,
Jun 27, 2019, 6:01:17 PM6/27/19
to Shrutarshi Basu, Pollen
I just use `raco pollen render -p` rather than plain `raco pollen render` in my Makefile, for the top directory in the project.

I'd guess this is more efficient than `make -j x` because all the files are inventoried and then doled out to each processor as fast as they can be rendered. But try it and let me know if that's not so.

Shrutarshi Basu

unread,
Jun 27, 2019, 10:33:00 PM6/27/19
to Matthew Butterick, Pollen
Hmm interesting, I reset the cache using `raco pollen reset` and then run `raco pollen render -p` as well as `make -B -j 4` (the -B option forces rebuild of all targets). Here's what I get:

raco pollen render -p  10.68s user 2.16s system 223% cpu 5.751 total

make -B -j 4  20.19s user 4.55s system 236% cpu 10.456 total

Looks like render -p is the way to go.

Thanks,
Basu

Matthew Butterick

unread,
Jun 27, 2019, 10:41:04 PM6/27/19
to Shrutarshi Basu, Pollen
BTW, it's wise to do `raco pollen setup` (which is parallel by default) before `raco pollen render -p`.


--
You received this message because you are subscribed to the Google Groups "Pollen" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pollenpub+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pollenpub/CA%2BYT8WiEp-vOusw7vHqszpqjY%3DVp7LnEgJmHLvcibRFnAQVUTg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Joel Dueck

unread,
Jun 28, 2019, 12:47:23 PM6/28/19
to Pollen
Take a look at one of my recent Pollen makefiles:  https://thelocalyarn.com/cgi-bin/yarncode/artifact/7a3d32d6264f1bcb

I define my dependencies such that if one of the core files (like pollen.rkt) has changed, it triggers a re-render of all documents with `raco pollen setup` and then `raco pollen render -p ...`.

The later dependencies check to see if individual Pollen articles (.poly.pm) have changed since their .html file was last rendered. If a render-of-everything has already been triggered above, then these will already be up to date at this point. But if not, any changed articles will be re-rendered singly, without parallel processing. (I don't use the -j switch when running make.)

So the basic idea is that I give make a very specific dependency tree, and rely on it to check those dependencies in a specific order, bringing in the new parallel setup and render when it's clear a complete rebuild is needed.

You also have to make sure that the makefile and the Pollen setup cache watchlist agree precisely on which are the "core" files that would trigger a complete rebuild.

Because I have some Pollen articles that depend on others, I also use tricks like "touching" the dependent articles during the evaluation of the `root` tag.

Shrutarshi Basu

unread,
Jun 28, 2019, 2:17:20 PM6/28/19
to Joel Dueck, Pollen
Thank you! This is very helpful. I'm still trying to figure out what the good balance between Makefiles and render -p. Some experimenting is in order.

--
You received this message because you are subscribed to the Google Groups "Pollen" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pollenpub+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pollenpub/d52ce987-16bf-4df1-be52-0b67450c8d22%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.
Reply all
Reply to author
Forward
0 new messages