does lazy evaluation determine which items are compiled?

[Jorge Peña]

I recently implemented an atom feed on my site. I wrote a post about it [ http://blaenkdenum.com/posts/post-feed-in-hakyll/ ] detailing how I did it.

I couldn't just take a snapshot after compiling each post, because the way I compile each post manipulates each post in such a ways that don't mesh well with feed readers -- i.e. adds a table of contents, syntax highlights codeblocks with Pygments, uses MathJax. So the solution I came up with was to define another pandoc compiler that did _not_ include these additional transformations, and that did not use MathJax in its writer options.

To accomplish this, I essentially created a snapshot of each post, and then I have a rule for a separate "feed" version of each post.

The reason I create a snapshot is because I want to run the item body through a filter in both cases (regular post and feed version), but then I want to compile them using different Pandoc Compilers (i.e. using a simpler one for the feed version). So in the feed version rule I load the snapshot I saved in the regular, no-version rule, after it had been run through the filter, and then compile that using the simpler Pandoc compiler.

The feed itself is then generated using the "feed" versions of the posts, using the typical:

> fmap (take 10) . recentFirst =<< loadAll "etc"

when I'm rendering the feed.

The problem of course is that I'm effectively compiling every post twice, albeit with different compilers each time. I couldn't figure out a better way to do this, so I was hoping that lazy evaluation would save the day by only compiling the ten most recent (according to the above code) posts using the simpler feed compiler. However, upon testing it seems that there's a (feed) version of every post being compiled, even if it's not ultimately included in the rendered atom feed.

Is this expected behavior? Or am I perhaps doing something wrong that's causing this to happen? Perhaps an implicit dependency is created somewhere that is forcing all of them to be compiled? What in the above code forces/causes all posts to be compiled, loadAll?

The relevant code is available here [ https://github.com/blaenk/blaenk.github.io/blob/source/src/Site.hs#L67-L109 ].

If this is expected behavior, and lazy evaluation doesn't save me from the unnecessary work, perhaps someone can propose a different way of doing this? It just seems like this approach might not scale well in the future if every single post has to be compiled twice.

Thanks!

[Roman Cheplyaka]

* Jorge Pe�a <jorge...@gmail.com> [2013-06-18 20:49:43-0700]

> The problem of course is that I'm effectively compiling every post twice,
> albeit with different compilers each time. I couldn't figure out a better
> way to do this, so I was hoping that lazy evaluation would save the day by
> only compiling the ten most recent (according to the above code) posts
> using the simpler feed compiler. However, upon testing it seems that
> there's a (feed) version of every post being compiled, even if it's not
> ultimately included in the rendered atom feed.

Lazy evaluation should not affect which side effects are performed.
(It is possible to do so, but it is considered a bad practice.)

Instead, side effects are typically controlled by monads. Which monadic
actions you sequence determines which side effects get performed at the
end.

Roman

[Jasper Van der Jeugt]

The following rule causes all feed posts to be compiled regardless of
whether they are used.

match postsPattern $ version "feed" $
compile ...

This behavior is mostly inherent to how Hakyll works, although in this
case there's room for an optimization which should be feasible to
implement.

Also note that in principle it should always only compile the latest
(and updated) posts twice. If Hakyll needs to recompile all posts each
time, something else might be wrong.

Peace,
Jasper

On Wed, Jun 19, 2013 at 12:23 PM, Roman Cheplyaka <ro...@ro-che.info> wrote:
> * Jorge Peña <jorge...@gmail.com> [2013-06-18 20:49:43-0700]

> --
> You received this message because you are subscribed to the Google Groups "hakyll" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to hakyll+un...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Jorge Peña]

Oh right, got it.

Yeah that's true. Actually I was forcing regeneration of everything because I have the latest commit hash in my layout's footer, so whenever I deploy I need to regenerate every page so that it can get the latest commit hash embedded. But I'm starting to think I should get rid of this to avoid having to do that each time. Then it wouldn't be that big of a deal to have to generate a feed version for every post, since it should only have to be done once.

That said, do you guys think my approach can be improved? If not I suppose I'll stick with it.

Thanks for the responses!

[Jasper Van der Jeugt]

The following context allows you to get the latest commit for just that item:

commitField :: Context String
commitField = field "commit" $ \item -> do
let fp = toFilePath $ itemIdentifier item
unixFilter "git" ["log", "-1", "--oneline", fp] ""

Not sure if you want the latest master commit everywhere -- I'll
ponder on that some more.

Peace,
Jasper

[Jorge Peña]

Thanks Jasper, indeed I was thinking perhaps I should switch to a per-post commit tag. This would mesh well with generation of posts, since when a post would require regeneration it would be likely that a commit would've been created for that change.

I ended up doing something like this. You guys can take a look here if you like, at the bottom in the post metadata section: http://blaenkdenum.com/posts/post-feed-in-hakyll/ This approach is a lot more semantic as well, I think.

Thanks :)