Confused about omitted-path

[Shrutarshi Basu]

Hello everyone,
I’m a little confused about the use of `omitted-path?`, and how paths
are handled in general.

Here is the setup: I have a project with a bunch of subdirectories.
Some of those subdirectories have JavaScript, or images, etc. These
directories *do not* need to be processed by `pollen render -s`, but
*do* need to be processed by `pollen publish`. As a start, I tried to
override `setup:omitted-path?` as directed in the documentation, and
printed out the paths that were being handled by the function. I
noticed some behavior that seems odd to me.

First, during `pollen render -s`, the paths getting sent to
`omitted-path?` seem to be paths of the subdirectories, not the files
that are being rendered. This seems a little odd to me: what if I want
some files in a directory to be rendered, and not others? But this is
fine for omitting image or script directories for my use case.

Second, during `pollen publish`, the paths sent to `omitted-path?` are
(1) the paths in the `publish-directory` that the files will get
copied to, not in the source directory, and (2) this time, the entire
paths to the target files, not just the directories. (1) makes it
harder to implement the functionality I want above, because I now I
need to think in terms of the target directory, not the current
project directory, and (2) seems inconsistent with what happens for
`pollen render`, so `omitted-path?` will have to handle both cases.

I’m not familiar with the internals of pollen, so maybe there is a
good reason for `omitted-path?` to be used this way. But here are some
suggestions for making things more uniform?

1. `omitted-path?` is always given the full source path for each
*file* being considered.
2. `omitted-path?` is called for *every* file that is being considered
for rendering, unless...
3. There is a `omitted-directory?` function that is called when pollen
is about to descend into a directory. If it returns true, the entire
directory is skipped, and the files and subdirectories in it do not
need to be processed by `omitted-path?`.
4. There is an optional argument to `omitted-path?` that describes
whether it is being called as part of a render or publish operation.
This lets things like image directories be skipped during render, but
included during publishing. Alternatively, render and publish could
call separate functions to check for omission (that might be cleaner
actually).

I’m interested in helping develop pollen, so I’m happy to try and make
these changes in a fork and send a pull request.

I know there’s a note in the docs saying that `publish` might need
improvement, so maybe my thoughts above relate to that?

For reference, my `omitted-path?` implementation is below. Do let me
know if I’m doing something wrong, or am just being ignorant.

Thanks,
Basu


;; Override Pollen parameters
(module setup racket/base
  (require racket/path pollen/setup)
  (provide (all-defined-out))

  (define publish-directory (expand-user-path "~/output/"))
  (define img-path (path->complete-path "images/"))
  (define js-path (path->complete-path "js/"))

  (define (omitted-path? path)
    (printf "Checking path: ~a\n" path)
    (define p-only (path-only (path->complete-path path)))
    (or (default-omitted-path? path)
        (equal? p-only img-path)
        (equal? p-only js-path)))
 )

[Shrutarshi Basu]

After digging into the source code, I’m realizing that my suggestions won’t work as I put them, mainly due to the presence of pagetrees. which can control which files get rendered. I’m wondering if for my needs it might be easier to build a simpler custom front-end rather than trying to change how Pollen behaves.

Also, can someone point me to where the functions `preproc-source?` and `pagetree-source?` live? I can’t seem to find them by grepping the pollen source code.

Thanks,

Basu

[Joel McCracken]

I don't have answers to your direct questions, but I can tell you that I encountered some similar issues and decided to just roll my own solutions for publishing.

Here I list all pollen files in my project for rendering:

https://gitlab.com/JoelMcCracken/joelmccracken.com/blob/88971b0c0a4f6fdc405c7f9a55bef4afc3875354/src/util.rkt#L27

Here is my build file that builds everything: https://gitlab.com/JoelMcCracken/joelmccracken.com/blob/88971b0c0a4f6fdc405c7f9a55bef4afc3875354/src/build.rkt#L88

I handle static JS files by just copying them manually to the build destination in a makefile: https://gitlab.com/JoelMcCracken/joelmccracken.com/blob/88971b0c0a4f6fdc405c7f9a55bef4afc3875354/src/Makefile

--
You received this message because you are subscribed to the Google Groups "Pollen" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pollenpub+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pollenpub/CA%2BYT8Wgd4Z3XP974Hx7_YrT9mXRVL4LTmWErmVtcgj2zReG9JQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Matthew Butterick]

Sorry about the delayed response — some of us were at Racket Week having our brains liquefied.

Here is the setup: I have a project with a bunch of subdirectories. Some of those subdirectories have JavaScript, or images, etc. These directories *do not* need to be processed by `pollen render -s`, but *do* need to be processed by `pollen publish`. As a start, I tried to override `setup:omitted-path?` as directed in the documentation, and printed out the paths that were being handled by the function. I noticed some behavior that seems odd to me. 

`pollen render` won't affect non-Pollen files, so it's safe to say `pollen render my-dir/*`.

`pollen publish`, by contrast, affects all files, because it's just a filesystem utility. The only Pollen-istic aspect of it is that it relies on predicates exported from a `setup` module.

`omitted-path?` is supposed to receive as input the complete path of every file Pollen tries to render or publish. If you think there's a bug, maybe you could show me a failing example?

After digging into the source code, I’m realizing that my suggestions won’t work as I put them, mainly due to the presence of pagetrees. which can control which files get rendered. I’m wondering if for my needs it might be easier to build a simpler custom front-end rather than trying to change how Pollen behaves.

Perhaps — `pollen setup` and `pollen render` and `pollen publish` were meant to be tools for the common cases. You can of course drive a render with a custom Racket script or Makefile.

FWIW I use pagetree sources at times not for navigation but strictly as a tool for scripting renders. That is, I might have a "utility.ptree" file that lists all the oddball source files (for instance, one that generates a payment-confirmation page, which is not in the default navigation flow) and then feed "utility.ptree" to `pollen render`.

Also, can someone point me to where the functions `preproc-source?` and `pagetree-source?` live? I can’t seem to find them by grepping the pollen source code.

They are generated by the `define-utility-functions` macro in `pollen/private/file-utils`.

be used this way. But here are some suggestions for making things more uniform? 

1. `omitted-path?` is always given the full source path for each *file* being considered. 
2. `omitted-path?` is called for *every* file that is being considered for rendering, unless... 

Per above, AFAIK that is how they should work.

3. There is a `omitted-directory?` function that is called when pollen is about to descend into a directory. If it returns true, the entire directory is skipped, and the files and subdirectories in it do not need to be processed by `omitted-path?`. 

I'm not clear how that improves on `omitted-path?` alone (also, I'm leery of having two functions with such overlapping juridsiction)

4. There is an optional argument to `omitted-path?` that describes whether it is being called as part of a render or publish operation. This lets things like image directories be skipped during render, but included during publishing. Alternatively, render and publish could call separate functions to check for omission (that might be cleaner actually). 

Certainly you can use the POLLEN environment variable from the outside to change the behavior of `omitted-path?` in render vs. setup.

POLLEN=BAR raco pollen setup

POLLEN=FOO raco pollen render

And then inside `omitted-path?`, you could test for the POLLEN environment variable and change the behavior accordingly.

[Shrutarshi Basu]

No worries on the delay. Hope Racket Week was fun. I considered going, but I don’t think I do enough Racket work yet to make it worthwhile.

Thanks for the advice, and pointing out the functions I was wondering about. I think the simplest thing for me would be to write some racket code to drive the render.

As for the use of `omitted-path?` here’s a minimal Pollen project: https://github.com/basus/pollen-bug

The `pollen.rkt` file overrides `omitted-path` to be print the inputs it gets. If you run `pollen render -s`, you should see that only the js directory is printed. But if you run `pollen publish`, then all the files get printed, but with the output location. This makes sense looking at the code: in render once inside a not-omitted directory, rendering happens according to the pagetree (existing or generated).

I don’t know if I would call this a bug and I’m not sure if anything needs to change in the Pollen interface. It’s just a behavior I wasn’t expecting, probably because I forgot about the role of pagetrees and don’t really use them

Hope that helps?

Thanks,

Basu

[Matthew Butterick]

On 07 16 19, at 2:50 PM, Shrutarshi Basu <s...@basus.me> wrote:

I don’t know if I would call this a bug and I’m not sure if anything needs to change in the Pollen interface. It’s just a behavior I wasn’t expecting, probably because I forgot about the role of pagetrees and don’t really use them

OK, I see what you mean. 

Part of the difference in output is attributable to the different set of files that `raco pollen render` and `raco pollen publish` take as input (`publish` operates on everything; `render`  only operates on things that look like Pollen sources)

However you're right that consistent with the docs, `publish` tests every directory and file against `extra-path?` and `omitted-path?`. Whereas `render` was inconsistent with the docs: it was only testing directories against `omitted-path?` (not files) and not using `extra-path?` at all. I agree this is a bug, and have pushed a fix. Thanks.

[Matthew Butterick]

I reverted part of this fix. The default behavior of `raco pollen render` is to render preproc and pagetree files in a certain directory. The problem is that if the `extra-path?` and `omitted-path?` predicates are applied at that moment, then nothing will be rendered (because by default, all Pollen source files, including preproc and pagetree files, test #true for `omitted-path?`, because they shouldn't be published.)

So now I'm wondering if the behavior you want is permanently in tension with this (long established) default behavior.

[Shrutarshi Basu]

No worries. I think it is probably better for the pollen project to maintain existing established behavior. For my part, I think I am going to write some simple driver code to work with my setup and requirements.

Thanks,

Basu