Infinite loops?

[Shrutarshi Basu]

I think I'm somehow creating infinite loops for two different use cases. First, I have a nested directory structure and I'm trying to run pollen in parallel subdirectory mode. I'm using `raco pollen -p -s . ` and here is the error I'm getting:

pollen: rendering generated pagetree for directory /Users/basus/src/basus/
pollen: rendered parallel on core 4 /template.html (21 ms)
pollen: rendered parallel on core 1 /error.html (222 ms)
pollen: rendered parallel on core 2 /index.html (271 ms)
pollen: rendering generated pagetree for directory /Users/basus/src/basus/about
pollen: rendering generated pagetree for directory /Users/basus/src/basus/css
pollen: rendering generated pagetree for directory /Users/basus/src/basus/images
pollen: rendering generated pagetree for directory /Users/basus/src/basus/js
pollen: rendering generated pagetree for directory /Users/basus/src/basus/posts
pollen: rendering generated pagetree for directory /Users/basus/src/basus/research
open-input-file: cannot open input file
  path: /Applications/Racket v7.2/collects/racket/compiled/match_rkt.zo
  system error: Too many open files; errno=24
  context...:
   default-load-handler
   standard-module-name-resolver
   namespace-module-instantiate!96
   for-loop
   [repeats 1 more time]
   run-module-instance!125
   for-loop
   [repeats 1 more time]
   run-module-instance!125
   do-dynamic-require5
open-input-file: cannot open input file
  path: /Applications/Racket v7.2/collects/racket/private/compiled/string_rkt.zo
  system error: Too many open files; errno=24
  context...:
   default-load-handler
   standard-module-name-resolver
   namespace-module-instantiate!96
   for-loop
   [repeats 1 more time]
   run-module-instance!125
   for-loop
   [repeats 1 more time]
   run-module-instance!125
   for-loop
   [repeats 1 more time]
   run-module-instance!125
   for-loop
   [repeats 1 more time]
   run-module-instance!125
   do-dynamic-require5
pollen: rendering generated pagetree for directory /Users/basus/src/basus/research/publications
open-input-file: cannot open input file
  path: /Applications/Racket v7.2/share/pkgs/scribble-lib/scribble/private/compiled/render-utils_rkt.zo
  system error: Too many open files; errno=24
  context...:
   default-load-handler
   standard-module-name-resolver
   namespace-module-instantiate!96
   for-loop
   [repeats 1 more time]
   run-module-instance!125
   for-loop
   [repeats 1 more time]
   run-module-instance!125
   for-loop
   [repeats 1 more time]
   run-module-instance!125
   for-loop
   [repeats 1 more time]
   run-module-instance!125
   do-dynamic-require5
dynamic-place: stdout dup failed
  system error: Too many open files; errno=24
  context...:
   /Applications/Racket v7.2/collects/racket/place.rkt:108:0: start-place
   /Users/basus/Library/Racket/7.2/pkgs/pollen/pollen/render.rkt:51:25: render-batch4
   /Users/basus/Library/Racket/7.2/pkgs/pollen/pollen/private/command.rkt:106:9: render-one-dir
   /Users/basus/Library/Racket/7.2/pkgs/pollen/pollen/private/command.rkt:127:15: for-loop
   /Users/basus/Library/Racket/7.2/pkgs/pollen/pollen/private/command.rkt:106:9: render-one-dir
   /Users/basus/Library/Racket/7.2/pkgs/pollen/pollen/private/command.rkt:127:15: for-loop
   /Users/basus/Library/Racket/7.2/pkgs/pollen/pollen/private/command.rkt:106:9: render-one-dir
   /Users/basus/Library/Racket/7.2/pkgs/pollen/pollen/private/external/logging.rkt:46:0
   (submod "/Users/basus/Library/Racket/7.2/pkgs/pollen/pollen/private/command.rkt" raco): [running body]
   temp37_0
   for-loop
   run-module-instance!125
   "/Applications/Racket v7.2/collects/raco/raco.rkt": [running body]
   temp37_0
   for-loop
   run-module-instance!125
   ...

Second, I am trying to override the setup module using the instructions described in the documentation. Here is the setup module I'm trying to write:

(module setup racket/base
  (provide (all-defined-out))
  (define publish-directory "./output")
  (define (omitted-path path)
    (print (path))
    (or (default-omitted-path path)
        (equal? path (path->complete-path "./images/"))
        (equal? path (path->complete-path "./js/"))
        )))

But when I run `raco pollen publish` I get the following output and then it gets stuck:

pollen: publishing from /Users/basus/src/basus to /Users/basus/Desktop/publish ...

Note that it doesn't seem to be picking up the new publish directory I specified above.

Unfortunately I'm still pretty new to Racket, so I'm not sure how to go about debugging this.

Thanks,
Basu

--

Shrutarshi Basu

Basus.me

The ByteBaker -- computer science is not about computers

@basus

[Shrutarshi Basu]

I've managed to fix the publish case so that I no longer see the error. It is not working the way I would like, but I will send a separate email with some thoughts on the `render` and `output` commands and their workings. Here is the code I currently have for my setup:

;; Override Pollen parameters
(module setup racket/base
  (require racket/path pollen/setup)
  (provide (all-defined-out))

  (define publish-directory (expand-user-path "~/output/"))

  (define (omitted-path? path)
    (define img-path (path->complete-path "images/"))
    (define js-path (path->complete-path "js/"))
    (define p-only (path-only path))
    (or (default-omitted-path? path)
        (equal? p-only img-path)
        (equal? p-only js-path)))
 )

I am still seeing the same problem with render

[Shrutarshi Basu]

After some more experimentation, it seems like the problem goes away if I drop the -p flag. So `raco pollen render -s .` works just fine. My hypothesis is that when running in parallel mode, the pollen process (?) opens a lot of files in parallel, which triggers the "Too many open files" error.

Thanks,
Basu

[Matthew Butterick]

Should -p take an argument to limit the max number of parallel jobs?

--
You received this message because you are subscribed to the Google Groups "Pollen" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pollenpub+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pollenpub/CA%2BYT8WhEWGs_kO-Fdp-VCQ9EsHtjw2zuwgbc-Ku2M3F0tu8TXQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Shrutarshi Basu]

That might work. It seems like the error is coming from somewhere in the Racket runtime when it tries to load a module, so I don't think there is an easy way to catch the error and cleanly deal with it.

Thanks,
Basu

[Matthew Butterick]

I've added a -j (or --jobs) option to `raco pollen render` that takes a certain number of parallel jobs as an argument.

So the existing

raco pollen render -p

Will, as usual, start a number of jobs equal to the number of processor cores, but the new

raco pollen render -j n

Will start n parallel jobs, for some integer n.

I'm not sure why Racket would throw a "too many open files" error, but maybe fewer jobs will help.

To be consistent, I've also added these switches to `raco pollen setup` and changed the default setup action to be non-parallel. So if you want to keep using parallel setup, you'll now have to do:

raco pollen setup -p

or

raco pollen setup -j n