raco distribute and runtime paths, conditioned on whether a package was installed at compile time

[Jon Zeppieri]

Background: the gregor date and time library depends on another
library called tzinfo to provide information from the systems zoneinfo
database. However, not all systems have a zoneinfo database installed
(notably, Windows systems, but also some minimal UNIX ones used in
Docker containers, for example). So there's another package called
tzdata which provides the zoneinfo database. On Windows, but not on
other platforms, tzdata is a dependency of tzinfo.

When you create a standalone executable with `raco exe` and `raco
distribute` what I want is for tzinfo to use the files from tzdata, if
the latter package is installed when the executable is built. The code
I'm using looks like this:
=====
;; If the tzdata package is installed, put its zoneinfo directory at
;; the head of the search path.
(define-runtime-path-list tzdata-paths
(match (find-relevant-directories '(tzdata-zoneinfo-module-path))
[(cons info-dir _)
(define path ((get-info/full info-dir) 'tzdata-zoneinfo-module-path))
(list path)]
[_
null]))

(match tzdata-paths
[(cons dir _)
(current-zoneinfo-search-path
(cons dir (current-zoneinfo-search-path)))]
[_
(void)])
=====

That is, the tzdata package defines, in an info.rkt file, a key named
`tzdata-zoneinfo-module-path`. I use that to determine if the package
is installed, and, if so, I create a runtime path for it and add that
path to the head of a list of paths to search for zoneinfo files.

The problem is that if a create a bundle for distribution with `raco
distribute` and try to use that bundle on a system where `tzdata` is
not installed, it doesn't work, since the conditional code to find the
`tzdata-zoneinfo-module-path` key fails, of course. So even though the
files are available in the bundle, they aren't used.

Is there a way around this problem? Maybe using `getinfo` is just a
bad approach, but I'm not sure what a good one would be.

- Jon

[Jon Zeppieri]

On Sun, Nov 10, 2019 at 3:26 PM Jon Zeppieri <zepp...@gmail.com> wrote:
> =====
> ;; If the tzdata package is installed, put its zoneinfo directory at
> ;; the head of the search path.
> (define-runtime-path-list tzdata-paths
> (match (find-relevant-directories '(tzdata-zoneinfo-module-path))
> [(cons info-dir _)
> (define path ((get-info/full info-dir) 'tzdata-zoneinfo-module-path))
> (list path)]
> [_
> null]))

I also tried doing the getinfo stuff at expansion time (so, at phase
2, when the RHS of `define-runtime-path` is running at phase 1, and at
phase 1 when it's running at phase 0 -- it does also run at phase 0,
right?), but that didn't help.

[Jon Zeppieri]

And I just tried this, with no luck:

(define-runtime-path-list tzdata-paths
(with-handlers ([exn:fail:filesystem:missing-module? (λ (_) null)])
(list
(resolved-module-path-name
((current-module-name-resolver) '(lib "tzdata/zoneinfo"
"tzinfo") #f #f #f)))))

The files did show up in the raco distribute bundle, but apparently
the code can't find them, which suggests that the
current-module-name-resolver isn't set to look for them there.

[Jon Zeppieri]

On Sun, Nov 10, 2019 at 7:38 PM Jon Zeppieri <zepp...@gmail.com> wrote:
>
> On Sun, Nov 10, 2019 at 4:45 PM Jon Zeppieri <zepp...@gmail.com> wrote:
> >
> > On Sun, Nov 10, 2019 at 3:26 PM Jon Zeppieri <zepp...@gmail.com> wrote:
> > > =====
> > > ;; If the tzdata package is installed, put its zoneinfo directory at
> > > ;; the head of the search path.
> > > (define-runtime-path-list tzdata-paths
> > > (match (find-relevant-directories '(tzdata-zoneinfo-module-path))
> > > [(cons info-dir _)
> > > (define path ((get-info/full info-dir) 'tzdata-zoneinfo-module-path))
> > > (list path)]
> > > [_
> > > null]))

I found a version that works, though I still wonder if there'a a
better approach:

=====
;; If the tzdata package is installed, put its zoneinfo directory at
;; the head of the search path.

(define-syntax (detect-tzdata stx)
(syntax-case stx ()
[(_)

(match (find-relevant-directories '(tzdata-zoneinfo-module-path))
[(cons info-dir _)

(let ([path ((get-info/full info-dir) 'tzdata-zoneinfo-module-path)])
#`(begin
(define-runtime-path tzdata-path '#,path)
(current-zoneinfo-search-path
(cons (simplify-path tzdata-path)
(current-zoneinfo-search-path)))))]
[_ #'(void)])]))

(detect-tzdata)

[Matthew Flatt]

At Sun, 10 Nov 2019 15:26:58 -0500, Jon Zeppieri wrote:
> That is, the tzdata package defines, in an info.rkt file, a key named
> `tzdata-zoneinfo-module-path`. I use that to determine if the package
> is installed, and, if so, I create a runtime path for it and add that
> path to the head of a list of paths to search for zoneinfo files.

At Mon, 11 Nov 2019 00:00:37 -0500, Jon Zeppieri wrote:
> I found a version that works, though I still wonder if there'a a
> better approach:

I'd say the problem is that "info.rkt" and `find-relevant-directories`
are set up as Racket-installation concepts. They're not made to play
well with standalone applications.

The most similar concept that plays well with standard applications is
libraries in the Racket "lib" directory. Having packages that supply
timezone information install the timezone files to Racket's "lib" (in
the same way that a package like "db-win32-x86_64" installs a library)
might be a better approach in the long run.

If I understand, these files would make more sense in a "share"
directory than in the "lib" directory. If that distinction seems
important, we could add a 'share mode to `define-runtime-path` that is
similar to 'lib mode.


I think your solution so far, which relies on compile-time checks, will
not work right with packages that are distributed in "built" form. That
creates a problem, in turn, for building a Racket distribution.

[Jon Zeppieri]

On Tue, Nov 12, 2019 at 2:09 PM Matthew Flatt <mfl...@cs.utah.edu> wrote:
>
> I'd say the problem is that "info.rkt" and `find-relevant-directories`
> are set up as Racket-installation concepts. They're not made to play
> well with standalone applications.
>
> The most similar concept that plays well with standard applications is
> libraries in the Racket "lib" directory. Having packages that supply
> timezone information install the timezone files to Racket's "lib" (in
> the same way that a package like "db-win32-x86_64" installs a library)
> might be a better approach in the long run.
>
> If I understand, these files would make more sense in a "share"
> directory than in the "lib" directory. If that distinction seems
> important, we could add a 'share mode to `define-runtime-path` that is
> similar to 'lib mode.
>
>
> I think your solution so far, which relies on compile-time checks, will
> not work right with packages that are distributed in "built" form. That
> creates a problem, in turn, for building a Racket distribution.
>

I hadn't thought of built packages, but, yes, that sounds like a problem.

I like the idea of installing the data to a "share" directory. I see
that the `info.rkt` file for a collection has a `copy-shared-files`
key. In my case, both the `tzinfo` package and the `tzdata` package
contribute to the `tzinfo` collection. So, two questions: (1) Would it
work just to add this to the collection-level `info.rkt` in the
`tzdata` package? (2) How do I locate these files after they're
copied? Is this what the (notional) 'share mode for
`define-runtime-path` would do?

- Jon

[Matthew Flatt]

At Tue, 12 Nov 2019 21:29:44 -0500, Jon Zeppieri wrote:
> I like the idea of installing the data to a "share" directory. I see
> that the `info.rkt` file for a collection has a `copy-shared-files`
> key. In my case, both the `tzinfo` package and the `tzdata` package
> contribute to the `tzinfo` collection. So, two questions: (1) Would it
> work just to add this to the collection-level `info.rkt` in the
> `tzdata` package?

Yes. I think you'd want to copy the "zoneinfo" directory that way.

> (2) How do I locate these files after they're
> copied? Is this what the (notional) 'share mode for
> `define-runtime-path` would do?

Yes. Although you can find the files using `find-share-dir` and/or
`find-user-share-dir`, adding a 'share mode to `define-runtime-path`
would make it possible for `raco distribute` to find and carry along a
directory/file when it's present, the same as 'so mode does.

(I confused 'lib mode with 'so mode before. I think you want something
like 'so mode.)

[Matthew Flatt]

At Tue, 12 Nov 2019 19:46:01 -0700, Matthew Flatt wrote:
> Although you can find the files using `find-share-dir` and/or
> `find-user-share-dir`, adding a 'share mode to `define-runtime-path`
> would make it possible for `raco distribute` to find and carry along a
> directory/file when it's present, the same as 'so mode does.

Added.

[Jon Zeppieri]

On Tue, Nov 12, 2019 at 10:13 PM Matthew Flatt <mfl...@cs.utah.edu> wrote:
>
> At Tue, 12 Nov 2019 19:46:01 -0700, Matthew Flatt wrote:

> > Although you can find the files using `find-share-dir` and/or
> > `find-user-share-dir`, adding a 'share mode to `define-runtime-path`
> > would make it possible for `raco distribute` to find and carry along a
> > directory/file when it's present, the same as 'so mode does.
>

> Added.
>

Wow, that was fast. Thanks!

- Jon

[James Platt]

Would this now be the preferred method to include a copy of the SQLite library. required by the DB module, to your distribution? The software I am working on requires a newer version of SQLite than is included in at least some of the operating systems we want to support and it's too much to expect end users to install it themselves.

https://docs.racket-lang.org/db/notes.html?q=sqlite#%28part._sqlite3-requirements%29

[Bogdan Popa]

James Platt writes:

> Would this now be the preferred method to include a copy of the SQLite
> library. required by the DB module, to your distribution? The
> software I am working on requires a newer version of SQLite than is
> included in at least some of the operating systems we want to support
> and it's too much to expect end users to install it themselves.

I released two packages[1][2] that distribute more recent versions of
SQLite3 to Linux and macOS a couple of months ago. I build the shared
libraries in CI (using GitHub Actions) then add the .so and .dylib files
to each package, using #lang info's `copy-foreign-libs'[3] to tell
Racket that it should add the libs to a folder where the ffi library
will know to look them up. The source code is here[4]. I'd be happy to
add more architectures and OSs if you need them.

Here's an example[5] where I use the linux package to get the tests for
`deta' passing on the package build server.

I hope that helps!

[1]: https://pkgd.racket-lang.org/pkgn/package/libsqlite3-x86_64-linux
[2]: https://pkgd.racket-lang.org/pkgn/package/libsqlite3-x86_64-macosx
[3]: https://docs.racket-lang.org/raco/setup-info.html?q=copy-foreign-libs#%28idx._%28gentag._22._%28lib._scribblings%2Fraco%2Fraco..scrbl%29%29%29
[4]: https://github.com/Bogdanp/racket-libsqlite3
[5]: https://github.com/Bogdanp/deta/blob/master/deta-test/info.rkt#L5

[James Platt]

On Nov 16, 2019, at 3:49 AM, Bogdan Popa wrote:

>
> James Platt writes:
>
>> Would this now be the preferred method to include a copy of the SQLite
>> library. required by the DB module, to your distribution? The
>> software I am working on requires a newer version of SQLite than is
>> included in at least some of the operating systems we want to support
>> and it's too much to expect end users to install it themselves.
>
> I released two packages[1][2] that distribute more recent versions of
> SQLite3 to Linux and macOS a couple of months ago. I build the shared
> libraries in CI (using GitHub Actions) then add the .so and .dylib files
> to each package, using #lang info's `copy-foreign-libs'[3] to tell
> Racket that it should add the libs to a folder where the ffi library
> will know to look them up. The source code is here[4]. I'd be happy to
> add more architectures and OSs if you need them.

Thanks. I think this is exactly what I need. Linux and macOS are good for now. We will need to add Windows support later but that won't be for a while yet. Eventually, we will want to support some NAS devices and limited mobile apps (Android and iOS) but these are not a priority.


James