racket2nix

[stewart mackenzie]

Hello,

We're partially through the development of a nix{os} utility which transforms an info.rkt into a nix expression.

https://github.com/clacke/racket2nix

It'll be helpful if other nixers/racketers could contribute or test the project!

kr/sjm 

[Claes Wallin]

Hi, Racketeers! I'm the one mainly working on this, and I could use some help right now. :-)

I have a concrete problem, and I've been banging my head against this for a bit, so I hope someone has some insights to share.

First some background:

The way Nix works is, each package has its own directory under /nix/store with an FHS-like structure. The directory is called /nix/store/<unique 32 character hash>-<package name>, so for example we would have a directory /nix/store/vawj9y05mlvaflflswd7f7ibw3kj4x92-base/share/racket/pkgs/base , where the info.rkt for base would reside.

Now, when I want to build another package I would need that package to find racket-lib (in the racket package itself) and base (in the package base). What I have been doing, just to get things working, is to `raco pkg install --scope-dir /nix/...mypkg/share/racket/pkgs /nix/...base/share/racket/pkgs/base`, and so on for each transitive dependency of mypkg, and then finally `raco pkg install --copy --scope-dir /nix/.../pkgs ./mypkg`.

If you're familiar with the bowels of racket packaging, you are probably already seeing a number of issues with this. It "works", as in, I manage to build packages that depend on other packages, but it seems that doing install on each transitive dependency creates quite a lot of unnecessary work, and using --scope-dir the way I am places links.rktd in an unexpected place. I'm also having trouble with packages that want to write to collects, or copy man pages.

To improve this I'd like to do a number of things:
1. The package I'm currently building probably needs to be the installation path for racket, so that I can use installation scope and have links.rktd and other things end up in the expected places.
2. As my dependencies and their dependencies have already been installed once, and I want to use them from their already-prepared destination, it should be possible to just merge together the links.rktd of my dependencies and add my own package to the list, and that should be my new links.rktd.

Are you following me so far, or am I already misunderstanding something?

I've finally found that the way to customize where racket finds things is to mess about with `racket -G` and the config.rktd file. I was hoping that I could just add the share/racket directories of my dependencies to 'links-search-files, but it doesn't seem that simple.

Here's what I'm currently getting when installing base:

++ /nix/store/hmdy6zq34vq0761l4lym9cl3piw9b6gc-racket-6.12/bin/racket -G /nix/store/m5msdbznjzjn0jkrknbk6cswybjqpi4m-base/etc/racket -U -X /nix/store/m5msdbznjzjn0jkrknbk6cswybjqpi4m-base/share/racket/collects -S /nix/store/hmdy6zq34vq0761l4lym9cl3piw9b6gc-racket-6.12/share/racket/collects -e '
(require setup/dirs)
(display (format "(current-library-collection-links): ~a~n" (current-library-collection-links)))
(display (format "(find-library-collection-links): ~a~n" (find-library-collection-links)))
(display (format "(find-links-file): ~a~n" (find-links-file)))
'
(current-library-collection-links): (#f /nix/store/hmdy6zq34vq0761l4lym9cl3piw9b6gc-racket-6.12/share/racket)
(find-library-collection-links): (#f /nix/store/hmdy6zq34vq0761l4lym9cl3piw9b6gc-racket-6.12/share/racket)
(find-links-file): /nix/store/m5msdbznjzjn0jkrknbk6cswybjqpi4m-base/share/racket/links.rktd
++ /nix/store/hmdy6zq34vq0761l4lym9cl3piw9b6gc-racket-6.12/bin/racket -G /nix/store/m5msdbznjzjn0jkrknbk6cswybjqpi4m-base/etc/racket -U -X /nix/store/m5msdbznjzjn0jkrknbk6cswybjqpi4m-base/share/racket/collects -S /nix/store/hmdy6zq34vq0761l4lym9cl3piw9b6gc-racket-6.12/share/racket/collects /nix/store/hmdy6zq34vq0761l4lym9cl3piw9b6gc-racket-6.12/share/racket/collects/raco/main.rkt pkg install --copy --deps fail --fail-fast --scope installation ./base
main.rkt pkg install: missing dependencies;
 specify `--deps search-auto' to install them, or
 specify `--deps search-ask' to be asked about installing them
  for package: ./base
  missing packages:
   racket-lib

/nix/store/m5msdbznjzjn0jkrknbk6cswybjqpi4m-base/etc/racket/config.rktd is:
#hash(
  (share-dir . "/nix/store/m5msdbznjzjn0jkrknbk6cswybjqpi4m-base/share/racket")
  (links-search-files . ( "/nix/store/hmdy6zq34vq0761l4lym9cl3piw9b6gc-racket-6.12/share/racket" ))
)


I would be very grateful for just a hint or nudge in the right direction. I'm happy to already have received a PR on my code style from Felleisen himself, and am open to suggestions on everything. I want this to be a tool that will benefit the community, not something you will have to fight to have any use for it, or something that is too obscure or idiosyncratic for anybody to be able to contribute.

--
   /c

[Anthony Carrico]

I use Nix, but mostly for Haskell currently, rather than Racket. I did
prototype a nix project with Racket. I don't have the sources at my
fingertips. I'll report if I look back and see anything helpful for you.

It looks like you are trying to tie into the Racket package system. We
didn't try to do so, we were just trying to go from a git clone to a
common development environment in one step.

I do remember modifying the racket derivation in the nixpkgs repo. I
don't think that got pushed upstream. The Racket derivation needed to be
parameterized to find the opengl libs/drivers. I also remember that I
couldn't get this to work reliably on any OS except NixOS. Any opengl
program was going to have similar trouble since the libs are a function
of the drivers which can't really be provided by Nix on non NixOS
platforms. This situation was more of a reflection on the state of
OpenGL than Nix in my opinion, but we ended up developing in a NixOs vm
for this reason only. I don't remember what other changes I made, this
was the biggest pain point. I did integrate with other foreign functions.

--
Anthony Carrico

[Matthew Flatt]

At Wed, 14 Feb 2018 03:44:44 -0800 (PST), Claes Wallin wrote:
> 1. The package I'm currently building probably needs to be the installation
> path for racket, so that I can use installation scope and have links.rktd
> and other things end up in the expected places.

That sounds right.

> 2. As my dependencies and their dependencies have already been installed
> once, and I want to use them from their already-prepared destination, it
> should be possible to just merge together the links.rktd of my dependencies
> and add my own package to the list, and that should be my new links.rktd.
>

> [...]

>
> I've finally found that the way to customize where racket finds things is
> to mess about with `racket -G` and the config.rktd file. I was hoping that
> I could just add the share/racket directories of my dependencies to
> 'links-search-files, but it doesn't seem that simple.

I think this is also the right idea, but I think you need to set
'pkg-search-dirs in parallel to 'links-search-files within
"config.rktd".

The two separate entries reflect how the package system is built on top
of the collection-link layer, but it has to keep additional information
that is specific to the package layer.

[Claes Wallin]

Thanks! I'll try that.

I thought packages were always looked up through the collections, but maybe that's a difference between package dependencies when installing and the way you look them up when you refer to them in the code?

--
   /c

[Claes Wallin]

On Thursday, February 15, 2018 at 3:01:08 AM UTC+8, Anthony Carrico wrote:

The Racket derivation needed to be
parameterized to find the opengl libs/drivers. I also remember that I
couldn't get this to work reliably on any OS except NixOS. Any opengl
program was going to have similar trouble since the libs are a function
of the drivers which can't really be provided by Nix on non NixOS
platforms.

Wow, ok.
 

This situation was more of a reflection on the state of
OpenGL than Nix in my opinion, but we ended up developing in a NixOs vm
for this reason only.

Yeah, that sounds like an oddity.

One nice thing you can do once you have racket2nix is that you can have your racket package generated as a nix package, and then override the nix package to have native dependencies, which I hope will be very helpful to people in your situation in the future.

--
   /c

[Claes Wallin]

On Thursday, February 15, 2018 at 9:18:14 AM UTC+8, Claes Wallin wrote:

On Thursday, February 15, 2018 at 3:16:24 AM UTC+8, Matthew Flatt wrote:

 

I think this is also the right idea, but I think you need to set
'pkg-search-dirs in parallel to 'links-search-files within
"config.rktd".

> Thanks! I'll try that.

It worked! So now it even complains when I try to install base against a full racket, that base already exists. This is a good thing. :-)

But when I'm installing against minimal I get:

/nix/store/rm4jgl2jhdmv0b2jlr2jmziih8clximf-racket-minimal-6.12/bin/racket -G /nix/store/gfwrnnq5ql1b35i03cs3xyzrflcrapfi-base/etc/racket -U -X /nix/store/gfwrnnq5ql1b35i03cs3xyzrflcrapfi-base/share/racket/collects -S /nix/store/rm4jgl2jhdmv0b2jlr2jmziih8clximf-racket-minimal-6.12/share/racket/collects
(current-library-collection-links): (#f /nix/store/rm4jgl2jhdmv0b2jlr2jmziih8clximf-racket-minimal-6.12/share/racket)
(find-library-collection-links): (#f /nix/store/rm4jgl2jhdmv0b2jlr2jmziih8clximf-racket-minimal-6.12/share/racket)
(find-links-file): /nix/store/gfwrnnq5ql1b35i03cs3xyzrflcrapfi-base/share/racket/links.rktd
(current-library-collection-paths): (/nix/store/gfwrnnq5ql1b35i03cs3xyzrflcrapfi-base/share/racket/collects /nix/store/rm4jgl2jhdmv0b2jlr2jmziih8clximf-racket-minimal-6.12/share/racket/collects)
(find-library-collection-paths): (/nix/store/gfwrnnq5ql1b35i03cs3xyzrflcrapfi-base/share/racket/collects)
(find-pkgs-dir): /nix/store/gfwrnnq5ql1b35i03cs3xyzrflcrapfi-base/share/racket/pkgs
(get-pkgs-search-dirs): (/nix/store/rm4jgl2jhdmv0b2jlr2jmziih8clximf-racket-minimal-6.12/share/racket/pkgs)
collection-file-path: collection not found
  collection: "setup"
  in collection directories:
   /nix/store/gfwrnnq5ql1b35i03cs3xyzrflcrapfi-base/share/racket/collects

 
So (current-library-collection-paths) reports both base/.../collects and racket-minimal-6.12/.../collects. And my little dump script, which printed that value, successfully requires setup/dirs, so I wonder why raco (or something) can't.

--
   /c

[stewart mackenzie]

Re opengl: I recall running into a similar problem, I eventually scrapped the opengl dependency but it might be possible to wrap the whole thing up in a makeWrapper then pass in the opengl executable path for non-nixos systems into it. 

The reason, I believe, is that opengl is packaged as a static library and not a dynamic lib.

[reilithion]

I'm excited for this project! As a user of both Racket and NixOS, I've been hoping for something like this to come along for a while. I wish I had more time to devote to helping it succeed. I'll probably poke at it a little with a small side project and try to give feedback on using it.

[Christopher Lemmer Webber]

Claes Wallin writes:

> On Sunday, February 11, 2018 at 9:43:34 PM UTC+8, stewart mackenzie wrote:
>>
>> Hello,
>>
>> We're partially through the development of a nix{os} utility which
>> transforms an info.rkt into a nix expression.
>>
>> https://github.com/clacke/racket2nix
>>
>> It'll be helpful if other nixers/racketers could contribute or test the
>> project!
>>
>
> Hi, Racketeers! I'm the one mainly working on this, and I could use some
> help right now. :-)
>
> I have a concrete problem, and I've been banging my head against this for a
> bit, so I hope someone has some insights to share.

This is great Claes, and as I said on IRC, I hope we can eventually
collaborate... as some others know I'm hoping to get more Racket
packages into Guix... which is like Nix, but written in Scheme!
But that work will probably build on experiences from this Nix work if I
get to it first at all (and it sounds like you might beat me to it
anyway) ;)

Thanks for working on this!

[johnbclements]

Test posting, please ignore, sorry....

[Claes Wallin]

On Tuesday, February 20, 2018 at 4:40:38 AM UTC+8, cwebber wrote:

This is great Claes, and as I said on IRC, I hope we can eventually
collaborate... as some others know I'm hoping to get more Racket
packages into Guix... which is like Nix, but written in Scheme!
But that work will probably build on experiences from this Nix work if I
get to it first at all (and it sounds like you might beat me to it
anyway) ;)

Thanks for working on this!

It sounded on IRC that you're pretty busy, so yeah, I'll probably get there first. :-)

Status update: racket2nix is now in a limited useful state! I'm able to build the drracket package without any error conditions occurring along the way, but:
 - The resulting drracket package is empty. :-D
 - I'm not actually building the packages, just installing them (raco pkg install --no-setup).
 - There is a lot of cleaning up to do in the nix expressions, the low-hanging fruit being e.g. redundant dependency listings -- drracket has over 1500 dependencies, but a simple sort -u on them reveals that they are actually only 135.
 - It probably doesn't put any stuff in .../bin .

My next step is dogfooding racket2nix: Put the logic in a library, generate a thin wrapper that calls it, add rackunit tests, make a man page, etc. Hopefully this will teach me enough about packaging that I can go back and work toward a working drracket package.

[Anthony Carrico]

On 02/16/2018 04:32 AM, stewart mackenzie wrote:
> Re opengl: I recall running into a similar problem, I eventually
> scrapped the opengl dependency but it might be possible to wrap the
> whole thing up in a makeWrapper then pass in the opengl executable path
> for non-nixos systems into it. 

Thanks. That might require some (dynamic) probing. IIRC the OpenGL
library/driver situation was different from distribution to distribution
(and just needlessly confusing in general). Maybe someone has worked up
a general solution to this particular problem, and if so Racket could
adopt that. Amazingly, Nix has already solved many such weird issues. If
this Racket/Nix effort goes forward, Racket users will probably need
some guidance (and discipline) regarding foreign libraries.

--
Anthony Carrico