finding the expander at run-time

[Jesse Alama]

I'm working on a #lang and having some trouble with run-time paths.

Before I desparately type increasingly improbable combinations of
tenuously related thing hoping that the next invocation will be the
right one, I thought I'd write here. I've tried looking around on the
mailing list archives for help, but I can't quite find what I'm looking
for. My problem is surely not new, so I apologize if I'm inadvertantly
re-asking a question with a well-known answer. If this has already been
definitively answered here or somewhere else, I'm happy with a link to
the discussion.

For context: the users of my #lang aren't Racket programmers, don't know
of Racket, and shoulnd't have to. I'm building a command line tool. It's
made with Racket, of course, but it's essential that this tool not
require DrRacket, and should not require command line invocations like
"racket my-cool-thing.rkt filename.whatever". It should work like this:

$ coolthing file.whatever

where coolthing is my compiled Racket program.

I get that there are ways of making executables, but I'm getting
confused about paths. Specifically, the path for my expander.

My code works as a proper #lang. That is, I can happily work with it in
DrRacket, and I get a REPL for free. That's really great. But, again,
there should be no (immediately visible) Racket dependency.

Here's what I've got:

* expander.rkt: the expander. Provides definitions according to which
raw S-expressions get their meaning.

* info.rkt: present, but contains -- to my knowledge -- nothing
especially relevant. (I almost omitted info.rkt from this list,
thinking it was definitely irrelevant, but then it occurred to me that
I may need to add some paths here.)

* grammar.rkt: Brag grammar (provides parse)

* secret.rkt: the principal program. Uses command-line, takes a filename
as its sole argument, parses it using grammar.rkt, and is intended to
execute the program along the lines of expander.rkt.

When I work in DrRacket, I know that a couple other files (main.rkt,
reader.rkt for read-syntax) are being used. I omitted them from this
list because (I think) they're not necessary for what I'm trying to
do. (But who knows: that may be where I'm wrong.)

In secret.rkt I've essentially got this:

(define program (parse path))
(parameterize ([current-namespace (make-base-empty-namespace)])
(namespace-require '(file "expander.rkt"))
(eval program))

This works at the command line (if I'm in the right directory) using

racket client.rkt whatever.foo

but fails when the program is compiled using raco exe:

open-input-file: cannot open module file
module path: /Users/jesse/secret/examples/expander.rkt
path: /Users/jesse/sources/secret/examples/expander.rkt
system error: no such file or directory; rktio_err=3

It's clear to my that this fails, and why. The expander is missing. If I
try to be more clever and replace the '(file "expander.rkt") with '(lib
...) -- so that it looks like this:

(define program (parse path))
(parameterize ([current-namespace (make-base-empty-namespace)])
(namespace-require '(lib "my-package/expander"))
(eval expanded))

then I get aother error:

standard-module-name-resolver: collection not found
for module path: (lib "secret/expander")
collection: "secret"
in collection directories:
/Users/jesse/secret/lib/plt/secret/collects

I've tried things like using ++lib with raco exe and ++collects-copy
with raco distribute, but I can't figure it out; I always get something
essentially equivalent to: "Can't find your expander.". I feel up
against the ropes: adding a main module to expander.rkt so that
command-line processing happens there (!). That feels pretty weird, but
maybe that would work. (Did I mention that I'm turning to increasingly
improbable ideas in my quest for a solution?) I'm sure I'm missing
something really simple here. Any ideas?

Jesse

[Jon Zeppieri]

On Sun, Aug 4, 2019 at 2:40 PM Jesse Alama <je...@lisp.sh> wrote:
> [...]

>
> In secret.rkt I've essentially got this:
>
> (define program (parse path))
> (parameterize ([current-namespace (make-base-empty-namespace)])
> (namespace-require '(file "expander.rkt"))
> (eval program))
>

I thought I knew the solution, but when I tried it, it didn't work.

I thought you needed to produce a runtime module path for the module
that you're `namespace-require`-ing:

(define-runtime-module-path-index expander-path expander-path "expander.rkt")

... and then use that in the `namespace-require`. But when I produce
an executable and run it, I always get a "require: unknown module"
error. (Also, it wasn't clear to me if this is supposed to work unless
you also use `raco distribute`, so I tried that. Same problem.)

So I'm also stumped.

[Dmitry Pavlov]

Hello,

Your task reminds me of the one I had some time ago, and the found
solution is still used to this day.
You can see the old thread here:
https://groups.google.com/forum/#!msg/racket-users/ATXEyp-4AJA/x0KHbeOhdFwJ

I am not sure it is compatible right away with what you are doing:
there is not "eval" there, but "eval-syntax"; also, dedicated
(require-input-port)
and (prefix-input-port) procedures, written by Matthew, are used to
exempt users of #lang line.

As for your problem with raco exe, see Matthew's suggestion to make a
(dummy) my-package/main
and then do ++lib my-package/main, maybe it will help in your case, too.

Regards,

Dmitry

[Jesse Alama]

On 5 Aug 2019, at 13:16, Dmitry Pavlov wrote:

> Your task reminds me of the one I had some time ago, and the found
> solution is still used to this day.
> You can see the old thread here:
> https://groups.google.com/forum/#!msg/racket-users/ATXEyp-4AJA/x0KHbeOhdFwJ
>
> I am not sure it is compatible right away with what you are doing:
> there is not "eval" there, but "eval-syntax"; also, dedicated
> (require-input-port)
> and (prefix-input-port) procedures, written by Matthew, are used to
> exempt users of #lang line.
>
> As for your problem with raco exe, see Matthew's suggestion to make a
> (dummy) my-package/main
> and then do ++lib my-package/main, maybe it will help in your case,
> too.

Thanks for helping out here! The thread your link to is really helpful.
The idea of sneaking in a #lang line is probably the path I'll explore
for now.

Perhaps one could consider automating this by doing adding a #:lang
keyword argument to dynamic-require? The idea would be that you'd
pretend that the form is written in some other language.