Correctly executing a source file from embedded Racket
Thomas Dickerson
I have modified the `run` function from the embedding example here (https://docs.racket-lang.org/inside/embedding.html) as follows:
|static int run(Scheme_Env *e, int argc, char *argv[]) {
| Scheme_Object *curout;
| int i;
| Scheme_Thread *th;
|
| mz_jmp_buf * volatile save, fresh;
|
| /* Declare embedded modules in "base.hpp": */
| declare_modules(e);
| /*** Alternatively (see email commentary below):
| scheme_set_collects_path(scheme_make_path(argv[1]));
| scheme_init_collection_paths(e, scheme_null);
| */
| curout = scheme_get_param(scheme_current_config(),
| MZCONFIG_OUTPUT_PORT);
|
| th = scheme_get_current_thread();
| for (i = 2; i < argc; i++) {
| save = th->error_buf;
| th->error_buf = &fresh;
|
| if (scheme_setjmp(*th->error_buf)) {
| th->error_buf = save;
| return -1; /* There was an error */
| } else {
| Scheme_Object *v, *a[2];
| v = scheme_load(argv[i]);
| scheme_display(v, curout);
| scheme_display(scheme_make_char('\n'), curout);
| th->error_buf = save;
| }
| }
|
| return 0;
|}
When I run my program, so that the file to be executed is in `argv[2]`, I get an error message that begins as follows:
|../libraries/dsl-exec/racket-src/gpsm-test.gpsm:1:0: #%top-interaction: unbound identifier;
| also, no #%app syntax transformer is bound
| at: #%top-interaction
Note that when I run programs written in my DSL with a standalone interpreter, this error does not appear, and I *explicitly* export `#%top-interaction`, `#%app`, `#%module-begin`, and `#%datum`.
I noticed some oddities when using `raco ctool`. For example, the documentation claims that a module specified for embedding with ++lib will embed all of its dependencies, but my #lang's reader submodule needed to be explicitly specified with its own ++lib argument. To see if this was a contributing factor, I also tried using a manual `scheme_set_collects_path` and `scheme_init_collections_paths` combo (with `argv[1]` pointing to the location of my collects dir on disk), but got the same error as with the embedded versions of the modules.
As a follow-up, using the manually-specified collections path variant of the code above, I also tried to execute the following simple Racket program (to isolate if the use of my #lang specifically was a contributing factor):
|#lang racket
|
|(display (+ 5 6))
|(newline)
The same error as before was immediately raised:
|racket-test.rkt:1:0: #%top-interaction: unbound identifier;
| also, no #%app syntax transformer is bound
| at: #%top-interaction
I assume I'm missing some crucial piece of the embedded Racket puzzle, but I have no idea what it is.
(I might also recommend that whatever this missing piece is be *clearly* documented in future editions of the documentation, since it seems like a fairly basic use-case).
Thanks,
~Thomas
Shu-Hung You
I think the program is more of Racket programming setup rather than
embedding Racket. When start a Racket program on our own, we have to
setup the environment correctly.
Here is my test program that can load an external program and the
accompanying commands (also attached at the end). It has been tested
the program on Mac OS X 10.11 with Racket head.
https://gist.github.com/shhyou/e424424236d1bc685fa002192221c29b
The program can also work for user-defined languages by adding ++lib
some-lang ++lib some-lang/lang/reader as usual.
Here are some thoughts:
* That error message looks like trying to run a program from an empty
namespace. For example,
$ cat a-module.rkt
#lang racket/base
123
$ cat run-from-empty-ns.rkt
#lang racket
(define host-ns (current-namespace))
(define ns (make-empty-namespace))
(parameterize ([current-namespace ns])
(namespace-attach-module host-ns ''#%builtin)
(load "a-module.rkt"))
$ racket run-from-empty-ns.rkt
a-module.rkt:2:0: #%top-interaction: unbound identifier;
at: #%top-interaction
As we can see, `scheme_namespace_require(scheme_intern_symbol("racket/base"));`
is actually missing from your setup program. This line requires racket
base and the usual #%app bindings. It's probably needed even if you're
loading your own DSL -- I suspect at some point racket/base will
install the right `load` function to make things work, and that this
will also affect loading custom #langs.
* Dependencies for main.rkt and lang/reader.rkt: I don't think there's
a conflict. The dependencies seem to refer to `require`s, like
racket/base -> racket/list, racket/...
* load: instead of using load, I would probably try to implement a
function in Racket that use `dynamic-require` to load other DSL
scripts. Then in the C part, we can `dynamic-require` that function to
load argv[1]. Of course, remember to add that module in raco ctool. An
example is given in load-file.rkt and call_racket_from_load.cpp in the
link.
I am not familiar with this; there could be a better way to
dynamically load a file.
Cheers,
Shu-Hung
# Commands:
$ raco ctool --c-mods base.c ++lib racket/base ++lib
racket/base/lang/reader ++lib racket/runtime-config
$ g++ -framework Foundation -I<RACKET-HEAD-REPO>/racket/include
-L<RACKET-HEAD-REPO>/racket/src/build/racket
-L<RACKET-HEAD-REPO>/racket/src/build/rktio call_racket.cpp -o
call_racket -lracket -lmzgc -lrktio -liconv
ld: warning: could not create compact unwind for _ffi_call_unix64:
does not use RBP or RSP based frame
$ ./call_racket "<RACKET-HEAD-REPO>/racket/collects"
In cpp main
Trying to load file from "a-module.rkt"
#<void>
In cpp: Instantiating the loaded module, `a-module`:
In Racket: module instantiated: ("this is in a function with" arg
"being the first argument")
12354
In cpp: get the value of `a-value`:
a-value is 12354
;; a-module.rkt
#lang racket/base
(provide (all-defined-out))
(define (a-binding-to-a-function x)
(list "this is in a function with" x "being the first argument"))
(printf "In Racket: module instantiated: ~s\n" (a-binding-to-a-function 'arg))
(define a-value 12354)
;; By default, this should print 12354
a-value
/* call_racket.cpp */
/*
raco ctool --c-mods base.c ++lib racket/base ++lib
racket/base/lang/reader ++lib racket/runtime-config
g++ -framework Foundation -I<RACKET-HEAD-REPO>/racket/include
-L<RACKET-HEAD-REPO>/racket/src/build/racket
-L<RACKET-HEAD-REPO>/racket/src/build/rktio call_racket.cpp -o
call_racket -lracket -lmzgc -lrktio -liconv
./call_racket "<RACKET-HEAD-REPO>/racket/collects"
*/
#include <scheme.h>
#include <iostream>
#include "base.c"
static int run(Scheme_Env *e, int argc, char *argv[])
{
Scheme_Object *curout;
int i;
Scheme_Thread *th;
mz_jmp_buf * volatile save, fresh;
declare_modules(e);
scheme_namespace_require(scheme_intern_symbol("racket/base"));
curout = scheme_get_param(scheme_current_config(),
MZCONFIG_OUTPUT_PORT);
th = scheme_get_current_thread();
{
scheme_set_collects_path(path);
/*
// init collection paths with `((dynamic-require 'racket/base 'list) argv-1)
Scheme_Object *rkt_list_sym[2], *args[1];
rkt_list_sym[0] = scheme_intern_symbol("racket/base");
rkt_list_sym[1] = scheme_intern_symbol("list");
args[0] = path;
Scheme_Object *paths = scheme_apply(scheme_dynamic_require(2,
rkt_list_sym), 1, args);
scheme_init_collection_paths(e, paths);
*/
scheme_init_collection_paths(e, scheme_null);
save = th->error_buf;
th->error_buf = &fresh;
if (scheme_setjmp(*th->error_buf)) {
th->error_buf = save;
return -1; /* There was an error */
} else {
Scheme_Object *load_result = scheme_load("a-module.rkt");
scheme_display(load_result, curout);
scheme_display(scheme_make_char('\n'), curout);
// Instantiate the loaded module
std::cout << "\nIn cpp: Instantiating the loaded module, `a-module`:\n";
scheme_eval_string("(dynamic-require ''a-module #f)", e);
// Instead of scheme_eval_string, we can also built the S-expression of
// (list 'quote 'a-module) then use scheme_dynamic_require
std::cout << "\nIn cpp: get the value of `a-value`:\n";
Scheme_Object *a_value = scheme_eval_string("(dynamic-require
''a-module 'a-value)", e);
std::cout << "a-value is " << SCHEME_INT_VAL(a_value) << "\n";
}
return 0;
}
std::cout << "In cpp main\n";
return scheme_main_setup(1, run, argc, argv);
> You received this message because you are subscribed to the Google Groups "Racket Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to racket-users...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
Thomas Dickerson
> As we can see, `scheme_namespace_require(scheme_intern_symbol("racket/base"));`
> is actually missing from your setup program. This line requires racket
> base and the usual #%app bindings. It's probably needed even if you're
> loading your own DSL -- I suspect at some point racket/base will
> install the right `load` function to make things work, and that this
> will also affect loading custom #langs.
You appear to be correct, but it strikes me as really odd that
(a) In a language which prides itself on metaprogramming, the interpreter namespace needs to be manually bootstrapped with a specific module.
(b) That the form the error takes is not an error with the requires/provides in the `main.rkt` for my #lang, but as an error in a module which is using it.
This code:
|#lang racket
|(require "Value-Forms.rkt"
| "Operator-Forms.rkt"
| "Variable-Forms.rkt")
|
|(provide
| #%module-begin
| #%top-interaction
| #%datum
| #%app
| (all-from-out "Value-Forms.rkt"
| "Operator-Forms.rkt"
| "Variable-Forms.rkt"))
Should definitely complain if its imports aren't available.
> * Dependencies for main.rkt and lang/reader.rkt: I don't think there's
> a conflict. The dependencies seem to refer to `require`s, like
> racket/base -> racket/list, racket/...
Relatedly, even if `main.rkt` and `lang/reader.rkt` are independent, it's really odd that my module beginning with `#lang racket` is insufficient for raco to pick up the racket dependencies.
> * load: instead of using load, I would probably try to implement a
> function in Racket that use `dynamic-require` to load other DSL
> scripts. Then in the C part, we can `dynamic-require` that function to
> load argv[1]. Of course, remember to add that module in raco ctool. An
> example is given in load-file.rkt and call_racket_from_load.cpp in the
> link.
>
> I am not familiar with this; there could be a better way to
> dynamically load a file.
>
Thanks, I'll think some more about this.
> # Commands:
> $ raco ctool --c-mods base.c ++lib racket/base ++lib
> racket/base/lang/reader ++lib racket/runtime-config
This appears to be precisely the necessary set of minimum ++libs. It *really* ought to be documented somewhere, in big bold letters.
Thanks for the help!
~Thomas
Matthew Flatt
> On Tuesday, July 25, 2017 at 5:52:45 PM UTC-4, Shu-Hung You wrote:>
> > As we can see,
> `scheme_namespace_require(scheme_intern_symbol("racket/base"));`
> > is actually missing from your setup program. This line requires racket
> > base and the usual #%app bindings. It's probably needed even if you're
> > loading your own DSL -- I suspect at some point racket/base will
> > install the right `load` function to make things work, and that this
> > will also affect loading custom #langs.
>
> You appear to be correct, but it strikes me as really odd that
> (a) In a language which prides itself on metaprogramming, the interpreter
> namespace needs to be manually bootstrapped with a specific module.
> (b) That the form the error takes is not an error with the requires/provides
> in the `main.rkt` for my #lang, but as an error in a module which is using it.
`dynamic-require`, but those are different. The `load` function,
including its C wrapper `scheme_load`, loads a top-level sequence of
forms in the top-level environment. For that to work, the top-level
environment needs to be populated with bindings.
You don't have to populate the top-level environment with
`racket/base`. You could instead use
scheme_namespace_require(scheme_intern_symbol("my-lang"))
where "my-lang" is your language's main module to populate the
environment from there. Still, this isn't really the right direction.
Meanwhile, since the top-level environment was empty, the message
didn't come from the inside of a module. There was no
`#%top-interaction` or `module` binding in the top-level environment,
so nothing in "gpsm-test.gpsm" was parsed as a module. Instead, the
error was attributed to a top-level form in "gpsm-test.gpsm" that could
not be given any meaning in the current top-level environment.
Overall, I think you want to `scheme_dynamic_require` instead of
`scheme_load`, just as Shu-Hung suggests. Using
`scheme_dynamic_require` corresponds to what `racket` does where you
provide it a file on the command line, and it avoids the top-level
environment and `load`.
I can see how the "Inside Racket" example lead you astray by showing a
traditional REPL, where `eval` and `load` come into play, and I'll look
at ways to improve the docs. But if your goal is to run modules, then
you can avoid all that trouble.
Going back to the original message on `++lib`, to add a little to
Shu-Hung's reply:
> I noticed some oddities when using `raco ctool`. For example, the
> documentation claims that a module specified for embedding with ++lib will
> embed all of its dependencies, but my #lang's reader submodule needed to be
> explicitly specified with its own ++lib argument.
reader. If you have
#lang racket
'hello
(module reader racket)
then the enclosing module doesn't import the `reader` submodule, so it
doesn't depend on it. (If you compile and load that module, then the
`reader` submodule isn't event loaded.)
So, if I understand the situation you're describing, that's why the
reader submodule didn't get pulled along when embedding the outer
module.
Thomas Dickerson
On Wednesday, July 26, 2017 at 9:15:12 AM UTC-4, Matthew Flatt wrote:
> You don't have to populate the top-level environment with
> `racket/base`. You could instead use
>
> scheme_namespace_require(scheme_intern_symbol("my-lang"))
>
> where "my-lang" is your language's main module to populate the
> environment from there. Still, this isn't really the right direction.
I actually tried this without much success, but I might still have been missing something.
> Overall, I think you want to `scheme_dynamic_require` instead of
> `scheme_load`, just as Shu-Hung suggests. Using
> `scheme_dynamic_require` corresponds to what `racket` does where you
> provide it a file on the command line, and it avoids the top-level
> environment and `load`.
Just tried this, and it worked great, thanks! I was having a little trouble getting the `scheme_eval` variant that constructed a `dynamic-require` form manually to work correctly, so knowing about `scheme_dynamic_require` is helpful.
> I can see how the "Inside Racket" example lead you astray by showing a
> traditional REPL, where `eval` and `load` come into play, and I'll look
> at ways to improve the docs. But if your goal is to run modules, then
> you can avoid all that trouble.
Thanks!
One more thing: in terms of repeatedly executing scripts, does it make sense to set up and tear down the interpreter every time? Or just swap in a fresh namespace?
Matthew Flatt
> One more thing: in terms of repeatedly executing scripts, does it make sense
> to set up and tear down the interpreter every time? Or just swap in a fresh
> namespace?
better.
Depending on how much you need to reset, you might be even better off
with `dynamic-rerequire` from `racket/rerequire`.
Shu-Hung You
I tried following Matthew's suggestions to attach my-lang and use
dynamic-require. Turns out that works pretty well! And then there's
even no need to setup collection path since everything loads from
compiled byte-code.
https://gist.github.com/shhyou/aa2adaf7e1b7d548783cee352c3230a9
The code is basically calling (dynamic-require `(file ,argv-1) #f) to
run the external script. At line 35, the code builds (list 'file
argv-1) and at line 41 the code dynamic-requires the script.
/* custom_dsl.cpp
$ raco ctool --cgc --c-mods base.c ++lib racket/base ++lib
racket/base/lang/reader ++lib racket/runtime-config ++lib my-lang
++lib my-lang/lang/reader
$ g++ -framework Foundation -I<RACKET-HEAD-REPO>/racket/include
-L<RACKET-HEAD-REPO>/racket/src/build/racket
custom_dsl -lracket -lmzgc -lrktio -liconv
$ ./custom_dsl ./a-module.rkt
*/
#include <scheme.h>
#include <iostream>
#include "base.c"
static int run(Scheme_Env *e, int argc, char *argv[])
{
Scheme_Object *curout;
int i;
Scheme_Thread *th;
mz_jmp_buf * volatile save, fresh;
/* Declare embedded modules in "base.c": */
declare_modules(e);
curout = scheme_get_param(scheme_current_config(),
MZCONFIG_OUTPUT_PORT);
th = scheme_get_current_thread();
th->error_buf = &fresh;
if (scheme_setjmp(*th->error_buf)) {
th->error_buf = save;
return -1; /* There was an error */
} else {
Scheme_Object *custom_dsl_path[2];
custom_dsl_path[0] = scheme_make_pair(
scheme_intern_symbol("file"),
scheme_make_pair(
scheme_make_utf8_string(argv[1]),
scheme_make_null()));
custom_dsl_path[1] = scheme_make_false();
Scheme_Object *custom_dsl = scheme_dynamic_require(2, custom_dsl_path);
}
return 0;
}
int main(int argc, char **argv) {
std::cout << "In cpp main\n";
return scheme_main_setup(1, run, argc, argv);
}
#lang my-lang
some 'DSL that turns everything into "quotes"
Cheers,
Shu-Hung
Thomas Dickerson
> At Wed, 26 Jul 2017 07:54:32 -0700 (PDT), Thomas Dickerson wrote:
> > One more thing: in terms of repeatedly executing scripts, does it make sense
> > to set up and tear down the interpreter every time? Or just swap in a fresh
> > namespace?
>
> Between those two options, a fresh namespace is almost certainly
> better.
> Depending on how much you need to reset, you might be even better off
> with `dynamic-rerequire` from `racket/rerequire`.
`dynamic-require` documentation says "Dynamically instantiates the module specified by mod in the current namespace’s registry at the namespace’s base phase, if it is not yet instantiated."
Is a module instantiation local to the current namespace? If so, it sounds like wiping the namespace is sufficient.
The core of my script execution code now looks like this:
| ret = boost::none;
| Scheme_Object *drArgs[2] = {scheme_make_path(filename), resultSym_};
| Scheme_Object *loadResult = scheme_dynamic_require(2, drArgs);
| if(SCHEME_CPTRP(loadResult)){
| Scheme_Object *resT = SCHEME_CPTR_TYPE(loadResult);
| if(acceptedTypeTag_ == resT){
| ret = (AcceptedType*)SCHEME_CPTR_VAL(loadResult);
| } else {
| std::cerr << "Wrong type-tag received" << std::endl;
| Scheme_Object *curout = scheme_get_param(scheme_current_config(), MZCONFIG_ERROR_PORT);
| std::cerr << "Got: ";
| scheme_display(resT, curout);
| scheme_display(scheme_make_char('\n'), curout);
| std::cerr << "Wanted: ";
| scheme_display(acceptedTypeTag_, curout);
| scheme_display(scheme_make_char('\n'), curout);
| }
| } else {
| std::cerr << "Result was not a cptr" << std::endl;
| }
I'm now working to understand the exception handling mechanism (reading through https://docs.racket-lang.org/inside/exceptions.html and the example code).
The documentation appears to be mostly centered on what to do if your C code is being called from Racket (rather than vice-versa).
Looking at this code specifically:
| mz_jmp_buf * volatile save, fresh;
| scheme_current_thread->error_buf = &fresh;
|
| if (scheme_setjmp(scheme_error_buf)) {
| /* There was an error or continuation invocation */
| if (scheme_jumping_to_continuation) {
| /* It was a continuation jump */
| scheme_longjmp(*save, 1);
|
| /* To block the jump, instead: scheme_clear_escape(); */
| } else {
| /* It was a primitive error escape */
| }
| } else {
| scheme_eval_string("x", scheme_env);
| }
| scheme_current_thread->error_buf = save;
I have questions about a few things:
(a) Why does the example check for an error *before* calling `scheme_eval_string`, rather than afterward? Is this just something weird with setjmp/longjmp?
(b) How do I handle the "primitive error escape" branch, assuming I want to print a useful error message, and continue executing gracefully? It looks like `scheme_clear_escape` may be useful for preventing the exception from propagating, but I'm not sure about that from the documentation (or how I'd be able to capture the error message from the exception).
(c) Is this style of error handling only necessary around blocks that are evaluating actual Racket code (via, e.g., `scheme_eval_string` or `scheme_dynamic_require`)? What about the `scheme_register_embedded_load` and `scheme_embedded_load` pair at the bottom of `declare_modules`? I ran my binary through the Unix `strings` command and the embedded data looks an awful lot like expanded S-expressions in plain text. Additionally, at what level of granularity should I structure the error handling? Does it need to be for *every* evaluation, individually? Or could I surround a whole block of evaluations with the same error handling code?
Thanks again,
~Thomas
Matthew Flatt
> Looking at this code specifically:
>
> | mz_jmp_buf * volatile save, fresh;
> | save = scheme_current_thread->error_buf;
> | scheme_current_thread->error_buf = &fresh;
> |
> | if (scheme_setjmp(scheme_error_buf)) {
> | /* There was an error or continuation invocation */
> | if (scheme_jumping_to_continuation) {
> | /* It was a continuation jump */
> | scheme_longjmp(*save, 1);
> |
> | /* To block the jump, instead: scheme_clear_escape(); */
> | } else {
> | /* It was a primitive error escape */
> | }
> | } else {
> | scheme_eval_string("x", scheme_env);
> | }
> | scheme_current_thread->error_buf = save;
>
> I have questions about a few things:
> (a) Why does the example check for an error *before* calling
> `scheme_eval_string`, rather than afterward? Is this just something weird with
> setjmp/longjmp?
setjmp() always returns 0, etc.
> (b) How do I handle the "primitive error escape" branch, assuming I
> want to print a useful error message, and continue executing
> gracefully? It looks like `scheme_clear_escape` may be useful for
> preventing the exception from propagating, but I'm not sure about
> that from the documentation (or how I'd be able to capture the error
> message from the exception).
an exception, then the exception was already printed and (probably) the
default error escape handler called `abort-current-continuation`.
Catching the value raised as an exception is not practical to do in C.
It's better to wrap up the work like `scheme_eval_string("x",
scheme_env)` in a thunk and call a Racket-implemented wrapper. Section
9.1 in "Inside Racket" shows one way to do that.
> (c) Is this style of error handling only necessary around blocks that
> are evaluating actual Racket code (via, e.g., `scheme_eval_string` or
> `scheme_dynamic_require`)? What about the
> `scheme_register_embedded_load` and `scheme_embedded_load` pair at
> the bottom of `declare_modules`? I ran my binary through the Unix
> `strings` command and the embedded data looks an awful lot like
> expanded S-expressions in plain text. Additionally, at what level of
> granularity should I structure the error handling? Does it need to be
> for *every* evaluation, individually? Or could I surround a whole
> block of evaluations with the same error handling code?
not raise an exception. Probably it's possible to construct a set of
embedded modules where there will be a declare-time error due to
conflicting or missing declarations, but I don't see how to make that
happen only sometimes.
The escape-catching pattern is needed anywhere that you don't want to
just exit/crash.
You can certainly call multiple `scheme_...` functions within a single
escape-handling block, including multiple calls to `scheme_eval_string`.
Thomas Dickerson
> Declaring (as opposed to instantiating) a compiled module will normally
> not raise an exception. Probably it's possible to construct a set of
> embedded modules where there will be a declare-time error due to
> conflicting or missing declarations, but I don't see how to make that
> happen only sometimes.
Good to know.
> The escape-catching pattern is needed anywhere that you don't want to
> just exit/crash.
>
> You can certainly call multiple `scheme_...` functions within a single
> escape-handling block, including multiple calls to `scheme_eval_string`.
Also good to know, thanks.
On Wednesday, July 26, 2017 at 11:09:48 AM UTC-4, Matthew Flatt wrote:
> At Wed, 26 Jul 2017 07:54:32 -0700 (PDT), Thomas Dickerson wrote:
> > One more thing: in terms of repeatedly executing scripts, does it make sense
> > to set up and tear down the interpreter every time? Or just swap in a fresh
> > namespace?
>
> Between those two options, a fresh namespace is almost certainly
> better.
For posterity, it's worth noting that you can use the Boost Coroutine library to implement this in C++ in a nice object-oriented fashion to make a loop that executes Racket tasks in a way that doesn't require bouncing your `main` through `scheme_main_setup` or equivalent, which is great if you're embedding a racket interpreter to be reused as an API rather than a one-off program.