Language-Specific Plugins: Toolbar button functionality to call drracket:eval:expand-program

[Kshitij Sachan]

I'm working on a DSL to represent 3D geometrical shapes. I'm adding a language-specific plugin (toolbar button) that should: 1) expand the source code to get a module object, 2) evaluate the module object, 3) dynamic-require that module object, 4) Create an OpenGL Context, 5) render this object using a C++ library that contains a function that takes in a module object as input and generates in the existing OpenGL context.

Suppose my geometrical shape code is as follows:

#lang hypothetical-lanugage

(hypothetical-code)

I have a couple questions on how to set this up:

1. The documentation on drracket:eval:expand-program is rather dense, and I don't understand what the parameters should be. Could you please provide me with a minimum working example of how to call this function?
2. Is the output of drracket:eval:expand-program directly suitable to be called as the argument to the eval function? If not, what kind of preprocessing do I need to do before calling eval?
   
3. Will the result of eval be a module object?
4. If I press the toolbar button twice, will dynamic-require work or do I need dynamic-rerequire? Essentially what I'm asking is do I get a clean evaluation context each time I press my toolbar button?
5. Is it good style for my button to save the source code each time I run it? If so, how can I do that?
6. Ideally, I'd like the OpenGL rendering to appear as a third window to accompany the interactions and definitions window. Is there any way to do that?

Thanks!

[Philip McGrath]

The functionality you describe—in particular, setting up clean evaluation contexts—sounds a lot like what the "Run" button already does. Unless you have some other notion of "running" in mind for programs in your DSL, I think what you describe could be accomplished nicely with a "main" submodule, which (as you may know) is run only when its enclosing module is invoked as a main program, not when it is required as a library. One benefit is that this would make this functionality independent of DrRacket.

To illustrate what I mean, your source program:

#lang hypothetical-lanugage
(hypothetical-code)

would be transformed by the reader into:

(module example hypothetical-lanugage

  (#%module-begin

    (hypothetical-code)))

Then, your `#%module-begin` macro might expand to something like:

(module example hypothetical-lanugage

  (#%plain-module-begin

    (provide shape)

    (define shape (hypothetical-code))

    (module* main racket/base

      (require (submod "..")

               hypothetical-lanugage/private/render)

      (render shape))))

What I'm most unsure of about your question is that you say you want to use a "C++ library that contains a function that takes in a module object as input." Do you mean that you want to use the Racket C API to create a new primitive module or something? That is not a very common thing to do. Racket modules are not first-class values. While you can produce a value representing the compiled form of a module (`compiled-module-expression?`) by doing something like:

(compile '(module foo racket/base))

that is rarely what you want to do. In particular, the result of calling `eval` is not a module (unless of course you do something like `(eval '(compile '(module foo racket/base)) (make-base-namespace))`.

Calling `eval` on a module form, compiled or otherwise, merely declares the module: it doesn't immediately do anything. For example, this expression:

(let ([ns (make-base-namespace)])
  (eval '(module effects racket/base
           (println "running"))
        ns)
  (println "module evaluated")
  (eval '(require 'effects) ns))

prints:

"module evaluated"
"running"

because the module isn't run until the `require` form is evaluated. Also, `eval` handles expanding the program already: manual expansion is only needed if you want to do some kind of static analysis.

-Philip

--
You received this message because you are subscribed to the Google Groups "Racket Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to racket-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/179b48af-cfe1-4d6a-8c11-6f27584a4129%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jack Firth]

This sounds very similar to pict3d, particularly if you adopt the approach suggested by Phillip of using a main submodule to render things instead of a drracket plugin. Is pict3d close to what you're trying to do? Not saying you should use pict3d or do things a similar way—I'm just curious about your use case.

[Arie Schlesinger]

Hi,

Are there racket notebooks like jupyter or swish for prolog ?

Thank you

--

[Neil Van Dyke]

Arie Schlesinger wrote on 6/26/19 1:55 AM:

> Are there racket notebooks like jupyter or swish for prolog ?

There's work on an IPython/Jupyter kernel for Racket, but I'd also like
to encourage someone to follow through on also adding a (separate) good
notebook mode to DrRacket, as discussed in this thread from December:
https://groups.google.com/forum/#!topic/racket-users/MsAh2aBU5Sw

A notebook mode is potentially a mostly-fun thing to implement, and
opens up a new way of using Racket.  (Imagine, for example, using a
notebook mode to explore what can be done with some language features or
in some domain, and then rapidly cleaning it up and fleshing out the
Markdown/Scribble bits, to turn it into a blog post and shareable
notebook.  This isn't especially new, but the conveniences of a notebook
mode could make a big difference in what you do, whether and how you
share it, and whether people see and play with it.)

[Thomas Dickerson]

Hi,

Chiming in here, because Kshitij is working on this project for me.

It sounds like Philip has answered (3) and most of (2).

On Tuesday, June 25, 2019 at 10:06:07 PM UTC-4, Philip McGrath wrote:

The functionality you describe—in particular, setting up clean evaluation contexts—sounds a lot like what the "Run" button already does. Unless you have some other notion of "running" in mind for programs in your DSL, I think what you describe could be accomplished nicely with a "main" submodule, which (as you may know) is run only when its enclosing module is invoked as a main program, not when it is required as a library. One benefit is that this would make this functionality independent of DrRacket.

A couple additional bits of context:

• The DSL he is working on is embedded in a C++ application and strongly typed, so we have several different notions of "running", which are orthogonal to DrRacket's run button:
1. The language is a language for describing parametric 3D models, not for rendering, and the final result of being evaluated by Racket is a module containing a provided result value, which is a pointer to a C++ object which represents some a low level program evaluated by our C++ runtime to produce a triangle mesh.
   
2. Our C++ applications obtain the result value from a DSL program by calling scheme_dynamic_require in the inner loop of a Racket interpreter we've stuffed into a coroutine.
3. When the toolbar button is pushed in DrRacket we want to evaluate the program in a fresh namespace, (dynamic-require ''module-name 'result), and then pass that result value into a C++ FFI function that will handle the C++ side of evaluation and doing the actual OpenGL calls to visualize it in the editor. This should have similar behavior to the run button from an evaluation standpoint, but after evaluating the program, enter a render loop until, e.g. the button is pushed again to stop rendering. Here, the DrRacket plugin is responsible for creating the OpenGL context, which could be in a separate window, but as Kshitij said it would be ideal if we could figure out how to add a pane below the REPL.
   

What is the "correct" way to get the current program source code from the DrRacket editor to pass into eval?

drracket:eval:expand-program seemed closest, given the type of the first argument, but the plugin documentation is not clear and we don't actually know how to get the relevant input port or text-pos.

On Wednesday, June 26, 2019 at 12:06:36 AM UTC-4, Jack Firth wrote:

This sounds very similar to pict3d, particularly if you adopt the approach suggested by Phillip of using a main submodule to render things instead of a drracket plugin. Is pict3d close to what you're trying to do? Not saying you should use pict3d or do things a similar way—I'm just curious about your use case.

 

pict3d looks quite interesting, but isn't quite what we want - see the earlier use case discussion in my reply to Philip.

[Philip McGrath]

If you're sure you want to get the raw program source and deal with it manually, you can use the method `get-definitions-text` on the DrRacket frame given as an argument to your button's callback function. This will give you the editor object for the definitions window. You can then use a method like `get-flattened-text` to get the source as a sting or (probably better) the function `open-input-test-editor` to get an input port directly. Before evaluating it, you need to convert it to an s-expression with `read` or, better, a syntax object with `read-syntax`. In either case, you will want to use `with-module-reading-parameterization` to set the reading parameters properly.

However, if I understand your requirements correctly, I still think expanding to a "main" submodule would be the best way to go. To be concrete, I'll use the example I gave before of a partially-expanded version of your program, but with `shape` renamed to `result` to match your description:

(module example hypothetical-lanugage

  (#%plain-module-begin

    (provide result)

    (define result (hypothetical-code))

    (module* main racket/base

      (require (submod "..")

               hypothetical-lanugage/private/render)

      (render result))))

On Wed, Jun 26, 2019 at 12:05 PM Thomas Dickerson <thomas_d...@alumni.brown.edu> wrote:

1. The language is a language for describing parametric 3D models, not for rendering, and the final result of being evaluated by Racket is a module containing a provided result value, which is a pointer to a C++ object which represents some a low level program evaluated by our C++ runtime to produce a triangle mesh.

 

Hopefully it's clear how this aspect of the expansion works: `(hypothetical-code)` evaluates to your foreign C++ representation just as it would in your current way of thinking about these programs.

2. Our C++ applications obtain the result value from a DSL program by calling scheme_dynamic_require in the inner loop of a Racket interpreter we've stuffed into a coroutine.

 

Again, this part works just fine. In particular, because of the way Racket's submodules work, the code in the "main" submodule does not run when you instantiate the parent module, whether via `scheme_dynamic_require`, `dynamic-require`, `require`, or some other way: the compiled form of the submodule need not even be loaded into memory. So you can use rendering code in the main submodule without creating a dependency for your main application.

3. When the toolbar button is pushed in DrRacket we want to evaluate the program in a fresh namespace, (dynamic-require ''module-name 'result), and then pass that result value into a C++ FFI function that will handle the C++ side of evaluation and doing the actual OpenGL calls to visualize it in the editor. This should have similar behavior to the run button from an evaluation standpoint, but after evaluating the program, enter a render loop until, e.g. the button is pushed again to stop rendering. Here, the DrRacket plugin is responsible for creating the OpenGL context, which could be in a separate window, but as Kshitij said it would be ideal if we could figure out how to add a pane below the REPL.

 

This is where I think using a submodule will be helpful. When you press the "Run" button in DrRacket, the "main" submodule is run for effect before starting the REPL. (Command-line `racket` will do similarly depending on the flags.) In the example expansion, `(render result)` would pass the value described by your DSL program to your rendering function and whatever FFI calls are needed. It could certainly open a new GUI window, and it could also return a value that renders as an embedded widget (snip) in the interactions pane. (See the plot library for an example.) I haven't looked at the DrRacket extension APIs enough to be sure whether you could open a new pane in the DrRacket window, but it certainly seems possible.

Using a "main" submodule should let you take advantage of DrRacket's programming-languages-as-operating-systems capabilities for managing the programs it runs with minimal additional work. DrRacket would take care of destroying your rendering context when you click "Run" again after changing your program, it could enforce memory limits, and its "Stop"/"Kill" button could terminate misbehaved rendering code. You could re-create all of these capabilities yourself, but that's a fair amount of work, and I'm not seeing a compelling reason for this to be an extralinguistic tool.

-Philip

[Robby Findler]

May I ask why you need to have the C++ code embed Racket instead of
the other way around (ie using the FFI)? Generally speaking, DrRacket
(and other Racket tools) are going to work better if they get to
"drive", I expect. (Put another way, I think Philip is giving you good
advice here, fwiw.)

Robby

> --
> You received this message because you are subscribed to the Google Groups "Racket Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to racket-users...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/279abcad-9dc7-4224-b6de-ef6fd0209a83%40googlegroups.com.

[Thomas Dickerson]

Thanks for the responses.

On Fri, Jun 28, 2019 at 5:50 PM Philip McGrath <phi...@philipmcgrath.com> wrote:

If you're sure you want to get the raw program source and deal with it manually, you can use the method `get-definitions-text` on the DrRacket frame given as an argument to your button's callback function. This will give you the editor object for the definitions window. You can then use a method like `get-flattened-text` to get the source as a sting or (probably better) the function `open-input-test-editor` to get an input port directly.

This is very helpful, thanks.

 

Again, this part works just fine. In particular, because of the way Racket's submodules work, the code in the "main" submodule does not run when you instantiate the parent module, whether via `scheme_dynamic_require`, `dynamic-require`, `require`, or some other way: the compiled form of the submodule need not even be loaded into memory. So you can use rendering code in the main submodule without creating a dependency for your main application.

Specifically, this means we can avoid byte-compiling and loading any graphics modules the main submodule depends on, yes?

 

It could certainly open a new GUI window, and it could also return a value that renders as an embedded widget (snip) in the interactions pane. (See the plot library for an example.)

Snips seem nice, but even browsing through the source for the 3d renderers for plot, I don't see any way to get an OpenGL context for a snip, since they appear to work on predefined DCs, rather than allowing you to construct a canvas.

 

I haven't looked at the DrRacket extension APIs enough to be sure whether you could open a new pane in the DrRacket window, but it certainly seems possible.

Are you aware of any projects that usefully manipulate the interactions pane that I could turn to for example code? It looks like it's possible to extend that class.

On Sun, Jun 30, 2019 at 11:44 AM Robby Findler <ro...@cs.northwestern.edu> wrote:

May I ask why you need to have the C++ code embed Racket instead of
the other way around (ie using the FFI)?

The short answer is this project is a very small part of a very large enterprise C++ code base, and it's not reasonable to rewrite every single program in our toolkit that needs to interact with our DSL to be a dynamic library with a Racket front-end.

 

Generally speaking, DrRacket (and other Racket tools) are going to work better if they get to
"drive", I expect.

Thankfully we're not trying to embed DrRacket - our applications are primarily command-line driven, so the editor enhancements are just for artist convenience.

The rest of the time, Racket happily sits in its own stack, and has no idea we've embedded in it (although since the time that was written, the Boost Coroutines library was deprecated, so we built our own).

[Philip McGrath]

On Mon, Jul 1, 2019 at 11:40 AM Thomas Dickerson <thomas_d...@alumni.brown.edu> wrote:

On Fri, Jun 28, 2019 at 5:50 PM Philip McGrath <phi...@philipmcgrath.com> wrote:

Again, this part works just fine. In particular, because of the way Racket's submodules work, the code in the "main" submodule does not run when you instantiate the parent module, whether via `scheme_dynamic_require`, `dynamic-require`, `require`, or some other way: the compiled form of the submodule need not even be loaded into memory. So you can use rendering code in the main submodule without creating a dependency for your main application.

Specifically, this means we can avoid byte-compiling and loading any graphics modules the main submodule depends on, yes?

I believe the main submodule will still be compiled the enclosing module is loaded from source, but that should be trivial: it really only needs a `require` and a function call. If the file has already been compiled to bytecode, the bytecode for the main submodule won't be loaded, nor for any of its transitive dependencies.

Depending on how you are distributing your code, you may or may not want to add an indirection via `dynamic-require` to prevent the distribution building tools from shipping graphics libraries that you only need for the main submodule. (Basically this would be the reverse of what `racket/runtime-path` does.) I haven't done much with the tools for distributing built artifacts beyond toy experiments, but I think some of them may be able to do that stripping automatically or with e.g. a command-line flag, without needing a `dynamic-require`. At the maximum, you could put the graphics modules in their own package, analogous to the `foo`+`foo-lib`+`foo-doc`+`foo-test` convention: in that case, the support library for your main submodule (what I called `hypothetical-lanugage/private/render`) would just be a stub to `dynamic-require` the `render` function or do something useful if the graphics package isn't installed, like print a message to standard error.

 

It could certainly open a new GUI window, and it could also return a value that renders as an embedded widget (snip) in the interactions pane. (See the plot library for an example.)

Snips seem nice, but even browsing through the source for the 3d renderers for plot, I don't see any way to get an OpenGL context for a snip, since they appear to work on predefined DCs, rather than allowing you to construct a canvas.

I haven't used OpenGL, from Racket or otherwise, but it might be possible to use a bitmap from `make-gl-bitmap` as a buffer for your OpenGL drawing and copy it to the canvas via `draw-bitmap`. If you have an existing foreign library that does the actual drawing, you can get a platform-specific pointer via `gl-context<%>`. 

 

I haven't looked at the DrRacket extension APIs enough to be sure whether you could open a new pane in the DrRacket window, but it certainly seems possible.

Are you aware of any projects that usefully manipulate the interactions pane that I could turn to for example code? It looks like it's possible to extend that class.

I can't think of any examples that manipulate the DrRacket window from within the program being evaluated. In general, DrRacket goes to a lot of effort to isolate itself from the programs it runs, which is usually a good thing. I suspect, though, that a solution would involve your support library cooperating with an extension set up by your #lang. Communication might need to go through a non-obvious channel like a logger: Alexis has a blog post about a wonderfully devious use for loggers as a communication mechanism, but they're also used less deviously by e.g. the Future Visualizer.

On the whole, though, either returning a snip or creating a new `frame%` would probably be easier. Using a new `frame%` would also be entirely independent of DrRacket: I expect it would work with racket-mode for Emacs, for example.

-Philip

[Robby Findler]

On Mon, Jul 1, 2019 at 10:40 AM Thomas Dickerson
<thomas_d...@alumni.brown.edu> wrote:
> On Sun, Jun 30, 2019 at 11:44 AM Robby Findler <ro...@cs.northwestern.edu> wrote:
>>
>> May I ask why you need to have the C++ code embed Racket instead of
>> the other way around (ie using the FFI)?
>
>
> The short answer is this project is a very small part of a very large enterprise C++ code base, and it's not reasonable to rewrite every single program in our toolkit that needs to interact with our DSL to be a dynamic library with a Racket front-end.

I think I'm missing a bit more context so I'll continue this line of
questions. The path that you seem to be heading down seems to be one
where you'll duplicate a lot of work that's already gone into
DrRacket, and it seems unlikely to me that it is the most productive.

Can you say a little more about what you want to do with DrRacket? I
guess there will be some program in your DSL that you'll want to work
with. Do you expect to be able to run these programs? If so, does it
make sense to run them when you aren't in the
embedded-in-the-huge-C++-code-base mode? How about compilation of the
programs (i.e. macro expansion)?

Robby

[Thomas Dickerson]

On Mon, Jul 1, 2019 at 2:23 PM Philip McGrath <phi...@philipmcgrath.com> wrote:

I believe the main submodule will still be compiled the enclosing module is loaded from source, but that should be trivial: it really only needs a `require` and a function call. If the file has already been compiled to bytecode, the bytecode for the main submodule won't be loaded, nor for any of its transitive dependencies.

Depending on how you are distributing your code, you may or may not want to add an indirection via `dynamic-require` to prevent the distribution building tools from shipping graphics libraries that you only need for the main submodule. (Basically this would be the reverse of what `racket/runtime-path` does.) I haven't done much with the tools for distributing built artifacts beyond toy experiments, but I think some of them may be able to do that stripping automatically or with e.g. a command-line flag, without needing a `dynamic-require`. At the maximum, you could put the graphics modules in their own package, analogous to the `foo`+`foo-lib`+`foo-doc`+`foo-test` convention: in that case, the support library for your main submodule (what I called `hypothetical-lanugage/private/render`) would just be a stub to `dynamic-require` the `render` function or do something useful if the graphics package isn't installed, like print a message to standard error.

I'll chew on this, but if the dependencies for the main submodule need to be available at the time that it is compiled, then the easiest thing may just be doing a dynamic-require.

 

 I haven't used OpenGL, from Racket or otherwise, but it might be possible to use a bitmap from `make-gl-bitmap` as a buffer for your OpenGL drawing and copy it to the canvas via `draw-bitmap`. If you have an existing foreign library that does the actual drawing, you can get a platform-specific pointer via `gl-context<%>`. 

That's an interesting possibility, and I shouldn't actually need the gl-context pointer, just to make it current before calling the foreign rendering code.

 

 I suspect, though, that a solution would involve your support library cooperating with an extension set up by your #lang.

Right, this is what we originally hand in mind.

 

On the whole, though, either returning a snip or creating a new `frame%` would probably be easier. Using a new `frame%` would also be entirely independent of DrRacket: I expect it would work with racket-mode for Emacs, for example.

frame% is portable, but also not a very nice UI. I'll see what can be done with a snip.

On Mon, Jul 1, 2019 at 2:56 PM Robby Findler <ro...@cs.northwestern.edu> wrote:

I think I'm missing a bit more context so I'll continue this line of
questions. The path that you seem to be heading down seems to be one
where you'll duplicate a lot of work that's already gone into

DrRacket, and it seems unlikely to me that it is the most productive

 

Perhaps you could elaborate on the mechanisms you think we would be duplicating, because those may provide either the entry points we're looking for, or clarify a potential miscommunication on our end.

 

Can you say a little more about what you want to do with DrRacket?

 

Currently, most of our DSL programs are constructed automatically from inferred geometry, and it's fine, because an artist/programmer doesn't need to do any design work.

Going forward, we would like to also be able to model reusable parametric object classes that we can plug in to our scenes, and here it would be very helpful to be able to visualize the geometry generated by our DSL programs interactively.

Rather than reinventing the wheel, and writing our own editor, we would like to bend DrRacket to our will for that purpose; however, the current "Run" button doesn't seem like quite the right UI for that.

Do you expect to be able to run these programs? If so, does it
make sense to run them when you aren't in the
embedded-in-the-huge-C++-code-base mode? How about compilation of the
programs (i.e. macro expansion)?

We've essentially phase shifted everything, so that Racket's Phase 0 is really Phase 1 in a DSL library implemented with C++, which provides our "core language", so the result of "running" the Racket program is a pointer to an opaque C++ type which represents the fully expanded syntax for our program, ready to be evaluated by the C++ geometry library.

Racket's Phase 1 handles type checking + inference of the surface language, converting pretty surface syntax into a lot of horrific FFI code, and providing a consistent API to the resulting modules to make it easy to grab the C++ representation of the DSL program.

We currently can then either just run this as a Racket program, which works, but is basically useless, or further stuff the Racket interpreter into a C++ library for reading, writing, and executing our DSL programs from elsewhere in the C++ code base. We would like to have a third option which is "allow DrRacket to invoke the final C++ evaluation step of the DSL, and then allow for interactive rendering of the resulting geometry, in a friendly UI".

 

[Robby Findler]

Sorry for letting this thread lapse. Wrt to your third option
mentioned below, would it work to make that option accessible via the
FFI? If so, then maybe you could make the "essentially phase shifted
everything" into "actually phase shifted everything" and then when you
get the result program have your #%module-begin call into the FFI to
actually run the program. Does that seem like a viable option?

Robby

[Thomas Dickerson]

On Tuesday, July 23, 2019 at 2:34:47 AM UTC-4, Robby Findler wrote:

Sorry for letting this thread lapse. Wrt to your third option
mentioned below, would it work to make that option accessible via the
FFI? If so, then maybe you could make the "essentially phase shifted
everything" into "actually phase shifted everything" and then when you
get the result program have your #%module-begin call into the FFI to
actually run the program. Does that seem like a viable option?

Yes - the architecture we eventually settled on is a main submodule that does roughly this to get a black-box value which we can pass back to a rendering function to produce a gl-bitmap, and stick it in a snip.

By the way - it seems you weren't the only person who doesn't use OpenGL very often from Racket, because gl-bitmap had bit-rotted somewhat, compared to a 'gl-styled canvas, but my PR got merged, and gl-bitmaps now support core contexts.

[Thomas Dickerson]

On Tuesday, July 23, 2019 at 12:35:27 PM UTC-4, Thomas Dickerson wrote:

Yes - the architecture we eventually settled on is a main submodule that does roughly this to get a black-box value which we can pass back to a rendering function to produce a gl-bitmap, and stick it in a snip.

Sorry for the double email, but for posterity I want to be slightly more specific here: our main submodule uses dynamic-require for all of the graphics/rendering functionality, to avoid saddling our embedded Racket used by C++ applications with extra dependencies.

[Robby Findler]

On Tue, Jul 23, 2019 at 11:35 AM Thomas Dickerson
<thomas_d...@alumni.brown.edu> wrote:
> Yes - the architecture we eventually settled on is a main submodule that does roughly this to get a black-box value which we can pass back to a rendering function to produce a gl-bitmap, and stick it in a snip.
>
> By the way - it seems you weren't the only person who doesn't use OpenGL very often from Racket, because gl-bitmap had bit-rotted somewhat, compared to a 'gl-styled canvas, but my PR got merged, and gl-bitmaps now support core contexts.

Thank you!

Robby

[Neil Van Dyke]

HN is currently commenting on one attempt to increase sharing of
notebook-oriented programming:

"Show HN: A tool to convert Jupyter notebooks to beautiful blogs"
https://news.ycombinator.com/item?id=20515880


Neil Van Dyke wrote on 6/26/19 2:31 AM: