[racket] examples of hooking a #lang into DrRacket?

[Matthew Butterick]

Pollen source files run fine in DrRacket. But the `#lang pollen` line shows up as red (though no error is reported in the bottom of the window) and no toolbar buttons are available. So something is not quite right.

I gather that this has something to do with telling DrRacket about language-specific capabilities. [1]

However, many of the custom #langs in Racket use the standard #lang s-exp syntax/module-reader, which seems to eliminate these problems from the start.

Q: To supplement the documentation, are there particular #langs that show examples of this kind of configuration file?

[1] http://docs.racket-lang.org/tools/Language-Specific_Capabilities.html?q=Customizing%20DrRacket%u2019s%20Behavior

[Matthew Flatt]

The most relevant section of the documentation is

http://docs.racket-lang.org/guide/language-get-info.html


The Scratchy language is intended to be an example:

https://github.com/mflatt/scratchy


You could also check "reader.rkt" as part of

http://queue.acm.org/downloads/2011/racket/6-color/README.txt
http://queue.acm.org/downloads/2011/racket/6-color/txtadv.rkt
http://queue.acm.org/downloads/2011/racket/6-color/world.rkt
http://queue.acm.org/downloads/2011/racket/6-color/lang/color.rkt
http://queue.acm.org/downloads/2011/racket/6-color/lang/reader.rkt

which is from "Creating Languages in Racket" in "Queue". The
"reader.rkt" module has the relevant part at the end:

;; DrRacket asks `get-info' for a 'color-lexer module:
(require racket/runtime-path)
(define-runtime-path color-lexer-path "color.rkt")
(define (get-info in mod line col pos)
(lambda (key default)
(case key
[(color-lexer)
(dynamic-require color-lexer-path 'color-lexer)]
[else default])))

The Scratchy example is more recent and I think better organized,
though.

____________________
Racket Users list:
http://lists.racket-lang.org/users

[Matthew Butterick]

Still having some trouble with this:

When I use '#lang scratchy' in DrRacket, the #lang line is black and the DrRacket toolbar buttons show up, as they should.

After some trial & error, what I found is that if I simply change pollen/main.rkt to use the Scratchy parser and reader, like so:

;;--------------------------------pollen/main.rkt
#lang racket/base

(require scratchy/scratchy)
(provide (all-from-out scratchy/scratchy))

(module reader racket/base
(require scratchy/reader)
(provide (all-from-out scratchy/reader)))


... then '#lang pollen' invokes the Scratchy parser & reader correctly (= it will run sample Scratchy code).

Unfortunately the '#lang pollen' line still shows up in red, and there are still no DrRacket toolbar buttons (and still no error message in the bottom of the window)

Since a brain transplant hasn't fixed the problem, what else might DrRacket be looking for to validate '#lang pollen'?

[Robby Findler]

I did these steps:

"raco pkg install scratchy"
mkdir pollen
raco pkg install --link pollen
cat <the code from your message> > pollen/main.rkt

and after that, drracket turns #lang pollen black for me.

Robby

[Matthew Butterick]

That's a helpful clue, but DrRacket still seems to be making some kind of mysterious & negative judgment about the package when it starts up:


1) I created a branch in my github repo called pollen/scratchy that only consists of the file shown below, and switched to that branch.

2) Consistent with your result, when I launch DrRacket, '#lang pollen' is now black and buttons show up. Scratchy code works. That's promising.

3) With DrRacket still running, I switch my github branch back to pollen/master.

4) '#lang pollen' stays black, and buttons still appear, though now it's behaving as pollen. This is good news, but ...

5) ... when I quit DrRacket and restart (staying on pollen/master), '#lang pollen' is red once again, and no buttons.

6) Moreover, switching to pollen/scratchy while DrRacket is running does not fix the problem.

7) But if I quit DrRacket again, staying with pollen/scratchy, and relaunch DrRacket, I return to the state described in step (2).


Sorry for the convoluted description. I'm curious if there's some extra package-validation step that DrRacket performs at startup that could account for this behavior, as that seems to be the pattern that triggers the problem.

[Robby Findler]

DrRacket doesn't do anything with packages during startup that's
related to this. It doesn't do anything with the language at startup
either, but it does cache information about the language to avoid
querying the language while you are editing the file. Editing the
"#lang" line invalidates this cache but otherwise DrRacket won't
notice changes to your language's implementation (well, it won't
notice changes that affect thing like the buttons and whatnot that
you're looking for -- running your program is a separate process).

What happens when you do

(read-language (open-input-port "#lang pollen"))

?

Robby

[Robby Findler]

Oh, sorry I meant:

(read-language (open-input-string "#lang pollen"))

Robby

On Sun, Jul 20, 2014 at 6:23 PM, Robby Findler

[Robby Findler]

It appears that read-language can return #f and what DrRacket does in
that case appears to be what you're seeing. Is that what you're
getting?

The docs say that read-language never returns #f (presumably it is
allowed to return #f when fail-thunk returns #f, but the relevant
fail-thunk here doesn't return #f) so I'm not sure if that's a docs
bug, in which case I should change drracket to cope with it or an
implementation bug.

But if I create a pollen directory, use 'rack pkg --link pollen' to
set it up and then put main.rkt with this content:

#lang racket/base
(module reader racket/base (provide read-syntax)
(define (read-syntax a b)
#'(module m racket/base)))

then I see this:

$ racket
Welcome to Racket v6.1.0.3.

> (read-language (open-input-string "#lang pollen"))

#f


Robby

[Matthew Butterick]

> It appears that read-language can return #f and what DrRacket does in
> that case appears to be what you're seeing. Is that what you're
> getting?

Your theory seems correct. Yes, Pollen is the only #lang returning #f for read-language, e.g.:

#lang racket/base
(map
(λ(lang) (read-language (open-input-string (format "#lang ~a" lang))))
'(racket racket/base scratchy scribble/manual scribble/text rackjure pollen))

'(#<procedure:language-info>
#<procedure:language-info>
#<procedure:...atchy/reader.rkt:26:2>
#<procedure:language-info>
#<procedure:language-info>
#<procedure:language-info>
#f)

Moreover, if I use your main.rkt, it triggers the same problem in DrRacket (#lang line is red, no toolbar buttons)

But beyond the issue of whether the docs or DrRacket is doing the right thing with the #f value, I'm unclear why Pollen is returning #f for read-language. One possibly unorthodox thing I'm doing here is that my #lang files rely on macro expansion. For instance, here's Pollen's main.rkt:

#lang racket/base
(require pollen/main-base)
(define+provide-module-begin-in-mode world:mode-preproc)

(module reader racket/base
(require pollen/reader-base)
(define+provide-reader-in-mode world:mode-auto))

... where `define+provide-module-begin-in-mode` is a macro that creates the #%module-begin macro, and `define+provide-reader-in-mode` is a macro that creates the reader.

This doesn't bother DrRacket when the file is being run, but perhaps this approach is incompatible with either read-language, or DrRacket's #lang validation?

[Robby Findler]

Yeah: I think that it is safe to say at this point that pollen doesn't need to change. Either read-language's docs (and various places in DrRacket and libraries) need to change or read-language needs to change and either way pollen will work afterwards. 

My guess is that your language is the only one that doesn't define a get-info function itself but I can't check that theory right now. 

You can probably make pollen work now by supplying a get-info proc that always returns the default value. (See the third and subsequent paragraphs here: http://docs.racket-lang.org/reference/Reading.html?q=read-language#%28def._%28%28quote._~23~25kernel%29._read-language%29%29 ).

Robby

[Matthew Flatt]

Currently, `read-language` does return #f when `get-info` is not
available. I'm not sure anymore whether that was intended, but since it
has worked that way for a while, and since the intent of
`read-language` is to report the `get-info` function, then I think the
way forward here is to fix the documentation.

It sounds like DrRacket is handling a `#f` result ok: it should only
support syntax coloring if a language specifies a 'color-lexer
configuration via `get-info`, and so if `get-info` is not provided,
then no 'color-lexer configuration is available. Or is `get-info`
provided and somehow not recognized?

[Robby Findler]

On Mon, Jul 21, 2014 at 12:53 AM, Matthew Flatt <mfl...@cs.utah.edu> wrote:
> Currently, `read-language` does return #f when `get-info` is not
> available. I'm not sure anymore whether that was intended, but since it
> has worked that way for a while, and since the intent of
> `read-language` is to report the `get-info` function, then I think the
> way forward here is to fix the documentation.

Okay.

> It sounds like DrRacket is handling a `#f` result ok: it should only
> support syntax coloring if a language specifies a 'color-lexer
> configuration via `get-info`, and so if `get-info` is not provided,
> then no 'color-lexer configuration is available. Or is `get-info`
> provided and somehow not recognized?

Syntax coloring is one thing and that's what's turning things red,
yes. But I think that a language that defines a get-info that returns
the default value for color-lexer should get the same behavior as when
read-language returns #f. The former currently gets the racket-lexer.
Do you think that behavior should change?

(DrRacket also consults the read-language returned get-info procedure
for other things, including the lack of buttons that Matthew B.
reported. As it happens, the "returns #f" path is currently going thru
the "raises exn" path, which is probably not what we want.)

[Matthew Flatt]

I guess I'm conflating "colors syntax" and "turns the #lang line
black". In the absence of `get-info`, it seems fine to turn the #lang
line red and default to S-expression coloring for the module content.

Thanks for reminding me of the original issue. I agree that a #f result
from `read-language` should leave a default set of buttons, including
"Run".

[Matthew Butterick]

I can confirm that when I add a get-info function to the Pollen reader, it does work as expected in DrRacket. So perhaps Pollen doesn't "need" to change, but it will ;) Thank you for the help.

FWIW here's the accumulation of stumbles that led to my mistake:

+ AFAICT the #lang tutorial in the docs refers to get-info but not read-language. [1] Thus I never knew that read-language existed or what it did.

+ Based on the tutorial example [1], I had incorrectly concluded that get-info was an arbitrary private name, rather than a special name sought by read-language. 

+ I also didn't appreciate that get-info is mandatory for proper DrRacket compatibility, not optional.

+ (get-info, of course, is also the name of a function in setup/getinfo [2])

 

+ When there is no get-info function, DrRacket visually suggests that an error has occurred (red #lang line, no toolbar buttons) but doesn't print the actual error in the window (creating mystery)

+ You say that DrRacket doesn't cache any #lang information at startup. For whatever reason, I continue to find that relaunching makes a difference (i.e., if the #lang doesn't provide get-info at the beginning of a DrRacket session, adding one *during* the session will not fix the error; it will be noticed only on relaunch). I can't account for this behavior, but I can reliably reproduce it. This issue tripped me up, however, as it made it seem that get-info was not making a difference, so I eliminated it as the solution too soon.

[1] http://docs.racket-lang.org/guide/language-get-info.html?q=%23lang&q=read-language

[2] http://docs.racket-lang.org/raco/getinfo.html?q=get-info&q=%23lang&q=read-language#%28def._%28%28lib._setup%2Fgetinfo..rkt%29._get-info%29%29

[Robby Findler]

Okay, I've pushed a change that addresses most of the concerns raised
in this thread.

And just to clarify, there should have been no difference between an
always-default-returning get-info procedure and just not defining one.
The push should hopefully fix that.

There are two issues that remain, I believe:

1) the problem you were seeing where DrRacket didn't show any buttons
is probably an unrelated bug. There is, I believe, a race-condition in
this code; I don't understand what makes it happen and haven't had
luck fixing it (I've fixed a few variations on this bug and thought
I'd found them all, but apparently not!).) DrRacket should always have
the Run and Stop buttons.

2)

On Mon, Jul 21, 2014 at 1:23 PM, Matthew Butterick <m...@mbtype.com> wrote:
> + You say that DrRacket doesn't cache any #lang information at startup. For
> whatever reason, I continue to find that relaunching makes a difference
> (i.e., if the #lang doesn't provide get-info at the beginning of a DrRacket
> session, adding one *during* the session will not fix the error; it will be
> noticed only on relaunch). I can't account for this behavior, but I can
> reliably reproduce it. This issue tripped me up, however, as it made it seem
> that get-info was not making a difference, so I eliminated it as the
> solution too soon.

Oh, of course, the namespace caches the module in the usual way.
DrRacket should probably manage this more explicitly (currently it
just loads these modules in without much ado) and possibly even
implement an explicit policy for evicting this cache to avoid having
to restart DrRacket when developing a new language. I'll leave that
for another day, tho.