Migrating from a "model-driven" language framework to Racket

[Guillaume Savaton]

I am a Racket beginner trying to create my own DSL.

As a long-time user of Xtext and other similar tools in the Eclipse ecosystem, I have come to Racket expecting that it would address similar concerns.

At the moment, I have mixed feelings: I find the metaprogramming facilities in Racket very effective, but at the same time I am struggling to achieve tasks that were supported natively by Xtext.

For those who don't know Xtext, here is a summary of how it works:

• A language project is based on a grammar with attribute annotations.
• The grammar is converted into a "metamodel", i.e. a set of classes where each grammar rule corresponds to a class.
• A parser is automatically generated. It can convert some source text into a "model", i.e. a set of instances of the classes from the metamodel.
• A model can be manipulated using Java APIs. Specialized languages are available to constrain a model, query it, transform it, or generate code using templates.

In Racket, I have started my language project by reproducing what I would have done in Xtext:

• I have created a grammar with bragg
• I have written a set of syntax classes that play the role of the metamodel
• Syntax objects play the role of the model, and I can get their attributes with syntax-parse
• I have written several macros that can generate Racket code in the simplest cases.

However, I miss some facilities that Xtext provides out-of-the-box:

• Racket syntax classes do not directly support inheritance.
  
• Syntax objects are not tied to syntax classes in a class-instance relationship, and I have to use syntax-parse every time I want to read an attribute.
• Xtext automatically creates child->parent references in the generated AST. In Racket, it seems that I cannot get the parent of a syntax object.
  
• Xtext provides a default mechanism for resolving named references, and a scoping API for languages that need specific scoping rules. The AST generated by Xtext is actually an object graph rather than a tree.
  

My main concern is about managing the scopes/lexical contexts in my language. I am still browsing the documentation but I have found no library or guide that addresses this issue.

The language examples that I have found are either too simple (their scoping rules can be easily mapped to those of Racket through macros), or use ad-hoc techniques, so that it is difficult to infer a general methodology.

So far, I have made two attempts to work around these issues: (1) by creating a metamodel-like data structure using Racket structs, and transforming syntax objects into struct instances; or (2) using syntax objects only and attaching context data to each of them as a syntax property.

Both have strengths and weaknesses, and I am still feeling that I am not using Racket with the right mindset.

I hope I have made my concerns clear. Maybe I can create a small example to further illustrate what I want to do and where I am stuck.

Have you experienced similar concerns in one of your projects?

What design patterns would you recommend ?

Do you know any well-commented real-life example that I could use for inspiration?

Thanks in advance for your answers.

Guillaume Savaton

N.B: I have also published a similar question at stackoverflow two weeks ago, but it still has no answer:

https://stackoverflow.com/questions/61622912/domain-specific-languages-in-racket-compared-to-model-driven-frameworks-such-as

[Stephen De Gabrielle]

Hi Guillaume

I don't think I can answer your questions, as I'm not a language developer, but this topic interests me;

What are the Racket Syntax Classes you have implemented? Can you provide an example?

I'm aware of syntax objects, as a specialised data structure for syntax manipulation, but despite the 'object' in the 'syntax-object' I don't

believe they are integrated into the main Racket class/object system.

https://docs.racket-lang.org/reference/syntax-model.html#%28tech._syntax._object%29

(This link also has a section on scopes, but I don't know if it is what you are looking for?)

Kind regards, 

Stephen

On Thursday, May 21, 2020 at 5:36:08 PM UTC+1, Guillaume Savaton wrote:

I am a Racket beginner trying to create my own DSL.

As a long-time user of Xtext and other similar tools in the Eclipse ecosystem, I have come to Racket expecting that it would address similar concerns.

At the moment, I have mixed feelings: I find the metaprogramming facilities in Racket very effective, but at the same time I am struggling to achieve tasks that were supported natively by Xtext.

For those who don't know Xtext, here is a summary of how it works:

• A language project is based on a grammar with attribute annotations.
• The grammar is converted into a "metamodel", i.e. a set of classes where each grammar rule corresponds to a class.
• A parser is automatically generated. It can convert some source text into a "model", i.e. a set of instances of the classes from the metamodel.
• A model can be manipulated using Java APIs. Specialized languages are available to constrain a model, query it, transform it, or generate code using templates.

In Racket, I have started my language project by reproducing what I would have done in Xtext:

• I have created a grammar with bragg
• I have written a set of syntax classes that play the role of the metamodel

Syntax Classes example? 

[Guillaume Savaton]

Hi Stephen.

Thanks for your answer.

Le vendredi 22 mai 2020 13:57:53 UTC+2, Stephen De Gabrielle a écrit :

What are the Racket Syntax Classes you have implemented? Can you provide an example?

The project consists in creating a simple hardware description language.

I am preparing a simplified version that could help clarify my questions.

I will add it to this conversation as soon as it is ready.

I'm aware of syntax objects, as a specialised data structure for syntax manipulation, but despite the 'object' in the 'syntax-object' I don't

believe they are integrated into the main Racket class/object system.

As far as I know, they are not.

A syntax object is basically an S-expression with additional information such as source location and syntax properties.

https://docs.racket-lang.org/reference/syntax-model.html#%28tech._syntax._object%29

(This link also has a section on scopes, but I don't know if it is what you are looking for?)

I think this page describes how scopes are implemented in the Racket language itself.

When implementing a DSL, this information is useful if I want to convert the constructs of my language into Racket forms that preserve the scoping semantics.

But is it always possible and is it always a good idea?

Another approach is to define my own scoping rules, so that I can perform semantic checks and code transformations before generating Racket forms.

So my question can be rephrased like this: "When implementing a DSL, can I reuse Racket's scope data structures, and is there an API for defining my own scoping rules?"

Cheers.

Guillaume

[Jens Axel Søgaard]

Hi Guillaume,

Thanks for taking the time to write this question.

...

> My main concern is about managing the scopes/lexical contexts in my language.
> I am still browsing the documentation but I have found no library or guide that addresses this issue.

> The language examples that I have found are either too simple (their scoping rules can be easily mapped to those of Racket through macros),
> or use ad-hoc techniques, so that it is difficult to infer a general methodology.

There are several approaches. As you have found out the simplest is to map language constructs to similar Racket forms.
If the scoping rules are different, you can consider implementing Racket forms with new scoping constructs first,
and then map your dsl to your new forms. 

> Maybe I can create a small example to further illustrate what I want to do and where I am stuck.

I think that's a good idea. It is much easier to provide feedback based on a concrete example. 


/Jens Axel

--
You received this message because you are subscribed to the Google Groups "Racket Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to racket-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/3ee19998-88bf-48f7-8129-c1be9419d4c9%40googlegroups.com.

--

--
Jens Axel Søgaard

[John Clements]

> ...

> So far, I have made two attempts to work around these issues: (1) by creating a metamodel-like data structure using Racket structs, and transforming syntax objects into struct instances; or (2) using syntax objects only and attaching context data to each of them as a syntax property.
> Both have strengths and weaknesses, and I am still feeling that I am not using Racket with the right mindset.

I think your (2) sounds like a lighter-weight solution. However, it definitely does seem as though much of the difficulty here is related to the differences between a more imperative and a more functional style. I think your idea of a simplified example—especially one illustrating the situations in which context information is required—would be an excellent idea!

John

[Guillaume Savaton]

Le mardi 26 mai 2020 03:59:59 UTC+2, johnbclements a écrit :

> So far, I have made two attempts to work around these issues: (1) by creating a metamodel-like data structure using Racket structs, and transforming syntax objects into struct instances; or (2) using syntax objects only and attaching context data to each of them as a syntax property.
> Both have strengths and weaknesses, and I am still feeling that I am not using Racket with the right mindset.

I think your (2) sounds like a lighter-weight solution. However, it definitely does seem as though much of the difficulty here is related to the differences between a more imperative and a more functional style.

I'm not sure that solution (2) is lighter. Maybe the weight is moved to another part of the implementation :)

I have set up an example at this address: https://gist.github.com/senshu/c6db95615b4b2567f168d6bfbe61655e

It is basically a very stripped-down hardware description language that borrows from VHDL and Verilog.

Like VHDL, the description of a circuit is split into an "entity" (the interface of the circuit) and an "architecture" (the implementation).

For the sake of simplicity, this example does not implement a complete model of computation.

The file "tiny-hdl-example-v1.rkt" implements a full adder and prints its truth table.

At this point, there is no name resolution, so I need to give redundant information in the "port-ref" expressions.

A better version of the language would allow to write:

(assign (h1 a) a)

instead of:

(assign (port-ref half-adder h1 a) (port-ref full-adder a))

because we know that

• h1 is an instance of half-adder-arch, that has the entity half-adder,
• the current architecture is full-adder-arch and its entity is full-adder

I hope it clarifies my current concerns.

Guillaume

[Sam Tobin-Hochstadt]

I think the best way to implement what you describe for a "better
version" is as follows:

Expand `(instance h1 half-adder-arch)` into something like

(define-syntax h1 (half-adder-arch-info))

Where `half-adder-arch-info` is an expansion-time structure describing
half-adders.
Then the `assign` macro can use `syntax-local-value` to get out that
information, see that it's a half-adder, and then generate the code
you've written manually.

This paper: https://arxiv.org/abs/1106.2578 talks about this technique
in more detail, focused on the implementation of `match-expander`s,
which work this way, but it's a common Racket technique and used for
structs, syntax classes, `for` transformers, and more.

Sam

(

> --
> You received this message because you are subscribed to the Google Groups "Racket Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to racket-users...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/a42ec2de-6441-483f-a05a-910c03c317e8%40googlegroups.com.

[John Clements]

Okay, I could just be putting my foot in my mouth, here, but it sounds like you’re describing the kinds of things that are typically done by a type-checker. Have you considered adding a type-checking pass? It would contain an environment that maps names like “h1” to “types” that indicate that h1 is an instance of half-adder-arch.

In your case, the type checker would also be “resolving” lightweight expressions like (assign (h1 a) a) into fully-decorated expressions like (assign (port-ref half-adder h1 a) (port-ref full-adder a)) .

Does this make sense?

John

[Guillaume Savaton]

Le mardi 26 mai 2020 21:41:14 UTC+2, Sam Tobin-Hochstadt a écrit :

I think the best way to implement what you describe for a "better
version" is as follows:

Expand `(instance h1 half-adder-arch)` into something like

  (define-syntax h1 (half-adder-arch-info))

I have seen this done somewhere else.

At the time, I found that it looked more like a hack to work around the limitations of syntax objects (compared to the model-drive approach).

But I also understand that it can be a flexible way to organize the expansion-time data that we need without creating a complete object graph.

I will have a closer look at this pattern.

This paper: https://arxiv.org/abs/1106.2578 talks about this technique
in more detail, focused on the implementation of `match-expander`s,
which work this way, but it's a common Racket technique and used for
structs, syntax classes, `for` transformers, and more.

Thanks for the reference.

I will have a look at it in the next few days.

Guillaume

[Guillaume Savaton]

Le mardi 26 mai 2020 21:45:21 UTC+2, johnbclements a écrit :

In your case, the type checker would also be “resolving” lightweight expressions like (assign (h1 a) a) into fully-decorated expressions like (assign (port-ref half-adder h1 a) (port-ref full-adder a)) .
Does this make sense?

Totally.

I have been browsing the Mini-Java example that was published for the 2016 Language Workbench Challenge, and this "decoration" phase happens in the typechecker module.

https://github.com/dfeltey/lwc2016/tree/master/mini-java

I have added two files to the gist at https://gist.github.com/senshu/c6db95615b4b2567f168d6bfbe61655e

• tiny-hdl-resolver.rkt implements the name resolution phase. It is freely inspired by what I found in the Mini-Java sources.
• tiny-hdl-example-v2.rkt is an updated example where the redundant information has been removed.

This small language still needs some semantic checking:

• An output port must be written exactly once in a given architecture.
• An input port cannot be assigned.

[Guillaume Savaton]

Several months have passed since I started this thread.

I have finally decided to write a blog series about this experiment.

There are seven posts planned, the first four of which are already published:

• General introduction
• Step 1: execution of a hand-written Racket example
• Step 2: code generation using macros
• Step 3: name resolution

In Step 3, you will find a reference to the paper "Macros for Domain-Specific Languages" by Michael Ballantyne, Alexis King and Matthias Felleisen.

In fact, the paper answers several questions that were raised in this discussion thread. It also addresses a concern that I had not considered at all, which is the possibility to create macro-extensible DSLs.

Though I used only a tiny part of their ideas, I'd like to thank Michael and Matthias for the friendly and enlightening conversation we had in May.