Feature request: lexer macro. Reloaded.
Igor Tkachev
Previous discussion of the subject ended with the conclusion that
there is no real need for this feature and if we still need it, raw
token macro is good enough.
Let me answer some questions of the previous discussion first.
http://nemerle.org/mailman/pipermail/devel-en/2007-February/003444.html
> Isn't raw token macro enough to do this?
The following example is compilable and does almost what is expected:
def doc = xml <doc><name>Last, First</name></doc>;
WriteLine(doc.InnerXml);
But if we change it a little bit, it stops compiling:
WriteLine((xml <doc><name>Last, First</name></doc>).InnerXml);
The problem is that the compiler decides by itself what token is a
stopper, and those rules can be different from DSL rules. Another
problem is that DSL can’t use {} and () on its own. The following
example is not compilable:
def doc = xml <doc>Hello, World! :)</doc>;
WriteLine(doc.InnerXml);
So, the answer to the question is No. Actually, the xml macro as it is
right now is pretty useless and can be used for demonstration and
education purposes only.
http://nemerle.org/mailman/pipermail/devel-en/2007-February/003446.html
> Could you give an example of construct, which MUST be present in the
> parsed language, but would be incorrect for Raw Tokens stream?
Xml macro above is one sample. Also I met some difficulties when I
started thinking of how we can implement LINQ extension. Same thing:
def a = from b in c orderby d, e select new { f; g }; - OK
def a = (from b in c orderby d, e select new { f; g }).ToString(); - Fails at the comma
As I said, the problem is a stopper token, which is selected by the compiler.
I spent some time analyzing compiler behavior and found out that there
is no way to solve this problem with current implementation, which
means we do not really have full support for embedded DSLs.
Actually, to bring full DSL flexibility into the compiler a macro has
to control the PreParse step. It should take tokens and return PExpr
(or modify the token stream).
I believe it can be done. We just need to decide if we need this
feature or not.
--
Best regards,
Igor mailto:i...@rsdn.ru
Kamil Dworakowski
> Actually, to bring full DSL flexibility into the compiler a macro has
> to control the PreParse step. It should take tokens and return PExpr
> (or modify the token stream).
>
> I believe it can be done. We just need to decide if we need this
> feature or not.
I'd like it. Should get token stream, eat how many tokens it likes, and
return PExpr (with remaining tokens?).
Michal Moskal
>
> Hi, devs!
>
> Previous discussion of the subject ended with the conclusion that
> there is no real need for this feature and if we still need it, raw
> token macro is good enough.
>
> Let me answer some questions of the previous discussion first.
>
>
> http://nemerle.org/mailman/pipermail/devel-en/2007-February/003444.html
> > Isn't raw token macro enough to do this?
>
> The following example is compilable and does almost what is expected:
>
> def doc = xml <doc><name>Last, First</name></doc>;
> WriteLine(doc.InnerXml);
>
> But if we change it a little bit, it stops compiling:
>
> WriteLine((xml <doc><name>Last, First</name></doc>).InnerXml);
>
> The problem is that the compiler decides by itself what token is a
> stopper, and those rules can be different from DSL rules. Another
> problem is that DSL can't use {} and () on its own.
Wouldn't it be enough to associated stoppers with keywords? So that
the macro introducing xml keyword would tell the parser to ignore
commas?
> The following
> example is not compilable:
>
> def doc = xml <doc>Hello, World! :)</doc>;
> WriteLine(doc.InnerXml);
>
> So, the answer to the question is No. Actually, the xml macro as it is
> right now is pretty useless and can be used for demonstration and
> education purposes only.
Hm, it wouldn't be enough here... I guess you're right.
>
>
> http://nemerle.org/mailman/pipermail/devel-en/2007-February/003446.html
>
> > Could you give an example of construct, which MUST be present in the
> > parsed language, but would be incorrect for Raw Tokens stream?
>
> Xml macro above is one sample. Also I met some difficulties when I
> started thinking of how we can implement LINQ extension. Same thing:
>
> def a = from b in c orderby d, e select new { f; g }; - OK
> def a = (from b in c orderby d, e select new { f; g }).ToString(); - Fails at the comma
>
> As I said, the problem is a stopper token, which is selected by the compiler.
>
> I spent some time analyzing compiler behavior and found out that there
> is no way to solve this problem with current implementation, which
> means we do not really have full support for embedded DSLs.
>
> Actually, to bring full DSL flexibility into the compiler a macro has
> to control the PreParse step. It should take tokens and return PExpr
> (or modify the token stream).
>
> I believe it can be done. We just need to decide if we need this
> feature or not.
>
>
> --
> Best regards,
> Igor mailto:i...@rsdn.ru
>
>
> >
>
--
Michał
Kamil Skalski
On Nov 1, 2:59 am, Igor Tkachev <i...@rsdn.ru> wrote:
> Hi, devs!
>
> Previous discussion of the subject ended with the conclusion that
> there is no real need for this feature and if we still need it, raw
> token macro is good enough.
>
> Let me answer some questions of the previous discussion first.
>
> http://nemerle.org/mailman/pipermail/devel-en/2007-February/003444.html
>
> > Isn't raw token macro enough to do this?
>
> The following example is compilable and does almost what is expected:
>
> def doc = xml <doc><name>Last, First</name></doc>;
> WriteLine(doc.InnerXml);
>
> But if we change it a little bit, it stops compiling:
>
> WriteLine((xml <doc><name>Last, First</name></doc>).InnerXml);
>
> The problem is that the compiler decides by itself what token is a
> stopper, and those rules can be different from DSL rules. Another
> problem is that DSL can't use {} and () on its own. The following
> example is not compilable:
>
> def doc = xml <doc>Hello, World! :)</doc>;
> WriteLine(doc.InnerXml);
>
> So, the answer to the question is No. Actually, the xml macro as it is
> right now is pretty useless and can be used for demonstration and
> education purposes only.
You can still escape the characters chosen by compiler. E.g.
def doc = xml <doc>Hello, World @:)</doc>;
should work. But I agree that the rules necessary to know for proper
escaping are a bit strange...
This is similar situation as in XSL, where you must sometimes put much
effort to escape xml characters in plain text used there.
We could however introduce a kind of special syntax for "raw" blocks
of text, something like <![CDATA[ .... ]]>
>
> http://nemerle.org/mailman/pipermail/devel-en/2007-February/003446.html
>
> > Could you give an example of construct, which MUST be present in the
> > parsed language, but would be incorrect for Raw Tokens stream?
>
> Xml macro above is one sample. Also I met some difficulties when I
> started thinking of how we can implement LINQ extension. Same thing:
>
> def a = from b in c orderby d, e select new { f; g }; - OK
> def a = (from b in c orderby d, e select new { f; g }).ToString(); - Fails at the comma
>
> As I said, the problem is a stopper token, which is selected by the compiler.
>
> I spent some time analyzing compiler behavior and found out that there
> is no way to solve this problem with current implementation, which
> means we do not really have full support for embedded DSLs.
>
> Actually, to bring full DSL flexibility into the compiler a macro has
> to control the PreParse step. It should take tokens and return PExpr
> (or modify the token stream).
>
> I believe it can be done. We just need to decide if we need this
> feature or not.
I don't like this, though I agree that this is doable and a way to go
if you want such feature.
How would you propose delimiting the raw text here?
Would it be specified in
macro xx(t : Token)
syntax ("start", t, "END")
or some other way to tell preparser how far should it go with eating
token stream? I wouldn't like it to use any special API, so it should
be declarative.
Kamil
Igor Tkachev
>> The problem is that the compiler decides by itself what token is a
>> stopper, and those rules can be different from DSL rules. Another
>> problem is that DSL can't use {} and () on its own.
> Wouldn't it be enough to associated stoppers with keywords? So that
> the macro introducing xml keyword would tell the parser to ignore
> commas?
I do not think so. Lets take linq for example:
def a = from c in SomeThing select c;
def a = (from c in SomeThing select c).ToString();
def a = from c in SomeThing select new { c.Field1, c.Field2 };
What keyword we should use here?
Igor Tkachev
> I don't like this, though I agree that this is doable and a way to go
> if you want such feature.
> How would you propose delimiting the raw text here?
> Would it be specified in
> macro xx(t : Token)
> syntax ("start", t, "END")
> or some other way to tell preparser how far should it go with eating
> token stream? I wouldn't like it to use any special API, so it
> should be declarative.
That is the problem. It can be done in declarative way.
IT
confusion.
Anyway, what are we going to do, guys?
Regards,
Igor
Kamil Skalski
supposed to feel like. The goal is to make things declarative.
And about the lexer customization - it is really not very elegant and
makes the language feel incredibly hacky. If programer hears that the
very token stream of program is defined arbitrarily by macro, it makes
him think about the language as a "worse than Perl" lexical hell.
Also, it is going to hit the performance a little bit, since you need
to lookup every token in a map checking if it is beginning of syntax
macro and then feed each next token into the plugin framework (though
probably this cost is very low compared to general slowness of
compiler).
I still don't understand why you need to have all this arbitrary power
for extending syntax. Why not simply say that:
{
def x = from x in c select x.foo, x.bar;
}
is valid and
{
def x = (from x in c select x.foo, y.foo);
}
is NOT.
This is rather low cost for staying in a reasonably well defined world
and way of defining syntax extensions.
2007/11/16, IT <igor.t...@gmail.com>:
--
Kamil Skalski
http://nazgul.omega.pl
VladD2
> right now is pretty useless and can be used for demonstration and
> education purposes only.
> As I said, the problem is a stopper token, which is selected by the compiler.
no RAW tokens!
RTM get grouped tokens (which results after PreParse step).
RTM should get truly RAW token STREAM.
In my option RTM should look like:
macro LinqQuery(tokens : IEnumerable[Token])
sintax("from")
{
...
where "from" is a start keyword.
VladD2
> the macro introducing xml keyword would tell the parser to ignore
> commas?
We overcomplicate this.
Lexer macro can recognize tokens itself.
VladD2
>
> def doc = xml <doc>Hello, World @:)</doc>;
def res = from customers c @, order o
where o.customer_id = c.customer_id
select o.SomeField @, c.SomeField...
It's joke?
VladD2
> macro xx(t : Token)
> syntax ("start", t, "END")
>
> or some other way to tell preparser how far should it go with eating
> token stream? I wouldn't like it to use any special API, so it should
> be declarative.
I think lexer macro should find it itself. We should only specify
"start token".
VladD2
> I don't know - this is very much against the way Nemerle always
> supposed to feel like. The goal is to make things declarative.
We only specify start token!
We need mechanism like RTM but without any restrictions.
RTM (or some like it) should not restrict we use coma or bracket in
way we need.
Kamil Skalski
At the level of { } brackets the separator token is ; not , so
the example look like
{
def res = from customers c , order o
where o.customer_id = c.customer_id
select o.SomeField, c.SomeField...;
def x = ..
}
I can't see any problem here. You would only need to escape , if you
put it into ( ) brackets, like
def res = ( from customers c @, order o
where o.customer_id = c.customer_id
select o.SomeField @, c.SomeField...);
which doesn't make any sense - the rule is simple: linq macro must be
used as "top-level" in { } scope.
Kamil Skalski
The point is that user should be able to predict the end of scope
where the macro's syntax is used. Now the declaration is very close to
what he can expect - if there is
syntax ("foo", "(", tokens, ")")
then you know that there will be
foo ( some tokens )
Владислав Чистяков
> where the macro's syntax is used.
What for this need?
> Now the declaration is very close to
> what he can expect - if there is
> syntax ("foo", "(", tokens, ")")
> then you know that there will be
> foo ( some tokens )
I think this restriction is unnecessary.
It killing possibility create on Nemerle DSL like LINQ :(.
Қɑil ଈ Skalski
>
> > The point is that user should be able to predict the end of scope
> > where the macro's syntax is used.
>
> What for this need?
To make code more readable.
>
> > Now the declaration is very close to
> > what he can expect - if there is
> > syntax ("foo", "(", tokens, ")")
> > then you know that there will be
> > foo ( some tokens )
>
> I think this restriction is unnecessary.
>
> It killing possibility create on Nemerle DSL like LINQ :(.
Those restrictions does not prevent anyone from writing Linq
expressions parser. AFAIK Linq expressions does not contain ; or
unbalanced )( in them, so they should be possible to parse using
current macro capabilities.
Dmitry Ivankov
More importantly, let's first look at some LINQ expressions and their supposed usage in Nemerle. (maybe put them somewhere in a temp wiki page)
I remember one example: "from A, B do C"
1) Can A, B, C be arbitrary expressions?
2) Can user always tell if it is ("from A, B do C") or ("from A", "B do C") ?
I haven't used LINQ, so I can't provide examples ;)
Қɑil ଈ Skalski
expressions, so it is not possible to use just a "flat" layout, like
we use in most macros, like:
syntax ("if", "(", cond, ")", expr1, "else", expr2)
or
syntax("for", "(", ....
so here "raw" token groups needs to be used, like:
macro fromLinq(tokens: Token)
syntax("from", tokens) {
// here we do manual parsing of tokens
}
if such an expression would be used in { } scope, then tokens will end
on first ; or closing } (of course at top level, so you can always
use ( .. {.. ;... } ... ) groups inside).
This is the most important fact - if Linq expressions does not contain
them (and AFAIK, they don't) and also round parenths. groups () are
conforming with "natural" shape (are , comma separated), then I can
see no problems.
The only real limitation of token groups is when you want to place
arbitrary texts as part of Nemerle source code - like token streams
with unbalanced parenthesis or ; , usage not consistent with core
Nemerle source code. In my opinion allowing embedding such chunks of
code as valid Nemerle source code is a bad practice. If you want it,
you should use separate files and then process them in your
application or alternatively build a macro, which will read them and
process at compile time.
DSLs are ok, but if you mix them with other languages you should
ensure that they have some minimum level of consistency. The current
design of macros is forcing this to some extent and though developers
should have as much choice as possible, I don't see a problem here -
the alternatives are so rich...
2007/12/8, Dmitry Ivankov <divan...@gmail.com>:
hi_octane
> I don't know - this is very much against the way Nemerle always
> supposed to feel like. The goal is to make things declarative.
> And about the lexer customization - it is really not very elegant and
> makes the language feel incredibly hacky. If programer hears that the
> very token stream of program is defined arbitrarily by macro, it makes
> him think about the language as a "worse than Perl" lexical hell.
> Also, it is going to hit the performance a little bit, since you need
> to lookup every token in a map checking if it is beginning of syntax
> macro and then feed each next token into the plugin framework (though
> probably this cost is very low compared to general slowness of
> compiler).
lexical trashcan. However, we have to accept industry changes that
come with new versions of MS languages somehow to be attractive
for .NET developers. If we drop linq or other language extensions, we
just loose audience and finally loose the language.
Maybe we can left lexer macros semi-declarative (for example by
specifying a starting keyword, and name of an extension method which
will act as a stopping rule)? And, in addition, limit usage of lexer
macros, for example allow them only for language developers (accept
only those defined in the core or macros namespaces or assemblies?).
This will leave us a complete control over such 'hacky' extensions,
still allowing accommodating required features.
hi_octane
>
> I can't see any problem here. You would only need to escape , if you
> put it into ( ) brackets, like
> def res = ( from customers c @, order o
> where o.customer_id = c.customer_id
> select o.SomeField @, c.SomeField...);
>
> which doesn't make any sense - the rule is simple: linq macro must be
> used as "top-level" in { } scope.
where linq queries enclosed with round brackets are useful. For
example, the code below taken from MSDN topic "select clause (C#
Reference)":
ContactInfo cInfo =
(from ci in app.contactList
where ci.ID == id
select ci).FirstOrDefault();
return cInfo;
I think developers will use linq queries in a way we can't predict and
I'm sure than MS will continue work on linq, extending number of
keywords and operations available (for example they can add some kind
of a "transaction{ }" statement, etc). Forcing people to learn some
rules that aren't clear even for those who experienced with Nemerle
can be the way to oblivion of (imho) the best language for .net.
We also must take into account that MS added just first two DSL
extensions to their languages (XML Literals and Linq). And both aren't
allowed by the Nemerle's lexer model. So we need to make lexer a bit
more flexible.
VladD2
> Wouldn't it be enough to associated stoppers with keywords? So that
> the macro introducing xml keyword would tell the parser to ignore
> commas?
extended lex/parse rule is done. And macro can't do anything.
BTW a DSEL (embedded DSL) like LINQ no have stopper tokens. Without
change macro logic we can't implement DSEL like LINQ (more precisely
only if we quot DSEL, but this is ugly way).
Vlad
kisloid
I think that it would be better to make possible macro system to get
raw tokens without stopper token (only start token and list of
tokens). Because implementation of Linq with current macros looks
unnatural and may scare people how come from C#. Also Linq is a
integral part of .Net platform supported by C# and VB. Furthermore in
the future MS can add new technology for example Qwer or Asdf and
people who use Nemerle must be ready to easily integrate it to
Nemerle.
Im my opinion current macro system is restricted for creating DSeL and
it must be extended as IT & Vlad says.
Yuri.
Қɑil ଈ Skalski
We could pass the reference to current group of tokens to the macro.
This way macro could "pass over" to all token groups within current
one, so e.g.
(from x,y select z,w where u)
which is groped into
( (from x) (y select z) (w where u) )
would be available to "from" macro as pointer to
"x" token, as well as to whole ( (from x) ... ) group.
2007/12/13, hi_octane <hi.octa...@gmail.com>:
>
> Kamil Skalski wrote:
> >
> > I can't see any problem here. You would only need to escape , if you
> > put it into ( ) brackets, like
> > def res = ( from customers c @, order o
> > where o.customer_id = c.customer_id
> > select o.SomeField @, c.SomeField...);
> >
> > which doesn't make any sense - the rule is simple: linq macro must be
> > used as "top-level" in { } scope.
>
> Well, I've just started working with linq in C#3, and found few cases
> where linq queries enclosed with round brackets are useful. For
> example, the code below taken from MSDN topic "select clause (C#
> Reference)":
>
> ContactInfo cInfo =
> (from ci in app.contactList
> where ci.ID == id
> select ci).FirstOrDefault();
>
> return cInfo;
>
> I think developers will use linq queries in a way we can't predict and
> I'm sure than MS will continue work on linq, extending number of
> keywords and operations available (for example they can add some kind
> of a "transaction{ }" statement, etc). Forcing people to learn some
> rules that aren't clear even for those who experienced with Nemerle
> can be the way to oblivion of (imho) the best language for .net.
Well, it is alway possible to implement given DSL inside compiler
itself. If as many mentions Linq is de-facto standard for .NET, we can
simply do it as all the others do it - implement it directly.
However I think it is still possible to do it using macros, which have
exactly the power they should have.
>
> We also must take into account that MS added just first two DSL
> extensions to their languages (XML Literals and Linq). And both aren't
> allowed by the Nemerle's lexer model. So we need to make lexer a bit
> more flexible.
> >
>
Владислав Чистяков
2007/12/15, Қɑil ଈ Skalski <kamil....@gmail.com>:
>...
> (from x,y select z,w where u)
> which is groped into
> ( (from x) (y select z) (w where u) )
>
> would be available to "from" macro as pointer to
> "x" token, as well as to whole ( (from x) ... ) group.
Maybe simply allow to create truly raw tokens macro (which take
IEnumerable [the Symbol])?
Understand, bad DSEL will not be used by people!
> Well, it is alway possible to implement given DSL inside compiler
> itself. If as many mentions Linq is de-facto standard for .NET, we can
> simply do it as all the others do it - implement it directly.
> However I think it is still possible to do it using macros, which have
> exactly the power they should have.
Of course, in language we can implement anything, but (in my option)
it's not Nemerle way.
People will be say: "And for what macros in Nemerle if even LINQ
hardcoded in compiler?".
And they will be right.
Қɑil ଈ Skalski
> Hi,
>
> 2007/12/15, Қɑil ଈ Skalski <kamil....@gmail.com>:
> >...
> > (from x,y select z,w where u)
> > which is groped into
> > ( (from x) (y select z) (w where u) )
> >
> > would be available to "from" macro as pointer to
> > "x" token, as well as to whole ( (from x) ... ) group.
>
> Maybe simply allow to create truly raw tokens macro (which take
> IEnumerable [the Symbol])?
>
> Understand, bad DSEL will not be used by people!
Why do you suppose that Linq implemented using current macros is a "bad DSL"?
I would just like to keep the rule, that Nemerle source code has well
balanced () { } [], because lack of this makes programs ugly and
unreadable.
DSL, which violates this rule should simlply not be used as embedded
into native source code, but be kept in separate files. Linq is not
the case and it can be implemented with this rule.
>
> > Well, it is alway possible to implement given DSL inside compiler
> > itself. If as many mentions Linq is de-facto standard for .NET, we can
> > simply do it as all the others do it - implement it directly.
> > However I think it is still possible to do it using macros, which have
> > exactly the power they should have.
>
> Of course, in language we can implement anything, but (in my option)
> it's not Nemerle way.
I agree.
> People will be say: "And for what macros in Nemerle if even LINQ
> hardcoded in compiler?".
> And they will be right.
>
DSL is violating lexing or preparser rules of compiler, then this is a
clear indication that this DSL should not be allowed to mix with
standard language. Otherwise you can also write programs in...
Whitespace ;) It has very loose requirement of embedded DSLs ;)
But once again, Linq is well-formed and does not require hardcoding.
Igor Tkachev
> Well, it is alway possible to implement given DSL inside compiler
> itself. If as many mentions Linq is de-facto standard for .NET, we
> can simply do it as all the others do it - implement it directly.
You can do it only by breaking (,) {;} rule, right? So, what's the
point to make such an exception for hardcoded linq implementation?
Lets break this rule for lex macros.
> However I think it is still possible to do it using macros, which have
> exactly the power they should have.
Unfortunately, current macros support is good for syntactical shortcuts
only, not for eDSL.
Владислав Чистяков
2007/12/21, Igor Tkachev <i...@rsdn.ru>: