[scala-ide-dev] XML colouring and document partitioning

[Matt Russell]

I've been looking at syntax colouring for embedded XML (e.g. to fix
ticket #2522), and making partitioning aware of XML regions appears to
be required to get this to work.

The standard way to do partitioning in Eclipse is to use a
FastPartitioner with a RuleBasedScanner. Individually, a rule can be
quite powerful -- for example, recognising a nested comment -- but
they do not seem to be designed to track state as you pass from
partition to partition. Unfortunately, this makes life difficult as at
the lexical level Scala does (I believe) need a small amount of
context to track when to switch back from XML to Scala or back from
Scala to XML. (For example, you can keep track of the nesting level of
XML tags and Scala braces.)

So I was considering whether we need a custom IDocumentPartitioner for
Scala to get this to work, and wondered if anyone else had thoughts on
the topic,

-- Matt

[Miles Sabin]

Do we really need to keep track of the nesting level?

Cheers,


Miles

--
Miles Sabin
tel: +44 7813 944 528
gtalk: mi...@milessabin.com
skype: milessabin
http://www.chuusai.com/
http://twitter.com/milessabin

[Matt Russell]

On May 6, 9:11 am, Miles Sabin <mi...@milessabin.com> wrote:
> Do we really need to keep track of the nesting level?

I believe so, although I'd be delighted to be wrong. For example,

<bar> <foo>{ 3 + 2 }</foo> + "wibble" </bar> + "wobble"
XXXXXXXXXXXSSSSSSSSSXXXXXXXXXXXXXXXXXXXXXXXXSSSSSSSSSSS

(Fixed width font useful here!)

Say, for simplicity, we had just two rules for two partitions, Scala
and XML. The XML rule could happily consume up to "{" and identify the
first partition as XML. Then, a Scala rule could consume to "}" and
identify that partition as Scala. At that point, the XML rule could
start reading XML again, but how would it know when to stop and switch
back to Scala? The only real indicator, as I see it, is that the XML
nesting bottoms out, but that requires remembering information from a
previous run of the XML rule.

-- Matt

[Miles Sabin]

Nice clear example ... thanks :-)

Yes, it looks like we do need to be able to detect that the XML has
bottomed out. Rather than remembering state across calls, presumably
we could re-establish the context from scratch? But I really don't
know what the pros and cons of either approach would be. Do you have
any other examples of IDocumentPartitioners maintaining state in the
way you're suggesting?

[Stefan Langer]

I tried doing something like this with a php partitioner but due to
time constraints I never finished it. It was part of the PHPEclipse
project, not sure if they still use it as it was a couple of years
ago. They have
similar requirements for html and embedded php. In addition to that
they also have javascript to worry about so looking at their
partitioner might proof fruitfull.

-Stefan

2010/5/6 Miles Sabin <mi...@milessabin.com>:

[Matt Russell]

On May 6, 10:31 am, Miles Sabin <mi...@milessabin.com> wrote:
> Yes, it looks like we do need to be able to detect that the XML has
> bottomed out. Rather than remembering state across calls, presumably
> we could re-establish the context from scratch?

We could reexamine the document from the beginning, which would be
pretty time consuming if we had to do it for each application of a
partitioning rule. Perhaps we could scan backwards to try and work out
the level of nesting, but that also sounds involved.

> But I really don't
> know what the pros and cons of either approach would be. Do you have
> any other examples of IDocumentPartitioners maintaining state in the
> way you're suggesting?

The Perl plugin uses a custom document partitioner, although I believe
it's primarily so they can farm out the partitioning work to a library
lexer.

http://www.google.com/codesearch?q=perlpartitioner

Stefan mentioned PHPEclipse, which I'll have a look at.

The FastPartitioner can be parameterised with an
IPartitionTokenScanner, which doesn't have to be rule based. Trouble
is that it can be run over an arbitrary region of the document, so
there's still the problem of working out the nesting context.

-- Matt

[Jin Mingjian]

this functionality will be nice for our user.  

This topic seem similar to the topic of "embed DSL language in one host language". But they still has some differences. 

The WTP/XML is seemly using one hand-made XML partitioner, although I am not much sure. I am not sure that the Scala parser/compiler can provide how kinds of AST infos. It is possible to consider to use two parsers, one for main Scala, one for xml constructs.

2010/5/6 Stefan Langer <mailto...@googlemail.com>

[Miles Sabin]

On Thu, May 6, 2010 at 1:14 PM, Jin Mingjian <jin...@gmail.com> wrote:
> I am not sure that the Scala parser/compiler can provide how
> kinds of AST infos. It is possible to consider to use two parsers, one for
> main Scala, one for xml constructs.

We really want as lightweight a solution as possible here, rather than
reusing scalac's parser. The latter isn't fast or error tolerant
enough for the job, unfortunately.