Style number assumptions for LexPerl variable interpolation

[KHMan]

Looks like I'll use 9 new styles to do basic variable
interpolation highlighting of $<identifier> and @<identifier> in
the various quote[-like] operator strings:

"" qq{} `` qx{} // m{} qr{} s{}{} <<HERE

For things like qq{}, interpolation highlighting is not done when
the a single quote delimiter is used. Skipping highlighting of
more complex forms like $a[1], $b->{key} etc. Too much of talk of
'heuristics' in the documentation... leaving that for another day.
Also skipping highlighting of interpolated special variables
except for the usual $_ and @_.

This is roughly the behaviour of the oldish gvim and Komodo I have
currently, although they do not seem to do variable interpolation
highlighting for <<HERE blocks.

For the new style numbers, the plan is they are offset in value
from the parent style by a fixed, identical constant, in case I
want to use that behaviour. If this assumption is not desirable,
let me know and I'll skip the use of this assumption.

--
Cheers,
Kein-Hong Man (esq.)
Kuala Lumpur, Malaysia

[KHMan]

On Aug 10, 10:15 am, KHMan wrote:
> Looks like I'll use 9 new styles to do basic variable
> interpolation highlighting of $<identifier> and @<identifier> in
> the various quote[-like] operator strings:
>
>      "" qq{} `` qx{} // m{} qr{} s{}{} <<HERE

Preliminary work code and test cases (simple variable interpolation
for double-quoted strings only):
http://sites.google.com/site/rubblepiles/scintilla-and-scite-stuff/LexPerl-interpolate.zip

[Neil Hodgson]

KHMan:

> Preliminary work code and test cases (simple variable interpolation
> for double-quoted strings only):

VarInterpolation doesn't need to be marked with SCI_METHOD since it
is only used locally. SCI_METHOD is there to force a particular
calling convention (stdcall) so that lexers can be used if they are
separately compiled from Scintilla with a different calling convention
as may be the case with external lexers. Forcing stdcall on a method
may stop compiler optimizations like using fastcall or thiscall.

Otherwise, it looks good.

Neil

[KHMan]

OK

> Otherwise, it looks good.

[Ahmad M. Zawawi]

Thanks for the great patch.

I took the liberty of including it in Wx::Scintilla 0.21_05 which is
found at:
https://metacpan.org/release/AZAWAWI/Wx-Scintilla-0.21_05

Now Padre (http://padre.perlide.org) and other Wx-based Perl editors
can utilize your awesome patch right away.

Note: 'perl -c perl-test-interpolation.pl' fails. Is it that intended?
--
Thanks,
Ahmad M. Zawawi

[KHMan]

On 8/12/2011 6:56 PM, Ahmad M. Zawawi wrote:
> Thanks for the great patch.
>
> I took the liberty of including it in Wx::Scintilla 0.21_05 which is
> found at:
> https://metacpan.org/release/AZAWAWI/Wx-Scintilla-0.21_05
>
> Now Padre (http://padre.perlide.org) and other Wx-based Perl editors
> can utilize your awesome patch right away.

Hey, not so fast. It's just a preview, I pinged and got the
feedback I need from Neil.

I've done "" qq{} `` qx{}, hope to finish it before the end of the
month. A few hundred more test cases to check...

The final to-be-applied version will be posted to the Scintilla SF
feature tracker. You can grab it once it gets into hg.

> Note: 'perl -c perl-test-interpolation.pl' fails. Is it that intended?

Hmmm... perhaps you should look at the contents of the file. :-)

[Ahmad M. Zawawi]

I released it in as a *developer* version to CPAN as part of the
transition from Scintilla 1.x to scintilla latest. So it is not a big
deal. I am also working on a Perl 6 lexer for Scintilla. Basically
we're backporting Wx::STC binding to use scintilla latest over older
wxWidgets 2.8.x platforms.

Please keep us up-to-date with LexPerl.cxx changes. As soon as I have
a working LexPerl6.cxx, i'll post it on this group.

--
Thanks,
Ahmad M. Zawawi

[KHMan]

On 8/12/2011 9:14 PM, Ahmad M. Zawawi wrote:
> I released it in as a *developer* version to CPAN as part of the
> transition from Scintilla 1.x to scintilla latest. So it is not a big
> deal. I am also working on a Perl 6 lexer for Scintilla. Basically
> we're backporting Wx::STC binding to use scintilla latest over older
> wxWidgets 2.8.x platforms.
>
> Please keep us up-to-date with LexPerl.cxx changes.

Easy, use automation -- make sure you're signed up for:
http://groups.google.com/group/scintilla-tracker

Variable interpolation is the last biggie that's missing from
LexPerl. I don't remember anything else that's really worth doing
for LexPerl...

> As soon as I have
> a working LexPerl6.cxx, i'll post it on this group.

Good to know, and looking forward to writing some Perl 6 on SciTE.
I never got started on doing anything serious on Perl 6. If you
have a good set of tests for Perl 6 highlighting, I'd be
interested to dabble in it in the (very vague) future. IMHO, it
was possible to get Perl 5 highlighted mostly correctly mainly by
amassing a large set of test cases.

[KHMan]

On 8/12/2011 9:14 PM, Ahmad M. Zawawi wrote:
>[snip]

> Please keep us up-to-date with LexPerl.cxx changes. As soon as I have
> a working LexPerl6.cxx, i'll post it on this group.

Interim code. Basic variable interpolation for:

"" qq{} `` qx{} // m{} qr{} s{}{}

Still to do: HERE docs.

http://sites.google.com/site/rubblepiles/scintilla-and-scite-stuff/LexPerl-interpolate.zip

[Ahmad Zawawi]

Cool update.

By the way, I started experimenting more with LexPerl.cxx improved
styles using Wx::Scintilla
http://padre.perlide.org/trac/browser/trunk/Wx-Scintilla/share/perl_editor.pl?rev=15491

One thing to note, $self->SetStyleBits(7) is needed otherwise we'll
have indicators all over the place. Am i doing something wrong here?
GetStyleBitsNeeded() still reports only 5 needed bits.

Regards,
Ahmad M. Zawawi

> --
> You received this message because you are subscribed to the Google Groups
> "scintilla-interest" group.
> To post to this group, send email to scintilla...@googlegroups.com.
> To unsubscribe from this group, send email to
> scintilla-inter...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/scintilla-interest?hl=en.
>
>

[KHMan]

On 8/14/2011 7:32 PM, Ahmad Zawawi wrote:
> Cool update.
>
> By the way, I started experimenting more with LexPerl.cxx improved
> styles using Wx::Scintilla
> http://padre.perlide.org/trac/browser/trunk/Wx-Scintilla/share/perl_editor.pl?rev=15491
>
> One thing to note, $self->SetStyleBits(7) is needed otherwise we'll
> have indicators all over the place. Am i doing something wrong here?
> GetStyleBitsNeeded() still reports only 5 needed bits.

Yes, I think. (I hack lexers only, I'm all at sea with the other
bits of Scintilla.)

[Neil Hodgson]

Ahmad Zawawi:

> One thing to note,  $self->SetStyleBits(7) is needed otherwise we'll
> have indicators all over the place. Am i doing something wrong here?
> GetStyleBitsNeeded() still reports only 5 needed bits.

SCI_GETSTYLEBITSNEEDED returns 8 for me. The number of bits is set
as the last parameter in the LexerModule constructor:
LexerModule lmPerl(SCLEX_PERL, LexerPerl::LexerFactoryPerl, "perl",
perlWordListDesc, 8);

Neil

[KHMan]

On Aug 14, 2:38 am, KHMan wrote:
> On 8/12/2011 9:14 PM, Ahmad M. Zawawi wrote:
> >[snip]
> > Please keep us up-to-date with LexPerl.cxx changes. As soon as I have
> > a working LexPerl6.cxx, i'll post it on this group.

Interim code. One or two more things to do before submitting to the
tracker.

http://sites.google.com/site/rubblepiles/scintilla-and-scite-stuff/LexPerl-interpolate.zip

Basic variable interpolation done:
"" qq{} `` qx{} // m{} qr{} s{}{} here-docs

One style added for tr{}{} y{}{} translations which do not have
interpolation.

Some colours adjusted for the SciTE properties file, but then I not
real picky. The adjustment is intended that any interpolated variable
in various types of quoted blocks is darkish-red. Users welcome to
propose changes if their eyes hurt... :-)

TODO:
(1) Special variables (other than $_ and @_ which already work). Will
need it otherwise users will be wondering why they do not work, except
for // m{} qr{} s{}{} (unless some kind of reliable ruleset can be
divined).
(2) ${bareword} @{bareword} if it does not involve ripping up too much
code. Elsewhere, things like "$a[$i]" looks quite nice already;
parsing the entire var spec seems like overkill.

[KHMan]

On Aug 15, 8:20 pm, KHMan wrote:
>[snipped all]

Interim code. A few more things to do and more testing. Special
variables are now interpolated for:
"" qq{} `` qx{}

http://sites.google.com/site/rubblepiles/scintilla-and-scite-stuff/LexPerl-interpolate.zip

The kinds of additional patterns recognized are:
${word} @{word} and things like $${word}
$#{word} @#{word} and things like $#${word}
${digit} @{digit}
$$ and most special variables with operator chars
$0 to $9 and things like $123 (perl seem to greedily lex)
$^A and other control char special variables

InterpolateSegment() is the new code for variable interpolation. Once
the others (// m{} qr{} s{}{} here-docs) are shifted too, the older
VarInterpolation() will be deleted. The new code is closer to the way
Perl parses quoted operators and is less likely to exhibit glitches.