[scintilla] Changes to lexers

[Neil Hodgson]

To fix some problems with lexing in Scintilla and to add new
capabilities, there are going to be some major changes. It is likely
these will go into the release after the next one. This release will
be called 2.20 as the changes are not completely backwards compatible.

Lexers and lexer options such as properties and keywords are
currently attached to the view object rather than the Document object
which can lead to arbitrary and confusing results when there are
multiple views (with different lexers or parameters) displaying one
document. Lexer state will be moved from the view object to the
document object. Applications may be affected because this leads to
lexer parameters having different lifetimes.

Some lexers may benefit from storing additional information about a
document. This is difficult with the current code so lexer modules
will be replaced with lexer objects that can contain any desired data.
Lexers will provide a factory function to create these lexer objects.
Some helper classes made it easy to change all the internal lexers to
work this way. External lexers will have to be modified.

A longer description of these changes along with pointers to
implementation code is at
http://www.scintilla.org/nulex.html

Neil

--
You received this message because you are subscribed to the Google Groups "scintilla-interest" group.
To post to this group, send email to scintilla...@googlegroups.com.
To unsubscribe from this group, send email to scintilla-inter...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/scintilla-interest?hl=en.

[Manos]

Hi Neil.

This action is a very nice choice.
It is perfect for me that I use splitting windows.

Manos.

[Michael Mullin]

Trying to update the cocoa makefiles and get the new code compiling on
OSX...

I think you may have a compilation problem w/ the LexState class in
ScintillaBase.cxx when compiling with -DSCI_LEXER

The LexState prototype declared at line 16 of ScintillaBase.h is
within the Scintilla namespace. The LexState declared @ line 472 is
not.

On line 493 the compiler doesn't know if you are defining the
constructor for the Scintilla::LexState or the global scope LexState.

Can be fixed either by moving the code in ScintillaBase.cxx lines 472-
>491 into ScintillaBase.h (just replace that prototype with the real
class declaration). Or by tacking on a Scintilla:: to LexState in
line 472.

[Michael Mullin]

Neil, I've worked on the cocoa makefiles and have gotten it compiling/
linking. The Cocoa Demo.app runs (some stylizing runtime issues-aka
no styling-... but it runs).

I've uploaded the modifications to http://groups.google.ca/group/scintilla-interest/web/nulex-CocoaMod.zip

Also, I used the most recent version of the cocoa-makefiles from the
scintilla-cocoa launchpad project (maintained by Mike Lischke).

[Simon Steele]

This looks like a great change to me, and fixes some annoyances
Programmer's Notepad had seen with the current implementation with two
use cases: external lexers, split views.

Thanks for all your hard work,

Simon.

[Neil Hodgson]

Michael Mullin:

> I think you may have a compilation problem w/ the LexState class in
> ScintillaBase.cxx when compiling with -DSCI_LEXER
>
> The LexState prototype declared at line 16 of ScintillaBase.h is
> within the Scintilla namespace.  The LexState declared @ line 472 is
> not.

OK. There were also some namespace issues using another compiler
with CharacterSet.cxx.

> Can be fixed either by moving the code in ScintillaBase.cxx lines 472-
>>491 into ScintillaBase.h (just replace that prototype with the real
> class declaration).

This exposes the class to other code when its really private to
ScintillaBase.

> Or by tacking on a Scintilla:: to LexState in
> line 472.

Adding Scintilla:: to LexState looks like the technique that went
bad with recent compilers. Instead wrapped LexState class definition
in namespace Scintilla.

> I've uploaded the modifications to http://groups.google.ca/group/scintilla-interest/web/nulex-CocoaMod.zip

I can't download that file - sometimes Google Groups is really
difficult to make work.

Neil

[Michael Mullin]

dang, I cant download it either, and like a moron I deleted it from my
hd... gaahhh!

I'll do it again tomorrow evening :(

On May 16, 9:03 pm, Neil Hodgson <nyamaton...@gmail.com> wrote:
> Michael Mullin:
>
> > I think you may have a compilation problem w/ the LexState class in
> > ScintillaBase.cxx when compiling with -DSCI_LEXER
>
> > The LexState prototype declared at line 16 of ScintillaBase.h is
> > within the Scintilla namespace.  The LexState declared @ line 472 is
> > not.
>
>    OK. There were also some namespace issues using another compiler
> with CharacterSet.cxx.
>
> > Can be fixed either by moving the code in ScintillaBase.cxx lines 472-
> >>491 into ScintillaBase.h (just replace that prototype with the real
> > class declaration).
>
>    This exposes the class to other code when its really private to
> ScintillaBase.
>
> > Or by tacking on a Scintilla:: to LexState in
> > line 472.
>
>    Adding Scintilla:: to LexState looks like the technique that went
> bad with recent compilers. Instead wrapped LexState class definition
> in namespace Scintilla.
>

> > I've uploaded the modifications tohttp://groups.google.ca/group/scintilla-interest/web/nulex-CocoaMod.zip

[Michael Mullin]

ok, all is not lost. Zip files are unable to be downloaded until
wednesday so says this google post.
http://groups.google.com/group/groupsknownissues/browse_thread/thread/bcdc8ad58494fae5/8c944fb0e0bd1a03?show_docid=8c944fb0e0bd1a03

Try downloading again on thursday or so.

[Neil Hodgson]

Michael Mullin:

> The Cocoa Demo.app runs (some stylizing runtime issues-aka
> no styling-... but it runs).

There now has to be a once-only explicit call to
Scintilla_LinkLexers() in library initialization to add all the
internal lexers to the lexer catalogue. This was previously done with
an implicit mechanism. If there is no library initialization function
then add it into the constructor of the Scintilla object with a static
guard to ensure it is only called once.

Neil

[Michael Mullin]

Neil... the zip file is downloadable now. Please try again.

[Michael Mullin]

"There now has to be a once-only explicit call to
Scintilla_LinkLexers() in library initialization to add all the
internal lexers to the lexer catalogue."

This works for the cocoa Demo.app

[Neil Hodgson]

Michael Mullin:

> Neil... the zip file is downloadable now.  Please try again.

OK, it downloaded OK. There are a number of problems in the code:

1) The make files include common.mk which has as its first target
'clean' thus the default build for the .mk files is to clean rather
than to build something (preferrably 'all'). The include statement in
make files is a simple textual inclusion - it can be quite difficult
to make multi-file make systems work well.

2) The list of lexers is hard-coded so require manual maintenance
whenever a new lexer is added to Scintilla. scintilla/src/LexGen.py is
responsible for automatically maintaining the lists of lexers in most
of the Windows and GTK+ make files.

3) Include statements have been added to header files. This should
never be done for core Scintilla files. Platform-specific modules not
maintained by me (like cocoa) may do this but only for files that are
not shared with other platforms. The main problem is that the file
Accessor.h has changed in a way that helps upgrade lexers without
changing their source code (since there are so many lexers) at the
cost of not continuing to work for other code. Accessor.h should no
longer be included by most code. The other change is that Document.h
depends on ILexer.h so ILexer.h should be included before Document.h.
After those changes it should not be necessary to add includes to any
of the lexlib files.

New download fixes include problems but not make file problems.
Also includes sending errors to view objects where possible and a new
LexerNoExceptions class which is a good base class for external
lexers.
http://www.scintilla.org/nulex.zip

[Michael Mullin]

Newest zip file compiles/runs properly for me on OSX 10.6.

Are you going to revision control these files?

If you are, might I suggest you look into the feature set of
Launchpad? I've found that it makes it extremely easy to spawn off &
merge experimentation/release branches. I'm not sure if sourceforge
has similar features. I've had a very easy time maintaining an
experimental branch of scintilla based off of Mike Lishke's scintilla-
cocoa trunk, and have had an easy time spawning experimental branches
of my own codeassistor source.

Sorry to sound like an advertisement for LP, but I've been very happy
with it so far.

> lexers.http://www.scintilla.org/nulex.zip

[Neil Hodgson]

Michael Mullin:

> Are you going to revision control these files?

These are included in a temporary git project at
http://github.com/nyamatongwe/ScintillaGIT

I am not sufficiently happy with git (or hg or bzr) to move to any of them.

[Manos]

Neil,

is it possible to do moreover changes in 2.20 version, like the
follow:
The current version send notification messages when the text is
changed.
But because I use splitting windows, I have to handle these messages
two times,
one message from each view.
As long as in splitting windows the two views share the same memory
pointer,
it should be preferable these messages to send from document, instead
from views.
In this way, users can handle notifications once to manage Undo-Redo,
text changed e.t.c.

Regards,
Manos.

[Neil Hodgson]

Manos:

> is it possible to do moreover changes  in 2.20 version, like the
> follow:
> The current version send notification messages when the text is
> changed.
> But because I use splitting windows, I have to handle these messages
> two times, one message from each view.
> As long as in splitting windows the two views share the same memory
> pointer, it should be preferable these messages to send from
> document, instead from views.

This would be a major change and wouldn't suit one of the reasons
for the notifications which is to redisplay modified data in each view
when changes occur.

> In this way, users can handle notifications once to manage Undo-Redo,
> text changed e.t.c.

If you don't want to receive notifications from one view then
change its modification event mask with SCI_SETMODEVENTMASK.

Neil

[Manos]

To disable notifications in one view is a mistake, because I 'll have
no events from
this view and I could not manage the changes when occur in that view.
For now, I have to check from which view has sent the notification.

Manos.

[Manos]

To understand what I means, because my English are poor, suppose that
I have
a text and I split this in two views.Of course, I have two views for
the same document.
When I select a part of text in the first view, with mouse for
example,
the second view does not affect and I can select with mouse another,
different, part of text.
In this case I have two different selected text for the same document.
Whether is the actual selected text ?
How can I solve this problem ?
The only way that I can do, is to take the selection from one view and
to send
the relevant message to the other view for update this.
But this way is not perfect.

Manos.

> >    Neil- Hide quoted text -
>
> - Show quoted text -

[Neil Hodgson]

Manos:

> a text and I split this in two views.Of course, I have two views for
> the same document.
> When I select a part of text in the first view, with mouse for
> example,
> the second view does not affect and I can select with mouse another,
> different, part of text.

This is the common behaviour for applications with multiple views.
For example, Visual Studio and Xcode both work this way.

> In this case I have two different selected text for the same document.
> Whether is the actual selected text ?

There are separate selections for each view.

> How can I solve this problem ?
> The only way that I can do, is to take the selection from one view and
> to send
> the relevant message to the other view for update this.

If you want to have both views always have the same selection then
you must copy changes in selection from one view to the other.

Neil

[Manos]

Neil.

The only way that I could handle selection changes is from
SCN_UPDATEUI notification.
I tried this but I failled because the above message is sending and in
text changes.
Therefor, I added another command: SCEN_SELMOFIFY.

void ScintillaWin::NotifySelChanged()
{
::SendMessage(::GetParent(MainHWND()), WM_COMMAND,
MAKELONG(GetCtrlID(), SCEN_SELMOFIFY),
reinterpret_cast<LPARAM>(MainHWND()));
}

I call this from Editor.cxx after SetSelection function:

void Editor::SetSelection(SelectionPosition currentPos_,
SelectionPosition anchor_) {
SelectionRange rangeNew(ClampPositionIntoDocument(currentPos_),
ClampPositionIntoDocument(anchor_));
if (sel.Count() > 1 || !(sel.RangeMain() == rangeNew)) {
InvalidateSelection(rangeNew);
}
sel.RangeMain() = rangeNew;
SetRectangularRange();
ClaimSelection();
NotifySelChanged();
}


From my application I handle the above message like:

LRESULT CChildFrame::OnSelChange(WORD wNotifyCode, WORD wID, HWND
hWndCtl, BOOL& bHandled)
{
LONG lStart,lEnd;
if(m_view1.m_hWnd == hWndCtl)
{
if(m_bSelPosted == FALSE)
{
lStart=::SendMessage(m_view1.m_hWnd, SCI_GETSELECTIONSTART, 0, 0);
lEnd=::SendMessage(m_view1.m_hWnd, SCI_GETSELECTIONEND, 0, 0);
m_bSelPosted = TRUE;
::SendMessage(m_view2.m_hWnd,SCI_SETSEL,lStart,lEnd);
}
else
m_bSelPosted = FALSE;
}
else if(m_view2.m_hWnd == hWndCtl)
{
if(m_bSelPosted == FALSE)
{
lStart=::SendMessage(m_view2.m_hWnd, SCI_GETSELECTIONSTART, 0, 0);
lEnd=::SendMessage(m_view2.m_hWnd, SCI_GETSELECTIONEND, 0, 0);
m_bSelPosted = TRUE;
::SendMessage(m_view1.m_hWnd,SCI_SETSEL,lStart,lEnd);
}
else
m_bSelPosted = FALSE;
}
bHandled=0;

return 0;
}

I added the boolean variable m_bSelPosted to avoid infinite loop.
This works fine.

Manos.

[Neil Hodgson]

Manos:

>                        lStart=::SendMessage(m_view1.m_hWnd, SCI_GETSELECTIONSTART, 0, 0);
>                        lEnd=::SendMessage(m_view1.m_hWnd, SCI_GETSELECTIONEND, 0, 0);
>                        m_bSelPosted = TRUE;
>                        ::SendMessage(m_view2.m_hWnd,SCI_SETSEL,lStart,lEnd);

You should think about other forms of selection like rectangular
and multiple selections and whether you want to support them.

Neil