4.0 plans

[Neil Hodgson]

4.0 is an opportunity to make some bigger changes. That could mean it will be longer before a 4.0 release or there could be multiple 4.0.x releases made which are considered plastic and a 4.1.0 release made once everything has solidified again.

Areas that may change include the platform code, the interface between platform and core code, and the interface between lexers and core code.

1) The Cocoa platform code may move to using Automatic Reference Counting (ARC).

2) The GTK+ platform code may remove support for older versions of GTK+. I think retaining support only for GTK+ 2.24 and not for any other 2.x releases would be good.

3) The GTK+ platform code could start supporting GTK+ 3.9x/4.x. GTK+ appears to be moving from drawing to a Cairo context to creating a tree of render nodes which will be drawn on the GPU. Converting Scintilla to this approach looks like it could be a big job and may be an opportunity for someone interested in this area to redesign graphics output on GTK+. This may require changing how core Scintilla interfaces with drawing.

4) The platform interface should be extended to allow autocompletion item selection to be reported as an event. This should be implemented in a sufficiently generic way that adding more events in the future doesn’t cause a hard break in compatibility.

5) Mouse and keyboard event methods on Editor should take a single ‘int modifiers’ parameter for the modifier keys instead of multiple bool arguments. The old methods should be removed.

6) Platform layers could all implement the "Fine Ticker" methods to simplify the code. Currently, Editor checks FineTickerAvailable and may fall back to SetTicking if they are not available. This code isn’t checked much as the in-repository platforms all implement Fine Ticker methods so the code may be decaying.

7) Unused methods in the platform interface such as Window::SetTitle and Platform::DBCSCharLength should be removed. Platform::IsDBCSLeadByte should be moved into core where it can be shared - there is already a variant of this in Document. Platform methods Minimum, Maximum, and Clamp should be replaced with the equivalent C++ std:: standard library calls. More cleanups like these where possible.

8) Methods could be added to the ILexer interface to allow more metadata reporting to applications.

9) There are 2 ILexer interface versions and 2 IDocument interface versions. These could be collapsed into 1 interface each way.

10) UTF-8 could be made the default encoding.

One change that I think is too big for now is to split the lexers from Scintilla into a separate project. This would simplify Scintilla’s version number story. Lexer updates often change behaviour in ways that can break current clients. For example, by dividing a lexical class in 2 such as splitting documentation comments off from comments. Or by changing folding to work on ‘(‘ and ‘)’. This can make it harder for projects to update to a new release - they may want some bug fixes but are scared off by the potential for unwanted styling changes. Its a long way from semver or other version numbering schemes. This could also make Scintilla a better candidate for packaging on Linux and may also allow one lexer library to be reused between GTK+, Qt, wxWidgets and Tk.

Neil

[Mike Lischke]

>
> 8) Methods could be added to the ILexer interface to allow more metadata reporting to applications.

It would be very helpful if there was a way to get styling informations for a piece of text. For my editor I have a reporting function which optionally includes the text of the editor that produces a web page. In order to maintain the same color coding in the web page as used in the editor I have to do a number of extra steps which also involve accessing the C++ code directly. An export-to-html Scintilla call would make this a lot simpler.

>
> 10) UTF-8 could be made the default encoding.

It would even be worth thinking about supporting only utf-8 (sorry, yes, I suggested that before). Documents with other encodings have to be converted first. It's a persistence/transport layer decision what encoding a document should use, not that of the editor.

Mike
--
www.soft-gems.net

[rhkr...@gmail.com]

On Friday, June 02, 2017 02:57:23 AM Neil Hodgson wrote:
> 4.0 is an opportunity to make some bigger changes. That could mean it
> will be longer before a 4.0 release or there could be multiple 4.0.x
> releases made which are considered plastic and a 4.1.0 release made once
> everything has solidified again.

Just a general comment, then one specific comment:

Scintilla never stands still (and that's a good thing)--I'm glad to see that
you have more plans for the future, and, as you seem to usually do, you share
them with your user / fellow developer community! Kudos!

--< good stuff snipped >--

> One change that I think is too big for now is to split the lexers from
> Scintilla into a separate project. This would simplify Scintilla’s version
> number story. Lexer updates often change behaviour in ways that can break
> current clients. For example, by dividing a lexical class in 2 such as
> splitting documentation comments off from comments. Or by changing folding
> to work on ‘(‘ and ‘)’. This can make it harder for projects to update to
> a new release - they may want some bug fixes but are scared off by the
> potential for unwanted styling changes. Its a long way from semver or
> other version numbering schemes. This could also make Scintilla a better
> candidate for packaging on Linux and may also allow one lexer library to
> be reused between GTK+, Qt, wxWidgets and Tk.

When / if you (or somebody else) gets around to making significant changes fo
the lexers (includling possibly making them a separate project), I'd suggest
that you consider creating / adopting a meta-language (I think that's the word
I want to use) that would let (advanced?) users create and modify lexers
without needing to understand C/C++ nor the Scintilla codebase.

The editors I've used for most of the last 17 years (Nedit and Kate) both do
that in one fashion or another, and, in addition, both allow regular
expressions in that language.

At some point in time, I may provide a little more description of what they
do, or some sample "lexers" although that's not what they call them.

[Jason Haslam]

> On Jun 2, 2017, at 1:17 AM, 'Mike Lischke' via scintilla-interest <scintilla...@googlegroups.com> wrote:
>>
>> 10) UTF-8 could be made the default encoding.
>
>
> It would even be worth thinking about supporting only utf-8 (sorry, yes, I suggested that before). Documents with other encodings have to be converted first. It's a persistence/transport layer decision what encoding a document should use, not that of the editor.

I agree, and I also put forth a similar argument previously. Supporting multiple encodings adds a lot of complexity at the platform layer. I don’t think it’s necessary.

Jason

[Lex Trotman]

Having the Scintilla buffer as UTF-8 does not stop the application
from supporting multiple encodings, that is how Geany works, no matter
what encoding the file is, the buffer is always UTF-8 and its
converted between them on load and unload.

Cheers
Lex

>
> --
> You received this message because you are subscribed to the Google Groups "scintilla-interest" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to scintilla-inter...@googlegroups.com.
> To post to this group, send email to scintilla...@googlegroups.com.
> Visit this group at https://groups.google.com/group/scintilla-interest.
> For more options, visit https://groups.google.com/d/optout.

[Matthew Brush]

On 2017-06-02 08:32 AM, rhkr...@gmail.com wrote:
> On Friday, June 02, 2017 02:57:23 AM Neil Hodgson wrote:

>> [...]

>> One change that I think is too big for now is to split the lexers from
>> Scintilla into a separate project. This would simplify Scintilla’s version
>> number story. Lexer updates often change behaviour in ways that can break
>> current clients. For example, by dividing a lexical class in 2 such as
>> splitting documentation comments off from comments. Or by changing folding
>> to work on ‘(‘ and ‘)’. This can make it harder for projects to update to
>> a new release - they may want some bug fixes but are scared off by the
>> potential for unwanted styling changes. Its a long way from semver or
>> other version numbering schemes. This could also make Scintilla a better
>> candidate for packaging on Linux and may also allow one lexer library to
>> be reused between GTK+, Qt, wxWidgets and Tk.
>
> When / if you (or somebody else) gets around to making significant changes fo
> the lexers (includling possibly making them a separate project), I'd suggest
> that you consider creating / adopting a meta-language (I think that's the word
> I want to use) that would let (advanced?) users create and modify lexers
> without needing to understand C/C++ nor the Scintilla codebase.
>
> The editors I've used for most of the last 17 years (Nedit and Kate) both do
> that in one fashion or another, and, in addition, both allow regular
> expressions in that language.
>
> At some point in time, I may provide a little more description of what they
> do, or some sample "lexers" although that's not what they call them.
>

Not sure if you've seen it or not, but there is:

https://foicica.com/scintillua/

It sounds more or less like what you're talking about. I haven't tried
writing a lexer for it personally (though I have made a simple editor
that used it) because I hate Lua and would rather program in C++ :)

Regards,
Matthew Brush

[rhkr...@gmail.com]

Thanks for the reply!

Yeah, I meant to mention Lua, just to exclude it, but forgot. It's not what
I'm looking for, for at least three reasons:

* It is a general purpose language, which requires a lot of learning before
you can use it specifically for the purpose of writing a lexer.

* This is a corollary of the first, but it is not a specific purpose language
designed to allow novices to write lexers fairly easily.

* Not all Scintilla based editors allow / support Lua or Lua based lexers
(e.g., Geany).

[Neil Hodgson]

Mike Lischke:

> It would be very helpful if there was a way to get styling informations for a piece of text. For my editor I have a reporting function which optionally includes the text of the editor that produces a web page. In order to maintain the same color coding in the web page as used in the editor I have to do a number of extra steps which also involve accessing the C++ code directly. An export-to-html Scintilla call would make this a lot simpler.

SciTE has export as HTML/PDF/RTF/LaTeX/XML and Copy as RTF commands that don’t have to use any special access and they are reasonably short. RTF is more commonly accepted as a clipboard format so that is what SciTE provides in around 300 lines.

> It would even be worth thinking about supporting only utf-8 (sorry, yes, I suggested that before).

I have answered this before too.

Neil

[Neil Hodgson]

Randy Kramer:

> When / if you (or somebody else) gets around to making significant changes fo
> the lexers (includling possibly making them a separate project), I'd suggest
> that you consider creating / adopting a meta-language (I think that's the word
> I want to use) that would let (advanced?) users create and modify lexers
> without needing to understand C/C++ nor the Scintilla codebase.

Some of the applications that use Scintilla include their own lexer definition language matched to the goals of those projects. The languages diverge significantly depending on their audience and I wouldn’t really want to decide between them.

Scintillua uses the existing loadable lexer feature to provide generic lexing and its possible other lexer definition languages could be added in the same way.

There are also many lexer/parser generators and libraries that could be used within Scintilla. Sometimes they require headers or libraries to be installed or copied and this makes use of multiple tools messy. Choosing a particular tool as a preferred secondary choice (after writing a custom lexer) could be helpful. My preference would be for a C++ library like PEGTL but that is nowhere near an end-user tool.
https://github.com/taocpp/PEGTL

Neil

[rhkr...@gmail.com]

Neil,

Thanks for the response!

I've gone to the web page you listed for PEGTL and may dig a little deeper
over time.

I may from time to time post messages to the list following up on this, maybe
giving examples of the nedit and kate approaches to lexing / styling.

regards,
Randy Kramer

[Neil Hodgson]

Randy Kramer:

> I've gone to the web page you listed for PEGTL and may dig a little deeper
> over time.

Scintillua also uses PEGs and its a common technique these days so worth learning.

> I may from time to time post messages to the list following up on this, maybe
> giving examples of the nedit and kate approaches to lexing / styling.

TextMate bundles are probably the most commonly available user-level language for highlighting.
http://manual.macromates.com/en/language_grammars#language_grammars

Neil

[Neil Hodgson]

> 1) The Cocoa platform code may move to using Automatic Reference Counting (ARC).

This has been committed. Its the same set of changes as was presented in the April 24 “Cocoa changes and ARC for 4.0” message.

This is a major change so there is a good chance of new bugs. If you use Cocoa, then test your application with this change.

https://sourceforge.net/p/scintilla/code/ci/2378244bf3c76f69ec359e05cba9b8486b8f93ec/

Neil

[dail8859]

> TextMate bundles are probably the most commonly available user-level language for highlighting.http://manual.macromates.com/en/language_grammars#language_grammars

When I first ran across this (and this is similar to what Sublime uses) I was a huge fan. For editors that support several of the lexers, it is hard to generically say e.g. "this is a comment"...instead you have to hard code which styles are comments for each language. For the TextMate bundles you have more meta information so you can handle languages and their features more generically. This does lead to the need of more memory and CPU power to parse files but IMOH is worth it. As I mentioned in another topic a while ago I started working on a project to read in the Sublime package syntax (its in YAML) and using Scintilla for the actual text drawing and got some good initial results.

I'm definitely not arguing to drop support for the current lexers, but depending on what kind of major changes would be made for 4.0+ I would love to see some sort of lexer definition language like other editors have.

Just my two cents :)

[Lex Trotman]

The problem with definition based lexers is the performance, its a
common comment from users about the performance difference between
Scintilla based editors like Geany and GTKSourceView based editors
like Gedit. Certainly for human generated files (say a few thousand
lines) it doesn't matter much, but although its not its major
use-case, a programmers editor should reasonably expect to be used to
open computer generated files (such as generated code, HTML or logs)
which can be huge and then the performance difference is very
noticeable. Scintilla would have to be prepared to field such
complaints with definition based lexers.

[Neil Hodgson]

dail8859:

> I'm definitely not arguing to drop support for the current lexers, but depending on what kind of major changes would be made for 4.0+ I would love to see some sort of lexer definition language like other editors have.

Interpreting a definition language increases the attack surface of Scintilla as would including a Lua interpreter. If such a thing were to be included, it should be possible to disable it at build time.

Not that the current lexers are immune from security bugs - its easy to produce out-of-bounds access in document parsing code. It would help security to have an option for moving the lexers into a heavily sandboxed process and I regret not having found the time to work on that.

Neil

[Paul K]

> When I first ran across this (and this is similar to what Sublime uses) I was a huge fan. For editors that support several of the lexers, it is hard to generically say e.g. "this is a comment"...instead you have to hard code which styles are comments for each language. For the TextMate bundles you have more meta information so you can handle languages and their features more generically. This does lead to the need of more memory and CPU power to parse files but IMOH is worth it. As I mentioned in another topic a while ago I started working on a project to read in the Sublime package syntax (its in YAML) and using Scintilla for the actual text drawing and got some good initial results. 

I'll chime in with my thoughts on this as I've been developing an IDE using Scintilla and have also recently added Scintillua support (https://studio.zerobrane.com).

- To me the main advantage is the ability to add new lexers and modify existing lexers without recompilation and re-distribution of binaries

- The lexers focus on implementing their own specific features with common elements being handled by the lexer module itself (also in Lua). This leads to more compact code; for example, the python lexer is 135 lines in Scintillua and 747 in Scintilla (I cannot comment on the feature parity though, but enabling folding by indentation is only one line of code in Scintillua).

- I'm familiar with LPEG, but it doesn't require any deep knowledge of it for basic functionality. In fact, if you look at a simple lexer, it looks similar to TextMate's description (https://foicica.com/hg/scintillua/file/a740cb28d0f5/lexers/latex.lua).

- Scintillua support lexer embedding; for example its html lexer embeds css/javascript/coffeescript lexers and allows to specify other lexers to embed. It also supports folding across embedded lexers.

- Colors and styles are implemented consistently across lexers. "comment" and "number" will be styled the same way across all lexers, but each lexer can also add a unique type/style.

- It does require Lua and Lpeg libraries, but it's a drop-in Scintilla lexer replacement and can be linked all together into one file (as far as I remember; my application already includes Lua and Lpeg, so I only bundle the scintillua code in the lexer).

- There are already more than 110 lexers with new formats being regularly added (https://foicica.com/scintillua/LexerList.html).

- I haven't noticed significant performance differences between Scintillua and Scintilla lexers, but I haven't done any tests/measurements, especially on large files. I also use LuaJIT for my application, which does a good job of optimizing frequently visited fragments, so this may have impact as well.

- Since scintillua provides its own style handling, it is not quite compatible with the one provided by Scintilla. As my application was already using Scintilla styles, I ended up using a simple mapping to configure styles myself and to bypass Scintillua's style configuration.

Overall, it's been pleasant experience so far.

Paul.

[Neil Hodgson]

The Cocoa platform layer was modernized using Xcode’s Edit | Convert | To Modern Objective-C Syntax command. Most of the options were turned on except for inferring properties (that is, converting methods that look like properties into properties) as that caused some poor results for methods that were in protocols. "Add attribute annotations” also produced some strange results so was also turned off. A couple of instances of transformations produced bad code so were reverted. One apparent bug was found in InfoBar setDisplay.

https://sourceforge.net/p/scintilla/code/ci/31466630bcccc16427a2f14598118cf97a7b161e/
https://sourceforge.net/p/scintilla/code/ci/7f58802368f4188d8a6cbb5406766307a479f6af/
https://sourceforge.net/p/scintilla/code/ci/fdca97e72d4e0602386415107c6fa0b66dbe137d/
https://sourceforge.net/p/scintilla/code/ci/e1301b1d2321757f33ebe91c08dcd156c1631627/

Running an automatic indenter to format code consistently was discussed in the "C++17 Intentions” thread so I ran astyle 3.01 over the cocoa directory. Some Objective-C options were added (last 5 options) for minor padding in method definitions and to align the ‘:’ in calls over multiple lines to be similar to the Xcode formatter:

--style=attach
--indent=force-tab=8
--keep-one-line-blocks
--pad-header
--unpad-paren
--pad-comma
--indent-cases
--align-pointer=name
--pad-method-prefix
--pad-return-type
--pad-param-type
--align-method-colon
--pad-method-colon=after

https://sourceforge.net/p/scintilla/code/ci/57988f82bf71e7beb183163d648143187eceab7f/

Neil

[Neil Hodgson]

> 2) The GTK+ platform code may remove support for older versions of GTK+. I think retaining support only for GTK+ 2.24 and not for any other 2.x releases would be good.

Looking for old #ifs in the GTK+ platform layer showed these:

/home/neil/merc/scintilla/gtk/PlatGTK.cxx:48:#if GTK_CHECK_VERSION(2,22,0)
/home/neil/merc/scintilla/gtk/PlatGTK.cxx:313:#if CAIRO_VERSION >= CAIRO_VERSION_ENCODE(1, 8, 0)
/home/neil/merc/scintilla/gtk/PlatGTK.cxx:540:#if CAIRO_VERSION >= CAIRO_VERSION_ENCODE(1, 2, 0)
/home/neil/merc/scintilla/gtk/PlatGTK.cxx:591:#if CAIRO_VERSION >= CAIRO_VERSION_ENCODE(1,6,0)
/home/neil/merc/scintilla/gtk/PlatGTK.cxx:709:#ifdef PANGO_VERSION
/home/neil/merc/scintilla/gtk/PlatGTK.cxx:900:#ifdef PANGO_VERSION
/home/neil/merc/scintilla/gtk/ScintillaGTK.cxx:88:#if GTK_CHECK_VERSION(2,20,0)
/home/neil/merc/scintilla/gtk/ScintillaGTK.cxx:234:#if GTK_CHECK_VERSION(2,20,0)
/home/neil/merc/scintilla/gtk/ScintillaGTK.cxx:325:#if GTK_CHECK_VERSION(2,20,0)
/home/neil/merc/scintilla/gtk/ScintillaGTK.cxx:364:#if GTK_CHECK_VERSION(2,20,0)
/home/neil/merc/scintilla/gtk/ScintillaGTK.cxx:390:#if GTK_CHECK_VERSION(2,20,0)
/home/neil/merc/scintilla/gtk/ScintillaGTK.cxx:2726:#if GTK_CHECK_VERSION(2,22,0)
/home/neil/merc/scintilla/gtk/ScintillaGTK.cxx:2810:#if GTK_CHECK_VERSION(2,22,0)

Removing all these checks and choosing the ‘if’ case of each (not the ‘else’) worked correctly for an old version of Mint with GTK+ 2.24.23. Its possible but unlikely that earlier releases of GTK+ 2.24 will have problems.
The Pango checks appear to be handling Pango before it exposed a version number and the Cairo checks also appear quite old as Cairo 1.8 was 2008.
There are also checks for accessibility support but these are checking GTK+ 3.x so don’t matter for removing 2.16-2.22.

The attached patch for this change is only 182 lines and I think it is reasonable to commit to the 4.x branch.

Neil

[Mike Lischke]

> https://sourceforge.net/p/scintilla/code/ci/31466630bcccc16427a2f14598118cf97a7b161e/
> https://sourceforge.net/p/scintilla/code/ci/7f58802368f4188d8a6cbb5406766307a479f6af/
> https://sourceforge.net/p/scintilla/code/ci/fdca97e72d4e0602386415107c6fa0b66dbe137d/
> https://sourceforge.net/p/scintilla/code/ci/e1301b1d2321757f33ebe91c08dcd156c1631627/
>
> Running an automatic indenter to format code consistently was discussed in the "C++17 Intentions” thread so I ran astyle 3.01 over the cocoa directory. Some Objective-C options were added (last 5 options) for minor padding in method definitions and to align the ‘:’ in calls over multiple lines to be similar to the Xcode formatter:
>
> --style=attach
> --indent=force-tab=8
> --keep-one-line-blocks
> --pad-header
> --unpad-paren
> --pad-comma
> --indent-cases
> --align-pointer=name
> --pad-method-prefix
> --pad-return-type
> --pad-param-type
> --align-method-colon
> --pad-method-colon=after
>
> https://sourceforge.net/p/scintilla/code/ci/57988f82bf71e7beb183163d648143187eceab7f/

Looks good so far, but 8 chars indentation is huge. A lot of wasted space and produces a lot of wrapping problems for nested code that produces longer lines. The original indentation was better to read IMO.

Mike
--
www.soft-gems.net

[Neil Hodgson]

> 7) Unused methods in the platform interface such as Window::SetTitle and Platform::DBCSCharLength should be removed. Platform::IsDBCSLeadByte should be moved into core where it can be shared - there is already a variant of this in Document. Platform methods Minimum, Maximum, and Clamp should be replaced with the equivalent C++ std:: standard library calls. More cleanups like these where possible.

An initial version of a patch implementing this is attached. It removes 15 methods from the Platform interface. Some of these are platform-independent and have been moved into the src directory. Others are no longer needed. The C++ standard library provides std::min and std::max. Since std::clamp is not available from all compilers quite yet, there is a temporary Sci::clamp function.

Platform::IsDBCSLeadByte was replaced with DBCSIsLeadByte in the DBCS.cxx file. This may be needed by other platform layers.

If maintainers of other platforms want to support both the old and new Platform interfaces with one file then there could be some extra help such as a #defined symbol added to Platform.h.

Neil

[Neil Hodgson]

Mike Lischke:

> Looks good so far, but 8 chars indentation is huge. A lot of wasted space and produces a lot of wrapping problems for nested code that produces longer lines. The original indentation was better to read IMO.

Its the same indentation as the platform-independent parts of Scintilla. Objective C often has verbose calls so maybe they should be spilt over multiple lines more often.

Neil

[Neil Hodgson]

> 5) Mouse and keyboard event methods on Editor should take a single ‘int modifiers’ parameter for the modifier keys instead of multiple bool arguments. The old methods should be removed.

Initial patch for this attached.

The key changes are in Editor.h with KeyDown, ButtonDown, and ButtonMove removed; ButtonUp is replaced with ButtonUpWithModifiers; and ButtonMoveWithModifiers gains a curTime parameter for consistency.

- int KeyDown(int key, bool shift, bool ctrl, bool alt, bool *consumed=0);
- virtual void ButtonDown(Point pt, unsigned int curTime, bool shift, bool ctrl, bool alt);
- void ButtonMoveWithModifiers(Point pt, int modifiers);
- void ButtonMove(Point pt);
- void ButtonUp(Point pt, unsigned int curTime, bool ctrl);
+ void ButtonMoveWithModifiers(Point pt, unsigned int curTime, int modifiers);
+ void ButtonUpWithModifiers(Point pt, unsigned int curTime, int modifiers);

The Notify* methods that took individual bool modifier flags have also been removed.

Neil

[Neil Hodgson]

> Looking for old #ifs in the GTK+ platform layer showed these:

> …

> The attached patch for this change is only 182 lines and I think it is reasonable to commit to the 4.x branch.

This is now committed and the documentation updated.
https://sourceforge.net/p/scintilla/code/ci/90741c1cb988ff0eae02774204a407417598e621/
https://sourceforge.net/p/scintilla/code/ci/7b24a8026c86ce0b4190a8ba0b811a96a99156e3/

The MinGW distribution of GCC on Windows has lagged a bit in supporting new versions of GCC and also newer Windows APIs such as DirectDraw. MinGW-w64 is often more current and now includes GCC 7.1. I am using the nuwen.net distribution at
https://nuwen.net/mingw.html

Neil

[Neil Hodgson]

> 6) Platform layers could all implement the "Fine Ticker" methods to simplify the code. Currently, Editor checks FineTickerAvailable and may fall back to SetTicking if they are not available. This code isn’t checked much as the in-repository platforms all implement Fine Ticker methods so the code may be decaying.

Proposed patch for this attached.

Neil

[Neil Hodgson]

> 4) The platform interface should be extended to allow autocompletion item selection to be reported as an event. This should be implemented in a sufficiently generic way that adding more events in the future doesn’t cause a hard break in compatibility.

This was committed in
https://sourceforge.net/p/scintilla/code/ci/7e28cdba6d61e090ac6f6627c855ffd2603508e4/

The implementation was the same as proposed in the “Adding autocompletion selection change event in 4.0” message but the Cocoa platform code was simplified.

Platform implementors should be aware that this is a breaking change and will require (fairly simple) changes to platform implementations.

Original message:
https://groups.google.com/d/topic/scintilla-interest/IGfErwTzA8g/discussion

Neil

[Neil Hodgson]

>> 5) Mouse and keyboard event methods on Editor should take a single ‘int modifiers’ parameter for the modifier keys instead of multiple bool arguments. The old methods should be removed.
>
> Initial patch for this attached.

This has now been committed.
https://sourceforge.net/p/scintilla/code/ci/af83baede4309993b2ef0aa4f30968148855bc11/

A small change was made on Windows where PointFromLong became PointFromLParam to avoid casts.

This is another change that will require work by platform layer implementors.

Neil

[Neil Hodgson]

>> 6) Platform layers could all implement the "Fine Ticker" methods to simplify the code. Currently, Editor checks FineTickerAvailable and may fall back to SetTicking if they are not available. This code isn’t checked much as the in-repository platforms all implement Fine Ticker methods so the code may be decaying.
>
> Proposed patch for this attached.

Committed. Also removes Editor::Tick which was no longer called - its functionality is now in TickFor.
https://sourceforge.net/p/scintilla/code/ci/3a180dc6e9c81c3fa8230249601f63da62506b4d/

Platform layers which didn’t update to FineTicker* before now need to do so. This is the last of the changes to the platform layer interface currently being considered although more changes may be proposed. Next area to examine is the lexer interface.

Neil

[Neil Hodgson]

>> 7) Unused methods in the platform interface such as Window::SetTitle and Platform::DBCSCharLength should be removed.

> …
> An initial version of a patch implementing this is attached. It removes 15 methods from the Platform interface…

This has now been committed as
https://sourceforge.net/p/scintilla/code/ci/7030530a9a0f4fc1203a04378e83d82f0c35e7a0/

The Platform::MouseButtonBounce method was also removed. This was implemented very early on GTK+1.x/Linux where mouse button clicks sometimes caused multiple events.

These changes will break other platform layer implementations which will require changes to compile. Most of the changes will be simple removal of unused methods.

Neil

[Neil Hodgson]

> 8) Methods could be added to the ILexer interface to allow more metadata reporting to applications.

Here is a set of proposed methods to add to ILexer for 4.0 to allow retrieval of information on the styles supported by a lexer:

int MaximumNamedStyle();
const char *NameOfStyle(int style);
const char *TagsOfStyle(int style);
const char *DescriptionOfStyle(int style);

NameOfStyle returns the identifier from Scintilla.iface and SciLexer.h like “SCE_C_REGEX”. This is minimally useful but is the most scriptable so can be easily mechanically generated. Useful as a backup if the other methods aren’t implemented by a lexer (yet).

TagsOfStyle returns a string with multiple (mostly) predefined descriptive words. Like "literal quoted regex”. A set of known useful tags should be published and used when reasonable. A lexer defining a new type of lexical class could add a tag that is not in the known tag list and the list should then be updated. User interfaces could show these tags (directly or with some reformatting) and could translate them. Open questions include precise syntax (mixed case for tags? dots or spaces between tags?…), whether the order matters, and whether each style of a lexer should be unique (perhaps appending a number if otherwise non-unique).

DescriptionOfStyle returns a better explanation for people unfamiliar with a language. “IDL UUID like uuid(ba209999-0c6c-11d2-97cf-00c04f8eea45)”, “Archaic Hollerith format string”, or “Nested HereDoc”. These are the most explanatory but more difficult to parse for automatic visual assignment. Applications are less likely to localize these into different languages. Its also more work to fill these in - perhaps they should only be used for unusual cases with standard descriptions based on TagsOfStyle used for the common cases.

TagsOfStyle appears to be the most important method but its usefulness will depend on how well the list of known tags covers the set of all possible styles, hitting common styles with common tags but allowing fine differentiation when needed.

Some lexers (like cpp) provide a second copy of the named styles to show areas of the text that are active or inactive based on the preprocessor. Should TagsOfStyle report for the inactive styles and should there be an “inactive” tag added for inactive styles and empty or “active” for active styles. These styles don’t have an identifier in SciLexer.h so NameOfStyle doesn’t have anything interesting to return - it could append “_INACTIVE”.

Some lexers allow expanding a style into substyles, such as dividing identifiers into several sets. Should TagsOfStyle handle this? Perhaps it could be TagsOfStyle(primary style) + “ “ + substyle number.

Neil

[Neil Hodgson]

> 10) UTF-8 could be made the default encoding.

A proposed patch is attached - its mostly documentation changes with a single line of code changed.

Neil

[Neil Hodgson]

>> 10) UTF-8 could be made the default encoding.
>
> A proposed patch is attached - its mostly documentation changes with a single line of code changed.

Committed.
https://sourceforge.net/p/scintilla/code/ci/8d56eaef4f0a8807f083d8a3e0d5f5738d43bd20/

Neil

[kugel]

I wonder if incorperating a better GTK+ platform interface, like http://codebrainz.github.io/GtkScintilla/ would be a good idea for 4.0? The current interface is still hard to use, and a more natural way (for GTK+ application developers) is highly welcome.


Am Freitag, 2. Juni 2017 08:57:29 UTC+2 schrieb Neil Hodgson:

  4.0 is an opportunity to make some bigger changes. That could mean it will be longer before a 4.0 release or there could be multiple 4.0.x releases made which are considered plastic and a 4.1.0 release made once everything has solidified again.

   Areas that may change include the platform code, the interface between platform and core code, and the interface between lexers and core code.

1) The Cocoa platform code may move to using Automatic Reference Counting (ARC).

2) The GTK+ platform code may remove support for older versions of GTK+. I think retaining support only for GTK+ 2.24 and not for any other 2.x releases would be good.

3) The GTK+ platform code could start supporting GTK+ 3.9x/4.x. GTK+ appears to be moving from drawing to a Cairo context to creating a tree of render nodes which will be drawn on the GPU. Converting Scintilla to this approach looks like it could be a big job and may be an opportunity for someone interested in this area to redesign graphics output on GTK+. This may require changing how core Scintilla interfaces with drawing.

4) The platform interface should be extended to allow autocompletion item selection to be reported as an event. This should be implemented in a sufficiently generic way that adding more events in the future doesn’t cause a hard break in compatibility.

5) Mouse and keyboard event methods on Editor should take a single ‘int modifiers’ parameter for the modifier keys instead of multiple bool arguments. The old methods should be removed.

6) Platform layers could all implement the "Fine Ticker" methods to simplify the code. Currently, Editor checks FineTickerAvailable and may fall back to SetTicking if they are not available. This code isn’t checked much as the in-repository platforms all implement Fine Ticker methods so the code may be decaying.

7) Unused methods in the platform interface such as Window::SetTitle and Platform::DBCSCharLength should be removed. Platform::IsDBCSLeadByte should be moved into core where it can be shared - there is already a variant of this in Document. Platform methods Minimum, Maximum, and Clamp should be replaced with the equivalent C++ std:: standard library calls. More cleanups like these where possible.

8) Methods could be added to the ILexer interface to allow more metadata reporting to applications.

9) There are 2 ILexer interface versions and 2 IDocument interface versions. These could be collapsed into 1 interface each way.

10) UTF-8 could be made the default encoding.

   One change that I think is too big for now is to split the lexers from Scintilla into a separate project. This would simplify Scintilla’s version number story. Lexer updates often change behaviour in ways that can break current clients. For example, by dividing a lexical class in 2 such as splitting documentation comments off from comments. Or by changing folding to work on ‘(‘ and ‘)’. This can make it harder for projects to update to a new release - they may want some bug fixes but are scared off by the potential for unwanted styling changes. Its a long way from semver or other version numbering schemes. This could also make Scintilla a better candidate for packaging on Linux and may also allow one lexer library to be reused between GTK+, Qt, wxWidgets and Tk.

   Neil

[Neil Hodgson]

IDocument has been changed for 4.0. IDocumentWithLineEnd merged into IDocument and StartStyling no longer has a mask argument. The version ID is now dvRelease4.
https://sourceforge.net/p/scintilla/code/ci/3c87cf6d25993328cb189f504a1e523148402173/

The only likely client impact is in lexers outside the main distribution which may need to fix their calls to StartStyling.

Neil

[Neil Hodgson]

kugel:

> I wonder if incorperating a better GTK+ platform interface, like http://codebrainz.github.io/GtkScintilla/ would be a good idea for 4.0? The current interface is still hard to use, and a more natural way (for GTK+ application developers) is highly welcome.

Integrating Scintilla more closely into GTK+, following its conventions for APIs and signals would certainly help GTK+ applications.

To include something similar to this in (or as) the standard GTK+ distribution would require that the additions not significantly impact the maintenance of the standard distribution or add much to the work required for customization by downstream projects.

To minimize work, such an addition should handle adding APIs in a fully automated way. Downstream projects should be able to add and remove lexers easily. Requiring complex toolsets like autotools for standard builds would be unfortunate.

GtkScintilla mentions helper functions and language bindings. Other language bindings have, in the past, been maintenance sinks and depend on language popularity. I would (mostly) like to avoid them. Helper functions are often limited in applicability, only working well for those scenarios encountered by their author.

The current Scintilla MIT-style license should be retained which means that the referenced GtkScintilla project can not be merged since it is LGPL.

Neil

[Neil Hodgson]

As part of the 4.0 changes, lexing interfaces have changed to using 64-bit Sci_Position and Sci_PositionU on 64-bit builds. This has meant changes to derived types and extra casting. The baan, fortran, gui4cli, srec, ihex, tehex, latex, mysql, oscript, python, ruby, rust, and tex lexers were updated as was the SparseState class so watch out for bugs caused by the changes.

https://sourceforge.net/p/scintilla/code/ci/59732c65082d9940fc5211ed4deaec47e20c56bf/
https://sourceforge.net/p/scintilla/code/ci/df1416e3ff3af7692e466468b04c569d33fb7e2e/

Scintilla is still limited to 2GB documents - these change sets may allow changes to maximum document size in the future.

Neil