Scintilla 5.2.0 released

[Neil Hodgson]

Scintilla 5.2.0 is now available from the scintilla.org web site.

This is a major release that can layout text in parallel using threads to strongly improve performance for very wide lines. Implemented on macOS, Win32 using DirectWrite, and GTK on X and Wayland (not Win32). Only enabled once applications call SCI_SETLAYOUTTHREADS.

Scintilla 5.2.0 is compatible with Qt 6.

A C++17 compiler is required to build Scintilla. Microsoft Visual C++ 2019, GCC 7.3, Clang 6.0, Xcode 9.2 Clang or newer will work. Some slightly older compilers may still work.

A list of changes is available on the history page.
http://www.scintilla.org/ScintillaHistory.html

Scintilla uses Mercurial (Hg) for source code control. The repository can be cloned with
hg clone http://hg.code.sf.net/p/scintilla/code scintilla

Thanks to the contributors of code and documentation and to the testers.

Neil

[Mitchell]

Hi Neil,

On Wed, 9 Feb 2022 18:13:36 +1100
"'Neil Hodgson' via scintilla-interest" <scintilla...@googlegroups.com> wrote:

> Scintilla 5.2.0 is now available from the scintilla.org web site.

After updating to 5.2.0, my GTK 2.0-based application is getting assertion failures on ScintillaGTK:2762.cxx: PLATFORM_ASSERT(rgnUpdate == nullptr);

hg bisect blames this revision:

Changeset: 9065 (5e067a868567) Implement more unique_ptr allocation wrappers and place in new Wrappers.h header.

It appears that ScintillaGTK::ExposeTextThis() is re-entrant. Here's a simplified version of my call stack:

(see my note between #19 and #56)

#0 Scintilla::Internal::ScintillaGTK::ExposeTextThis(_GtkWidget*, _GdkEventExpose*) (this=0x555555ab4d40, ose=0x7fffffff74a0)
at scintilla/gtk/ScintillaGTK.cxx:2762
#1 0x00005555558573c5 in Scintilla::Internal::ScintillaGTK::ExposeText(_GtkWidget*, _GdkEventExpose*, Scintilla::Internal::ScintillaGTK*) (widget=0x555555aaab10, ose=0x7fffffff74a0, sciThis=0x555555ab4d40) at scintilla/gtk/ScintillaGTK.cxx:2791
[...]
#13 0x00007ffff7ad812d in gdk_window_process_updates () at /usr/lib/x86_64-linux-gnu/libgdk-x11-2.0.so.0
#14 0x000055555585194a in Scintilla::Internal::ScintillaGTK::ScrollText(long) (this=0x555555ea65c0, linesToMove=-3)
at scintilla/gtk/ScintillaGTK.cxx:1084
#15 0x0000555555799908 in Scintilla::Internal::Editor::ScrollTo(long, bool) (this=0x555555ea65c0, line=3, moveThumb=true)
at scintilla/src/Editor.cxx:974
#16 0x00005555557b3d9b in Scintilla::Internal::Editor::WndProc(Scintilla::Message, unsigned long, long)
(this=0x555555ea65c0, iMessage=Scintilla::Message::SetFirstVisibleLine, wParam=3, lParam=0) at scintilla/src/Editor.cxx:5993
#17 0x00005555558191b9 in Scintilla::Internal::ScintillaBase::WndProc(Scintilla::Message, unsigned long, long)
(this=0x555555ea65c0, iMessage=Scintilla::Message::SetFirstVisibleLine, wParam=3, lParam=0)
at scintilla/src/ScintillaBase.cxx:1130
#18 0x00005555558512d5 in Scintilla::Internal::ScintillaGTK::WndProc(Scintilla::Message, unsigned long, long)
(this=0x555555ea65c0, iMessage=Scintilla::Message::SetFirstVisibleLine, wParam=3, lParam=0) at scintilla/gtk/ScintillaGTK.cxx:935
#19 0x0000555555858518 in scintilla_send_message(ScintillaObject*, unsigned int, uptr_t, sptr_t)
(sci=0x555555cc3220, iMessage=2613, wParam=3, lParam=0) at scintilla/gtk/ScintillaGTK.cxx:3111
[... my application handles the Scintilla notification and calls back into Scintilla ...]
#56 0x0000555555860771 in scintilla_marshal_VOID__INT_BOXED
(closure=0x555555ad2210, return_value=0x0, n_param_values=3, param_values=0x7fffffffa840, invocation_hint=0x7fffffffa7c0, marshal_data=0x0) at scintilla/gtk/scintilla-marshal.c:117
#57 0x00007ffff7690802 in g_closure_invoke () at /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0
#58 0x00007ffff76a4814 in () at /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0
#59 0x00007ffff76afbbe in g_signal_emit_valist () at /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0
#60 0x00007ffff76b00f3 in g_signal_emit () at /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0
#61 0x00005555558520a9 in Scintilla::Internal::ScintillaGTK::NotifyParent(Scintilla::NotificationData) (this=0x555555ab4d40, scn=...)
at scintilla/gtk/ScintillaGTK.cxx:1163
#62 0x00005555557a1a64 in Scintilla::Internal::Editor::NotifyUpdateUI() (this=0x555555ab4d40) at scintilla/src/Editor.cxx:2441
#63 0x000055555579d4e6 in Scintilla::Internal::Editor::Paint(Scintilla::Internal::Surface*, Scintilla::Internal::PRectangle)
(this=0x555555ab4d40, surfaceWindow=0x555555c801d0, rcArea=...) at scintilla/src/Editor.cxx:1760
#64 0x0000555555857291 in Scintilla::Internal::ScintillaGTK::ExposeTextThis(_GtkWidget*, _GdkEventExpose*)
(this=0x555555ab4d40, ose=0x7fffffffb370) at scintilla/gtk/ScintillaGTK.cxx:2770
#65 0x00005555558573c5 in Scintilla::Internal::ScintillaGTK::ExposeText(_GtkWidget*, _GdkEventExpose*, Scintilla::Internal::ScintillaGTK*) (widget=0x555555aaab10, ose=0x7fffffffb370, sciThis=0x555555ab4d40) at scintilla/gtk/ScintillaGTK.cxx:2791

Do you have any idea what could be going wrong here and why that change is to blame?

Cheers,
Mitchell

[Neil Hodgson]

Mitchell:

> After updating to 5.2.0, my GTK 2.0-based application is getting assertion failures on ScintillaGTK:2762.cxx: PLATFORM_ASSERT(rgnUpdate == nullptr);
>
> hg bisect blames this revision:
>
> Changeset: 9065 (5e067a868567) Implement more unique_ptr allocation wrappers and place in new Wrappers.h header.

I can’t see that change causing the problem. I had considered using unique_ptr for rgnUpdate but didn’t because its a more complex situation due to being different types - a GdkRegion on GTK 2 and cairo_rectangle_list on GTK 3.

> It appears that ScintillaGTK::ExposeTextThis() is re-entrant.

The same pattern of reentrance should have happened before if the app is calling SetFirstVisibleLine with effect (not as a no-op to the same line). I hadn’t considered reentrance here and the correct behaviour to handle reentry would be to save the rgnUpdate into a local at the start of ExposeTextThis and restore it at the end.

Neil

[Mitchell]

Hi Neil,

On Tue, 15 Feb 2022 16:33:20 +1100

"'Neil Hodgson' via scintilla-interest" <scintilla...@googlegroups.com> wrote:

See attached patch against 5.2.0 for this correct behavior.

Cheers,
Mitchell

[Neil Hodgson]

Mitchell:

See attached patch against 5.2.0 for this correct behavior.

   Committed as

https://sourceforge.net/p/scintilla/code/ci/58bb44ceae61a61b3ccf8e570cba2a74456c2b18/

   Neil

[Mitchell]

Hi Neil,

On Wed, 9 Feb 2022 18:13:36 +1100

"'Neil Hodgson' via scintilla-interest" <scintilla...@googlegroups.com> wrote:

> Scintilla 5.2.0 is now available from the scintilla.org web site.
>
> This is a major release that can layout text in parallel using threads to strongly improve performance for very wide lines. Implemented on macOS, Win32 using DirectWrite, and GTK on X and Wayland (not Win32). Only enabled once applications call SCI_SETLAYOUTTHREADS.
>
> Scintilla 5.2.0 is compatible with Qt 6.
>
> A C++17 compiler is required to build Scintilla. Microsoft Visual C++ 2019, GCC 7.3, Clang 6.0, Xcode 9.2 Clang or newer will work. Some slightly older compilers may still work.

I am no longer able to build with MinGW-w64 (cross-compile for Windows from Linux). I am getting an "error: 'std::thread' has not been declared" error in EditView.cxx. After some research among a bunch of random threads online, it looks like std::thread will only work with MinGW-w64 if it has been compiled with POSIX threading (as opposed to native win32 threading). This is probably not very desirable. If I compile my (GTK) application using MinGW-w64 with POSIX threads, it compiles, but Scintilla views are completely white. I don't know the cause of this and I didn't want to spend too much time going down a rabbit hole.

Maybe being able to #ifdef out the new threading capabilities would be desirable?

Cheers,
Mitchell

[Neil Hodgson]

Mitchell:

I am no longer able to build with MinGW-w64 (cross-compile for Windows from Linux).

   MinGW-w64 9.2.0 works on Windows 10 and does run multithreaded. I don’t know which version of threads that has but its from a basic install without installing any extra thread library.

   Reinstalled from https://nuwen.net/mingw.html which has g++ 11.2 and it multithreads too.

I am getting an "error: 'std::thread' has not been declared" error in EditView.cxx.

   On Fedora 35, with mingw64-gcc-c++, compiling works but linking fails with ‘cannot find -lpthread’. 

   The only direct use of std::thread is std::thread::hardware_concurrency() which could be #ifed to 1.

   #ifing all the threading and locking is going to make it much more difficult to implement features.

   Neil

[Mitchell]

Hi Neil,

On Wed, 16 Feb 2022 17:13:57 +1100

"'Neil Hodgson' via scintilla-interest" <scintilla...@googlegroups.com> wrote:

> Mitchell:
>
> > I am no longer able to build with MinGW-w64 (cross-compile for Windows from Linux).
>
> MinGW-w64 9.2.0 works on Windows 10 and does run multithreaded. I don’t know which version of threads that has but its from a basic install without installing any extra thread library.
>

> Reinstalled from https://nuwen.net/mingw.html <https://nuwen.net/mingw.html> which has g++ 11.2 and it multithreads too.

>
> > I am getting an "error: 'std::thread' has not been declared" error in EditView.cxx.
>
> On Fedora 35, with mingw64-gcc-c++, compiling works but linking fails with ‘cannot find -lpthread’.

Okay, I am building with MinGW-w64 7.3 on an Ubuntu 18.04 LTS image. Maybe a more recent version of MinGW has the stuff I need.

> The only direct use of std::thread is std::thread::hardware_concurrency() which could be #ifed to 1.

There is another thing, std::future<void>, that my MinGW is complaining about, so a simple #if would not work here.

> #ifing all the threading and locking is going to make it much more difficult to implement features.

I understand.

I tried building my application with POSIX-threaded-MinGW with pre-Scintilla 5.2.0. It builds and runs fine. However, switching back to Scintilla 5.2.0 gives me blank Scintilla views. Something's changed. I'll have to dig further, but it will take time.

Cheers,
Mitchell

[Neil Hodgson]

Mitchell:

> There is another thing, std::future<void>, that my MinGW is complaining about, so a simple #if would not work here.

There are three std:: elements needed for the multithreading feature: atomic, mutex, and async. Any modification for missing features should identify which of these is unavailable to minimize the changes.

async is defined in <future> which includes future. async is the core that may spin up threads although an implementation of async could avoid threads by only providing launch::deferred (which EditView uses when threads==1).

If async is unavailable then the main if (…threads…) loop would be #ifed.

Neil

[Mitchell]

Hi Neil,

On Wed, 16 Feb 2022 08:57:11 -0500
Mitchell <li...@triplequasar.com> wrote:

> Hi Neil,
>
> On Wed, 16 Feb 2022 17:13:57 +1100
> "'Neil Hodgson' via scintilla-interest" <scintilla...@googlegroups.com> wrote:
>
> > Mitchell:
> >
> > > I am no longer able to build with MinGW-w64 (cross-compile for Windows from Linux).
> >
> > MinGW-w64 9.2.0 works on Windows 10 and does run multithreaded. I don’t know which version of threads that has but its from a basic install without installing any extra thread library.
> >
> > Reinstalled from https://nuwen.net/mingw.html <https://nuwen.net/mingw.html> which has g++ 11.2 and it multithreads too.
> >
> > > I am getting an "error: 'std::thread' has not been declared" error in EditView.cxx.
> >
> > On Fedora 35, with mingw64-gcc-c++, compiling works but linking fails with ‘cannot find -lpthread’.
>
> Okay, I am building with MinGW-w64 7.3 on an Ubuntu 18.04 LTS image. Maybe a more recent version of MinGW has the stuff I need.

I tried updating my toolchain to Ubuntu 20.04's MinGW 7.0.0 and GCC 9.3.0, but cross-compiling for Windows from Linux still fails with the same errors about std::mutex not naming a type, std::thread not being declared, and std::future<void> being an incomplete type.

Cheers,
Mitchell

[Neil Hodgson]

Mitchell:

> I tried updating my toolchain to Ubuntu 20.04's MinGW 7.0.0 and GCC 9.3.0, but cross-compiling for Windows from Linux still fails with the same errors about std::mutex not naming a type, std::thread not being declared, and std::future<void> being an incomplete type.

Attached is a patch that implements a SCI_DISABLE_THREADS pre-processor symbol. Its not tested much and should be minimized - if headers are available then they should be outside the #if. Particularly atomic although its often included by other std headers. Its unlikely I (or others) will be testing any configurations without threading so it would be the responsibility of people that want to disable threads to maintain the non-threaded implementation.

Multiple-core processors are now included in the cheapest systems and threads are a central part of current platforms. Scintilla will increasingly use threads to improve performance and this will spread through the code. Even modules that do not start or control threads may see atomic and mutex inside data structures and function declarations.

It may be reasonable to wrap std::mutex in a class that has a null implementation choice. Something similar was investigated during the threading work to use std::shared_mutex on platforms where that is available as it performs better. shared_mutex isn’t available on older operating systems - it was implemented for Win32 in Vista (2007) and macOS 10.12 (2016). atomic may not be worth wrapping as that may stop the compiler seeing through the code to inline and optimize.

Neil

[Mitchell]

Hi Neil,

On Sat, 26 Feb 2022 09:50:57 +1100

"'Neil Hodgson' via scintilla-interest" <scintilla...@googlegroups.com> wrote:

Thanks for your efforts. Despite not being able to compile with MinGW, I would NOT encourage you to commit this patch. As you said, threading is quite common and Scintilla should benefit from it. It's on the onus of developers (i.e. me) to figure out why Scintilla is not compiling with threads on a particular toolchain. I think was hoping more that you had an idea of what could be wrong than to actually patch it out of Scintilla.

Cheers,
Mitchell

[Neil Hodgson]

Mitchell:

t… the onus of developers (i.e. me) to figure out why Scintilla is not compiling with threads on a particular toolchain. I think was hoping more that you had an idea of what could be wrong than to actually patch it out of Scintilla.

   Another approach would be for you to see if the threading works when building with MSYS2 then use a GitHub action to build binaries on a GitHub Win32 server.

   Neil

[Mitchell]

Hi,

I have resolved the compiler issue and am writing in case anyone else experiences it too.

It appears Debian-based distributions provide two versions of the MinGW
compilers: those with native Win32 threads and those with POSIX threads
(pthreads). The native thread version (which is the default) simply
does not have a C++11 threads implementation, which is why I was
getting those compiler errors. There is a third-party project[1] that
provides C++ headers that work with the native thread version of MinGW.
Scintilla compiles with that compiler version and those headers (albeit
with some extra #includes in Scintilla code). I haven't tested if it
runs though. If you don't want to add an external dependency, you can
compile Scintilla using the POSIX thread version of MinGW. You will
need to either distribute a pthread dll with your application or
statically link in libstdc++. In my case, I distribute my application
as a Win32 Gtk application and my Gtk runtime comes with a
libwinpthread-1.dll which may be sufficient. (I also static link
libstdc++ so I'm honestly not sure what's strictly required.)

[1]: https://github.com/meganz/mingw-std-threads

Cheers,
Mitchell