Qt + Tcl in the different threads

[Alex (Oleksandr) Malyushytskyy]

We are trying to develop TCL based application which allows to display and hide Qt GUI using tcl commands while mostly always providing user interaction through terminal.
Coming from Qt world I also want the Gui to be responsive and that pushed me to solution having Qt and TCL in the different threads.
The task of starting Qt and Tcl was trivial. I created instance of QApplications,
And implemented run command in QThread subclass:

void BackendClient::run()
{
{
QMutexLocker locker(&mutex);
// Initialization of IGuiNotifier
....;
}

Tcl_Main(m_argc, m_argv, Tcl_AppInit);
}

But now I need to implement communication between the backend and gui threads.

1. The backend-> gui is relatively easy (or at least I think it is right now). I am planning to have implement IGuiNotifier interface which is going to be QObject subclass, make it available to the our base class Command implementation and each command can simply call the notify function of this object , which will use queed blocking connection to notify gui (setState for example).
2. The gui - backend is unclear to me. Ideally it should be post message type communication, but I am not sure how to implement it.
Problem is in the fact that tcl thread runs its own the event loop right now.
So the only approach which comes to my mind is to inject in the loop mutex and use it as a guard to force tcl event loop to create and execute the command.

But https://www.tcl.tk/doc/howto/thread_model.html says "Tcl scripts in different threads can communicate by posting scripts onto the event queue of an interpreter in a different thread. This can be synchronous, where you wait for the result, or asynchronous, where your thread does not wait for the other thread to evaluate its script." Unfortunately I can't find example how to post that event, can it be done from non-tcl related thread?
Also since I am completely new to tcl is it viable solution? Is there better pattern?

Best regards,
Alex

[Donal K. Fellows]

On Friday, 1 October 2021 at 01:58:50 UTC+1, al...@flex-logix.com wrote:
> We are trying to develop TCL based application which allows to display and hide Qt GUI using tcl commands while mostly always providing user interaction through terminal.
> Coming from Qt world I also want the Gui to be responsive and that pushed me to solution having Qt and TCL in the different threads.
> The task of starting Qt and Tcl was trivial. I created instance of QApplications,
> And implemented run command in QThread subclass:
>
> void BackendClient::run()
> {
> {
> QMutexLocker locker(&mutex);
> // Initialization of IGuiNotifier
> ....;
> }
>
> Tcl_Main(m_argc, m_argv, Tcl_AppInit);
> }
>
> But now I need to implement communication between the backend and gui threads.
>
> 1. The backend-> gui is relatively easy (or at least I think it is right now). I am planning to have implement IGuiNotifier interface which is going to be QObject subclass, make it available to the our base class Command implementation and each command can simply call the notify function of this object , which will use queed blocking connection to notify gui (setState for example).
> 2. The gui - backend is unclear to me. Ideally it should be post message type communication, but I am not sure how to implement it.
> Problem is in the fact that tcl thread runs its own the event loop right now.
> So the only approach which comes to my mind is to inject in the loop mutex and use it as a guard to force tcl event loop to create and execute the command.

Providing you don't have to pass too much data about (you know your application much better than I do!) then you can use a pipe to pass messages from one thread to another. (You'd probably want a second pipe to send things in the other direction, and to make sure that access to them on at least the Tcl side is non-blocking.)

Pipes are pretty easy, and are supported by all common operating systems.

It's also possible to transfer other IO channels between threads, and the reflected channels of Tcl 8.5 are set up to hide the communication details. You could then use the basic Tcl IO C API straight from the Qt code without the complexity of pipes and yet have things end up correctly in Tcl in another thread. (Assuming the Tcl thread is in the event loop.)

> But https://www.tcl.tk/doc/howto/thread_model.html says "Tcl scripts in different threads can communicate by posting scripts onto the event queue of an interpreter in a different thread. This can be synchronous, where you wait for the result, or asynchronous, where your thread does not wait for the other thread to evaluate its script." Unfortunately I can't find example how to post that event, can it be done from non-tcl related thread?
> Also since I am completely new to tcl is it viable solution? Is there better pattern?

Otherwise, the easiest way to send messages is to have a little helper interpreter IN THE QT THREAD that does the details of the inter-thread communications (using the implementation provided in the thread package). Yes, you can do the messaging at the low level as well — at that level, messages are C structures allocated using the system memory allocator so that they can be allocated in one thread and freed in another — but the details are fairly complicated, especially for bidirectional communications. It's better to reuse the existing code in the thread package or the reflected channel implementation!

You're quite likely going to end up with a small Tcl interpreter in the Qt thread. It doesn't really need to do all that much after things are set up.

Donal.

[briang]

If you do want to pursue having a Tcl interpreter in the Qt thread, seriously consider this:
https://ssl.webpack.de/www.eurotcl.eu/presentations/EuroTcl2019-Griffin-TclQtEventLoop.pdf

(from EuroTcl 2019: https://ssl.webpack.de/www.eurotcl.eu/pastevents.html#eurotcl2019)

-Brian

[Alex (Oleksandr) Malyushytskyy]

Alex (Oleksandr) Malyushytskyy
8:41 PM (19 minutes ago)
to comp.lang.tcl

Thanks Donal and Brian for input,

>> Yes, you can do the messaging at the low level as well — at that level, messages are C structures allocated using the system memory allocator so that they can be allocated in one thread and freed in another — but the details are fairly complicated, especially for bidirectional communications.

I was going to implement the backend->gui communication using Qt utilities - signals and slots with queued connections. This is partially done already so the backend gets a Qt object subclass pointer which implements the custom interface every command should when changing UI related data.
I could also use the same mechanism for TCL-> gui communication, but for that I need to run the Qt event loop in the thread I call TCL_MAIN.
That gets a bit complicated since this is not even the QT main thread.
I would not throw away the idea of posting message and would greatly appreciate examples for creating and sending messages with C API.

>> Pipes are pretty easy, and are supported by all common operating systems.

I have to say I had no much experience with pipes, but as far as I know there is no cross-platform implementation and what is more important using pipes will require the call to read from pipe in the event loop and event loop basically will have to non-stop reading from the pipe even though nothing is sent most of the time.
I might be wrong, but if I am right, I see no advantages using pipe over adding mutex in TCL standard event loop and just add the list of strings/commands gui wants the main loop to execute (before processing the other events)

>>It's also possible to transfer other IO channels between threads, and the reflected channels of Tcl 8.5 are set up to hide the communication details. You could then use the basic Tcl IO C API straight from the Qt code without the complexity of pipes and yet have things end up correctly in Tcl in another thread. (Assuming the Tcl thread is in the event loop.)

I have to admit I fail to understand how redirection/reflected channels can help me to pass the data from Qt/Gui thread to TCL. (Basically gui should pass strings to TCL ). But I failed to find a C API example using the feature, and do not think I understand it completely, so I will try to look at it tomorrow.

>> Otherwise, the easiest way to send messages is to have a little helper interpreter IN THE QT THREAD that does the details of the inter-thread communications (using the implementation provided in the thread package).

1. You are mentioning a helper interpreter IN THE QT THREAD, but how to have the interpreter help me to communicate?

2. Does having an interpreter in the main thread means that QT EVENT LOOP will have to be merged (like Brian says?)
2.It seems that the recommendation is using TCL in 2 threads. Main thread is gui thread , tcl is used there solely for communication, the second one will be for user input/output in the terminal (no Qt), It also will execute all the tcl commands.
But I can't find an example on how to use "the implementation provided in the thread package" on a C api level, is there anything available I am missing?


Thanks and best regards!
Alex

[briang]

I think it's a mistake to call Tcl_Main() from within a Qt program. You have to decide if this is a Qt application or a Tcl application. Choose one, then use the other toolkit as a dependent subsystem. What does this mean? In the case of a Qt application, create a Tcl interpreter, and initialize it according to the needs of your Qt application. This is different from calling Tcl_Main(), because Tcl_Main() basically assumes "ownership" of everything. Once you have the interpreter, you can call Tcl_Eval*() from Qt (i.e., your C++ code) to run Tcl commands, scripts, etc. Also note that there should only be 1 Tcl interpreter running in a thread, and calls to the interp should only be done from the same thread.

> >> Pipes are pretty easy, and are supported by all common operating systems.
> I have to say I had no much experience with pipes, but as far as I know there is no cross-platform implementation and what is more important using pipes will require the call to read from pipe in the event loop and event loop basically will have to non-stop reading from the pipe even though nothing is sent most of the time.
> I might be wrong, but if I am right, I see no advantages using pipe over adding mutex in TCL standard event loop and just add the list of strings/commands gui wants the main loop to execute (before processing the other events)

Actually, both Tcl and Qt provide cross-platform abstractions to pipes, files, and sockets. Using the Tcl or Qt API's for open/close/read/write operations should make life easier.

Anonymous pipes is how Tcl threads communicate between Tcl interps running in different threads. It's how thread::send sends messages from one thread to another at the Tcl scripting level. Of course I am over simplifying the underlying implementation :)

> >>It's also possible to transfer other IO channels between threads, and the reflected channels of Tcl 8.5 are set up to hide the communication details. You could then use the basic Tcl IO C API straight from the Qt code without the complexity of pipes and yet have things end up correctly in Tcl in another thread. (Assuming the Tcl thread is in the event loop.)
> I have to admit I fail to understand how redirection/reflected channels can help me to pass the data from Qt/Gui thread to TCL. (Basically gui should pass strings to TCL ). But I failed to find a C API example using the feature, and do not think I understand it completely, so I will try to look at it tomorrow.
> >> Otherwise, the easiest way to send messages is to have a little helper interpreter IN THE QT THREAD that does the details of the inter-thread communications (using the implementation provided in the thread package).
> 1. You are mentioning a helper interpreter IN THE QT THREAD, but how to have the interpreter help me to communicate?

Create a Tcl interp in your "Tcl" thread. Create a helper Tcl Interp in your main Qt thread. From Qt code, call Tcl_Eval*() with a script that will take in a message string and send it to the other Tcl interp using the thread::send or similar thread supported communication mechanisms. The "Tcl" thread interp can reply using the same thread::send back to the Qt tcl interp. In the Qt tcl interp, define a new C implemented command that takes the string response and pass it back to whatever Qt object needs the response.

The tricky part is that the Qt tcl interp won't have a running event loop. It will work for simple cases with carefully placed "update" calls, but this is very fragile.

>
> 2. Does having an interpreter in the main thread means that QT EVENT LOOP will have to be merged (like Brian says?)
> 2.It seems that the recommendation is using TCL in 2 threads. Main thread is gui thread , tcl is used there solely for communication, the second one will be for user input/output in the terminal (no Qt), It also will execute all the tcl commands.

The beauty of merging the Tcl & Qt event loop is that you don't really need 2 threads to handle tcl scripting in the Qt application, unless you want to for some other reason. The event loop merge is actually just the Qt event loop. It simply replaces the Tcl native event loop details with Qt event loop details. The terminal input/output simply becomes another Qt event that Qt response to, and it just means that Qt ends up calling the Tcl interp to handle the specific event tied to the Tcl interpreter.

> But I can't find an example on how to use "the implementation provided in the thread package" on a C api level, is there anything available I am missing?

I'm not sure. I suspect that assuming Threads are required in order to integrate Tcl into a Qt application is misguided. Threads are often used in places that aren't necessary.

For any GUI application, the event loop is critical, independent of the presence of threads. Integrate the event loop, then just use Tcl as you would any other C function.

[Alex (Oleksandr) Malyushytskyy]

Thanks Bryan,

>> I'm not sure. I suspect that assuming Threads are required in order to integrate Tcl into a Qt application is misguided. Threads are often used in places that aren't necessary.
For any GUI application, the event loop is critical, independent of the presence of threads. Integrate the event loop, then just use Tcl as you would any other C function.

Using threads are inspired not by the need of integration.
The backend will run long time processing tasks.
Having such tasks in the thread is the only real option to keep gui responsive.
The only alternative of periodically from the code process events causing a lot of issues, is not reliable and impossible to implement without putting a ton of calls into unrelated code.
The thread is a need , not a desire,

>> I think it's a mistake to call Tcl_Main() from within a Qt program. You have to decide if this is a Qt application or a Tcl application. Choose one, then use the other toolkit as a dependent subsystem. What does this mean? In the case of a Qt application, create a Tcl interpreter, and initialize it according to the needs of your Qt application. This is different from calling Tcl_Main(), because Tcl_Main() basically assumes "ownership" of everything. Once you have the interpreter, you can call Tcl_Eval*() from Qt (i.e., your C++ code) to run Tcl commands, scripts, etc. Also note that there should only be 1 Tcl interpreter running in a thread, and calls to the interp should only be done from the same thread.

For our application in a regular workflow GUI is an important , but a "secondary" function.
Users may choose to show it or not from tcl.
In fact it has to run on Linux without X present and the user may not be able to start gui.
So it is natural to say that TCL should be the main.
But if we want to show gui and immediately face the problem: QWidgets have to be in the main thread.
So if the system is capable of showing gui, we need to put Qt in the main thread.
Do we want tcl in the same with GUI thread? No, because 90% of the commands are long running.

>> I think it's a mistake to call Tcl_Main() from within a Qt program.

Do you see any specific problem?

As far as I understand there are only 3 problems to solve:
1. Make tcl and Qt run in different threads (done)
2. TCL to GUI notifications ( see no problem)
Backend has a pointer to interface and QObject class implementing it sends signals connected with queued blocking connection.
So every command calling it will wait for gui to process it.
3. Gui to tcl notifications (this is what I try to solve now):
Gui has to post the text commands to the tcl thread.
Gui accesses the data to be shown through implemented by the backend interface.
Access to such data is controlled by the state of the document,
which every tcl command changing the state should set, but there is no knowledge about tcl in the gui and vice versa.

Do you see any pitfalls in such a design?
Otherwise why is it a mistake?

Best regards,
Alex

[briang]

On Monday, October 4, 2021 at 9:19:17 AM UTC-7, al...@flex-logix.com wrote:
> Thanks Bryan,
> >> I'm not sure. I suspect that assuming Threads are required in order to integrate Tcl into a Qt application is misguided. Threads are often used in places that aren't necessary.
> For any GUI application, the event loop is critical, independent of the presence of threads. Integrate the event loop, then just use Tcl as you would any other C function.
> Using threads are inspired not by the need of integration.
> The backend will run long time processing tasks.
> Having such tasks in the thread is the only real option to keep gui responsive.
> The only alternative of periodically from the code process events causing a lot of issues, is not reliable and impossible to implement without putting a ton of calls into unrelated code.
> The thread is a need , not a desire,
> >> I think it's a mistake to call Tcl_Main() from within a Qt program. You have to decide if this is a Qt application or a Tcl application. Choose one, then use the other toolkit as a dependent subsystem. What does this mean? In the case of a Qt application, create a Tcl interpreter, and initialize it according to the needs of your Qt application. This is different from calling Tcl_Main(), because Tcl_Main() basically assumes "ownership" of everything. Once you have the interpreter, you can call Tcl_Eval*() from Qt (i.e., your C++ code) to run Tcl commands, scripts, etc. Also note that there should only be 1 Tcl interpreter running in a thread, and calls to the interp should only be done from the same thread.
> For our application in a regular workflow GUI is an important , but a "secondary" function.
> Users may choose to show it or not from tcl.
> In fact it has to run on Linux without X present and the user may not be able to start gui.
> So it is natural to say that TCL should be the main.
> But if we want to show gui and immediately face the problem: QWidgets have to be in the main thread.
> So if the system is capable of showing gui, we need to put Qt in the main thread.
> Do we want tcl in the same with GUI thread? No, because 90% of the commands are long running.
> >> I think it's a mistake to call Tcl_Main() from within a Qt program.
> Do you see any specific problem?

Ok, so in your case, using Tcl_Main() is appropriate since this is fundamentally a Tcl application with an optional Qt GUI.
I don't see any specific problem so far. The main thread runs Tcl with a command line user interface, and an optional graphical user interface, both can be active at the same time assuming an integrated event loop. The business logic end of the application runs in a separate thread in all cases.

>
> As far as I understand there are only 3 problems to solve:
> 1. Make tcl and Qt run in different threads (done)
> 2. TCL to GUI notifications ( see no problem)
> Backend has a pointer to interface and QObject class implementing it sends signals connected with queued blocking connection.
> So every command calling it will wait for gui to process it.
> 3. Gui to tcl notifications (this is what I try to solve now):
> Gui has to post the text commands to the tcl thread.
> Gui accesses the data to be shown through implemented by the backend interface.
> Access to such data is controlled by the state of the document,
> which every tcl command changing the state should set, but there is no knowledge about tcl in the gui and vice versa.

A workable solution that I can imagine would be:
1) Main thread has a Tcl interp (call it Tcl-0). Tcl-0 manages the command line interface on the terminal.
2) A separate thread is created for the Document operations, where all the hard work is done. In this thread another Tcl interpreter is created (call it Tcl-Doc).
3) Tcl-0 can then communicate with the Document thread using Tcl's thread commands. User commands typed into the terminal would be read by Tcl-0, which, in turn, would pass the operation onto Tcl-Doc and wait for a response.
4) When the Qt GUI is created, Qt signals/slots can be used, or, the Qt code can call Tcl_Eval(Tcl-0, "string command") in the main thread, and in turn, Tcl-0 will forward the operation to Tcl-Doc, if that is required by the operation.
5) Alternatively, Qt can send a signal to the Document thread, to a specially designed object that will take a "string" argument, and turn around and pass the "string" to the Tcl-Doc interpreter via Tcl_Eval().

The bits you have to invent is the set of custom Tcl commands that interface with the document business logic. A subset of these are for direct Qt/Tcl interactions, and the other set are for User command-line interactions. And there's likely some overlap between the two.
There's also no restriction in having a request come through the Tcl command chain, and having the Document logic reply via a Qt signal back to the GUI.

>
> Do you see any pitfalls in such a design?

The pitfall in the solution I outlined above is it will work well with an integrated event loop, otherwise, all the Tcl operations will have to be restricted to non-event based operations, treating Tcl as purely functional sequential code. Any attempt to use any event operations will lead to bad behavior or deadlocks.

> Otherwise why is it a mistake?

My original assumption was that this is a Qt application, i.e., Qt is the overall top controller for the application, so using Tcl_Main() would be inappropriate in this scenario, in my opinion. But my assumption was wrong, sorry.

-Brian

>
> Best regards,
> Alex