Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

Single step NRE / green thread

98 views
Skip to first unread message

Sebastian Biały

unread,
Nov 20, 2018, 4:39:33 AM11/20/18
to
Hello.

I'm trying to execute NRE Tcl using some sort of functions to single
step. I need this to implement "green thread" or "fiber" concept that
allows me to interleave tcl execution flow with other green threads in
same real thread.

I was able to setup interpreter to allow me to properly execute NRE
version of tcl, however there is an obstacle.

First of all, I need to use tclInt.h because stuff I need is internal.
Suprising since part of api is public in Tcl_*.

Secondly, TclNRRunCallbacks( ... ); does while(everything) loop by its
own. There is no "single step" version that I know that can pull
*single* NRE callback, execute it and return and I can control when next
single step takes place.

I can probably write by own version of TclNRRunCallbacks without loop,
seems to be trivial. But on other hand this not seems to be right due to
heavy use of tcl internals. Why there is loop inside TclNRRunCallbacks
that prevents me to control execution of NRE code? Is there other
official way to run single step using Tcl_* api or at least TclInt.h?

There is simple example that I wrote just to play around with this idea:

#include "tcl.h"
#include "tclInt.h"

#include <string>
#include <iostream>
#include <assert.h>

int main()
{
auto interp = Tcl_CreateInterp();

Interp *iPtr = ( Interp * ) interp;
auto topcb = TOP_CB( interp );

std::string command( "set i 1\nwhile {$i <= 5} { puts $i\nincr i }" );
auto script = Tcl_NewStringObj( command.c_str(), command.size() );
auto result = Tcl_NREvalObj( interp, script, 0 );

assert( result == TCL_OK );

// I do not want this!
result = TclNRRunCallbacks( interp, 0, topcb );

// I want something like this
//while ( !interp->emptyStack() )
//{
// Tcl_NRSingleStep( ... )
// ...my own green thread code here ...
//}

std::cout << Tcl_GetStringResult( interp ) << std::endl;
assert( result == TCL_OK );

Tcl_DeleteInterp( interp );
return 0;
}

Sebastian Biały

unread,
Nov 20, 2018, 5:39:03 AM11/20/18
to
On 2018-11-20 10:39, Sebastian Biały wrote:
> I can probably write by own version of TclNRRunCallbacks without loop,

Here it is. Not perfect, just to illustrate how it can be done in,
probably, wrong way. Still, no luck with linking some internal stuff:

#include "tcl.h"
#include "tclInt.h"

#include <string>
#include <iostream>
#include <assert.h>

class TclGreenThread
{
public:
TclGreenThread( Tcl_Interp *_interp );

bool next();

private:
Interp *m_interp;
Tcl_Interp *m_tcl_interp;
};

TclGreenThread::TclGreenThread( Tcl_Interp *_interp )
: m_interp( (Interp *) _interp )
, m_tcl_interp( _interp )
{
if ( *( m_interp->result ) != 0 )
{
( void ) Tcl_GetObjResult( _interp );
}
}

bool
TclGreenThread::next()
{
int result = 0;

auto callbackPtr = TOP_CB( m_interp );
if ( callbackPtr == nullptr )
return false;

auto procPtr = callbackPtr->procPtr;
TOP_CB( m_interp ) = callbackPtr->nextPtr;
result = procPtr( callbackPtr->data, m_tcl_interp, result );

// error LNK2001: unresolved external symbol _tclFreeObjList
// TCLNR_FREE( m_interp, callbackPtr );

return true;
}

int main()
{
auto interp = Tcl_CreateInterp();

Interp *iPtr = ( Interp * ) interp;
auto topcb = TOP_CB( interp );

std::string command( "set i 1\nwhile {$i <= 5} { puts $i\nincr i }" );
auto script = Tcl_NewStringObj( command.c_str(), command.size() );
auto result = Tcl_NREvalObj( interp, script, 0 );

assert( result == TCL_OK );

TclGreenThread thread( interp );

while ( thread.next() )
;

Gerald Lester

unread,
Nov 20, 2018, 9:28:20 AM11/20/18
to
On 11/20/2018 04:39 AM, Sebastian Biały wrote:
> On 2018-11-20 10:39, Sebastian Biały wrote:
>> I can probably write by own version of TclNRRunCallbacks without loop,
>
> Here it is. Not perfect, just to illustrate how it can be done in,
> probably, wrong way. Still, no luck with linking some internal stuff:
>...

You may want to create a TIP and have this as a proff of concept
implementation.


--
+----------------------------------------------------------------------+
| Gerald W. Lester, President, KNG Consulting LLC |
| Email: Gerald...@kng-consulting.net |
+----------------------------------------------------------------------+

Sergey G. Brester

unread,
Nov 21, 2018, 6:36:52 AM11/21/18
to
The problem is that with this solution (breaking out from TclNRRunCallbacks in "unexpected" way) you may get some unpredictable after-effects, thus it is IMHO a bit too aggressive.

So you can simply use Tcl_NRAddCallback to create your own callbacks to do single steps in the main cycle of TclNRRunCallbacks without any "dirty" hacks.

Thus why not something like this.

static int
NRPostInvokeHidden(
ClientData data[],
Tcl_Interp *interp,
int result)
{
Thread *thread = (Thread *)data[0];
void *arg2 = data[1];
// .... put your own green thread code here ...

// create new callback if needed:
//Tcl_NRAddCallback(interp, MyNRInvokeStep, thread, arg2, NULL, NULL);
return TCL_OK;
}

Tcl_NRAddCallback(interp, MyNRInvokeStep, thread, arg2, NULL, NULL);

Sergey G. Brester

unread,
Nov 21, 2018, 6:50:40 AM11/21/18
to
It had a typo (copy-pasted):

/s/NRPostInvokeHidden/MyNRInvokeStep

pooryorick

unread,
Nov 21, 2018, 7:39:46 AM11/21/18
to
I'm not sure what you're getting at here. Tcl already has coroutines and an event loop. {ycl coro call} and friends provide additional utilities.

--
Yorick

Sebastian Biały

unread,
Nov 22, 2018, 3:15:20 AM11/22/18
to
On 2018-11-21 12:36, Sergey G. Brester wrote:
> So you can simply use Tcl_NRAddCallback to create your own callbacks to do single steps in the main cycle of TclNRRunCallbacks without any "dirty" hacks.

No, this is actually opposite to whole idea. I cannot have TCL stuff on
stack while working with green treads. I need to *return* from TCL to do
next iteration with different green thread. TCL cannot be sheduler for
others, since it is possible that I do want to play with *two* TCLs at once.

Sebastian Biały

unread,
Nov 22, 2018, 3:22:27 AM11/22/18
to
On 2018-11-21 13:39, pooryorick wrote:
> I'm not sure what you're getting at here. Tcl already has coroutines and an event loop. {ycl coro call} and friends provide additional utilities.

Let's say that I need TCL coroutine at C++ level. Not at TCL level. And
it is *ALMOST* ready except that TCl is trying to give me a favor with
this internal loop that I do not want.

And I need *true* stackless TCL with single step function. Currently
using TclNRRunCallbacks I cannot control TCL flow in any way except
hacks like boost::context to force stack unwind and start next green
thread. But boost:context is too heavy to use frequently, whole idea of
green thread is to minimize stack/registers switching.

And Yes, I would like to preeempt many TCL threads and other threads in
same real thread. That is why I need TCL to return to me every time
single step is executed.

Donal K. Fellows

unread,
Nov 22, 2018, 6:10:28 AM11/22/18
to
On 20/11/2018 09:39, Sebastian Biały wrote:
> I'm trying to execute NRE Tcl using some sort of functions to single
> step. I need this to implement "green thread" or "fiber" concept that
> allows me to interleave tcl execution flow with other green threads in
> same real thread.

I'm trying to figure out why you're going to all this work, since you
could instead just run a Tcl interpreter in a real thread of its own.

But apart from that basic question, you have a separate problem that the
granularity level that you're injecting at isn't designed for what
you're trying to do. The NR callbacks you're (ab)using aren't at all a
one-to-one match with commands; a command may generate no callbacks at
all (e.g., if it is compiled to bytecode that's pretty likely) or it may
use multiple callbacks, and those won't be always obvious ones from an
external perspective. (For example, TclOO uses several stacked callbacks
to handle call data-structure memory management.) Also, the callbacks
are managed in a stack (but not the C stack); if you're adding your own
callbacks, they might not get called for a while (and some operations
such as traces or scripted-IO callbacks run in ways that *really*
shouldn't be suspended; the C stack will actually contain real stack
frames while they're running because we can't wrap our heads around
fixing them).

You _might_ get better behaviour if you also install an interpreter
command stepping trace (most easily done at the Tcl level, with [trace
add execution] with, e.g., the enterstep mode; I think there's a way to
do it via the API but that's not something I've ever explored) as that
will have the side effect of forcing things into single-step mode
instead of using heavy bytecoding, but it has the side effect of greatly
slowing Tcl down *precisely because* it disables heavy bytecoding. :-)

Donal.
--
Donal Fellows — Tcl user, Tcl maintainer, TIP editor.

Sebastian Biały

unread,
Nov 22, 2018, 9:20:00 AM11/22/18
to
On 2018-11-22 12:10, Donal K. Fellows wrote:
> I'm trying to figure out why you're going to all this work, since you
> could instead just run a Tcl interpreter in a real thread of its own.

In case of real thread I need to support all kind of magical
synchronization between threads. Green threads does some interesting suff:

a) no syncing needed via OS thread primitives
b) possibility to suspend execution at will
c) possibility to create debugger in embedded application
d) serialize internal stack(s) some day for real suspend (yes, fairly
tales but still possible)

> But apart from that basic question, you have a separate problem that the
> granularity level that you're injecting at isn't designed for what
> you're trying to do. The NR callbacks you're (ab)using aren't at all a
> one-to-one match with commands; a command may generate no callbacks at
> all (e.g., if it is compiled to bytecode that's pretty likely) or it may
> use multiple callbacks, and those won't be always obvious ones from an
> external perspective.

I'm aware of that. Actually I did some testing of this code and it seems
to be perfectly okay with granularity level. This is what I expect.

> (For example, TclOO uses several stacked callbacks
> to handle call data-structure memory management.) Also, the callbacks
> are managed in a stack (but not the C stack); if you're adding your own
> callbacks

I'm not adding my own callbacks for now. Just using what came with TCL.

>, they might not get called for a while (and some operations
> such as traces or scripted-IO callbacks run in ways that *really*
> shouldn't be suspended; the C stack will actually contain real stack
> frames while they're running because we can't wrap our heads around
> fixing them).

That is perfectly okay. I do understand all problems with green threads
/ coroutines. It does not makes them less interesting to me since this
is proper solution in my case. I have no way to explain this in details
since my code is closed source. Believe however that I do understand
what I'm trying to accomplish.

> You _might_ get better behaviour if you also install an interpreter
> command stepping trace

*Slow*. Already using this. Also, this does not *returns* to me at all
with is a point in this discussion. I'm just getting some kind of
callback with unknown C-stack and only option I have is to use
boost::context continuations to run away from this. Both tracing and
boost::context are slow. This is *NOT* an option for single-step solution.

Doing single step seems to be fast and proper way to handle full return
from C-stack as frequent as possible.

Actually whole point of this is this ridiculous loop inside
TclNRRunCallbacks. I do not need this, but there is no way around it
[1]. It would be interesting to have TclNRRunSingleCallback instead.
This would solve many problems with really using TCl as
coroutine/continuation from C code. Yes, just this small change would be
sufficient. TclNRRunCallbacks can be where it is for rest of 99.9% cases.



[1] I did some research over the years and this loop can be found in
many different language engines I've tried. Surprising, since many of
them can be converted to continuations by *removing* this useless loop
just like that without loosing any features. I still do not understand
why this loop is so important for everyone that is came for free with
soooo many languages ... :).
0 new messages