Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

One interp, many threads

76 views
Skip to first unread message

Roger Oberholtzer

unread,
May 30, 2011, 3:25:23 AM5/30/11
to
I have an application that runs as many threads. It is a C application
that uses Tcl/Tk very heavily to manage a GUI. It does this both with
the Tcl C interface functions, and with running scripts.

One thread runs the Tk event loop and manages the display and user
input. The other threads would like to do various things in the
display as well (but not get user input - that stays in the main
thread). My current solution is a bit of a hack that involves threads
waiting for access to a socket that the event loop thread reads from,
and executes the commands found. This solves the issue of each thread
contributing Tcl scripts to be executed by the main thread. However, I
cannot help but think that my solution is not the best solution. Are
there any packages that anyone knows about that manage this sort of
thing?

While on the topic, my Tcl/Tk is compiled to support threads. Does
that mean that I can, say, call Tcl_SetVar() from two different C
threads? On different variables, of course. But in the same interp. I
have been doing this for a while and have seen no problems. But
something tells me I have just been lucky.

Gerald W. Lester

unread,
May 30, 2011, 9:07:35 AM5/30/11
to

No you may not!

The rule is an interpreter lives and can be called in only one thread.

A thread, however, can have multiple interpreters.

--
+------------------------------------------------------------------------+
| Gerald W. Lester, President, KNG Consulting LLC |
| Email: Gerald...@kng-consulting.net |
+------------------------------------------------------------------------+

Roger Oberholtzer

unread,
May 31, 2011, 4:30:10 AM5/31/11
to
On May 30, 3:07 pm, "Gerald W. Lester" <Gerald.Les...@KnG-

Consulting.net> wrote:
> On 5/30/11 2:25 AM, Roger Oberholtzer wrote:
>
>
>
>
>
>
>
>
>
> > I have an application that runs as many threads. It is a C application
> > that uses Tcl/Tk very heavily to manage a GUI. It does this both with
> > the Tcl C interface functions, and with running scripts.
>
> > One thread runs the Tk event loop and manages the display and user
> > input. The other threads would like to do various things in the
> > display as well (but not get user input - that stays in the main
> > thread). My current solution is a bit of a hack that involves threads
> > waiting for access to a socket that the event loop thread reads from,
> > and executes the commands found. This solves the issue of each thread
> > contributing Tcl scripts to be executed by the main thread. However, I
> > cannot help but think that my solution is not the best solution. Are
> > there any packages that anyone knows about that manage this sort of
> > thing?
>
> > While on the topic, my Tcl/Tk is compiled to support threads. Does
> > that mean that I can, say, call Tcl_SetVar() from two different C
> > threads? On different variables, of course. But in the same interp. I
> > have been doing this for a while and have seen no problems. But
> > something tells me I have just been lucky.
>
> No you may not!
>
> The rule is an interpreter lives and can be called in only one thread.

Or you need to serialize access to the shared interpreter, which is
what I do with the scripts. As all the threads need to share the top
level window, this becomes a necessity.

I expected that in the second part of my question I have just been
lucky (the SetVar() calls). There are only a few of these in a thread.
But it only takes one...

Since I plan on improving this part of the code (even though it really
does work well), I thought I would ask it there were already some tcl-
specific tools in this area. That is the reason for my question.

> A thread, however, can have multiple interpreters.

Since interpreters cannot share a single top level window, one must be
creative.


>
> --
> +------------------------------------------------------------------------+
> | Gerald W. Lester, President, KNG Consulting LLC                        |

> | Email: Gerald.Les...@kng-consulting.net                                |
> +------------------------------------------------------------------------+

Googie

unread,
May 31, 2011, 10:06:23 AM5/31/11
to
On 30 Maj, 15:07, "Gerald W. Lester" <Gerald.Les...@KnG-

Consulting.net> wrote:
> The rule is an interpreter lives and can be called in only one thread.
>
> A thread, however, can have multiple interpreters.

In my opinion, this is stupid threads model. Somebody might feel
offended, but here're facts:

1. It makes harder to accomplish some sort of stuff (example: Tk usage
from this discusson, but not only this one),

2. It's always possible to "emulate" current model of threads using
model known from Java or C++ (shared environment), but it's impossible
to do it in opposite way (at least not an easy way... one must
serialize data across interpreters, or do other, heavy workarounds).

Redesigning things to fit better for Tcl threads model is not always
the answer.

Regards,
Googie

Donal K. Fellows

unread,
May 31, 2011, 10:34:43 AM5/31/11
to
On May 31, 3:06 pm, Googie <pawelsal...@gmail.com> wrote:
> In my opinion, this is stupid threads model. Somebody might feel
> offended, but here're facts:
>
> 1. It makes harder to accomplish some sort of stuff (example: Tk usage
> from this discusson, but not only this one),

Yes, but it imposes heavy penalties in terms of locking. Any resource
that is shared (especially for any kind of update, including cleanup)
between threads *needs* a lock somewhere to prevent concurrent
modifications. Since memory areas are one of the things that count as
resources, the easiest way to handle this is to use a single global
lock. That *really* hurts performance. (Yes, you can use smaller locks
but then you've got to get it right and that's not at all easy,
especially as nothing tells you if you've got it wrong; you just get
random problems like deadlocks, lethal embraces, crashes, etc. What's
more, they'll probably all go away once you attach a debugger, making
them true Heisenbugs...)

> 2. It's always possible to "emulate" current model of threads using
> model known from Java or C++ (shared environment), but it's impossible
> to do it in opposite way (at least not an easy way... one must
> serialize data across interpreters, or do other, heavy workarounds).

So we don't do emulation of concurrent modification crashes. :-) Or
maybe we can just claim that everything's all Turing-equivalent anyway
and stop worrying. :-)

Seriously though, a lot of workloads don't need to share lots of state
all over the place. Those work just fine. The large shared state codes
do have issues too; they're hard to get right except in special cases
and don't scale well to larger than one machine. Message passing is
saner for most things, for all its restrictions.

It's possible that certain operations ought to be special-cased
though. Things to support image processing would be good candidates
there.

> Redesigning things to fit better for Tcl threads model is not always
> the answer.

Yet the alternatives have their own horrible problems too.

Donal.

Googie

unread,
May 31, 2011, 1:27:35 PM5/31/11
to
I agree that alternatives have their issues too, but about the part:

On 31 Maj, 16:34, "Donal K. Fellows"


<donal.k.fell...@manchester.ac.uk> wrote:
> > 2. It's always possible to "emulate" current model of threads using
> > model known from Java or C++ (shared environment), but it's impossible
> > to do it in opposite way (at least not an easy way... one must
> > serialize data across interpreters, or do other, heavy workarounds).
>
> So we don't do emulation of concurrent modification crashes. :-) Or
> maybe we can just claim that everything's all Turing-equivalent anyway
> and stop worrying. :-)

Let me put this in other words.

As for now we have thread working in its isolated environment (let's
call it MODEL A):
+ No concurrent modifications take place.
- Informations between threads are shared using messages (even huge
data sets...).

This model cannot be "made" to work as MODEL B (below). This is what
we get and we have to deal with it.

And there's a MODEL B, known from Java or C++, where threads share
same environment:
- concurrent modifications are everyday issue (crashes, deadlocks).
+ passing information/objects/huge_data from thread to thread is done
simply by accessing a variable.

This model CAN be "made" to work as MODEL A by creating new
interpreter with [interp create] and assigning it for the thread -
it's just a couple of code lines.

Conclusion: MODEL A forces developer to use only MODEL A, while MODEL
B gives chance to use any of models, as somebody pleases, depending on
developer needs or mood.

Lets summarise: in MODEL B - if someone don't want to deal with the
synchronization stuff - he doesn't need to, but he has a possibility
to write applications with better performance.

I think of relation between MODEL B and MODEL A a little like as of
relation between TclOO and Itcl (or other TclOO based OO framework).

Regards,
Googie

tomk

unread,
May 31, 2011, 1:46:29 PM5/31/11
to

Perhaps I'm missing something (I'm sure I am) but it seems to me that
if you want to use MODEL B you just use tequila on the variables you
want to share. If I recall correctly all you have to do is declare the
variables to be shared so program conversion should be easy. Tequila
does sharing through a ip address and port so it scales to multiple
machines.
tomk

Googie

unread,
May 31, 2011, 6:58:37 PM5/31/11
to
On May 31, 7:46 pm, tomk <krehbiel....@gmail.com> wrote:
> Perhaps I'm missing something (I'm sure I am) but it seems to me that
> if you want to use MODEL B you just use tequila on the variables you
> want to share. If I recall correctly all you have to do is declare the
> variables to be shared so program conversion should be easy. Tequila
> does sharing through a ip address and port so it scales to multiple
> machines.

Well, it's just another way to pass messages between threads anyway.
The MODEL B assumes that data is placed in memory and accessed at same
block of memory from multiple threads, so there's no need to copy/
message it (but there's mentioned concurrent access/modification
issue).

Regards,
Googie

Christian Gollwitzer

unread,
Jun 1, 2011, 2:24:06 AM6/1/11
to
Am 01.06.11 00:58, schrieb Googie:

> Well, it's just another way to pass messages between threads anyway.
> The MODEL B assumes that data is placed in memory and accessed at same
> block of memory from multiple threads, so there's no need to copy/
> message it (but there's mentioned concurrent access/modification
> issue).
Do you know about tsv::set ?
Christian

Uwe Klein

unread,
Jun 1, 2011, 3:08:42 AM6/1/11
to
Roger Oberholtzer wrote:
> I have an application that runs as many threads. It is a C application
> that uses Tcl/Tk very heavily to manage a GUI. It does this both with
> the Tcl C interface functions, and with running scripts.

Does each thread work on "his" set of widgets ( graph, canvas .. )?

uwe

Googie

unread,
Jun 1, 2011, 3:31:46 AM6/1/11
to
On 1 Cze, 08:24, Christian Gollwitzer <aurio...@gmx.de> wrote:
>   Do you know about tsv::set ?

Yes, but still, this requires developer to copy data into new variable
(created by tsv). Some datas are huge and it's not efficient to copy
them. Other datas depend on packages loaded into interpreter (like Tk,
Itcl, or many others), so in some cases it's not even possible to
accomplish task using tsv or any other workaround.

Regards,
Googie

Roger Oberholtzer

unread,
Jun 1, 2011, 8:13:11 AM6/1/11
to

Generally this is so. But it is not so clear cut. There are some
interactions.

The application controls many data acquisition devices. Each device is
handled in a thread. (We also had a version where the devices were
managed via asynchronous signals when data arrived - but the issue is
the same.) In the case of, say, a GPS receiver, we have to show
location data as it arrives. This location information is used by the
operator to make various decisions, and thus should be responsive. The
GPS thread updates some variables via the C interface. No one else
will update those variables. The main thread calls the Tk event loop,
which will detect the changes in appropriate variables and update the
display accordingly. All this is working very well.

That is the simple case. Using the GPS example, the program can also
react automatically to location, causing things to happen that may, in
their turn, update widgets other then the GPS ones. For example, the
location may trigger a report that requires update of information from
many other threads. This is all coordinated by the simple mechanism of
all the threads only updating variables via the C interface, and the
Tk event loop in the main thread causing the display update.

I plan to review this code to make sure we are not violating any
assumptions. When doing so, I would like know if there is an existing
solution to these things in Tcl. I think I see that there is none. So
any work we do will not be re-inventing something.

Donal K. Fellows

unread,
Jun 1, 2011, 8:32:02 AM6/1/11
to
On Jun 1, 8:31 am, Googie <pawelsal...@gmail.com> wrote:
> Yes, but still, this requires developer to copy data into new variable
> (created by tsv). Some datas are huge and it's not efficient to copy
> them. Other datas depend on packages loaded into interpreter (like Tk,
> Itcl, or many others), so in some cases it's not even possible to
> accomplish task using tsv or any other workaround.

If it's a big enough issue, write some C code that provides shared
access to a common memory area under the protection of a concealed
lock. That is, there are cases where Tcl's simple message passing
model doesn't work very well, but they're almost always better solved
by an extension instead of trying to convert the whole of Tcl into
another thread model. Thread models are subtle things!

Donal.

Uwe Klein

unread,
Jun 1, 2011, 10:37:10 AM6/1/11
to
Roger Oberholtzer wrote:
> On Jun 1, 9:08 am, Uwe Klein <u...@klein-habertwedt.de> wrote:
>>Does each thread work on "his" set of widgets ( graph, canvas .. )?
>>
>>uwe
>
Roger:

>
> Generally this is so. But it is not so clear cut. There are some
> interactions.
>
> The application controls many data acquisition devices. Each device is
> handled in a thread. (We also had a version where the devices were
> managed via asynchronous signals when data arrived - but the issue is
> the same.) In the case of, say, a GPS receiver, we have to show
> location data as it arrives. This location information is used by the
> operator to make various decisions, and thus should be responsive. The
> GPS thread updates some variables via the C interface. No one else
> will update those variables. The main thread calls the Tk event loop,
> which will detect the changes in appropriate variables and update the
> display accordingly. All this is working very well.

I have written and currently running a ?similar? application.

Nmea messages from various sources. ( AIS, GPS, Log, Speed, ... )
All messages go by way of expect_background through a single instance
of expect without hassle. ( reading via fileevent with linebuffering
would work too )
Sentences are written to their respective variables.
i.e. GPGLL,... goes into
set ::GP(GLL,last,sentence) $sentence
set ::GP(GLL,last,pit) [pit]
........ separate content into array items ..
incr ::GP(GLL,cnt)

Processing is triggered by variable traces.
i.e. the AIS message processor for VDM, VDO is triggered by updates
on AI(VDM,cnt) and AI(VDO,cnt)

Maybe you just don't need threads?

uwe

Gerald W. Lester

unread,
Jun 1, 2011, 10:53:14 AM6/1/11
to
On 6/1/11 7:13 AM, Roger Oberholtzer wrote:
> On Jun 1, 9:08 am, Uwe Klein<u...@klein-habertwedt.de> wrote:
>> Roger Oberholtzer wrote:
>>> I have an application that runs as many threads. It is a C application
>>> that uses Tcl/Tk very heavily to manage a GUI. It does this both with
>>> the Tcl C interface functions, and with running scripts.
>>
>> Does each thread work on "his" set of widgets ( graph, canvas .. )?
>>
>> uwe
>
> Generally this is so. But it is not so clear cut. There are some
> interactions.
>
> The application controls many data acquisition devices. Each device is
> handled in a thread. (We also had a version where the devices were
> managed via asynchronous signals when data arrived - but the issue is
> the same.) ....

>
> I plan to review this code to make sure we are not violating any
> assumptions. When doing so, I would like know if there is an existing
> solution to these things in Tcl. I think I see that there is none. So
> any work we do will not be re-inventing something.

If instead you received the data in Tcl via file events (socket goes
readable) you have no issues.

I've done this very successfully and it eliminates the needs for threads and
takes less resources (generally you get a *large* performance/scalability
boost doing this).

Your design model is based on C/C++/Java -- it is a poor design model for Tcl.

--
+------------------------------------------------------------------------+
| Gerald W. Lester, President, KNG Consulting LLC |

| Email: Gerald...@kng-consulting.net |
+------------------------------------------------------------------------+

Googie

unread,
Jun 3, 2011, 11:02:54 AM6/3/11
to
On 1 Cze, 14:32, "Donal K. Fellows" <donal.k.fell...@manchester.ac.uk>
wrote:

> If it's a big enough issue, write some C code that provides shared
> access to a common memory area under the protection of a concealed
> lock. That is, there are cases where Tcl's simple message passing
> model doesn't work very well, but they're almost always better solved
> by an extension instead of trying to convert the whole of Tcl into
> another thread model. Thread models are subtle things!
>
> Donal.

I know that converting thread model is something unlikely to be done
and I understand reasons for that.
Anyway, I consider existing model as one of major Tcl disadventages
(even though I love Tcl).

Regards,
Googie

Donal K. Fellows

unread,
Jun 3, 2011, 11:07:28 AM6/3/11
to
On Jun 3, 4:02 pm, Googie <pawelsal...@gmail.com> wrote:
> Anyway, I consider existing model as one of major Tcl disadventages
> (even though I love Tcl).

I consider it to be an advantage; as it means that we have next to no
lock contention in Tcl programs. This is a colossal advantage,
especially for programs that don't have this fixation on big lumps of
shared state.

Donal.

Roger Oberholtzer

unread,
Jun 4, 2011, 6:21:40 AM6/4/11
to

Speed is the issue. These are top of the line inertial navigation
systems with gyroscopes, accelerometers and GPS receivers with
omnistar corrections. Some data is tagged with this information. Also,
locations trigger events. It is imperative that the data be acted on
as soon as possible. The time it takes the data to work through the
serial drivers in to our C application (blocking read calls seem to be
the quickest responders on Linux) is barely acceptable (and is also
being reviewed for ways to improve it). But any delay beyond that is
not good. When a vehicle is travelling at 90 km/h, a 1/10 sec delay is
2.5 meters. This is larger than the typical location error from our
devices. Also, much of the data we collect is provided by devices that
do not lend themselves to Tk event loops or Tcl access. Of course we
could wrap everything up in code available from Tcl. But the benefit
is unclear as there would still be a very significant amount of C
required. So, we have decided that the most sane use for us in this
all is to use Tk as a GUI, and have the C code set Tcl variables and
call the Tk event loop to keep the display snappy. This allows us to
have a very flexible GUI that is easily modifiable in Tcl scripts.

>
> uwe

Uwe Klein

unread,
Jun 4, 2011, 7:03:55 AM6/4/11
to
Roger Oberholtzer wrote:
> Speed is the issue. These are top of the line inertial navigation
> systems with gyroscopes, accelerometers and GPS receivers with
> omnistar corrections. Some data is tagged with this information. Also,
> locations trigger events. It is imperative that the data be acted on
> as soon as possible. The time it takes the data to work through the
> serial drivers in to our C application (blocking read calls seem to be
> the quickest responders on Linux) is barely acceptable (and is also
> being reviewed for ways to improve it).

What magnitude in delays and responsiveness are you talking about?
( I have used jim ( minitcl interpreter) for sub millisecond responsive
ness on Linux using RT scheduling and the userland interrupt patch.)

Still poll/select would be available on the C-side.
( It is obvious I see threads as an often times
suboptimal solution.)

Have you looked into RT addons for Linux?

In my experience the primary issue to decide is
which things need RT responsiveness and which don't.

I have had good results with small ( in complexity) state machines
doing stuff on the RT side and a slower controlling
app in userland.

uwe

0 new messages