Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

Thread package is amazing at solving problems

246 views
Skip to first unread message

lmn...@gmail.com

unread,
Jan 13, 2018, 8:54:40 PM1/13/18
to
The tcl thread package is the cats meow, but doesn't seem to get talked about?

Is there something shinier or newer?

Am I late to the party?

Here is my take:

I have found the thread package highly useful to solve problems -- but had to overcome the learning curve and trials and tribulations to get to that point.

I suppose this is limiting the usage and adoption on a wider scale?

However, the obstacle seems easily overcome if community pages existed that shared some basic concepts and conventions (example code) so that folks get this info/training upfront. (Paves an easier path forward to adoption and rollout)

For example: I now routinely leverage threading to solve problems 10x faster with 10x less code and complications vs any other methods.

Given this backdrop...

Questions:

Is there an active tcl thread community page with resources for ramping up on tcl threads?

Is there a shinier newer thing?

Is this sentiment regarding under utilized and underappreciated usedulness of the top thread package shared? (I could be wrong?)

I would like to see tcl thread go viral (as it deserves)...and I will surely contribute whatever I can.


My motivation is two fold:
1. I've plateaued on the subject and crave a vibrant community with advanced subject discussions available to tap into.

2. I like what I like and I want others to like it too. Its awesome!

Jim

Gerald Lester

unread,
Jan 14, 2018, 1:13:19 AM1/14/18
to
Ah, what types of issues did you run into --threading in Tcl is very
straight forward.

That being said, it is also not needed many times because the Tcl event
loop is so powerful as to make threading not needed where it would be in
most other languages.


--
+----------------------------------------------------------------------+
| Gerald W. Lester, President, KNG Consulting LLC |
| Email: Gerald...@kng-consulting.net |
+----------------------------------------------------------------------+

Alexandru

unread,
Jan 14, 2018, 3:14:08 AM1/14/18
to
How can I use the event loop instead of threading? Isn't threading's main purpose do parallel computing?

lmn...@gmail.com

unread,
Jan 14, 2018, 11:24:19 AM1/14/18
to
Hi Gerald,

Can you share a link to resources you found helpful or elaborate on your remark regarding tcl threads being straight forward?

Here is the question I am hoping your remark was aimed at:

Question: Is there an active community page with resources for ramping up?
I personally did not find much.

And to answer your question, regarding what types of issues were encountered on the journey to becoming an avid thread user...

Here goes:

Obstacles to getting ramped-up on tcl threads and making it far enough to reach production code vs the grave yard:

1. Missing the 'hello world' sample code for tcl threading.

2. Missing example of how to wait for all threads to complete, then proceed.
P.s. At this point corner cases can be mentioned, there are going to be unique corner cases related to threads (or at a minimum come up a lot more in threading context than you may have experienced previously) -- so lets mention them early-on...

3. thread pools, okay I've read the man page, but how to actually use it now. Get me started...

We are missing example code on thread pools.

And -- there is more code to using a pool than simply creating the pool and submitting stuff to it -- you need stuff around this. And you need an explanation of what's happening (this helps with debug and knowing your code is solid when you think you're 'done' writing and reviewing and testing).

Users can build up from the example code...but right now they have to figure out examples -- which is the obstacle.

Remember we all usually come here thinking about our actual problem, so the better we provide the samples -- the more adopters there will be and users will quickly map their particular problems and put threads into action in real production code -- with confidence.

4. How to debug a thread failure -- the techniques and methods of tackling a failing thread is largely ignored.
And debugging threads is definitely not like debugging non-threaded stuff. Whether you're debugging dev code or in production -- there are considerations and techniques to making this work reliably.

So -- we need to share that -- the techniques and strategies that work well with thread debugging...

5. TSV (thread save variables) -- a) hello world example; b) introduce via sample code a problem being solved. That opens up the world to seeing how tcl solves it. AND -- it's quite amazing (and safe). And while I've got your attention I'll tell you about good practices and what to avoid, etc...

6. All the above suggestions cover the new adopters and ground work -- but then you need a vibrant community of users to support the advanced users -- and that's when the real magic happens. You'll see the whole subject explode with users moving from beginner to advanced. And the package being used by more than just those old-timer tcl gurus.


Is this all altruistic non-sense?
Even though what I talk about sounds like it benefits other people more than myself -- believe me this is not altruistic on my part.

My motive:
My motive is to tap into such a community if doing the above would lead to an aggregation of information that grows to support more advanced levels.

In other words, it would be helpful to see more posts on this message board involving threads or to see the birthing of dedicated page(s) on the topic, etc...I've gone about as far as I can with the command pages.

Jim

Rich

unread,
Jan 14, 2018, 11:51:42 AM1/14/18
to
lmn...@gmail.com wrote:
> Question: Is there an active community page with resources for
> ramping up? I personally did not find much.

There is a fair amount of info re. Threads on the Tcl wiki
(http://wiki.tcl.tk or http://wiki.tcl-lang.org - both URL's go to the
same wiki). Did you search/look there?

> And to answer your question, regarding what types of issues were
> encountered on the journey to becoming an avid thread user...
>
> [most of the list of obstacles snipped]

> 5. TSV (thread save variables)

TSV stands for "(T)hread (S)hared (V)ariable)" The S is for "Shared"
not "save".

> Is this all altruistic non-sense?
> Even though what I talk about sounds like it benefits other people
> more than myself -- believe me this is not altruistic on my part.
>
> My motive:
> My motive is to tap into such a community if doing the above would
> lead to an aggregation of information that grows to support more
> advanced levels.

Tcl is wholly community driven. What that really means is if someone
from the community does not volunteer to do the work, the work likely
is not going to get done.

Since you stated in your OP that you had made it past some of these
obstacles, then you would be in a *very* good position to enhance the
documentation/examples around the Tcl threading subsystem. One place
where you can do that is the Tcl wiki. It is open to editing by
anyone. So either find a thread page that needs help and start helping
it (i.e., add the content/explanation/examples you find are missing).
Or, start creating one or more new thread pages that explain what you
have learned (so others can then leverage your learning) and adding the
examples you did not find so the next community member has a stonger
footing to start upon.

> In other words, it would be helpful to see more posts on this message
> board involving threads or to see the birthing of dedicated page(s)
> on the topic, etc...I've gone about as far as I can with the command
> pages.

This is not a "message board". It is a Usenet news group. And posts
here *only* arrive by community members making those posts. So if you
have useful info, post it. If you have questions, post them (but for
questions it would probably be best not to include 10 in a single post,
one or maybe two per post is fine). But if you want to see more
threads posts, then someone in the community (i.e., you, and others)
needs to post those to the group. Note however that it is *very*
useful to the community if, after posting info/questions, that you
take initative to move the relevant bits off the various articles in
the group and add them to the wiki so they are easier to find later
when someone else is looking for the same information.

Also, proper quoting is good (note what I have done above vs. your
replies which omit all the relevant context). This being a Usenet
group, the google groups interface is *not* the best way to read nor
post and you would be well suited to abandon that awful google groups
interface and instead get a real Usenet newsreader
(https://en.wikipedia.org/wiki/List_of_Usenet_newsreaders) and register
an Eternal September (https://www.eternal-september.org/) or AIOE
(https://news.aioe.org/) account (both are free) to read/post to the
group.

lmn...@gmail.com

unread,
Jan 14, 2018, 12:40:34 PM1/14/18
to
I have searched and use tcl/wiki a lot, unfortunately to no avail on this topic.

I suppose I wanted to make sure I just didn't overlook something so I asked here hoping to get similar confirmation. For such a useful topic -- I'm a little surprised.


I agree with you that the tcl/wiki will be the appropriate place to document all the awesomeness of tcl threads to achieve the goals I mentioned -- but before I started willy nilly posting about threads.

It occurs to me to gauge efficacy of such an endevour by first asking a couple questions of the existing community (here) and use the answers to see if that was indeed a likely output that could be achieved:

1. Is there an active tcl thread community page with resources for ramping up on tcl threads?

(We/I will wait a little longer and see if we get more responses)

2. Is there a shinier newer thing?

(Gerald mentioned event loops, but question back to him on that is what about parallel/processing. We'll let that discussion play out).

(Lets not stab ourselves in the foot here -- I would hate to have people read this post and think that threads are subjugate to eventloops and therefor unnecessary. The thread package makes tcl a superior tool for parallel processing vs 'any' other languages).
We, here in the tcl community, are a smart bunch so we know that already --
but anybody new -- now you know that too).

3. Is this sentiment regarding under utilized and underappreciated usefulness of the tcl thread package shared? (I could be wrong?)

It would be encouraging if we talked about all the places that threading is being used in the wild...that bolsters the notion that community work on this is worthwhile.

Jim

PS. I apologize for the google groups interface. I've tried the usenet stuff and ended up back at google. I know this is a religious topic here. Sorry about that. I'm much more religious about tcl if that is any consolation. I will attemp to use a reader if you provide a specific suggestion. I'm on a windows/pc, browser based preferred, but can download a specific program is that's 'the best option'.

Rich

unread,
Jan 14, 2018, 1:32:27 PM1/14/18
to
lmn...@gmail.com wrote:
> I have searched and use tcl/wiki a lot, unfortunately to no avail on
> this topic.

Then there is an opening there to help increase the community
documentation around this topic.

> It occurs to me to gauge efficacy of such an endevour by first asking
> a couple questions of the existing community (here) and use the
> answers to see if that was indeed a likely output that could be
> achieved:

That is fair.
>
> 1. Is there an active tcl thread community page with resources for
> ramping up on tcl threads?

None that I am aware of - but I am but one poster here to Usenet.

> 2. Is there a shinier newer thing?
>
> (Gerald mentioned event loops, but question back to him on that is
> what about parallel/processing. We'll let that discussion play out).

Event loops are older than threads in Tcl. Event loops do not provide
true /parallel/ processing (as in using 8 CPU's 100% simultaneously),
as they are single threaded. But they do provide /perceived/ parallel
processing in that the program can be doing something in the background
while the user interacts with the foreground. But for an entire set of
problems, merely being able to do something asynchronously in the
background, even without using plural CPU's simultaneously, is a net
win.

> Lets not stab ourselves in the foot here -- I would hate to have
> people read this post and think that threads are subjugate to
> eventloops and therefor unnecessary.

Different use situations. Threads have a use, the event loop has a use
(and is central to any Tk GUI). Some problems can be solved by either
mechanism, some problems better lend themselves to one or the other.

> The thread package makes tcl a superior tool for parallel processing
> vs 'any' other languages.

Tcl's threads have some advantages that make them superior, and some
disadvantages that hurt (depending upon your needs). Tcl's threads are
have more in common with processes than what other languages call
Threads (Tcl's threads are isolated from each other, often referred to
as the "apartment" model of threading). That isolation is a boon in
very many situations (because a whole host of synchronization problems
simply do not exist) but a detriment in others (when plural threads
need to very closely share the same dataset).

> 3. Is this sentiment regarding under utilized and underappreciated
> usefulness of the tcl thread package shared? (I could be wrong?)

No idea. I've built event loop based background task code, and I've
built Threads based parallel code for my own uses. I use the one that
seems, to me, at the time to be the more appropriate one to utilize to
achieve the task I'm trying to achieve.

> It would be encouraging if we talked about all the places that
> threading is being used in the wild...that bolsters the notion that
> community work on this is worthwhile.

That's hard to say, because some problems lend themselves well to a
multi-processing approach, some do not. Depends upon the objective.
For two of the threaded bits of code I've built, one of them was an
embarassingly parallel task (run tDom's xml validator on a file tree of
several tens of thousands of xml docs to verify they all met validated
against a DTD). Each doc is 100% independent of any other, and since I
had an 8-core CPU in the machine I was using, a thread pool of 8
threads, with a master thread slicing up ranges of file names and
sending those into the pool, worked very well to fairly well saturate
the 8 cores (and the disk too for that matter, as in the end disk seek
time became toe biggest time consumer).

The other big threaded bit of code I have written was for what is
essentially a job scheduler type app, and there I threaded it because I
wanted the different I/O portions (fetch data from a http api,
extract/rearrange the fetched data on disk, move the rearranged data to
a job input directory, fetch completed jobs from a job output
directory, etc.) to be synchronous in the Tcl code [1], but parallel to
each other. This one, however, was parallelization around I/O wait
time instead of parallelization for saturating all CPU's in a multi-cpu
system.

> PS. I apologize for the google groups interface. I've tried the
> usenet stuff and ended up back at google. I know this is a religious
> topic here. Sorry about that. I'm much more religious about tcl if
> that is any consolation. I will attemp to use a reader if you
> provide a specific suggestion. I'm on a windows/pc, browser based
> preferred, but can download a specific program is that's 'the best
> option'.

I don't have any windows Usenet reader suggestions, other than I have
heard of folks using Thunderbird to some success to read Usenet groups.
I myself use Linux for everything (no ms windows on *any* of my
machines) and with Linux several good Usenet news readers generally
already come installed out of the box (depending upon one's choice of
Linux distribution).

[1] at the time, I didn't have ready access to 8.6 or its coroutines to
use to make an event loop variant, so threads were the easy way to
'parallelize' the multiple different I/O waits

Gerald Lester

unread,
Jan 14, 2018, 2:59:08 PM1/14/18
to
That is what threading should be used for, but...

In many other languages it is used to not block the "main" thread while
"waiting" on I/O or GUI input -- Tcl's event loop does a great job on this.

Gerald Lester

unread,
Jan 14, 2018, 3:29:21 PM1/14/18
to
On 01/14/2018 10:24 AM, lmn...@gmail.com wrote:
> Hi Gerald,
>
> Can you share a link to resources you found helpful or elaborate on your remark regarding tcl threads being straight forward?
>
> Here is the question I am hoping your remark was aimed at:
>
> Question: Is there an active community page with resources for ramping up?
> I personally did not find much.
>
> And to answer your question, regarding what types of issues were encountered on the journey to becoming an avid thread user...
>
> Here goes:
>
> Obstacles to getting ramped-up on tcl threads and making it far enough to reach production code vs the grave yard:

Thread are not in the grave yard, they are used by many people --
including me. It is just that they tend to be used for where someone
needs parallel processing or have a blocking API.


> 1. Missing the 'hello world' sample code for tcl threading.

If by, "Hello World" you mean a GUI with maybe a clock updating every
second and a button that exits it -- that would be a misuse of
threading, that is a perfect fit for using the event loop instead.

Please give a "specification" for what you want as an intro sample code.

> 2. Missing example of how to wait for all threads to complete, then proceed.
> P.s. At this point corner cases can be mentioned, there are going to be unique corner cases related to threads (or at a minimum come up a lot more in threading context than you may have experienced previously) -- so lets mention them early-on...

Main thread:
set cond [::thread::cond create]
set slaveThreadCount 0

##
## Create slave, send it $cond, incr slave count, then...
##
set mutex [thread::mutex create]
while {$slaveThreadCount) {
thread::cond wait $cond $mutex
incr slaveThreadCount -1
}

## All slaves done

Slave Thread:
## Done, so
thread::cond notify $cond
## do other cleanup and then destroy the thread

Alternatively, slaveThreadCount could be a TSV and decremented in the
slave prior to doing the notify.

Or Alternatively, the master could just vwait on slaveThreadCount and
have the slaves, when the are about to finish up, do a thread::send to
the master the following {incr slaveThreadCount -1)

Thee is likely several more ways, those are the ones that spring to mind
right now.

>
> 3. thread pools, okay I've read the man page, but how to actually use it now. Get me started...
>
> We are missing example code on thread pools.
>
> And -- there is more code to using a pool than simply creating the pool and submitting stuff to it -- you need stuff around this. And you need an explanation of what's happening (this helps with debug and knowing your code is solid when you think you're 'done' writing and reviewing and testing).

Ah, you test and debug the code that will run in the thread pool as
though it is the only thing in the world -- since as far as it is
concerned it is.

To me the man page says what is happening -- what is missing? what are
you looking for?

> Users can build up from the example code...but right now they have to figure out examples -- which is the obstacle.
>
> Remember we all usually come here thinking about our actual problem, so the better we provide the samples -- the more adopters there will be and users will quickly map their particular problems and put threads into action in real production code -- with confidence.
>
> 4. How to debug a thread failure -- the techniques and methods of tackling a failing thread is largely ignored.
> And debugging threads is definitely not like debugging non-threaded stuff. Whether you're debugging dev code or in production -- there are considerations and techniques to making this work reliably.

Debugging code that will run in a Tcl Thread is *EXACTLY* like debugging
non-thread code. How is it "different"?

>
> So -- we need to share that -- the techniques and strategies that work well with thread debugging...
>
> 5. TSV (thread save variables) -- a) hello world example; b) introduce via sample code a problem being solved. That opens up the world to seeing how tcl solves it. AND -- it's quite amazing (and safe). And while I've got your attention I'll tell you about good practices and what to avoid, etc...

Again, specify a TSV introductory sample program?

> 6. All the above suggestions cover the new adopters and ground work -- but then you need a vibrant community of users to support the advanced users -- and that's when the real magic happens. You'll see the whole subject explode with users moving from beginner to advanced. And the package being used by more than just those old-timer tcl gurus.
>
>
> Is this all altruistic non-sense?
> Even though what I talk about sounds like it benefits other people more than myself -- believe me this is not altruistic on my part.
>
> My motive:
> My motive is to tap into such a community if doing the above would lead to an aggregation of information that grows to support more advanced levels.
>
> In other words, it would be helpful to see more posts on this message board involving threads or to see the birthing of dedicated page(s) on the topic, etc...I've gone about as far as I can with the command pages.

This is a useneet newsgroup, not a message board.

Feel free to create/edit pages on wiki.tcl.tk on what you have
experienced.

Ask *concreate* questions here on c.l.t.

Peter Dean

unread,
Jan 14, 2018, 5:50:01 PM1/14/18
to
On 15-Jan-18 3:40 AM, lmn...@gmail.com wrote:
> PS. I apologize for the google groups interface. I've tried the usenet stuff and ended up back at google. I know this is a religious topic here. Sorry about that. I'm much more religious about tcl if that is any consolation. I will attemp to use a reader if you provide a specific suggestion. I'm on a windows/pc, browser based preferred, but can download a specific program is that's 'the best option'.
>

I use thunderbird on windows and linux. I didn't know about the free
usenet when I took out a subscription with astraweb.com many years ago
but at $10 for 25gb it's gonna last me several lifetimes. It has posts
going back to 2003.

jsunth...@gmail.com

unread,
Jan 15, 2018, 6:52:29 AM1/15/18
to
Jim, would be possible to write this kind of command?

proc noop {args} {}
proc threadit {num args} {LITERAL BLACK MAGIC NUM TIMES}



Jim

unread,
Jan 15, 2018, 4:40:45 PM1/15/18
to
On 1/15/2018 3:52 AM, jsunth...@gmail.com wrote:
> Jim, would be possible to write this kind of command?
>
> proc noop {args} {}
> proc threadit {num args} {LITERAL BLACK MAGIC NUM TIMES}

Absolutely! Yes, great question.

There is one such implementation, that I am aware of, but it is
implemented as a command embedded in a commercial tool.

The community here (me inclusive), can develop our own 'parallel_job <>'
command and post it freely to the community to use and that would
immediately be useful I think.

In a nutshell, here is what we need to do:

1. Write a procedure called 'parallel_job <cmds>' that takes a list of
commands and executes them in threads. Provide some configurability for
generally useful things.

Configuration options might be:

parallel_conf -use_tpool <0|1> 0=default
parallel_conf -tpool_min_workers <N> N=0 default
parallel_conf -tpool_max_workers <N> N=1 default
parallel_conf -initcmd {}
parallel_conf -exitcmd {}
parallel_conf -keep_alive <0|1> 0=default
parallel_conf -initcmd_env_import <0|1> 1=enabled=default; 0=disabled
This -initcmd_env_import feature automates the 'tedious' setup
of the thread. It copies what I have found to be the most
"standard/typical" things we need initialized in each thread. Stuff
like: auto_path, existing proc definitions -- users typically want to
have their threads 'just like their main' thread. We can do that for
the most part* For Gerald (we provide the 0=disable option).

parallel_conf -other options?

PARALLEL_JOBS <cmds> interface might be:

parallel_jobs {
{blackmagic -input <file1> -output <file1.out>}
{blackmagic -input <file2> -output <file2.out>}
{blackmagic -input <file3> -output <file3.out>}
...
}
parallel_jobs {
{analyze_data -set <A> -output <setA.out>}
{analyze_data -set <B> -output <setB.out>}
{analyze_data -set <C> -output <setC.out>}
...
}
parallel_jobs {
{file copy <input_fileX> <output_location>; cmd ...}
{file copy <input_fileY> <output_location>; cmd ...}
...
}

Command Description:
The parallel_jobs procedure takes a list of commands and evaluates
them individually in threads. This is a blocking procedure and returns
after all commands have been executed and threads have been closed
(automatically handled by the parallel_jobs command).

Each line is an element in a tcl list. The code will be passed to a
thread where it will be 'eval'uated by the tcl interpreter -- so any
standard tcl can be passed as the list element.

Depending on how the user configures the settings (see parallel_conf
for details) -- the threads will be done in a pool, or each element
creates a separate thread.

Note: Because threads are not initialized -- users can a) pass an
'initcmd' through the configuration interface which will execute the
'initcmd' at the startup of each thread; b) pass the initialization
sequence in the <cmd> itself; or c) eliminate any startup/initialization
dependency (the practical ability to do this depends on the particulars
of each problem).

Example demonstrating (b): passing the initialization sequence in the
<cmd> itself:

parallel_jobs {
{source init.tcl; blackmagic -input <file1> -output <file1.out>}
{source init.tcl; blackmagic -input <file2> -output <file2.out>}
{source init.tcl; blackmagic -input <file3> -output <file3.out>}
}

In this example, ./init.tcl is a routine responsible for initializing
the thread with everything that it needs to evaluate the remaining tcl
code of the <cmd>. For example update 'auto_path' so that 'blackmagic'
procedure is automatically found in the search path.

Remember threads are uninitialized (dumb) -- so we have to tell it
everything it needs to walk, crawl and run -- just like you need to do
when you start a tcl shell for the first time (this can be tedious) --
(see parallel_conf -initcmd_env_import option for a possible helper
routine we can bake to relieve some of this tedious work)...

STDOUT Considerations:
My 'blackmagic' likes to print messages, so how do you deal with that?

Threads are going to scramble everything to stdout as execution is
asynchronous and unordered (as desired).

Also, if tcl throws errors or exceptions for ANY REASON -- those are
going to get thrown to stdout in the same manner

See #1,2,3, and 7 below for more on dealing with STDOUT thread issues.

Here are a couple things to keep in mind when coding 'blackmagic':

1. print useful messages indicating what the <input file> name was at
the time the exception or error was thrown. The last thing you need is
a cryptic 'cannot open file' message and have no idea which file it was
referring to. Or depending on what blackmagic does not be able to trace
which through which file it was that caused the problem.

2. time stamps -- stdout is going to look terrible. give yourself a
fighting change and add timestamps to messages so you might be able to
relate them or stitch a series of events together. remember -- if
you're doing parallel processing, chances are you're doing something
that takes a long time -- so this can be useful despite any initial
skepticism.

3. whatever can go wrong -- will go wrong. So harden your code -- write
every check you can, check check and re-check -- for example: check
file read permissions before opening, print a message and return if not,
check write permissions, print a message and return, wrap 'open' calls
with catch and 'close' calls with catch.

4. don't wait to let tcl throw an exception, those will leave you with
cryptic messages and think that threads are unstable.

5. if you are absolutely confident that 'open' worked and 'close'
worked you are going to narrow down the issues and get to the real issue
very quickly. For example: I thought the network was stable -- I
actually assumed it was -- until it turns out -- it wasn't stable much
to my surprise. In other words: Jezz -- I thought there was no problem
with file write permissions, until there were...hmm...I'll be damned,
users do the darndest things. Well, why did you do that?

6. Parallel processing results in utilization of the compute resource
going up dramatically vs non parallel processing. You might run into
things you never saw before, ulimit issues (user has max number of open
file limits exceeded), you're chuggling along, you can open 1k files no
problem but 1k+1 files and all of a sudden you core dump and 'cannot
open file' message appears (re-read #3 above). Disk i/o will go up
(stall the network or machine, hang the system or did it run with no
problem?) I've had all 3 cases. The filesystem and network usage
should go up -- you're going to find out what the network can actually
provide, some disks have network limits, some are high performance and
mirrored or low performance, you might compete with other i/o on the
network or same disk with other jobs -- as a result you might find
unstable mounts in your network that you never even noticed before
(re-read #3 above).

7. Log messages: There is a solution to out of order log/stdout
messages but I have not written the code or thought about how to solve
it actually -- so I'm hoping somebody that has might join in and
contribute that code to this effort. I just know its possible and would
love to collaborate with somebody on solving it.

We haven't talked about:
a. tsv (thread shared variables) -- I also refer to them as safe
variables because they are also safe.
b. how to use tsv to your benefit (above) and to solve a wider range of
problems.
c. assumptions and limitations of what this style of api can do.
(what's feasible and what's not).

TSV is a very useful tool that expands this topic even farther.

But I'll pause here.

Jim

Rich

unread,
Jan 15, 2018, 5:09:35 PM1/15/18
to
Jim <lmn...@gmail.com> wrote:
> 7. Log messages: There is a solution to out of order log/stdout
> messages but I have not written the code or thought about how to solve
> it actually -- so I'm hoping somebody that has might join in and
> contribute that code to this effort. I just know its possible and would
> love to collaborate with somebody on solving it.

If you are willing to:
1) use tcllib's logger
2) have the main thread be responsible for the actual logging I/O

Then you can configure logger in the main thread as you prefer, and
each child thread can then do a thread::send to the main thread to
output log information.

If you don't want the child to wait for the log output to complete, the
thread::send can be done with the -async option.


Alternately, you can also configure one extra child thread as the
"logger" thread and have all the other threads do thread::send to that
'logging' thread. This is if you don't want the main thread to have to
be busy with logging.



two...@gmail.com

unread,
Jan 15, 2018, 9:10:48 PM1/15/18
to
On Saturday, January 13, 2018 at 5:54:40 PM UTC-8, lmn...@gmail.com wrote:

> However, the obstacle seems easily overcome if community pages existed that shared some basic concepts and conventions (example code) so that folks get this info/training upfront. (Paves an easier path forward to adoption and rollout)

I too would love to see a few small examples of threads.

The (only) time I feel the need for threads is when my events involve timed delays. I’ve written a “wait” proc using vwait and a unique global variable generator to allow for multiple waits concurrently and not hang the main; however, it still does have problems, in particular with nested vwaits.

---------------

The below program has 2 buttons, b1/b2 just [puts] the time before and after a delay. One would normally expect b1 to take 5 seconds and b2 to take 10 seconds. Button 3 lists the pending after's.

Pushing b1 then (quickly) b2 produces different behavior than pushing b2 and then b1.

This is the documented behavior of vwait, and is the problem with this example.

I think it would be cool to see this implemented using threads to get the expected results.

------------


button .bb1 -text "b1" -command {b1}
button .bb2 -text "b2" -command {b2}
button .bb3 -text "show" -command {showit}

pack .bb1 .bb2 .bb3 -fill both -expand true
console show ;# To use the console on unix/linux see wiki https://wiki.tcl.tk/786

proc uniqkey { } {
set key [ expr { pow(2,31) + [ clock clicks ] } ]
set key [ string range $key end-8 end-3 ]
set key [ clock seconds ]$key
return $key
}

proc wait { ms } {
set uniq [ uniqkey ]
set ::__sleep__tmp__$uniq 0
after $ms set ::__sleep__tmp__$uniq 1
vwait ::__sleep__tmp__$uniq
unset ::__sleep__tmp__$uniq
}

proc b1 {} {
puts "[clock format [clock seconds]] in b1 before "
wait 5000
puts "[clock format [clock seconds]] in b1 after "
}

proc b2 {} {
puts "[clock format [clock seconds]] in b2 BEFORE"
wait 10000
puts "[clock format [clock seconds]] in b2 AFTER"
}
proc showit {} {
foreach item [after info] {
puts "after $item = [after info $item]"
}
}


Rich

unread,
Jan 15, 2018, 10:07:47 PM1/15/18
to
two...@gmail.com wrote:
> On Saturday, January 13, 2018 at 5:54:40 PM UTC-8, lmn...@gmail.com wrote:
> The (only) time I feel the need for threads is when my events involve
> timed delays. I?ve written a ?wait? proc using vwait and a unique
> global variable generator to allow for multiple waits concurrently
> and not hang the main; however, it still does have problems, in
> particular with nested vwaits.
>
> ---------------
>
> The below program has 2 buttons, b1/b2 just [puts] the time before
> and after a delay. One would normally expect b1 to take 5 seconds
> and b2 to take 10 seconds. Button 3 lists the pending after's.
>
> Pushing b1 then (quickly) b2 produces different behavior than pushing
> b2 and then b1.
>
> This is the documented behavior of vwait, and is the problem with
> this example.
>
> I think it would be cool to see this implemented using threads to get
> the expected results.

The reason you get the differing effect you see is that vwaits nest,
and an outer vwait can't "go" until the inner vwait is finally
released.

When you do b1->b2 you have a 5 second wait with a nested 10 second
wait. The outer one can't be released until the inner 10 second wait
is done.

But when you do b2->b1 you have an outer 10 second wait with a nested 5
second wait. The nested wait will finish before the outer, and it will
return. That frees the outer to return when its time expires.

Making this change:

--- t.orig 2018-01-15 22:02:33.216932534 -0500
+++ t 2018-01-15 22:01:15.915031033 -0500
@@ -5,7 +5,6 @@
button .bb3 -text "show" -command {showit}

pack .bb1 .bb2 .bb3 -fill both -expand true
-console show ;# To use the console on unix/linux see wiki https://wiki.tcl.tk/786

proc uniqkey { } {
set key [ expr { pow(2,31) + [ clock clicks ] } ]
@@ -24,14 +23,12 @@

proc b1 {} {
puts "[clock format [clock seconds]] in b1 before "
- wait 5000
- puts "[clock format [clock seconds]] in b1 after "
+ after 5000 {puts "[clock format [clock seconds]] in b1 after "}
}

proc b2 {} {
puts "[clock format [clock seconds]] in b2 BEFORE"
- wait 10000
- puts "[clock format [clock seconds]] in b2 AFTER"
+ after 10000 {puts "[clock format [clock seconds]] in b2 AFTER"}
}
proc showit {} {
foreach item [after info] {

Makes the behavior of b1->b2 and b2->b1 identical.

While it likely does not fit your real use case, for timed delays, it
is often easier to use after's ability to schedule a script to be run
later than to try to build a vwait based 'wait' while the event loop
continues.

Alternately, if you really need a 'stop and wait here while the event
loop continues before continuing' then that would be a perfect
situation for using a coroutine.

two...@gmail.com

unread,
Jan 16, 2018, 3:32:05 AM1/16/18
to
On Monday, January 15, 2018 at 7:07:47 PM UTC-8, Rich wrote:

> Alternately, if you really need a 'stop and wait here while the event
> loop continues before continuing' then that would be a perfect
> situation for using a coroutine.

Sure, for a "hello world" type of program, it would be easy to fix it using several techniques.

But suppose you had to interleave 20 http requests with varying delays (an androwish tcl program I wrote to control my Roku device). It's much easier to write (and especially to read) using sequential wait commands.

Can my example be done using tcl threads? Can one even use vwait within threads? Would it require a different kind of "wait" proc?


Naked after calls (e.g. after 5000) hang the gui, so that is why I chose to use vwait. Then I ran into the vwait nesting problem.

Or is there a better way to write a "wait" procedure, either within threads or just the event loop?





Rich

unread,
Jan 16, 2018, 5:49:21 AM1/16/18
to
two...@gmail.com wrote:
> On Monday, January 15, 2018 at 7:07:47 PM UTC-8, Rich wrote:
>
>> Alternately, if you really need a 'stop and wait here while the event
>> loop continues before continuing' then that would be a perfect
>> situation for using a coroutine.
>
> Sure, for a "hello world" type of program, it would be easy to fix it
> using several techniques.
>
> But suppose you had to interleave 20 http requests with varying
> delays (an androwish tcl program I wrote to control my Roku device).
> It's much easier to write (and especially to read) using sequential
> wait commands.

Which is why I also said: "While it likely does not fit your real use
case," but that sentence did not get quoted.

> Can my example be done using tcl threads?

If the example "Roku controller" is a gui that you don't want to hang
while the http requests run, you can run the http requests in a second
thread, and run them synchronously if you like.

The http package also supports background operation via the event loop
with the -command option. You get a callback when the http operation
completes (or times out, if you also specify -timeout). Of course to
directly use it without coroutines you have to structure your code in
callback style (i.e., you do lose the linear code aspect of synchronous
code).

> Can one even use vwait within threads?

Yes, but it only vwaits that one thread, not any other thread.

> Would it require a different kind of "wait" proc?

No, but depending on just what you are trying to do, a 'different' wait
proc might be in order in any case.

> Naked after calls (e.g. after 5000) hang the gui, so that is why I
> chose to use vwait. Then I ran into the vwait nesting problem.

Yes, that's their definition, and since a Tk GUI is already event loop
based, it is often possible to restructure your flow to also be event
based and not need to 'after X' wait anywhere.

> Or is there a better way to write a "wait" procedure, either within
> threads or just the event loop?

Without coroutines, then event loop and call-back style is better than
trying to shoe-horn a 'wait here' into a sequential code flow. But
that then breaks your logic up into small snippets and puts the control
flow into the event loop, which can make it more difficult to reason
about.

If you want to keep the synchronous style, but have asynchronous
operation (event loop) then using 8.6 and coroutines are the superior
way to go.

Donal K. Fellows

unread,
Jan 16, 2018, 8:11:10 AM1/16/18
to
On 16/01/2018 10:49, Rich wrote:
> If you want to keep the synchronous style, but have asynchronous
> operation (event loop) then using 8.6 and coroutines are the superior
> way to go.

I can also recommend using coroutines for this sort of thing. They let
you get sensible control flow with very little script-level overhead, so
your code writes pretty much the same (OK, different commands but the
same pattern) as a synchronous wait yet without the bad side effects.

Donal.
--
Donal Fellows — Tcl user, Tcl maintainer, TIP editor.

two...@gmail.com

unread,
Jan 16, 2018, 1:03:22 PM1/16/18
to
On Tuesday, January 16, 2018 at 5:11:10 AM UTC-8, Donal K. Fellows wrote:

> I can also recommend using coroutines for this sort of thing. They let
> you get sensible control flow with very little script-level overhead, so
> your code writes pretty much the same (OK, different commands but the
> same pattern) as a synchronous wait yet without the bad side effects.
>

Interesting. Could you provide an example that demonstrates how one might write a co-routine with a wait, in as few lines as possible? And maybe also for contrast, one that does it via threads?

Both would be most appreciated.

Co-routines could possibly cure one minor problem I have in my roku controller code. I have a button that invokes a 2 minute skip (by sending 12 right, and one select - each right skips 10 seconds, and select starts it playing again). If I hit the 2 minute skip a second time before the first one is done, it interjects the second skip in the middle of the first one. That actually still works, but I've wondered about how to best synchronize things.

Rich

unread,
Jan 16, 2018, 1:30:12 PM1/16/18
to
two...@gmail.com wrote:
> On Tuesday, January 16, 2018 at 5:11:10 AM UTC-8, Donal K. Fellows wrote:
>
>> I can also recommend using coroutines for this sort of thing. They let
>> you get sensible control flow with very little script-level overhead, so
>> your code writes pretty much the same (OK, different commands but the
>> same pattern) as a synchronous wait yet without the bad side effects.
>>
>
> Interesting. Could you provide an example that demonstrates how one
> might write a co-routine with a wait, in as few lines as possible?

Ok, you did say "as few lines as possible":

1) Install tcllib (saves you from having to re-invent the wheel here)
2) package require coroutine [1] in your script
3) add "coroutine::util after <delay>" where you want to wait [2]

> And maybe also for contrast, one that does it via threads?

Threads would not (normally) make the main routine wait [2], so you'll need
to be quite a bit specific as to what kind of wait you want. If you
want this:

compute-something
wait-here-for-3-minutes
compute-something-else

Then your best choices are:

1) standard after - but event loop pauses
2) the above tcllib cooroutine bits

> Both would be most appreciated.
>
> Co-routines could possibly cure one minor problem I have in my roku
> controller code. I have a button that invokes a 2 minute skip (by
> sending 12 right, and one select - each right skips 10 seconds, and
> select starts it playing again). If I hit the 2 minute skip a second
> time before the first one is done, it interjects the second skip in
> the middle of the first one. That actually still works, but I've
> wondered about how to best synchronize things.

That one is easy, although I am assuming you mean a tk or ttk "button"
and not an lirc remote control when you say "button".

Do one of these:

1) First thing in the command executed by the button's -command script,
set the button to disabled state. For Tk that would be ".b configure
-state disabled" (assuming the button was named .b). Then, last thing
before the script is done and exits, reset the -state back to "normal"
state. Tk (or Ttk) buttons that are in 'disabled' state do not respond
to clicks, which is one purpose for the disabled state, to provide a
form of exclusive locking when necessary. The other purpose, of
course, is to indicate "this button not available now".

2) Add a boolean flag variable to your command script that says "I'm
currently running". Init the flag to 0. First thing in the command
script, check if the flag is 1, and return if so. Otherwise, set the
flag to 1, process the rest of the script, then as the last thing
before the script completes, set the flag back to 0.

#1 above is the easier one to do, because all the support is already
present in the Tk library.


[1] https://core.tcl.tk/tcllib/doc/trunk/embedded/www/tcllib/files/modules/coroutine/tcllib_coroutine.html

[2] there's a bit more than this, you also have to "coroutine::util
create" the coroutine as well.

[3] 'waiting' would be possible by calling the thread in synchronous
mode, but doing that defeats the main purporse for threads (background
async. work), and still gives you a 'locked GUI' on the thread that is
doing the wait.

two...@gmail.com

unread,
Jan 16, 2018, 3:09:39 PM1/16/18
to
On Tuesday, January 16, 2018 at 10:30:12 AM UTC-8, Rich wrote:


Thanks for the detailed reply.


>
>
> That one is easy, although I am assuming you mean a tk or ttk "button"
> and not an lirc remote control when you say "button".

Yes, I am referring to tk/ttk buttons in my program. The roku has an http
interface that's easy to use, but provides no feedback and if you send them
too quickly, it just ignores the later ones. Hence, the need for my program
to wait. I determined the timing by trial and error. There's an android store
app that I monitored to see the http requests it was sending, but otherwise
there's no documentation I could find. Still, it mostly works.

>
> Do one of these:
>
> 1) First thing in the command executed by the button's -command script,
> set the button to disabled state. For Tk that would be ".b configure
> -state disabled" .....

Actually, I want both button pushes to trigger, just that the second would
wait until the first finished.

That's where I figured threads would help. In my younger days, I think that would have required a mutex or a monitor. It's been a long time. Not sure what the proper terminology is today.



>
>
> [1] https://core.tcl.tk/tcllib/doc/trunk/embedded/www/tcllib/files/modules/coroutine/tcllib_coroutine.html
>
> [2] there's a bit more than this, you also have to "coroutine::util
> create" the coroutine as well.

I looked at that page, but unfortunately it does not include any examples. I'm
not an expert tcl programmer, just use it for casual programming to create
scripts so I don't have to work so hard. And I'm 71 years old, and new stuff
is getting harder to digest, without some simple examples. So please forgive me.


>
> [3] 'waiting' would be possible by calling the thread in synchronous
> mode, but doing that defeats the main purporse for threads (background
> async. work), and still gives you a 'locked GUI' on the thread that is
> doing the wait.

What I figured was that starting a thread would free up the GUI. I think
perhaps I don't understand enough to be able to use them.

I also don't want to hijack the OP's thread. In it, he was looking for some simple thread examples. I posted my example thinking it could be converted with just a few commands. Sadly, I think it's a bit more difficult than I thought.

Gerald Lester

unread,
Jan 16, 2018, 3:51:38 PM1/16/18
to
Have a "queue" of roku events (i.e. a list, or better yet use TclLib's
struct::queue).

The buttons first get the size of the queue, then they put one or more
commands in the queue. If the queue size was zero (0), they call the
ProcessCommands command.

ProcessCommands would first check to see if a command ws active, if none
was active it would take a command off of the queue and call it. Each
command would mark that a command as active, do the http call (with a
-command CommandFinished and an [after time RaiseTimeout]).

The CommandFinished would clear the command active flag and if the queue
size is not zero, call ProcessCommands.

The RaiseTimeout does whatever is appropriate if the Ruko did not
respond in a timely fashion.

BTW, can you share your app (like maybe via http://chiselapp.com/) -- I
have a couple of Rukos and would be interested in it (and maybe making
some improvements to share back).

Bob Jolliffe

unread,
Jan 16, 2018, 4:13:21 PM1/16/18
to
On the subject of documentation of "event based" vs "threaded" vs "co-routines" there is a very nice piece of documentation that Arjen Markus has done here (http://www.magicsplat.com/articles/queueing.html). Using each of these paradigms to illustrate the solution to the same problem. By no means a "hello world" example, but I something I have found useful.

Debates around threads vs events are legendary and at one point in time took on an near religious fervour. My own take is that:
1. synchronous code can be easier to read than asynchronous code
2. it can thus also easily become overused : having many more threads than you have CPU cores quickly becomes counter-productive in terms of performance overhead.
3. thread-pools are almost always a good idea for multi-threaded programs to stop programs getting carried away with unrestrained thread creation. Java programs used to be notorious for this.
4. bugs in threaded programs can be really hard to pin down
5. it's not always an either-or proposition. Having a pool of threads managing event driven IO tasks can be a really good architecture

I must confess that tcl co-routines intrigue me, though I am not sure I completely have my head around them ie. understanding the sort of patterns where a co-routine is the right thing to use instead of a thread. I suspect it comes down to enjoying the readability of synchronous style code without incurring the burden of managing the synchronisation problems which lurk beneath many threaded approaches. Taking the phrase of the OP, maybe the "cat's meow" is in fact to be found in these tcl co-routines.

It's wonderful to have all these tools in the toolbox.

Cheers
Bob

On Sunday, 14 January 2018 01:54:40 UTC, lmn...@gmail.com wrote:
> The tcl thread package is the cats meow, but doesn't seem to get talked about?
>
> Is there something shinier or newer?
>
> Am I late to the party?
>
> Here is my take:
>
> I have found the thread package highly useful to solve problems -- but had to overcome the learning curve and trials and tribulations to get to that point.
>
> I suppose this is limiting the usage and adoption on a wider scale?
>
> However, the obstacle seems easily overcome if community pages existed that shared some basic concepts and conventions (example code) so that folks get this info/training upfront. (Paves an easier path forward to adoption and rollout)
>
> For example: I now routinely leverage threading to solve problems 10x faster with 10x less code and complications vs any other methods.
>
> Given this backdrop...
>
> Questions:
>
> Is there an active tcl thread community page with resources for ramping up on tcl threads?
>
> Is there a shinier newer thing?
>
> Is this sentiment regarding under utilized and underappreciated usedulness of the top thread package shared? (I could be wrong?)
>
> I would like to see tcl thread go viral (as it deserves)...and I will surely contribute whatever I can.
>
>
> My motivation is two fold:
> 1. I've plateaued on the subject and crave a vibrant community with advanced subject discussions available to tap into.
>
> 2. I like what I like and I want others to like it too. Its awesome!
>
> Jim

two...@gmail.com

unread,
Jan 16, 2018, 4:47:24 PM1/16/18
to
On Tuesday, January 16, 2018 at 12:51:38 PM UTC-8, Gerald Lester wrote:

> BTW, can you share your app (like maybe via http://chiselapp.com/) -- I
> have a couple of Rukos and would be interested in it (and maybe making
> some improvements to share back).
>

I will post a new thread (no pun) with the code one can use to send commands
to a roku.

Rich

unread,
Jan 16, 2018, 4:51:38 PM1/16/18
to
Bob Jolliffe <bobjo...@gmail.com> wrote:
> I must confess that tcl co-routines intrigue me, though I am not sure
> I completely have my head around them ie. understanding the sort of
> patterns where a co-routine is the right thing to use instead of a
> thread.

One analogy that fits ok is to think of a coroutine as a special type
of proc where you can ask Tcl to suspend execution of the proc in such
a way that you can later restart _at the point you left off_ in the
proc. As part of suspending the proc, you also get to return a value
out to your caller.

This example is from Tcl's coroutine manpage (with a bit of annotation):

# just a basic proc
proc allNumbers {} {
yield
set i 0
while 1 {
yield $i
incr i 2
}
}

# create a "special proc"
coroutine nextNumber allNumbers

# call the "special proc" multiple times
for {set i 0} {$i < 10} {incr i} {
puts "received [nextNumber]"
}

Notice that the "proc" is an infinite loop (while 1 {...}) with a
special command inside the loop (yield).

The 'yield' command is how you ask Tcl to "stop running this proc here,
and do a 'return' of this value (the $i in the example).

The way you "jump back in to the middle of things" is by calling the
proc multiple times (the for loop at the bottom calls the proc ten
times in a row).

Normal procs start executing again at their first line each time you
call them.

"Special" coroutine procs jump back into the middle of where they were,
and continue running from that point. So for this example, the second
call to "nextNumber" causes the proc to restart as if the 'yield' had
just finished running and nothing special had happened.

> I suspect it comes down to enjoying the readability of synchronous
> style code without incurring the burden of managing the
> synchronisation problems which lurk beneath many threaded approaches.

That is one of the benefits. Another benefit is that they can let you
write event driven code that looks like basic straight sequential code
on the surface, but that is really event driven instead. This avoids
having to snake your control flow through multiple different procs that
each jump around to another in some callback. The wiki has a page for
a coroutine enabled command line that is not too hard to follow at
http://wiki.tcl-lang.org/24060.

The main command line looks like this:

proc repl {} {
while 1 {
set cmd [get-command]
set code [catch { uplevel #0 $cmd } result opts]
if {$code == 1} {
puts [dict get $opts -errorinfo]
} else { puts $result }
}
}

Which looks like it is an infinite loop that waits for get-command to
return something (and if written normally, *is* such an infinite loop).
But because it is inside a coroutine, the loop gets put aside until
there is input to process.

> Taking the phrase of the OP, maybe the "cat's meow" is in fact to be
> found in these tcl co-routines.

They can make some rather snakey looking control flow paths look
straight and narrow instead.

Uwe Klein

unread,
Jan 17, 2018, 11:01:22 AM1/17/18
to
Am 14.01.2018 um 18:40 schrieb lmn...@gmail.com:
> I have searched and use tcl/wiki a lot, unfortunately to no avail on this topic.


http://wiki.tcl.tk/1904

Uwe
0 new messages