starkit, Thread, source, vfs
Krys
I have a problem with Starpack. I create a starkit, which has:
3 files
1th files - create 3 threads and source 2-nd file
2-nd file - sources a 50 files
Each threads(with main program) source 2-nd file and call function form
2-nd file.
My application crash few seconds after start - System Error
(Windows), breach memory (Linux). This is not repeatability wrong.
This application is correct, if i run in standard tcl (no tclkit).
I try use: my tclkit, dqkit, basekit (ActiveState) [multithreading]
I think problem is source files in child Thread, but I don't sure...
My question
Is Metakit thread-safety ?
Is VFS thread-safety ?
Krys
Eckhard Lehmann
Krys wrote:
> This application is correct, if i run in standard tcl (no tclkit).
> I try use: my tclkit, dqkit, basekit (ActiveState) [multithreading]
Hmm, (at least) one Tclkit is missing in your list: that from
http://e-lehmann.de/index.php/tcltk/tclkits-and-extensions/
But I don't believe that it will behave different from the others.
Do you load files from a C extension anywhere - or does anything happen
that is not Tcl_FS* aware?
If you're building Tclkit by yourself anyway, you could build it with
debugging symbols (--enable-symbols) and then attach gdb (or a friend)
to see what exactly happens...
> My question
> Is Metakit thread-safety ?
> Is VFS thread-safety ?
Normally yes... I don't know exactly but I had no problems so far.
Eckhard
Michael Schlenker
Those are only then thread aware if you compiled them with
--enable-threads, otherwise random things may happen, this should be the
standard at least for dqkit, maybe for the AS basekit.
Depending on what kind of things you do, there may be other issues (i
don't think it is a good idea to write to a metakit file from different
threads at the same time, but don't know for sure.)
Do you use any other binary extension?
Michael
Krys
> Krys wrote:
>
>>This application is correct, if i run in standard tcl (no tclkit).
>>I try use: my tclkit, dqkit, basekit (ActiveState) [multithreading]
>
>
> Hmm, (at least) one Tclkit is missing in your list: that from
> http://e-lehmann.de/index.php/tcltk/tclkits-and-extensions/
>
> But I don't believe that it will behave different from the others.
So I tried with this version also. It behaves similarly.
I tried:
1. my tclkit 1 (tcl8.4.11+ Thread262)
2. my tclkit 2 (tclkit8.4.13+thread265)
I build my tclkit with the newest libraries.
3. dqkita (dqkit 8.4.13)
4. basekit (ActiveState) (w wersji 8.4 i 8.5)
5. http://e-lehmann.de/index.php/tcltk/tclkits-and-extensions/
(version 8.4.13sh 8.4.13win)
> Do you load files from a C extension anywhere - or does anything happen
> that is not Tcl_FS* aware?
> If you're building Tclkit by yourself anyway, you could build it with
> debugging symbols (--enable-symbols) and then attach gdb (or a friend)
> to see what exactly happens...
Ok, thank you. I will try debug problem.
Krys
Krys
Michael Schlenker wrote:
> Those are only then thread aware if you compiled them with
> --enable-threads, otherwise random things may happen, this should be the
> standard at least for dqkit, maybe for the AS basekit.
Yes, I bulid tclkit with --enable-threads. Normal threads act correct
in my starpack. This is complex application with a lot files (a lot of
threads, which source this same files). I described a example script. I
isolation my problem and write a test starpack, which I describe in
firsth post.
> Depending on what kind of things you do, there may be other issues (i
> don't think it is a good idea to write to a metakit file from different
> threads at the same time, but don't know for sure.)
Maybe, I dont know :)
> Do you use any other binary extension?
what do you by this understand ?
Thank You
Krys
Michael Schlenker
If you use any packages which are written in C those (like for example
TclVFS or Mk4Tcl) those need to be compiled for threading to pick up the
correct definitions of some functions from tcl.h. (In TEA compliant
packages this means you have to specify --enable-threads when
configuring the extension). If you only have pure Tcl code with no dll
or .so files added you should have no problems at that point (if you use
'load' anywhere or use package require, which invokes a load somewhere,
you are in trouble if the loaded package is not thread safe.
There are some differences between standard Tcl and a Tclkit, mainly in
the startup and initialization code, things like finding encodings,
timezone data, script libraries and seeding the auto_path.
The best thing would be if you can get a backtrace of the crash for a
symbols enabled built, to see where the crash happens.
Michael
Bryan Oakley
> Hello
>
> I have a problem with Starpack. I create a starkit, which has:
> 3 files
> 1th files - create 3 threads and source 2-nd file
> 2-nd file - sources a 50 files
> Each threads(with main program) source 2-nd file and call function form
> 2-nd file.
> My application crash few seconds after start - System Error
> (Windows), breach memory (Linux). This is not repeatability wrong.
> This application is correct, if i run in standard tcl (no tclkit).
> I try use: my tclkit, dqkit, basekit (ActiveState) [multithreading]
>
> I think problem is source files in child Thread, but I don't sure...
I was able to get a similar crash on windows with a starkit, threads and
multiple scans through the vfs. Jeff Hobbs was able to reproduce it and
told me he had a fix; not sure of the status of that fix, and I'm not
sure if this is the same problem or not.
My problem was that within a thread we 'package require'd a package that
had a conditional dependency on another package we didn't have. Each
failed 'package require' caused a re-scan of the vfs looking for the
missing package. It was during this process that we would get our crash
almost without fail. When I rearranged some code and removed the
dependency, and thus eliminated the multiple passes through the vfs, the
crashes went away.
Krys
Bryan Oakley wrote:
> I was able to get a similar crash on windows with a starkit, threads and
> multiple scans through the vfs. Jeff Hobbs was able to reproduce it and
> told me he had a fix; not sure of the status of that fix, and I'm not
> sure if this is the same problem or not.
>
> My problem was that within a thread we 'package require'd a package that
> had a conditional dependency on another package we didn't have. Each
> failed 'package require' caused a re-scan of the vfs looking for the
> missing package. It was during this process that we would get our crash
> almost without fail. When I rearranged some code and removed the
> dependency, and thus eliminated the multiple passes through the vfs, the
> crashes went away.
In my example script, I don't use package require within threads.
Thank You.
Krys
Krys
Michael Schlenker wrote:
> If you use any packages which are written in C those (like for example
> TclVFS or Mk4Tcl) those need to be compiled for threading to pick up the
> correct definitions of some functions from tcl.h. (In TEA compliant
> packages this means you have to specify --enable-threads when
> configuring the extension). If you only have pure Tcl code with no dll
> or .so files added you should have no problems at that point (if you use
> 'load' anywhere or use package require, which invokes a load somewhere,
> you are in trouble if the loaded package is not thread safe.
Yes, I use Mk4Tcl, I build this component with --enabled-threads. I
think dqkit and basekit (ActiveState) is build with --enable-threads.
My starpack build with runtime:tclkit, or dqkit, or
basekit(ActiveState), or Extension from
http://e-lehmann.de/index.php/tcltk/tclkits-and-extensions/ it crash.
> The best thing would be if you can get a backtrace of the crash for a
> symbols enabled built, to see where the crash happens.
Ok, thank you.
Krys
Krys
firsth file: program.tcl
second file: function.tcl
third file: this is only procedure, which I don't use in my script.
This is only for source.
==== program.tcl =====
package provide program 1.0
package require Thread
set homedir [file dirname [info script]]
cd $homedir
source sources/function.tcl
thread::create {
source sources/function.tcl
wypisz
thread::wait
}
thread::create {
source sources/function.tcl
after 10 wypisz
wypisz
thread::wait
}
proc wypisz1 {} {
puts "inna funkcja"
after 10 wypisz1
}
after 10 wypisz
wypisz
wypisz1
for {set i 0} {$i<10000} {incr i} {
puts "sources"
source sources/function.tcl
}
thread::wait
====function.tcl====
proc wypisz {} {
puts "nr watku [thread::id]"
tsv::set test test "test"
set test [tsv::get test test]
puts $test
after 0 wypisz
}
proc bigSources {} {
for {set i 0} {$i<10} {incr i} {
for {set i 0} {$i<50} {incr i} {
puts "sources"
source util${i}.tcl
}
}
}
bigSources
puts !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!Koniec
And this example, act correct in standard tcl, and crashed in starpack.
Eckhard Lehmann
Krys wrote:
> In my example script, I don't use package require within threads.
That doesn't matter. [package re] mostly sources files or loads
libraries... so it's basically the same as [source]'ing them directly
(plus some autoloading magic that [package re] implements).
Eckhard
Jeff Hobbs
The answer to the last is no. Tcl's core vfs is, but tclvfs was most
certainly not, which is the basis for starkits. I have corrected this
and the next release of ActiveTcl with basekits should work fine for you.
--
Jeff Hobbs, The Tcl Guy, http://www.activestate.com/
Bryan Oakley
Strictly speaking, "package require" has nothing to do with it. It was a
trigger that caused the vfs to be accessed more than necessary. This
access (either glob, source or maybe open) is what caused the crash.
The point being, there appears to be a bug with accessing the same vfs
directory and/or file from more than one thread.
Krys
proc source {f} {
set fd [thread::send [tsv::get mainThread mainThread] [list open
$f]]
set context [thread::send [tsv::get mainThread mainThread] [list
read $fd]]
thread::send [tsv::get mainThread mainThread] [list close $fd]
uplevel 1 $context
}
This procedure to improve situation, but not all. Sometimes my starpack
still crash, but rare.
Thank You
Krys
Krys
>
> The answer to the last is no. Tcl's core vfs is, but tclvfs was most
> certainly not, which is the basis for starkits. I have corrected this
> and the next release of ActiveTcl with basekits should work fine for you.
>
> --
>
> Jeff Hobbs, The Tcl Guy, http://www.activestate.com/
Is version available tclvfs on
http://tclvfs.cvs.sourceforge.net/tclvfs/tclvfs/ (files source) maybe
is already thread-safety ?
Krys