Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
vanillawish: compile files for use inside a package
221 views
Skip to first unread message
Manfred Rosenberger
Jul 3, 2019, 4:52:34 PM7/3/19
to
I have a package "testPkg" and a testscript "test.tcl" in the same directory with this structure:
---------------------------
.../myDir
./testPkg/
./pkgIndex.tcl
./testPkg.tcl
./lib/
./lib.tcl
./test.tcl
./vanillawish-e5dc71ed9d-win64.exe
---------------------------
I want to compile the file lib.tcl
... using: vanillawish-e5dc71ed9d-win64.exe
vanillawish-e5dc71ed9d-win64.exe builtin:TDK/compiler testPkg/lib/*.tcl
---------------------------
these are my files
pkgIndex.tcl:
package ifneeded testPkg 0.00 "
[list source [file join $dir testPkg.tcl]];
[list source [file join $dir lib lib.tcl]];
"
testPkg.tcl
#
# lib.tcl
#
package provide testPkg 0.00
namespace eval testPkg {
variable packageHomeDir [file dirname [file normalize [info script]]]
}
lib.tcl
#
# lib.tcl
#
proc testPkg::report {} {
variable packageHomeDir
package require Tk
set b [button .b -text "press" -command "exit"]
pack $b
puts "\n"
puts " \$packageHomeDir -> $packageHomeDir"
puts "\n"
}
test.tcl
#
# test.tcl
#
set BASE_Dir [file normalize [file dirname [lindex $argv0]]]
lappend auto_path $BASE_Dir
package require testPkg
testPkg::report
---------------------------
run the script
tclsh test.tcl
with the files gives the result :
-> $packageHomeDir -> .../myDir/testPkg
---------------------------
now I want to compile the file "lib.tcl" and replace it inside the package with the compiled version ...
... but on Windows10/64bit the package crashes
... so what do I have to do, to create a package existing of compiled libraries?
thanks in advance, Manfred
---------------------------
.../myDir
./testPkg/
./pkgIndex.tcl
./testPkg.tcl
./lib/
./lib.tcl
./test.tcl
./vanillawish-e5dc71ed9d-win64.exe
---------------------------
I want to compile the file lib.tcl
... using: vanillawish-e5dc71ed9d-win64.exe
vanillawish-e5dc71ed9d-win64.exe builtin:TDK/compiler testPkg/lib/*.tcl
---------------------------
these are my files
pkgIndex.tcl:
package ifneeded testPkg 0.00 "
[list source [file join $dir testPkg.tcl]];
[list source [file join $dir lib lib.tcl]];
"
testPkg.tcl
#
# lib.tcl
#
package provide testPkg 0.00
namespace eval testPkg {
variable packageHomeDir [file dirname [file normalize [info script]]]
}
lib.tcl
#
# lib.tcl
#
proc testPkg::report {} {
variable packageHomeDir
package require Tk
set b [button .b -text "press" -command "exit"]
pack $b
puts "\n"
puts " \$packageHomeDir -> $packageHomeDir"
puts "\n"
}
test.tcl
#
# test.tcl
#
set BASE_Dir [file normalize [file dirname [lindex $argv0]]]
lappend auto_path $BASE_Dir
package require testPkg
testPkg::report
---------------------------
run the script
tclsh test.tcl
with the files gives the result :
-> $packageHomeDir -> .../myDir/testPkg
---------------------------
now I want to compile the file "lib.tcl" and replace it inside the package with the compiled version ...
... but on Windows10/64bit the package crashes
... so what do I have to do, to create a package existing of compiled libraries?
thanks in advance, Manfred
s.effe...@googlemail.com
Jul 4, 2019, 4:34:40 AM7/4/19
to
I don't experience a crash; however, I'm on Win 7 and didn't follow your steps precisely.
Nevertheless, I think I found a problem with the current compiler. I came up with the following minimal script called lib.tcl:
namespace eval testPkg {}
proc testPkg::report {} {
variable x
puts ""
}
If I start vanillawish and try to compile and run this, I get:
% package require compiler
1.7.1
% compiler::compile lib.tcl
% source lib.tbc
expected integer but got "X!z" at or near "X!z
2
+?!
2
13!
3
x
1
Dv
x
0"
I get the same error if I start the compiler using builtin:compiler lib.tcl.
I don't get the error if I use ActiveState's original TDK compiler for 8.5.
I also don't get an error if I modify the script in some way, e.g. if I initialise the (unused) variable:
namespace eval testPkg {}
proc testPkg::report {} {
variable x 0
puts ""
}
I also don't get an error if I omit the puts (but keep x uninitialized):
namespace eval testPkg {}
proc testPkg::report {} {
variable x
}
I'll put it on my watch list (no, no THAT watch list; the interesting problems watch list). If I don't find anything further enlightening, I'll file a ticket ... er ... where? ActiveTcl's GitHub repo was a one-shot thing. Is AndroWish the place where it really comes to life?
Nevertheless, I think I found a problem with the current compiler. I came up with the following minimal script called lib.tcl:
namespace eval testPkg {}
proc testPkg::report {} {
variable x
puts ""
}
If I start vanillawish and try to compile and run this, I get:
% package require compiler
1.7.1
% compiler::compile lib.tcl
% source lib.tbc
expected integer but got "X!z" at or near "X!z
2
+?!
2
13!
3
x
1
Dv
x
0"
I get the same error if I start the compiler using builtin:compiler lib.tcl.
I don't get the error if I use ActiveState's original TDK compiler for 8.5.
I also don't get an error if I modify the script in some way, e.g. if I initialise the (unused) variable:
namespace eval testPkg {}
proc testPkg::report {} {
variable x 0
puts ""
}
I also don't get an error if I omit the puts (but keep x uninitialized):
namespace eval testPkg {}
proc testPkg::report {} {
variable x
}
I'll put it on my watch list (no, no THAT watch list; the interesting problems watch list). If I don't find anything further enlightening, I'll file a ticket ... er ... where? ActiveTcl's GitHub repo was a one-shot thing. Is AndroWish the place where it really comes to life?
Manfred Rosenberger
Jul 4, 2019, 10:47:02 AM7/4/19
to
oh, thanks for testing
... It feels good not to be alone in the universe
... It feels good not to be alone in the universe
s.effe...@googlemail.com
Jul 4, 2019, 3:26:10 PM7/4/19
to
Found something deep in the tclcompiler module ...
In cmpWrite.c, function A85EncodeBytes.
I prepare bytesPtr (which points to 4 bytes in memory) to point to the value 0x84000000 which is -2080374784 in decimal.
There are two consecutive "if (word < 0) ..." in the function. The value of "word" after the first if is 544172512, the second if should never be entered. However, with optimizations of -O2 or above the code is executed! Depending on the specific version of MinGW I get different results for word then: -2147483648 (MinGW8.1.0) or -1126247488 (MinGW5.1.0). A negative value is clearly wrong to proceed with. The final result is that the generated .tbc file contains most likely some NULL bytes (it does for me). The encodeMap array is accessed with a negative index but it is usually surrounded by NULL bytes, therefore the likely result of NULL bytes in the .tbc file.
A quick workaround is to declare "word" as volatile. Will perhaps look at what MinGW's assembler output later this week or month to see what's going wrong.
Also, I don't really understand the comment /* Because some don't support unsigned long */. Wow, there must be some really special compilers out there. Some of the stuff in the source could really take advantage of unsigned arithmetic (or a rewrite in general, some parts are offensively ugly). And how old is this module? It's written in K&R style.
Well, that's about the problem I found. I have no idea if it is related to Manfred's problem, but if there's one problem with MinGW there may well be lurking more.
Hey Manfred ... can you look into your generated .tbc files if they contain NULL bytes? Would be a hint.
In cmpWrite.c, function A85EncodeBytes.
I prepare bytesPtr (which points to 4 bytes in memory) to point to the value 0x84000000 which is -2080374784 in decimal.
There are two consecutive "if (word < 0) ..." in the function. The value of "word" after the first if is 544172512, the second if should never be entered. However, with optimizations of -O2 or above the code is executed! Depending on the specific version of MinGW I get different results for word then: -2147483648 (MinGW8.1.0) or -1126247488 (MinGW5.1.0). A negative value is clearly wrong to proceed with. The final result is that the generated .tbc file contains most likely some NULL bytes (it does for me). The encodeMap array is accessed with a negative index but it is usually surrounded by NULL bytes, therefore the likely result of NULL bytes in the .tbc file.
A quick workaround is to declare "word" as volatile. Will perhaps look at what MinGW's assembler output later this week or month to see what's going wrong.
Also, I don't really understand the comment /* Because some don't support unsigned long */. Wow, there must be some really special compilers out there. Some of the stuff in the source could really take advantage of unsigned arithmetic (or a rewrite in general, some parts are offensively ugly). And how old is this module? It's written in K&R style.
Well, that's about the problem I found. I have no idea if it is related to Manfred's problem, but if there's one problem with MinGW there may well be lurking more.
Hey Manfred ... can you look into your generated .tbc files if they contain NULL bytes? Would be a hint.
undro...@gmail.com
Jul 6, 2019, 4:49:10 AM7/6/19
to
Stephan,
I believe you found it. Based on your explanation I've patched cmpWrite.c in
https://www.androwish.org/home/info/d74c9c42c0b57a11
and now have it working in Windows 32 bit (which BTW was as far as I've
tested the only affected platform, Windows 64 bit and Linux were fine).
Thanks,
Christian
I believe you found it. Based on your explanation I've patched cmpWrite.c in
https://www.androwish.org/home/info/d74c9c42c0b57a11
and now have it working in Windows 32 bit (which BTW was as far as I've
tested the only affected platform, Windows 64 bit and Linux were fine).
Thanks,
Christian
Manfred Rosenberger
Jul 6, 2019, 12:44:27 PM7/6/19
to
How can i check my .tbc file for a Null byte inside?
Manfred
s.effe...@googlemail.com
Jul 6, 2019, 6:16:11 PM7/6/19
to
Manfred,
the .tbc file is a text file. You will not really understand the contents, many small chunks that are A85 encoded. Nevertheless a text file. Open it with some not too simple text editor that will show glyphs for unusual characters like NULL.
Christian,
just for style, don't you think this chunk looks more pleasing to the eye than the original?
toEmit[0] = EN(word % 85UL); word /= 85UL;
toEmit[1] = EN(word % 85UL); word /= 85UL;
toEmit[2] = EN(word % 85UL); word /= 85UL;
toEmit[3] = EN(word % 85UL); word /= 85UL;
toEmit[4] = EN(word);
Compare this to:
toEmit[4] = EN(word / (85UL * 85 * 85 * 85));
word %= (85UL * 85 * 85 * 85);
toEmit[3] = EN(word / (85UL * 85 * 85));
word %= (85UL * 85 * 85);
toEmit[2] = EN(word / (85UL * 85));
word %= (85UL * 85);
toEmit[1] = EN(word / 85UL);
word %= 85UL;
toEmit[0] = EN(word);
And indeed, it's only a 32 bit problem combined with an optimizer bug. If sizeof(long) == 8, so not on Windows, the value will never be negative in the first place and everything's OK.
the .tbc file is a text file. You will not really understand the contents, many small chunks that are A85 encoded. Nevertheless a text file. Open it with some not too simple text editor that will show glyphs for unusual characters like NULL.
Christian,
just for style, don't you think this chunk looks more pleasing to the eye than the original?
toEmit[0] = EN(word % 85UL); word /= 85UL;
toEmit[1] = EN(word % 85UL); word /= 85UL;
toEmit[2] = EN(word % 85UL); word /= 85UL;
toEmit[3] = EN(word % 85UL); word /= 85UL;
toEmit[4] = EN(word);
Compare this to:
toEmit[4] = EN(word / (85UL * 85 * 85 * 85));
word %= (85UL * 85 * 85 * 85);
toEmit[3] = EN(word / (85UL * 85 * 85));
word %= (85UL * 85 * 85);
toEmit[2] = EN(word / (85UL * 85));
word %= (85UL * 85);
toEmit[1] = EN(word / 85UL);
word %= 85UL;
toEmit[0] = EN(word);
And indeed, it's only a 32 bit problem combined with an optimizer bug. If sizeof(long) == 8, so not on Windows, the value will never be negative in the first place and everything's OK.
undro...@gmail.com
Jul 7, 2019, 1:21:27 AM7/7/19
to
Stephan,
you're right. Finally, I got in check-in
https://www.androwish.org/home/info/ab2836e6c1dc53b8
for (i=0 ; i < 5 ; i++) {
toEmit[i] = EN(word % 85UL);
word /= 85UL;
}
Somehow citing T.S. Eliot:
"I grow old ... I grow old
I shall not write my loops unrolled"
you're right. Finally, I got in check-in
https://www.androwish.org/home/info/ab2836e6c1dc53b8
for (i=0 ; i < 5 ; i++) {
toEmit[i] = EN(word % 85UL);
word /= 85UL;
}
Somehow citing T.S. Eliot:
"I grow old ... I grow old
I shall not write my loops unrolled"
Manfred Rosenberger
Jul 7, 2019, 7:00:26 AM7/7/19
to
Am Sonntag, 7. Juli 2019 00:16:11 UTC+2 schrieb s.eff...@googlemail.com:
> Manfred,
> the .tbc file is a text file. You will not really understand the contents, many small chunks that are A85 encoded. Nevertheless a text file. Open it with some not too simple text editor that will show glyphs for unusual characters like NULL.
Stephan,
> Manfred,
> the .tbc file is a text file. You will not really understand the contents, many small chunks that are A85 encoded. Nevertheless a text file. Open it with some not too simple text editor that will show glyphs for unusual characters like NULL.
yes, I understood it in the same way. in notepad++ I can not see any NULL - character. What I can see are the CR and LF delimiter.
... so does it look like, that my .tbc file is OK but I can not load this file on any other reason?
Manfred
s.effe...@googlemail.com
Jul 7, 2019, 6:12:11 PM7/7/19
to
Wow, we're code golfing now ...
It seems that AndoWish is now officially the repo with the most active patches and users of TDK.
Manfred, we're still talking about the simple test from the beginning? Paste the .tbc here and I'll try it on my system.
It seems that AndoWish is now officially the repo with the most active patches and users of TDK.
Manfred, we're still talking about the simple test from the beginning? Paste the .tbc here and I'll try it on my system.
Manfred Rosenberger
Jul 7, 2019, 6:44:48 PM7/7/19
to
if {[catch {package require tbcload 1.6} err] == 1} {
return -code error "[info script]: The TclPro ByteCode Loader is not available or does not support the correct version -- $err"
}
tbcload::bceval {
TclPro ByteCode 2 0 1.7 8.6
1 0 11 5 0 0 8 0 4 1 1 -1 -1
11
(<E<!(H&s!+-!!
1
z
1
+!
5
x
4
,CHr@
x
15
8#;EF)5Nb3N&O+Ev7E-
x
0
p
6 0 54 6 0 0 24 0 3 6 6 -1 -1
54
Ib4.!%!dpL-tdW!-`8X!(9N<!9^w!!+?!!!i5|dv.Z/s!*ZyTv2Q/X!-BW<!-l4pv#!!
6
68#:v;B!
6
OX35w23!
6
x
14
ld^LCBFc78R4Lt6R%%
x
0
x
4
DP)*F
x
1
+!
x
4
LdV<+
x
27
LdV<+LdV<+%=sp@eF5SAV?lSAa0DE+WHFw
0
0
0 1
14
ld^LCBFc78R4Lt6R%%
0 0 0
x
15
rpwhC;Z2b3<?<+EfqT+
0
0
}
====== -------------------------
I have to check my code once again ... it currently works ...
... but what happend so far ... ?
s.effe...@googlemail.com
Jul 8, 2019, 5:44:38 AM7/8/19
to
I sourced the .tbc file which was OK. Running the proc however crashed my shell. I disassembled the report proc and found this at the beginning:
(0) push1 0 # "packageHomeDir"
(2) variable %v13498293
(7) unsetArrayStk 132
(9) nop
This is wrong, it should rather look like this:
(0) push1 0 # "packageHomeDir"
(2) variable %v0
(7) nop
(8) nop
(9) nop
I patched your .tbc file. Replace this cryptic line:
Ib4.!%!dpL-tdW!-`8X!(9N<!9^w!!+?!!!i5|dv.Z/s!*ZyTv2Q/X!-BW<!-l4pv#!!
With this line (5 characters after the leading Ib4.! replaced):
Ib4.!Pu(EK-tdW!-`8X!(9N<!9^w!!+?!!!i5|dv.Z/s!*ZyTv2Q/X!-BW<!-l4pv#!!
I could then run your .tbc file! This means that you're affected by the tclcompiler bug and also that NULL bytes are just likely outcomes but not necessarily the only outcome.
Try yourself, I'm sure the patch will work for you.
(0) push1 0 # "packageHomeDir"
(2) variable %v13498293
(7) unsetArrayStk 132
(9) nop
This is wrong, it should rather look like this:
(0) push1 0 # "packageHomeDir"
(2) variable %v0
(7) nop
(8) nop
(9) nop
I patched your .tbc file. Replace this cryptic line:
Ib4.!%!dpL-tdW!-`8X!(9N<!9^w!!+?!!!i5|dv.Z/s!*ZyTv2Q/X!-BW<!-l4pv#!!
Ib4.!Pu(EK-tdW!-`8X!(9N<!9^w!!+?!!!i5|dv.Z/s!*ZyTv2Q/X!-BW<!-l4pv#!!
I could then run your .tbc file! This means that you're affected by the tclcompiler bug and also that NULL bytes are just likely outcomes but not necessarily the only outcome.
Try yourself, I'm sure the patch will work for you.
Manfred Rosenberger
Jul 8, 2019, 5:22:53 PM7/8/19
to
0 new messages