Installing and using TDK from git

[henning.s...@gmail.com]

Hi,
I am trying to use the TDK provided on git hub (https://github.com/ActiveState/tdk). I'm running it on windows from a git bash shell, which apperently knows the configure command (I guess it uses MINGW in the backgroud).

First, I tried simply sourcing the app/tlcapp/entry.tcl. It then tells me that it can't find the compiler package.
I figured it must be in lib/tclcompiler. So I ran the configure script in this folder. Configure tells me: "WARNING: Can't find Tcl configuration definitions".

So my question is: Am I on the right track? And if so, how do I resolve this configuration issue?

Regards,
Henning

[stefan]

https://github.com/ActiveState/tdk clearly lacks basic build and deployment instructions (README.md, wiki?). Maybe file a GitHub issue also?

[Mike Griffiths]

I asked the same question on the Wiki, and we're definitely not the only ones having this problem.

There might be a file somewhere that I(/we all) haven't found which is meant to be the main entry point for it all, but I think it's just not meant to be run from the source like this, and there's a wrapper for the whole thing that's missing from the github repo.

[Mark Case]

I got it working and posted my findings on the wiki.

[Dave]

On 4/9/2018 11:14 PM, Mark Case wrote:
> I got it working and posted my findings on the wiki.
>

I took a look here: https://wiki.tcl.tk/3960 -- would that be right?
Could you post a URL to your findings?
Thanks.

--
computerjock AT mail DOT com

[Mark Case]

Yes, that’s the correct page - I posted as JMC. I’m going to try to find time within the next day or two to go through the code and document my changes.

[Dave]

On 4/10/2018 6:45 AM, Mark Case wrote:
> Yes, that’s the correct page - I posted as JMC. I’m going to try to find time within the next day or two to go through the code and document my changes.
>

Win7x64 using msys2 and the mingw gcc compiler
I removed the tdk_license stuff
Configure for lib/tclcompiler worked
Make fails with the compiler error:

gcc -DPACKAGE_NAME=\"tclcompiler\" -DPACKAGE_TARNAME=\"tclcompiler\"
-DPACKAGE_VERSION=\"1.7.1\" -DPACKAGE_STRING=\"tclcompiler\ 1.7.1\"
-DPACKAGE_BUGREPORT=\"\" -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1
-DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1
-DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1
-DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -DNO_VALUES_H=1 -DHAVE_LIMITS_H=1
-DNO_SYS_WAIT_H=1 -DNO_DLFCN_H=1 -DHAVE_SYS_PARAM_H=1
-DUSE_THREAD_ALLOC=1 -D_REENTRANT=1 -D_THREAD_SAFE=1 -DTCL_THREADS=1
-DTCL_WIDE_INT_TYPE=long\ long -DHAVE_STRUCT_STAT64=1 -DHAVE_LSEEK64=1
-DUSE_TCL_STUBS=1 -I"C:/Users/imdave/Downloads/tcl/tcl8.6.8/generic"
-I"C:/Users/imdave/Downloads/tcl/tcl8.6.8/unix"
-I/c/Users/imdave/Desktop/TDK/work/PREFIX/include -pipe -O2 -Wall
-Wno-implicit-int -c `echo cmpWPkg.c` -o cmpWPkg.o
cmpWPkg.c:2:20: fatal error: cmpInt.h: No such file or directory
#include "cmpInt.h"

I've searched the TDK source tree and, just in case, the Tcl source
tree. There's no cmpInt.h to be found

[s.effe...@googlemail.com]

> cmpWPkg.c:2:20: fatal error: cmpInt.h: No such file or directory
> #include "cmpInt.h"
>
> I've searched the TDK source tree and, just in case, the Tcl source
> tree. There's no cmpInt.h to be found
>
> --
> computerjock AT mail DOT com

TDK really needs build instructions. They were probably more concerned about removing code that reveals how the licensing algorithm works. Until then, take a look at the byte code loader here:

https://github.com/ActiveState/teapot/tree/master/lib/tbcload

[jmc....@gmail.com]

That header is part of the tbcload build. The Active State Tcl distributions include that header, so you could try downloading the latest ActiveTcl and then referencing its include folder in the tclcompiler Makefile, i.e. -I/c/ActiveTcl/include.

Another option is to download BAWT 0.7.3, which packages everything you need to build Tcl 8.6.8. It also include tbcload, so you could reference that folder in the tclcompiler Makefile, i.e. -I/c/BAWT-0.7.3/BawtBuild/Windows/x86/Release/Build/tbcload.

[Dave]

I found tbcload here (https://github.com/ActiveState/teapot) in what
appears to be a fairly recent version of teapot. So I've managed to
compile TDK's tclcompiler, tclparser and the missing tbcload. Trying a
simple test, the compiler does not work on windows -- either 32- or
64-bit -- compiled against Tcl 8.6.8. I compiled it on Ubuntu (against
tcl 8.6.4) and it does work.

It seems that there are NUL characters embedded in the compiler output
on windows. I don't know if that is a 8.6.8 vs. 8.6.4 issue or if it's a
windows issue. I will need to do some more debugging when I get a chance.

Once that's done, then the issue will be how to actually use TDK since
it's not clear how to build the installer and installed objects referred
in the docs.

[Dave]

I spent the day debugging my compile of TDK on windows.

Turns out there's most likely a compiler bug in gcc 6.3.0 and 7.3.0 --
from what I can tell, using -O2 resulted in some very similar code in
lib/tclcompiler/cmpWrite.c being optimized out. The assembler source is
a pain to read but it sure looked like the -O2 .s file optimized out a
small section of code. Ubuntu (which I've installed as a virtualbox VM)
is still using gcc 5.4.0 and that version of gcc does not exhibit the bug.

I've now managed to compile tclcompiler, tclparser and tbcload.

I've played around with the tclcompiler and it is pretty neat.

Next I figured I'd try one of the TDK apps. I chose the "check" app
since that seemed like it might be the easiest. I finally got it to run
with all it's "package required" working but it fails with the message:

can't read "starkit::topdir": no such variable

I'm guessing that TDK also needs starkit stuff (which was not included
on github). I'll make a further guess that all the packaging scripts
and/or installation scripts required by TDK were also not included on
github.

Just my opinion, but I'd say that the portion TDK dumped on github is
not at all usable in it's current state.

I've never messed with starkits and would not know how to proceed at
this point. The only Tcl script I have that I ever considered using
starkit also uses the img package and I am under the impression that
binary packages must be compiled into starkits in order to use them --
is that true?

[Harald Oehlmann]

Am 17.04.2018 um 04:57 schrieb Dave:
> can't read "starkit::topdir": no such variable

This variable is defined after a
package require starkit
starkit::startup
which are normally the first lines in the file "main.tcl" in the lowest
folder (typically app.vfs) of the application.
It contains the folder of the exe program (app.exe) which was started.

I personally run my starkit programs (and I suppose most do it) also
with wish by something like:
wish.exe -f app.vfs/main.tcl
Then you need an emulator like:
if {[catch {
package require starkit
starkit::startup
} ]} {
namespace eval starkit {set topdir .}

}

> I've never messed with starkits and would not know how to proceed at
> this point. The only Tcl script I have that I ever considered using
> starkit also uses the img package and I am under the impression that
> binary packages must be compiled into starkits in order to use them --
> is that true?

No, binary packages must not be included into a starkit.
But there might be additional code to make binary packages loadable
within a starkit, e.g. copying the binary package to temp or somewhere
else and then loading it.

---

As a side note, I thank you all for your efforts to make the tcldevkit
run. I would appreciate a fossil repository where you all feedback the
code and eventual guidance documents.

I am personally specially interested in tbcload / tbccompiler and to get
it as bundled package. Donald correctly stated that he wants to see some
action here to keep it run.

Thank you,
Harald

[Dave]

On 4/17/2018 1:20 AM, Harald Oehlmann wrote:
> No, binary packages must not be included into a starkit.
> But there might be additional code to make binary packages loadable
> within a starkit, e.g. copying the binary package to temp or somewhere
> else and then loading it.

Which is why I've never messed with these wrapper things. I use the jpeg
and icon things in the img package. Why tcl does not natively support
today's most universal image format (jepg) is a mystery to me. If one
>really< is concerned about memory usage (today?) then make the support
configurable.

[Dave]

There's probably not much more I can do with this.

If anyone is interested in my local fossil repository with my build, I
have put it on a file sharing site (it will last a week or so). There
were only a few changes to a couple of .c files to eliminate compiler
warnings, and I replaced all the autoconf stuff with current versions
based on tcl.m4 from 8.6.8, and I included tbcload from what appears to
be a fairly recent tclpro project on sourceforge.

Here's the link (good for a week): https://transfernow.net/32fgr5g7ql5t

It builds on win7 with a current version of MSYS2
(https://sourceforge.net/projects/msys2/) and with 32-/64-bit gcc 7.3.0
from MinGW gcc (https://sourceforge.net/projects/mingw-w64/) with win32
threading. I use the MinGW compilers to eliminate any dependency on
cygwin or any of the libgcc*.dll's and gcc's libpthread.dll.

[Donal K. Fellows]

On 17/04/2018 07:20, Harald Oehlmann wrote:
> No, binary packages must not be included into a starkit.

I thought that one of the USPs of the Tcl VFS system was that it *can*
load binary packages out of a starkit. It's the dependent dynamic
libraries (and any embedded executables) that need to be extracted first.

Donal.
--
Donal Fellows — Tcl user, Tcl maintainer, TIP editor.

[Christian Gollwitzer]

Am 17.04.18 um 20:28 schrieb Donal K. Fellows:

> On 17/04/2018 07:20, Harald Oehlmann wrote:
>> No, binary packages must not be included into a starkit.
>
> I thought that one of the USPs of the Tcl VFS system was that it *can*
> load binary packages out of a starkit. It's the dependent dynamic
> libraries (and any embedded executables) that need to be extracted first.

Yes, that is correct - regular binary packages can be wrapped in a
starkit and "package require"'d as usual. Maybe one has to adjust
auto_path in order for package require to find it, but usually it works.
Under the hood, indeed Tcl copies out the .dll (or .so, .dylib) into the
temp folder and loads it from there.

Of course, if that library then is linked to a dependent one, it does
not work because Tcl does not introspect the package to see if it needs
additional dependent libraries. However in th ecase of Img, the
libraries (libjpeg, libpng, libtiff) have been wrapped up in away that
they can be loaded.

Christian

[Dave]

I did a quick google for "tcl startkit source" and I get dozens of leads
whose pages contain even more links to various "kits" -- I could spend
days following the links. No wonder I gave up on this stuff years ago
when I first heard of it.

Is anyone aware of a source for a >recent< "starkit" source package and
how to >build< it using the latest Tcl?

Apparently one needs a "sdx" application. I don't want someone else's
"sdx" -- I want to build it from source so I know >exactly< what's
included. Is anyone aware of a source for that information?

[Mike Griffiths]

sdx is, itself, just a starkit - so you can grab it and use it to unpack itself to get the Tcl code (and binary packages) involved.

[Dave]

That is interesting. I will give that a try. Thank you.

[MartinLemburg@Siemens-PLM]

AFAIK JPEG and other formats and their libraries have incompatible licenses.
And reimplementing may cause legal issues as well.

For UI PNG is completely sufficient, since its transparency helps a lot, when designing UI icons. The lossless compression helps in times of desktop-zoom, when UI images gets scaled automatically.
JPEG images scaled to match the desktop-zoom are often very ugly - depending on the compression level/quality.

[Dave]

On 4/19/2018 8:26 AM, MartinLemburg@Siemens-PLM wrote:
> Am Dienstag, 17. April 2018 17:34:26 UTC+2 schrieb Dave:
>> On 4/17/2018 1:20 AM, Harald Oehlmann wrote:
>>> No, binary packages must not be included into a starkit.
>>> But there might be additional code to make binary packages loadable
>>> within a starkit, e.g. copying the binary package to temp or somewhere
>>> else and then loading it.
>>
>> Which is why I've never messed with these wrapper things. I use the jpeg
>> and icon things in the img package. Why tcl does not natively support
>> today's most universal image format (jepg) is a mystery to me. If one
>> >really< is concerned about memory usage (today?) then make the support
>> configurable.
>
> AFAIK JPEG and other formats and their libraries have incompatible licenses.
> And reimplementing may cause legal issues as well.

Then just make it a configure option to link against the officially
distributed JPG library -- just like about every other program that
deals with JPG's. Do a locate for libjpeg -- it's all over the place on
windows >and< the compiled libjpeg is distributed with the various
programs that use it. On unix, I'll bet close to 100% of systems already
have libjpeg installed.

>
> For UI PNG is completely sufficient, since its transparency helps a lot, when designing UI icons. The lossless compression helps in times of desktop-zoom, when UI images gets scaled automatically.
> JPEG images scaled to match the desktop-zoom are often very ugly - depending on the compression level/quality.

For UI, yes, you're right. However if you want a Tcl program to deal
with JPG's (your mother probably even knows what a JPG is) you have to
either convert it on the fly using >another< software package or else
research, locate, and install the img package. Without doing one of the
above one cannot even write a simple Tcl program to browse ones photo
collection >nor< can one distribute such a Tcl program without providing
instructions on installing img or (maybe?) have one of these starkits
>embed< the img package -- which does contain a version of libjpeg.

I'm just saying that it's a glaring omission to not have native jpeg
support IMO.

[Christian Gollwitzer]

Am 19.04.18 um 15:26 schrieb MartinLemburg@Siemens-PLM:

> Am Dienstag, 17. April 2018 17:34:26 UTC+2 schrieb Dave:
>> On 4/17/2018 1:20 AM, Harald Oehlmann wrote:
>>> No, binary packages must not be included into a starkit.
>>> But there might be additional code to make binary packages loadable
>>> within a starkit, e.g. copying the binary package to temp or somewhere
>>> else and then loading it.
>>
>> Which is why I've never messed with these wrapper things. I use the jpeg
>> and icon things in the img package. Why tcl does not natively support
>> today's most universal image format (jepg) is a mystery to me. If one
>> >really< is concerned about memory usage (today?) then make the support
>> configurable.
>
> AFAIK JPEG and other formats and their libraries have incompatible licenses.
> And reimplementing may cause legal issues as well.

No, that's not an issue here. Both the IJG JPEG library as well as
libpng have very liberal licenses (like the 3-clause BSD). And there
exists an extremely impressive from-scratch implementation by Sean
Barrett, which supports several standard image formats (including JPEG
and PNG) in a single C header file of only 7.5 kLOC:
https://github.com/nothings/stb
(There is more like this, e.g. a truetype renderer, a zlib
implementation, an ogg Vorbis decoder etc., all very impressive in small
pure C libraries)

>
> For UI PNG is completely sufficient, since its transparency helps a lot, when designing UI icons.

That is indeed a point in favour of PNG. The next evolution would be
support for a vector format like SVG.

Christian

[Peter Dean]

On 19/04/18 09:01, Dave wrote:
>> sdx is, itself, just a starkit - so you can grab it and use it to
>> unpack itself to get the Tcl code (and binary packages) involved.
>>
>
> That is interesting. I will give that a try. Thank you.

There's a starkit chapter in Brent Welch's tcl book which may help you.

http://www.beedub.com/book/4th/Starkit.pdf