configure tcl8.6.8 to compile sqlite command line tool

[Adam Jensen]

Hi,

I am compiling Tcl 8.6.8 from source. Is there a way to indicate, perhaps
through ./configure that the SQLite command-line interface should be
built and installed? The directory:

tcl8.6.8/pkgs/sqlite3.21.0/compat/sqlite3

Contains the files:

shell.c spaceanal.tcl sqlite3.c sqlite3ext.h sqlite3.h

The file "shell.c" is present so I guess that's a good sign.

https://www.sqlite.org/howtocompile.html

I suppose it could be done manually but if a more clean approach is
possible then that's the way to go.

Any ideas, hints, pointers, descriptions, or instructions will be greatly
appreciated!

Thanks

[Roderick]

On Mon, 24 Sep 2018, Adam Jensen wrote:

> I suppose it could be done manually but if a more clean approach is
> possible then that's the way to go.

If you want sqlite, then more clean is to download sqlite, read
the instructions to compile it and do it. But ...

https://www.sqlite.org/howtocompile.html

... you will have to do it manually! :)

[Adam Jensen]

That's easy enough but the goal is to have an SQLite command-line
interface that has exactly the same capabilities and behavior as the
sqlite that is built for tcl. This means it should be the same version
that is compiled and linked with the same flags. My current solution,
which is very sloppy and barely a workable solution, is to capture the
compiler flags for sqlite during the tcl build then use those flags to
compile the sqlite shell. It ends up looking like this:

tar xzf tcl8.6.8-src.tar.gz -C "$MAAPSS/bld/"
cd "$MAAPSS/bld/tcl8.6.8/unix"
./configure --prefix="$MAAPSS/usr" --enable-threads --enable-64bit --
enable-threadsafe --enable-dynamic-extensions --enable-fts5 --enable-rtree
\
PKG_CONFIG_PATH="$MAAPSS/usr/lib/pkgconfig" LDFLAGS="-L$MAAPSS/usr/lib"
CPPFLAGS="-I$MAAPSS/usr/include" \
CFLAGS="-march=native -O2" CXXFLAGS="-march=native -O2"
make
make install
cd "$MAAPSS/bld/tcl8.6.8/pkgs/sqlite3.21.0/compat/sqlite3"
gcc -DHAVE_READLINE=1 -DPACKAGE_NAME=\"sqlite\" -DPACKAGE_TARNAME=\"sqlite
\" -DPACKAGE_VERSION=\"3.21.0\" -DPACKAGE_STRING=\"sqlite\ 3.21.0\" \
-DPACKAGE_BUGREPORT=\"\" -DPACKAGE_URL=\"\" -DBUILD_sqlite=/\*\*/ -
DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1
\
-DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1
-DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -DHAVE_LIMITS_H=1 \
-DHAVE_SYS_PARAM_H=1 -DUSE_THREAD_ALLOC=1 -D_REENTRANT=1 -D_THREAD_SAFE=1
-DTCL_THREADS=1 -DSQLITE_THREADSAFE=1 -DUSE_TCL_STUBS=1 \
-DUSE_TCLOO_STUBS=1 -DMODULE_SCOPE=extern\ __attribute__\(\(__visibility__
\(\"hidden\"\)\)\) -DHAVE_HIDDEN=1 -DHAVE_CAST_TO_UNION=1 \
-D_LARGEFILE64_SOURCE=1 -DTCL_WIDE_INT_IS_LONG=1 -DUSE_TCL_STUBS=1 -
DHAVE_MALLOC_H=1 -DHAVE_FDATASYNC=1 -DHAVE_USLEEP=1 -DHAVE_STRCHRNUL=1 \
-DHAVE_LOCALTIME_R=1 -DHAVE_GMTIME_R=1 -DHAVE_MALLOC_USABLE_SIZE=1 -
DHAVE_UTIME=1 -DHAVE_FLOCK=1 -DHAVE_READLINK=1 -DHAVE_LSTAT=1 \
-DHAVE_PREAD=1 -DHAVE_PREAD64=1 -DHAVE_PWRITE=1 -DHAVE_PWRITE64=1 -
DHAVE_DECL_STRERROR_R=1 -DHAVE_STRERROR_R=1 -DSQLITE_ENABLE_DBSTAT_VTAB=1
\
-DSQLITE_ENABLE_FTS3_PARENTHESIS=1 -DSQLITE_ENABLE_FTS4=1 -
DSQLITE_ENABLE_FTS5=1 -DSQLITE_ENABLE_COLUMN_METADATA=1 -
DSQLITE_ENABLE_JSON1=1 \
-DSQLITE_3_SUFFIX_ONLY=1 -DSQLITE_ENABLE_RTREE=1 -
DSQLITE_ENABLE_UPDATE_DELETE_LIMIT=1 -DSQLITE_LIKE_DOESNT_MATCH_BLOBS=1 -
DSQLITE_UNTESTABLE=1 \
-DSQLITE_OMIT_DEPRECATED=1 -DSQLITE_OMIT_LOOKASIDE=1 -
DSQLITE_SECURE_DELETE=1 -DSQLITE_SOUNDEX=1 -DSQLITE_USE_ALLOCA=1 -
DSQLITE_WIN32_NO_ANSI=1 \
-DSQLITE_WIN32_GETVERSIONEX=0 -DSQLITE_API=MODULE_SCOPE -DSQLITE_EXTERN= -
I"$MAAPSS/bld/tcl8.6.8/pkgs/sqlite3.21.0/generic" \
-I"$MAAPSS/bld/tcl8.6.8/generic" -I"$MAAPSS/bld/tcl8.6.8/pkgs/
sqlite3.21.0/.." -march=native -O2 -pipe -m64 -O2 -fomit-frame-pointer \
-DNDEBUG -Wall -fPIC -I"$MAAPSS/usr/include" shell.c sqlite3.c -L"$MAAPSS/
usr/lib" -I"$MAAPSS/usr/include" -I"$MAAPSS/usr/include/readline" \
-I"$MAAPSS/usr/include/ncurses" -lpthread -ldl -lreadline -lncurses -lm -
o sqlite3
cp sqlite3 "$MAAPSS/usr/bin/"
cd "$MAAPSS/bld/tcl8.6.8/unix"
make clean

This works but it's very manual in the sense that if the Tcl ./configure
options for SQLite change then the compiler flags for building the sqlite
shell are no longer valid and will need to be manually extracted from the
tcl build log and inserted into the above script blurb. But it is
currently working and it does meet the very basic requirements that there
is an SQLite command-line interface that has the exact same behavior as
the Tcl-SQLite extensions.

[Roderick]

On Tue, 25 Sep 2018, Adam Jensen wrote:

> That's easy enough but the goal is to have an SQLite command-line
> interface that has exactly the same capabilities and behavior as the
> sqlite that is built for tcl.

If you download the whole tcl fossil repository, you will see that
sqlite source is not there, it is only in the ready distribution.
You should get it then separately from the fossil repository of sqlite.
Then you compile the same version with the same capabilities as both,
command line and tcl extension.

Rodrigo

[Adam Jensen]

I would rather use tdbc::sqlite3 than tclsqlite and for a variety of
other reasons my current approach seems preferable but thanks for
suggesting an alternative.

A funny aside, while looking into the alternative approach, this happened
on Ubuntu Bionic Beaver:

fossil clone http://core.tcl.tk/tcl/ tcl.fossil
Round-trips: 149 Artifacts sent: 0 received: 164064
Clone done, sent: 38579 received: 154156127 ip: 74.208.161.176
Rebuilding repository meta-data...
*** buffer overflow detected ***: fossil terminated
Aborted (core dumped)

wtf?

[nemethi]

Download a more recent fossil version from

https://www.fossil-scm.org/fossil/uv/download.html

--
Csaba Nemethi http://www.nemethi.de mailto:csaba....@t-online.de

[Adam Jensen]

Yep, the most recent version, fossil-src-2.7.tar.gz, built from source,
doesn't core dump during the clone operation.

[Roderick]

On Wed, 26 Sep 2018, Adam Jensen wrote:

> Yep, the most recent version, fossil-src-2.7.tar.gz, built from source,
> doesn't core dump during the clone operation.

You can also clone the fossil repository with fossil.

[Adam Jensen]

I am not sure if using fossil-scm makes much sense for what I am doing.
The goal of this phase of my project is to create a set of scripts that
will fetch and compile an application platform infrastructure (for unix-
like systems) so that everyone using and developing the application is
operating with the same platform. I couldn't find anything like a current
release for tcllib, tklib, and some other Tcl components and extensions
so it looks like it might be necessary to include fossil as part of the
application's platform infrastructure. It's not a big deal other than, at
the moment, it seems like it complicates things in an unnecessary and
wasteful way: automated builds of the application platform infrastructure
shouldn't need to fetch and store the entire development history of
various software components in order to get access to a specific version
of those software components. If anyone knows how to get fossil to a
shallow clone of a repository such that it contains only a specific
version or tag or ID, if that is possible, I would like to hear about it.

Side not: Once the application platform infrastructure is compiled from
source, the application itself will consist of text files. Even though
the project has Tcl, Tk and SQLite at the core on the client side, and Tcl
projects seem to gravitate towards fossil, I've been evaluating svn
(subversion) as a way to distribute the text files (application scripts),
and it looks like the way to go for this project.

[Rich]

Adam Jensen <han...@riseup.net> wrote:
> I couldn't find anything like a current release for tcllib,

Top hit on google search (for me) of "tcllib":

https://www.tcl.tk/software/tcllib/

Right column, second link: *Download distributions*

Goes to a Fossil page where releases are provided, top link is:

Current: 1.19 (Feb 16, 2018) Details

Details link goes to a Fossil page where one can download a zip
snapshot or tar.gz snapshot from either a github or source forge mirror
without needing a fossil login.

> tklib,

Same process for Tklib as above, results in a very similar destination
where a zip or tar.gz of the current release can be downloaded.

> and some other Tcl components and extensions so it looks like it

"other components and extensions" might just be somewhere, only a bit
of digging required to uncover them.

[Adam Jensen]

Super cool. Well, it is for tcllib. Following a similar path to find a
stable release for tklib, the most recent looks to have happened on March
25, 2013.

https://core.tcl.tk/tklib/event/1728da5a7473a65d4c87bae4b2f88adfcb7562e5

It looks like there has been plenty of development activity since then:

https://core.tcl.tk/tklib/timeline

I did find that clicking the link of the last check-in ID leads to a page:

https://core.tcl.tk/tklib/info/7bbd844ae7128034

that includes a link to a tarball:

https://core.tcl.tk/tklib/tarball/7bbd844ae7/
Tk+Library+Source+Code-7bbd844ae7.tar.gz

How functional that version is, I have no idea - I seriously doubt it has
been tested in any significant way. It is a way to assure a specific
version and it avoids the fossil dependency and inefficiency but since
there doesn't appear to be any tested, stable releases for tklib it might
make more sense to use the fossil method so an update and checkout can
happen easily when problems with tklib are found. <heavy sigh>

[Michael Soyka]

If I understand what you're looking for, this worked for me:
fossil clone http://core.tcl.tk/tklib tklib.fossil

[Adam Jensen]

My current process looks like this:

#!/usr/bin/env bash
source globals.sh
cd "$MAAPSS/bld/repos"
FILE=tklib.fossil
if [ ! -f $FILE ]; then
echo "Fetching $FILE"

fossil clone http://core.tcl.tk/tklib tklib.fossil

fi
DIRECTORY="$MAAPSS/bld/tklib"
if [ -d "$DIRECTORY" ]; then
cd "$MAAPSS/bld/tklib"
fossil update
fossil checkout
else
mkdir -p "$MAAPSS/bld/tklib"
cd "$MAAPSS/bld/tklib"
fossil open "$MAAPSS/bld/repos/tklib.fossil"
fi
cd "$MAAPSS/bld/tklib"
./configure --prefix="$MAAPSS/usr" --with-tclsh="$MAAPSS/usr/bin/tclsh8.6"
make
make install
make clean


Once things settle the above process will change a bit so that the
checkout is for a specific revision ID. The goal is for the application
platform infrastructure to be consistent among all of the users/
developers and to have a known state that can be referenced.

[Rich]

One of the disadvantages of the 'git model' [1] of distributed source
control is that the concept of a "release tarball" has morphed into
"check it out from the repository".

It is unfortunately how things have shifted. The end result is the
same (a stable 'item' that can be downloaded) but the method of
obtaining the 'item' is not the same as before.

But I can also see the advantages from the maintainers perspective.
The repository already contains all the components, the interface
provides a "download a bundle from this tag/checkpoint/reference" so
there is a path to not needing to roll up a release bundle, not needing
double storage space to hold it, etc.

[1] used in a generic sense

[Rich]

This line depends on 'globals.sh' being in the same directory with your
script. Might not be a problem for your particular environment, but it
does make this script's execution dependent upon its current working
directory.

You might also consider the 'set -e' option to cause the script to
abort if anything below returns an error code, instead of blindly
trying to continue forward.

Your script could also be replaced by a makefile, from which you'd get
automatic handling of your 'if [ ! -f' blocks as part of how make
handles dependency targets.

[Adam Jensen]

On Wed, 26 Sep 2018 23:50:22 +0000, Rich wrote:

> This line depends on 'globals.sh' being in the same directory with your
> script. Might not be a problem for your particular environment, but it
> does make this script's execution dependent upon its current working
> directory.
>
> You might also consider the 'set -e' option to cause the script to abort
> if anything below returns an error code, instead of blindly trying to
> continue forward.
>

Thanks. I put a 'set -e' in 'globals.sh' a couple of days ago. Currently,
these shell scripts for fetching and compiling software are something of
a first draft, little more that notes that I am generating while I
explore the issues and develop the build process.

> Your script could also be replaced by a makefile, from which you'd get
> automatic handling of your 'if [ ! -f' blocks as part of how make
> handles dependency targets.

I am finding it to be very convenient to be able to copy snippets from a
text file and paste them into a shell. I am seriously considering that
the finished product might have a similar style. Consequently, the same
information will likely be repeated in several places throughout the
scripts which complicates maintenance and refinement but I think it will
make exploration and extension more convenient and accessible.

[Adam Jensen]

If a specific version can be transferred rather than the entire
development history - in git I think this is called a shallow clone; in
svn it's not an issue at all - and if there is a release engineering
process that regularly identifies a stable, tested version and tags it,
then incorporating something like that into one's infrastructure is
smooth sailing. Of course, the engineering process and the related social
organization and tooling involved in building and distributing reliable
infrastructure components has a lot of overhead. Sometimes it's easier or
more desirable to keep things loose and less structured - a kind of
exploration laboratory. Some refactoring and sober engineering is
probably required to move laboratory explorations into the development of
infrastructure components. (My impression is that fossil is best suited
to fringe laboratory-like explorations that regularly blink in and out of
existence).

[Roderick]

On Thu, 27 Sep 2018, Adam Jensen wrote:

> If a specific version can be transferred rather than the entire
> development history -

With the same fossil executable you can start a web server that
offers a tar.gz ball or zip file. You can also use a normal web server.

Start "fossil ui" and go to

http://localhost:8080/download

Rodrigo

[Rich]

Adam Jensen <han...@riseup.net> wrote:
> On Wed, 26 Sep 2018 23:46:23 +0000, Rich wrote:
>
>> One of the disadvantages of the 'git model' [1] of distributed
>> source control is that the concept of a "release tarball" has
>> morphed into "check it out from the repository".
>>
>> It is unfortunately how things have shifted. The end result is the
>> same (a stable 'item' that can be downloaded) but the method of
>> obtaining the 'item' is not the same as before.
>>
>> But I can also see the advantages from the maintainers perspective.
>> The repository already contains all the components, the interface
>> provides a "download a bundle from this tag/checkpoint/reference" so
>> there is a path to not needing to roll up a release bundle, not
>> needing double storage space to hold it, etc.
>>
>> [1] used in a generic sense
>
> If a specific version can be transferred rather than the entire
> development history - in git I think this is called a shallow clone;

For projects that have their repo. on Github, (and have uploaded the
relevant tags to Github) one can select a specific tag or branch and
download a zip file containing just the files comprising that exact
version (i.e., the delivery of a "release tarball" is by asking Github
to create a "release tarball" on the fly from the item with tag X).

I think the other github competitors provide a similar feature.

Fossil also provides a similar feature. The Tcllib release tarball I
referenced a few messages back in this thread is created on the fly by
Fossil from the development history storage, but contains only the
files as they existed at that tag/branch point, not the whole history.

> in svn it's not an issue at all - and if there is a release
> engineering process that regularly identifies a stable, tested
> version and tags it, then incorporating something like that into
> one's infrastructure is smooth sailing.

Incorporating the github/fossil links, provided that the URL's are
stable, and can be retreived without 'logging in' first, into an
automated setup is also easy. Where it gets complicated is when those
URL's "hide" behind a "log into your github (fossil) account first"
because that puts a barrier towards an automated curl/wget solution
that would have worked with a plain download link. [1]

> (My impression is that fossil is best suited to fringe
> laboratory-like explorations that regularly blink in and out of
> existence).

Well, if you read the descriptions of it, that is not the intent,
although much like Git it can be used in that manner if one likes. But
the web pages describing it and its features make statements about
using enduring file formats that will still be accessible in 100
years, so transient items that appear and as quickly disappear is not
the aim for Fossil at all.


[1] And often the reason given for the hiding of the release tarball
generation behind a login page is that generating a release tarball on
the fly is an intensive operation and so there should be some barrier
to triggering it at an excessive rate. Which is reasonable, but also
an indication that while an "on-the-fly" generator can almost
substitute for a plain download link, it is not at all idential in
every way to a pre-generated tar.gz sitting under an Apache or Nginx
webserver.

[Adam Jensen]

Hi, Rodrigo.

At this point, I think fossil will need be incorporated not only into the
build process for my application's platform but also into the platform.
It wasn't a simple decision.

It looks like some components of the platform might need to be updated or
downgraded more regularly than the more stable components. For example,
tklib: the last release was Tklib-0.6 on Mar 25, 2013. I think that
project just doesn't test and tag releases anymore. In cases like that, I
guess the thing to do is maintain a clone of their repository in my
application's platform so there can be a convenient mechanism in-place to
modify that component if problems arise. Cloning and maintaining fossil
repositories within the distributed platform is not a big deal for small
projects like tklib but the tcl repository is about 225MB and takes quite
a bit of time to clone. I am also a little spooked that the packaged
version of fossil on the latest Ubuntu dumped core while cloning the tcl
repository. While that doesn't happen with the latest version of fossil,
built from source, it still suggests there might be a fragility and/or
limitation there that might be a hassle to live with, but making do with
what's available can have its own charm.

Since I've pretty much committed to incorporating fossil into the
platform and maintaining several repositories within the platform, I will
probably follow your earlier advice and build tcl and its various
components (e.g., tdbc::sqlite3) from the repositories.

Anyway, thanks for engaging in this thread. I appreciate the opportunity
for conversation.

[Adam Jensen]

On Thu, 27 Sep 2018 11:04:44 +0000, Rich wrote:

> the web pages describing it and its features make statements about using
> enduring file formats that will still be accessible in 100 years

Having the entire history of a project in a single database file that can
be copied onto a thumb-drive and put into a box in the closet and
forgotten about but then accessed ten years later with the current
versions of sqlite and fossil are the characteristics that make it seem
to me like an ideal tool for fringe explorations, whimsical experiments
and other low-ceremony projects.

[Rich]

Hmm, same description, two *very different* net interpretations.

My take on Fossil's "single DB file, accessible still in 100 years" is
that it is good for something that would be getting updated across a
good portion of that timeframe because it would mean that one would not
need to be continually exporting/importing it to/from the latest flavor
of the day (and enduring the inevitable impedance miss-matches that
come from the export/import process).

I.e., it could be stuck with, throughout all of the "flavors of the
day" rising up and dying back down, without needing to be changed (and
still being accessible).

[Adam Jensen]

Fossil puts an impressive collection of project functions into a single
executable: DVCS, issue tracker, web site, wiki, blog, etc., and then all
of the data is in a single, enduring database file. That's everything in
two files. It seems like the goal is to make the system administration
side of things - installation, backup & restore, etc. - easy and low cost.

In my experience a high-priority, high-resource project will attract
people who want to fuss over it and be involved in the meticulous
curation of the artifacts, analysis of the processes, management of the
operations, and planning of the strategies. Accommodating this activity
typically involves plenty of customization and transformation of the
infrastructure as the project develops.

I am a little curious about fossil's internals - I haven't looked into it
yet. There might be some useful technology in there that could
potentially be pulled out as a collection of SQLite extensions and Tcl
extensions that could then be used in applications well beyond a revision
control system. For example, how in fossil are multiple databases
synchronized? There might be some novel technology involved in the way
fossil does this that could be extracted and made available as a much
more generic Tcl-SQLite capability.