ANNOUNCE: Magicsplat Tcl/Tk 8.6.7-1.7.0 for Windows

[pal...@yahoo.com]

Announcing the release of the Magicsplat Tcl/Tk 8.6.7 for Windows binary distribution. See http://www.magicsplat.com/tcl-installer/index.html for details, included packages and downloads.

/Ashok

[Harald Oehlmann]

Am 06.09.2017 um 06:40 schrieb pal...@yahoo.com:
> Announcing the release of the Magicsplat Tcl/Tk 8.6.7 for Windows binary distribution. See http://www.magicsplat.com/tcl-installer/index.html for details, included packages and downloads.
>
> /Ashok
>

Ashok,
great work, thank you.

It is compiled with "Visual Studio 2015".

I personally would enjoy, that the x86 version would also install on x64
computers. I still only use 32 bit even on 64 bit windows.

Congratulations,
Harald

[Ashok]

Well, there are a few reasons why I prevent installing the 32-bit
version on 64-bit systems.

First, and most important, is laziness! It would add yet another
combination (32 on 64) that would need to be tested on the different
Windows versions and there are already too many.

Secondly, there are some subtle issues with path management, COM
registration etc. because of Windows's WoW64 virtualization that make me
avoid this configuration.

I'm curious as to why you prefer 32-bit. Do you find memory usage
significantly better as to make a difference? In fact on my 64-bit
system, I find the 64-bit tclsh uses less memory and commit than the
32-bit version. That is however without loading any data which will
obviously favour the 32-bit version.

/Ashok

[Harald Oehlmann]

Am 10.09.2017 um 12:54 schrieb Ashok:
> Well, there are a few reasons why I prevent installing the 32-bit
> version on 64-bit systems.
>
> First, and most important, is laziness! It would add yet another
> combination (32 on 64) that would need to be tested on the different
> Windows versions and there are already too many.
>
> Secondly, there are some subtle issues with path management, COM
> registration etc. because of Windows's WoW64 virtualization that make me
> avoid this configuration.
>
> I'm curious as to why you prefer 32-bit. Do you find memory usage
> significantly better as to make a difference? In fact on my 64-bit
> system, I find the 64-bit tclsh uses less memory and commit than the
> 32-bit version. That is however without loading any data which will
> obviously favour the 32-bit version.

Lazyness, never looked to 64 bit, sorry...

Thank you anyway, great !

Harald

[jm....@orens.fr]

Le mercredi 6 septembre 2017 06:40:45 UTC+2, pal...@yahoo.com a écrit :
> Announcing the release of the Magicsplat Tcl/Tk 8.6.7 for Windows binary distribution. See http://www.magicsplat.com/tcl-installer/index.html for details, included packages and downloads.
>
> /Ashok

Thank you very much, Ashok, for these binaries.

However, on Windows7 after switching to 64bit new release, we observe serious performance regression related to Tktreectrl and SQLite packages.

Does anybody knows why Tktreectrl package has bumped to version 2.4.2 ? (last version advertised on Tktreectrl'site is 2.4.1).

On SQLite's side, our previous version originated from compiled source in Androwish "Bonfire night"

Trying to understand the cause of observed perf regression, below is comparison of compile time options as returned from SQLite command "PRAGMA compile_options" :

Magicsplat 1.7 (SQLite 3.20.0) Androwish "Bonfire night" ((SQLite 3.15.1)

COMPILER=msvc-1911 COMPILER=gcc-4.7.0 20120120 (experimental)
ENABLE_COLUMN_METADATA ENABLE_COLUMN_METADATA
ENABLE_DBSTAT_VTAB ENABLE_DBSTAT_VTAB
ENABLE_FTS3
ENABLE_FTS3_PARENTHESIS ENABLE_FTS3_PARENTHESIS
ENABLE_FTS4 ENABLE_FTS4
ENABLE_FTS5 ENABLE_FTS5
ENABLE_JSON1
ENABLE_RTREE ENABLE_RTREE
ENABLE_UPDATE_DELETE_LIMIT ENABLE_UPDATE_DELETE_LIMIT
LIKE_DOESNT_MATCH_BLOBS LIKE_DOESNT_MATCH_BLOBS
UNTESTABLE (ex OMIT_BUILTIN_TEST) OMIT_BUILTIN_TEST
OMIT_DEPRECATED OMIT_DEPRECATED
OMIT_LOOKASIDE OMIT_LOOKASIDE
SECURE_DELETE SECURE_DELETE
SOUNDEX SOUNDEX
SYSTEM_MALLOC
THREADSAFE=1 THREADSAFE=1
USE_ALLOCA

Unfortunately I haven't found any "SYSTEM_MALLOC" and "USE_ALLOCA" related information on SQLite'site.

Last minor observation : the choice of 64bit version in the installer seems to installs allso 32bit version of Tcl+Tk binaries (but I may be wrong).

Thanks.

-jm

[jm....@orens.fr]

Le mercredi 6 septembre 2017 06:40:45 UTC+2, pal...@yahoo.com a écrit :

> Announcing the release of the Magicsplat Tcl/Tk 8.6.7 for Windows binary distribution. See http://www.magicsplat.com/tcl-installer/index.html for details, included packages and downloads.
>
> /Ashok

New try presenting SQLite's compile option differences in Magicsplat vs Androwish "Bonfire night":

=================================
Magicsplat :

COMPILER=msvc-1911
ENABLE_COLUMN_METADATA
ENABLE_DBSTAT_VTAB

ENABLE_FTS3_PARENTHESIS
ENABLE_FTS4
ENABLE_FTS5

ENABLE_RTREE
ENABLE_UPDATE_DELETE_LIMIT
LIKE_DOESNT_MATCH_BLOBS
UNTESTABLE (ex OMIT_BUILTIN_TEST)
OMIT_DEPRECATED
OMIT_LOOKASIDE
SECURE_DELETE
SOUNDEX

THREADSAFE=1


==================================
Andowish "Bonfire night" :


COMPILER=gcc-4.7.0 20120120 (experimental)
ENABLE_COLUMN_METADATA
ENABLE_DBSTAT_VTAB
ENABLE_FTS3
ENABLE_FTS3_PARENTHESIS
ENABLE_FTS4
ENABLE_FTS5
ENABLE_JSON1
ENABLE_RTREE
ENABLE_UPDATE_DELETE_LIMIT
LIKE_DOESNT_MATCH_BLOBS
OMIT_BUILTIN_TEST
OMIT_DEPRECATED
OMIT_LOOKASIDE
SECURE_DELETE
SOUNDEX
SYSTEM_MALLOC
THREADSAFE=1
USE_ALLOCA



-jm

[Ashok]

The easy question first :-) The treectrl is from the tktreectrl source
repository on SourceForge, not the release 2.4.1 file since the
repository has some additional bug fixes.

Regarding sqlite performance, I'm afraid cannot help there as all I do
is package binaries using the default build flags. The possible causes
for the differences might be

- Tcl versions (very unlikely to make a difference)

- Different SQLite versions and flags, quite possible that this might
make a difference. I don't know SQLite at all but I can imagine a flag
like USE_ALLOCA would make a difference to performance.

- Compiler difference between gcc and MS VC. Also possible. Current MS
VC is supposedly very competitive but one never knows.

Only way to tell I suppose is to compile the older sqlite version with
MS VC and see if it improves performance. The ALLOCA would still be
missing though!

Sorry, not much help but one workaround might be to replace the sqlite
in the magicsplat distro with the one from androwish and see if you get
the performance back. I believe the package would be compatible.

/Ashok

[jm....@orens.fr]

Le mercredi 6 septembre 2017 06:40:45 UTC+2, pal...@yahoo.com a écrit :

> Announcing the release of the Magicsplat Tcl/Tk 8.6.7 for Windows binary distribution. See http://www.magicsplat.com/tcl-installer/index.html for details, included packages and downloads.
>
> /Ashok

Thanks Ashok

Regarding 2.4.2 version of TkTreeCtrl, I've posted a bug report on sf.net :
https://sourceforge.net/p/tktreectrl/bugs/93/

Regarding SQlite, I've switched back to Androwish' binaries (wich is about 2x faster (raw but constant observation) for unknown reason)


-jm

[Ashok]

jm, I can try a rebuild of treectrl with a different set of compiler
flags. Would you be able to test it to see if it fixes the performance?

/Ashok

[jm....@orens.fr]

Yes Ashok, I'll be glad to test a new binary. Thanks.

-jm

[Ashok]

Could you please download and try the following versions:

https://1drv.ms/u/s!AglNWLb1CeJFgeo9aMgLTNO2daQ-vQ - this is 2.4.2
recompiled with a different set of compiler optimization flags

https://1drv.ms/u/s!AglNWLb1CeJFgeo-swoB9zsJXgufDQ - this is 2.4.1
compiled with Visual Studio 2017 (like the rest of magicsplat distro)

I'm hoping this will help narrow down the possible cause of slow down.

(Needless to say, please verify multiple tktreectrl packages are not all
in auto_path when you test).

Thanks

/Ashok

[Ashok]

Actually, please hold off for the moment from trying these files. I've
discovered another issue in the Tcl makefiles that I'm looking at.

/Ashok

[jm....@orens.fr]

In the meantime, if you are interested, below is a quick perf comparaison between SQLite's binaries of Androwish (3.15.1) vs Magicsplat (3.20.0) vs BAWT (3.20.0).
The mesures are roughly constant and are made on the same long calculation process (95% at SQL level with SQLite). (TkTreeCtrl is not involved here) :

Magicsplat (3.20.0) : 4 min 28 s
BAWT (3.20.0) : 2 min 16 s
Androwish (3.15.1) : 2 min 08 s

And below are the compile options of BAWT's SQLite library :

COMPILER=gcc-4.9.2

ENABLE_COLUMN_METADATA
ENABLE_DBSTAT_VTAB

ENABLE_FTS3_PARENTHESIS
ENABLE_FTS4
ENABLE_FTS5

ENABLE_JSON1
ENABLE_RTREE
ENABLE_UPDATE_DELETE_LIMIT
LIKE_DOESNT_MATCH_BLOBS
UNTESTABLE

OMIT_DEPRECATED
OMIT_LOOKASIDE
SECURE_DELETE
SOUNDEX

THREADSAFE=1

USE_ALLOCA


As an aside remark, since tDOM 0.9.0 package has integrated JSON processing, I think it's a good idea to activate SQLite's JSON's fonctions with ENABLE_JSON1 compile option.

-jm

[Ashok]

Ok, I'm hoping the following will fix sqlite. Please download
https://1drv.ms/u/s!AglNWLb1CeJFgeo_4awjflkXkBkewQ

This is a sqlite build that should *replace* the one in the distribution
(as a check, the new dll size is just over 1MB compared to the existing
one which is 1.2+)

Eagerly awaiting to see if this performs any better...

/Ashok

PS If your benchmark code is self-contained, I'd appreciate a copy if
possible.

[jm....@orens.fr]

Hello Ashok,
With the new Magisplat's SQLite 3.20.0 compiled version, running the same program in the same environment, settings etc. gives 2min 10s (!!) +- 2s.

BUT for all these measures, I forgot to run SQLite's "PRAGMA ANALYZE" command, wich gather internal stats inside the db for improved perf.

After running "PRAGMA ANALYZE" below are a new set of timings (same environment etc. and with the new Magisplat's compiled version) :

Androwish (3.15.1) + "PRAGMA ANALYZE" : 1 min 12s
Magicsplat (3.20.0) + "PRAGMA ANALYZE" : 1 min 05s
BAWT (3.20.0) + "PRAGMA ANALYZE" : 1 min 04s

The "benchmark" is actualy a test db version of our production payroll system with around 600 people processed.


I will try to cook sommething for quickly benchmarking SQlite with Tcl, and report here around next tuesday : stay tuned :)
(for serious measurements regarding SQLite, I guess Mr Hipp of SQLite should have what's needed)

-jm

[Ashok]

Glad that cleared up. tktreectrl also likely is the same issue of the
optimization level not being set. I've sent an email to the tcl core
mailing list as to the compiler options being used. Assuming it was only
an oversight, and not intentional, I hope to have an updated
distribution in a week or so.

I'm very glad you reported this as it impacts anyone doing Tcl and
package builds with the Microsoft tool set.

And I look forward to your quick benchmark for future regression
testing. As you said, SQLite does have a comprehensive test suite but it
is not really suitable for testing from a "generic" Tcl as it relies on
custom test harnesses afaict.

/Ashok

[Ashok]

jm

I've released a recompiled version - 8.6.7-1.7.1 - which I hope fixes
the SQLite performance issue. Would appreciate if you try it out and let
me know.

Thanks

/Ashok

[jm....@orens.fr]

High, Ashok,

Will try your script today or tomorrow. Will report ASAP.

Test script is done : around 400 LOC with comments (gave me more work than I thought).
It exercises some parts of both SQLite + Tcl : runs in around 1 min on my PC (with transactions activated + confortable SQLite's memory cache).
If wanted, can be a basis for pursuing more deep explorations / testings : you'll see.

Where can I upload it ?

Thanks

-jm

[Ashok]

jm,

I don't have an upload area as such. Could you either attach it as a
ticket at https://sourceforge.net/p/magicsplat/tickets or email it to
the address palmtcl at yahoo.

Thanks much, I think it will be a big help in verification in the future.

/Ashok

[jm....@orens.fr]

Le mardi 19 septembre 2017 12:28:53 UTC+2, Ashok a écrit :

Hello Ashok

I've installed Magicsplat version 1.7.1 wich runs very fine with SQLite :
Runing with same long running script as before gives :

Magicsplat 1.7.1 : 1 min 1s (+- 2s)
vs
Magicsplat 1.7.0 with recompiled Sqlite lib of sept 15 : 1 min 6s(+- 2s)

Apart Sqlite perf improvements, are there other changes in 1.7.1 vs 1.7.0 ?

Thank you again for this.

-jm

[Ashok]

No functional changes but several extensions which did not have
optimization turned on by default are now optimized.

Treectrl also one those in this category so curious as to whether you
see that perform better as well.

Regards,

/Ashok

[jm....@orens.fr]

Hello Ashok,

There are still some pb with TkTreeCtrl when individual items are filled with heavy individual payload.

Below is a comparison of timings with 3 different kinds of payload. All payloads come from SQLite storage.
Payload 1 = 1000 TkTreeCtrl items filled from 1000 records comprised of 35 fields containing only numbers or text < 20 car.
Payload 2 = 6 TkTreeCtrl items filled from 6 records comprised of 16 fields whith one of them containing strings of around 1800 car. (other fields are the same as Payload 1)
Payload 3 = 2 TkTreeCtrl items filled from 2 records comprised of 7 fields
with one of them containing a blob of around 25 Ko (other fields are the same as Payload 1)

Comparison :

BAWT compile (TkTreeCtrl version 2.4.1)
Payload 1 : 3 s. (+- 1s.)
Payload 2 : < 1 s.
Payload 3 : < 1 s.

SourceForge compile (TkTreeCtrl version 2.4.1)
-> the same as BAWT

MagicSplat 1.7.1 compile (TkTreeCtrl version 2.4.2)
Payload 1 = 4 s. (+- 1s.)
Payload 2 = 10 s. (+- 1s.) <--
Payload 3 = 7 s. (+- 1s.) <--


Regards,

-jm

[Ashok]

I'll follow up on email
/Ashok