ANNOUNCE: Stubs: A dynamic linking mechanism for Tcl, Tk and extensions.
Paul Duffin
==============================================================
What does it do ?
-----------------
1) Adds support for backlinking to all platforms, thus making it possible for
static executables to dynamically load extensions on any platform.
2) Eliminates all the operating system specific problems associated with
dynamically linking an extension to Tcl/Tk and any other library which
exports a stub interface. This may sound like a wild claim but it is true
simply because the operating system does not have to do anything.
3) Extensions and libraries built with different compilers will work together
even if it is not possible to link them normally. The only requirement for
them to be able to work together is that function calls are compatible.
4) As extensions do not have a hard coded reference to the library it is
possible to use them with any library which is compatible. e.g. an extension
which was originally built for Tcl 8.0 should work with Tcl 8.1, and an
extension built for Tcl 8.1 could work with Tcl 8.0.
How does it do it ?
-------------------
Basically the stub mechanism provides a cross platform dynamic linking
mechanism using tables of function pointers.
o A library which wishes to provide a stub interface populates a function
table with the addresses of its interface functions.
o A pointer to that table is then made available to any extension which needs
to access that library's interface.
o Once an extension has obtained the pointer it simply uses it to get the
address of the function it wants to call and calls it.
o At no time in this process does the operating system become involved.
To see the differences take a look at the following which describes what happens
when Tcl loads two different extensions.
o Loading an extension which is linked directly to Tcl.
1) Tcl asks the operating system to load the extension.
2) The operating system loads the extension and then tries to resolve any
undefined symbols. This process is very operating system dependent but
involves one or more of the following steps.
1) Resolve any symbols which are defined in the current process
context. (Backlinking).
2) Find and load any libraries that the extension is dependent on.
This involves searching paths defined through a variety of
operating system dependent methods.
3) Resolve any symbols which are defined in the new libraries.
3) Tcl then calls the extension's initialisation entry point.
o Loading an extension which uses Tcl's stub interface.
1) Tcl asks the operating system to load the extension.
2) The operating system does so, resolving any non Tcl symbols.
3) Tcl calls the extension's initialisation entry point.
4) The extension obtains the pointer(s) to Tcl's stub table(s) and uses
that to call Tcl.
Where is it ?
-------------
ftp://www.neosoft.com/pub/tcl/sorted/packages-8.0/devel/StubPatch804.tar.gz
How to apply it ?
-----------------
The StubPatch804.tar.gz file contains the following files.
Tcl_StubPatch.diff The patch for Tcl.
Tk_StubPatch.diff The patch for Tk.
README This file.
Unpack them as usual. e.g.
gunzip -c StubPatch804.tar.gz | tar -xvf -
These patches should only be applied to vanilla 8.0.4.
Move into the Tcl directory which contains the generic directory and apply the
patch making sure to specify -p1. e.g.
cd Tcl8.0.4
patch -p1 < ../Tcl_StubPatch.diff
And do the same for Tk. e.g.
cd Tk8.0.4
patch -p1 < ../Tk_StubPatch.diff
You are now free to build it.
How to build it ?
-----------------
o Unix
o Tcl
There is one new configure option for Tcl.
--enable-stub
If this option is disabled (default) then you will simply get a
normal tcl installation plus a few enhancements.
If it is enabled then Tcl will provide a stub interface for
extensions to use. It will also build an archive library called
libtclstub*.a which is used to initialise and access Tcl's stub
interface.
You should delete the following files (if they exist) if you have
changed the value of this option.
tclBasic.o
o Tk
There are two new configure options for Tk; both of the require that Tcl
is built with stub support.
--enable-tcl-stub
If this option is disabled then Tk will link to Tcl in the normal
way.
If it is enabled then Tk will use Tcl's stub interface.
You should delete the following files (if they exist) if you have
changed the value of this option.
tkMain.o tkConsole.o
--enable-stub
If this option is disabled then Tk will not provide a stub
interface.
If it is enabled then Tk will provide a stub interface for
extensions to use. It will also build an archive library called
libtkstub*.a which is used to initialise and access Tk's stub
interface.
You should delete the following files (if they exist) if you have
changed the value of this option.
tkWindow.o
o Windows
o Tcl
Two new makefiles have been provided, one for Borland and one for
Visual C++.
makefile.vc.stub (The Visual C++ version)
makefile.bc.stub (The Borland version)
These makefiles build a version of Tcl which provides a stub
interface for extensions to use and will also build a static
library tclstub80.lib which is used to initialise and access Tcl's
stub interface.
You should delete the following files if they exist when building
with these makefiles for the first time.
tclBasic.obj
o Tk
Four new makefiles have been provide, two for Borland and two for
Visual C++.
makefile.vc.stub (The Visual C++ version)
makefile.bc.stub (The Borland version)
These makefiles build a version of Tk which uses Tcl's stub
interface.
You should delete the following files if they exist when building
with these makefiles for the first time.
tkMain.obj tkConsole.obj
(Visual C++ users will need to delete all .obj files)
makefile.vc.stub2 (The Visual C++ version)
makefile.bc.stub2 (The Borland version)
These makefiles build a version of Tk which uses Tcl's stub
interface and also provides one of its own, it will build a static
library tkstub80.lib which is used to initialise and access Tk's
stub interface.
You should delete the following files if they exist when building
with these makefiles for the first time.
tkMain.obj tkConsole.obj tkWindow.obj
o Macintosh
At the moment there is no 'makefile' available to build a Macintosh version.
However the code has been designed to work with the Macintosh and therefore
it should not be too difficult to create one. If anyone would like to try
then I will be happy to provide them with some more details about what needs
to be done.
Compatability of the stub interfaces with the standard interfaces ?
-------------------------------------------------------------------
The stub interfaces of Tcl and Tk provide access to almost all of the external
and internal functions both generic and platform dependent. The exceptions to
these rules are ones which it makes no sense to make available through the stub
interface. These include all Tcl command functions plus the following functions:
Tcl_Main
Tcl_AppInit
Tk_Main
Neither Tcl nor Tk provides access to the global variables which are available
with the standard interface. This is for two reasons, the first being that it
is bad practice to access global variables and the second being that they could
only really be accessed using macros which can be very dangerous, especially in
C++.
On Windows and Macintosh the emulated X calls are included as part of the Tk
stub interface.
How to enable an extension to use a stub interface ?
----------------------------------------------------
Using Tcl's stub interface.
1) Before the extension calls any Tcl function is must call Tcl_Required. This
is essentially the same as Tcl_PkgRequire except that the package name is
implicitly "Tcl". As this function is supported by both the standard and the
stub interface the call to it does not need to be conditional. The stub
interface version checks that stubs are supported and initialises the stub
table pointers.
2) The makefile needs to be modified to use the stub library (libtclstub*.a)
instead of the standard library, and the compile flags need to be changed
to define ACCESS_TCL_THROUGH_STUB.
3) An inline versions of the stub interface is available, it uses macros in C
and inline functions in C++ to access Tcl's interface. These will be used
if INLINE_TCL_STUB is defined before including tcl.h, alternatively you
can include tclStubInls.h yourself. The stub library is still needed to
ensure that the stub initialisation is done correctly.
Using Tk's stub interface.
1) Before the extension calls any Tk function it must call either Tk_Required
or Tk_Present. These are essentially the same as Tcl_PkgRequire and
Tcl_PkgPresent respectively except that the package name is implicitly "Tk".
These functions are implemented as a macro in the standard version of the
interface because Tk may not be present when they are called. The stub
interface version checks that stubs are supported and initialises the stub
table pointers.
2) The makefile needs to be modified to use the stub library (libtkstub*.a)
instead of the standard library, and the compile flags need to be changed
to defined ACCESS_TK_THROUGH_STUB.
Inline versions of the stub interfaces are available, they uses macros in C
and inline functions in C++ to access the library's interface. The Tcl inline
interface is used if INLINE_TCL_STUB is defined before including tcl.h, or you
can include tclStubInls.h directly. Similarly for Tk. The stub library is still
needed to ensure that the stub initialisation is done correctly.
Unix users should take a look at the tclConfig.sh and tkConfig.sh file to see
what variables are available to help configuring stubs.
How to create an extension with a different compiler ?
------------------------------------------------------
All that is needed to do this is to build the stub library with your compiler.
e.g. assuming that you have a Borland compiler and want to build an extension
which works with a Visual C++ built core you need to do the following.
1) Get a patched version of the source and make tclstub80.lib. e.g.
make -f makefile.vc.stub tclstub80.lib
2) Build your extension making sure that you use the Borland version of the
stub library.
Although it is possible to link objects built with different compilers together
in most cases, each combination of compilers requires a different process to do
it and starts to get very complicated when you have to link an extension to two
or more libraries which have been built with a combination of compilers and
compiler versions.
This method should work for all combinations of compilers because it only
requires that function calls are compatible; if they are not then the compilers
cannot be used together anyway.
Changes
-------
The following lists the changes made to Tcl.
o Improved the support for AIX shared libraries. By making use of
export/import files I was able to create Tcl and Tk libraries which are
shared objects rather than archives containing shared objects. This makes
it possible to dynamically load Tk on AIX now. Extensions which are
correctly configured should not need to make any changes, extensions which
are also libraries will need to create an export file and use that instead
of the library when linking. See the Tk configure.in file to see how it
does it.
o Extended the package mechanism to support stubs, and added the ability to
determine whether a package has been loaded without causing it to be loaded.
See the PkgRequire.3 man page for more details.
o Added ByteArray type. There is a major problem when porting extensions which
wish to handle binary data from 8.0 to 8.1. In 8.0 they can use string
objects to hold the binary data, unfortunately this will not work on 8.1
because the internal representation of strings is now UTF. Binary data in
8.1 is handled using ByteArray objects. Therefore I have added them to
improve the upward compatibility.
o For each function which takes a variable number of arguments I have added
a function which takes a va_list instead. This is so that I can create
wrapper functions around them. These new functions names are the original
function names appended with "VA"
The following lists the changes made to Tk.
o Modified Tk_Main and TkConsoleCreate to work properly with stubs. The
problem is that those functions and the other Tk functions that they use
should not actually be in the standard Tk library because they are only
really needed by applications. The problem that they cause with stubs is
that they are called before the Tk initialisation function is called and
therefore the pointers to the Tcl stub tables have not been properly set up
in the Tk library. The solution was to make them take a Tcl_Interp parameter
(which is used to get the Tcl stub table pointers) and to get them to
initialise the pointers as well. The change to Tk_Main is both binary
compatible and source compatible but the change to TkCreateConsole is not
because no interface is provided for those functions.
Future
------
There is one more thing that I have left to do which will make this mechanism
even more useful; that is to make the libraries use their stub interface to
access their own functions. This will allow dynamic extensions to change the
behavior of the libraries, either to fix a problem or to add some function.
Plus Patch
----------
The plus patches also have a stub interface available for them although it only
supports a subset of Tcl functions and one Tk function. They are available from
<http://home.wxs.nl/~nijtmans/plus.html#files80>
Acknowledgements
----------------
Jean-Claude Wippler <j...@equi4.com>
For the initial seed idea from which this all came from and also for
his continued support and feedback during development.
Jan Nitjmans <Jan.Ni...@wxs.nl>
For his ideas and feedback which improved the final result and made
me think.
Contact
-------
Paul Duffin <pdu...@mailserver.hursley.ibm.com>
--
Paul Duffin
DT/6000 Development Email: pdu...@hursley.ibm.com
IBM UK Laboratories Ltd., Hursley Park nr. Winchester
Internal: 7-246880 International: +44 1962-816880
Alexandre Ferrieux
>
> Stubs: A dynamic linking mechanism for Tcl, Tk and extensions.
> ==============================================================
>
Wow !!! This is a milestone in Tcl's history.
Would Scriptics be so kind as to push *this* to top priority into its
evaluation queue ? Such an impressive amount of work shouldn't be
allowed to be overlooked; inclusion in 8.0.x would help !
> There is one more thing that I have left to do which will make this mechanism
> even more useful; that is to make the libraries use their stub interface to
> access their own functions. This will allow dynamic extensions to change the
> behavior of the libraries, either to fix a problem or to add some function.
Even better, doing this for the Tcl library itself would open an easy
path to loadable binary patches (think of what the fileevent patches
would look like if the notifier were dynamized this way...)
> The plus patches also have a stub interface available for them although it only
> supports a subset of Tcl functions and one Tk function. They are available from
> <http://home.wxs.nl/~nijtmans/plus.html#files80>
Attn other extension writers: please please please follow Ian's pioneer
initiative !
-Alex
Paul Duffin
>
> Paul Duffin wrote:
> >
> > Stubs: A dynamic linking mechanism for Tcl, Tk and extensions.
> > ==============================================================
> >
>
> Would Scriptics be so kind as to push *this* to top priority into its
> evaluation queue ? Such an impressive amount of work shouldn't be
> allowed to be overlooked; inclusion in 8.0.x would help !
>
That would be nice wouldn't it.
> > There is one more thing that I have left to do which will make this mechanism
> > even more useful; that is to make the libraries use their stub interface to
> > access their own functions. This will allow dynamic extensions to change the
> > behavior of the libraries, either to fix a problem or to add some function.
>
> Even better, doing this for the Tcl library itself would open an easy
> path to loadable binary patches (think of what the fileevent patches
> would look like if the notifier were dynamized this way...)
>
I prefer to call them dynamic patches as opposed to binary patches. In
my mind a binary patch is one you apply to a binary file to modify it,
whereas a dynamic patch is one that happens at runtime.
> > The plus patches also have a stub interface available for them although it only
> > supports a subset of Tcl functions and one Tk function. They are available from
> > <http://home.wxs.nl/~nijtmans/plus.html#files80>
>
> Attn other extension writers: please please please follow Ian's pioneer
> initiative !
>
> -Alex
Creating a stub interface for an extension can take a long time, is
tedious and very difficult to get right. This is one reason why Jan is
only supporting a subset of the functions, he has had to do it all by
hand.
Right from the start I recognised that this would be a problem so I have
written a couple of Tcl scripts which can analyse C source code and
automatically generate the tables, stub library source, macros, inline
functions etc. If an extension writer which provides a C interface would
like to add a stub interface to their code I would recommend that they
get in touch with me and I will see what I can do to either give them
the tools or do the generation myself.
Paul Duffin
==============================================================
What does it do ?
-----------------
ftp://www.neosoft.com/pub/tcl/sorted/packages-8.0/devel/StubPatch804.tar.gz
o Unix
o Tcl
o Tk
o Windows
o Tcl
o Tk
o Macintosh
Tcl_Main
Tcl_AppInit
Tk_Main
Changes
-------
Future
------
There is one more thing that I have left to do which will make this
mechanism even more useful; that is to make the libraries use their stub
interface to access their own functions. This will allow dynamic extensions
to change the behavior of the libraries, either to fix a problem or to add
some function.
Plus Patch
----------
The plus patches also have a stub interface available for them although it
only supports a subset of Tcl functions and one Tk function. They are
available from
<http://home.wxs.nl/~nijtmans/plus.html#files80>
Acknowledgements
----------------
Jean-Claude Wippler <j...@equi4.com>
For the initial seed idea from which this all came from and also
for his continued support and feedback during development.
Jan Nitjmans <Jan.Ni...@wxs.nl>
For his ideas and feedback which improved the final result and made
me think.
Contact
-------
Paul Duffin <pdu...@mailserver.hursley.ibm.com>
--
Paul Duffin
DT/6000 Development Email: pdu...@hursley.ibm.com
IBM UK Laboratories Ltd., Hursley Park nr. Winchester
Internal: 7-246880 International: +44 1962-816880
[[Send Tcl/Tk announcements to tcl-an...@mitchell.org
Send administrivia to tcl-announ...@mitchell.org
Announcements archived at http://www.findmail.com/list/tcl_announce/
The primary Tcl/Tk archive is ftp://ftp.neosoft.com/pub/tcl/ ]]