TIP #115: Making Tcl Truly 64-Bit Ready
Donal K. Fellows
TIP #115: MAKING TCL TRULY 64-BIT READY
=========================================
Version: $Revision: 1.1 $
Author: Donal K. Fellows <donal.k.fellows_at_man.ac.uk>
State: Draft
Type: Project
Tcl-Version: 9.0
Vote: Pending
Created: Wednesday, 23 October 2002
URL: http://purl.org/tcl/tip/115.html
WebEdit: http://purl.org/tcl/tip/edit/115
Post-History:
-------------------------------------------------------------------------
ABSTRACT
==========
This TIP proposes changes to Tcl to make it operate more effectively on
64-bit systems.
RATIONALE
===========
It is a fact of life that 64-bit platforms are becoming more common.
While once the assumption that virtually everything was a 32-bit
machine (where not smaller) was valid, this is no longer the case.
Particularly on modern supercomputers (though increasingly in
workstations and high-end desktop systems too), the amount of memory
that the machine contains is exceeding 2GB, and the need to address
very large amounts of memory is certainly there in scientific and
engineering applications. And where they lead, consumer systems will
probably follow too.
At the moment, Tcl is ill-prepared for this. In particular, the type
used for expressing sizes of entities in Tcl (whether strings, lists or
undifferentiated blocks of memory) is _int_ (and cannot be made into an
_unsigned int_ in most of those places where it is not already an
unsigned value) but on the majority of 64-bit platforms this is still a
32-bit type, which is a major restriction. However, on the vast
majority of those platforms _long_ is a 64-bit type, and so a suitable
replacement. (The exception to this is the Alpha, but that is unusual
in that both _int_ and _long_ are 64-bit types there, meaning that the
platform will be unaffected by such an alteration.)
DETAILS OF CHANGES
====================
The key changes will be to change the lengths of the following types
from _int_ to _long_ in all appropriate places (maintaining the
_unsigned_ where it already exists; most length types in Tcl have to be
signed because negative values are used to flag that the length should
be figured out from the data itself.)
* _Tcl_Obj_ - the _length_ member. (Potentially the _refCount_
member needs updating as well, but that's less critical.)
* _Tcl_SavedResult_ - the _appendAvl_ and _appendUsed_ members.
* _Tcl_DString_ - the _length_ and _spaceAvl_ members.
* _Tcl_Token_ - the _size_ and _numComponents_ members.
* _Tcl_Parse_ - the _commentSize_, _commandSize_, numWords_,
_numTokens_ and _tokensAvailable_ members.
* _CompiledLocal_ - the _nameLength_ member.
* _Interp_ - the _appendAvl_, _appendUsed_ and _termOffset_
members.
* _List_ - the _maxElemCount_ and _elemCount_ members.
* _ByteArray_ - the _used_ and _allocated_ members.
* _SortElement_ - the _count_ member.
* _SortInfo_ - the _index_ member.
* _CopyState_ - the _toRead_ and _total_ members.
* _GetsState_ - the _rawRead_, _bytesWrote_, _charsWrote_ and
_totalChars_ members.
* _ParseInfo_ - the _size_ member.
* _String_ - the _numChars_ member (see also the _TestString_
structure.)
Changes to the bytecode-related structures might be worthwhile doing
too, though there are more backward-compatibility issues there.
These changes will force many of the types used in the public API to
change as well. Notable highlights:
* _Tcl_Alloc_ will now take an _unsigned long_.
* _Tcl_GetByteArrayFromObj_ will now take a pointer to a _long_.
* _Tcl_GetStringFromObj_ will now take a pointer to a _long_.
* _Tcl_ListObjLength_ will now take a pointer to a _long_.
* _Tcl_GetUnicodeFromObj_ will now take a pointer to a _long_.
In the internal API, the following notable change will happen:
* _TclGetIntForIndex_ will now take a pointer to a _long_.
There are probably other similar API changes required.
WHAT THIS TIP DOES NOT DO
===========================
This TIP does not rearrange structure orderings. Although this would be
very useful for some common structures (notably _Tcl_Obj_) if the
common arithmetic types were smaller than the word size, it turns out
that the changes in types required to deal with larger entities will
make these rearrangements largely unnecessary and/or pointless.
(Inefficiency in statically-allocated structures won't matter as the
number of instances will remain comparatively small, even in very large
programs.)
COPYRIGHT
===========
This document has been placed in the public domain.
-------------------------------------------------------------------------
TIP AutoGenerator - written by Donal K. Fellows
[[Send Tcl/Tk announcements to tcl-an...@mitchell.org
Announcements archived at http://groups.yahoo.com/group/tcl_announce/
Send administrivia to tcl-announ...@mitchell.org
Tcl/Tk at http://tcl.tk/ ]]
Ray Strode
> and so a suitable replacement. (The exception to this is the Alpha,
> but that is unusual in that both _int_ and _long_ are 64-bit types
there,
> meaning that the platform will be unaffected by such an alteration.)
64-bit SPARC long is 32-bit (and long long is 64 bit).
--Ray
Chang Li
> This TIP proposes changes to Tcl to make it operate more effectively on
> 64-bit systems.
>
What are the real benefits to be the true 64bit? A 4G list is not a good
data structure.
Chang
j...@mrc-lmb.cam.ac.uk
> 32-bit type, which is a major restriction. However, on the vast
> majority of those platforms _long_ is a 64-bit type, and so a suitable
> replacement. (The exception to this is the Alpha, but that is unusual
> in that both _int_ and _long_ are 64-bit types there, meaning that the
> platform will be unaffected by such an alteration.)
This is possibly OS dependent, but Digital Unix (subsequently renamed to
Compaq Tru64 Unix) does not have int as 64-bit. Rather we have:
int 32
long 64
Also you'll find that most 64-bit systems chickened out of making long 64bit
and instead broke the ANSI spec (apart from perhaps C9x) by creating a new
type "long long". Ie:
int 32
long 32
long long 64
Yes the Alpha probably is the exception, but in my mind DEC did it right. It
broke lots of (badly written) code, but in the long run it was a sensible
decision - who wants multiple 32-bit types?
One definite suggestion is that you use a typedef. Indeed size_t springs to
mind as that already exists as a type used by malloc, file offsets, etc.
> WHAT THIS TIP DOES NOT DO
> ===========================
>
> This TIP does not rearrange structure orderings. Although this would be
> very useful for some common structures (notably _Tcl_Obj_) if the
> common arithmetic types were smaller than the word size, it turns out
> that the changes in types required to deal with larger entities will
> make these rearrangements largely unnecessary and/or pointless.
> (Inefficiency in statically-allocated structures won't matter as the
> number of instances will remain comparatively small, even in very large
> programs.)
There's still some small improvement here to consider. Eg Tcl_Obj starts with:
typedef struct Tcl_Obj {
int refCount; /* When 0 the object will be freed. */
char *bytes; /* This points to the first byte of the
...
Unless we're changing refCount to be size_t then this still has 4 bytes of
padding. Also it's very easy to get many objects, so it does all add up.
A change of sizes will surely break the stub interface, so if we're doing that
it's time to bite the bullet and put as many changes in at once (although
admittedly as separate TIPs to keep things clean).
Anyway, thanks for the tip - my comments are meant to be constructive as I like
the idea.
James
--
James Bonfield (j...@mrc-lmb.cam.ac.uk) Fax: (+44) 01223 213556
Medical Research Council - Laboratory of Molecular Biology,
Hills Road, Cambridge, CB2 2QH, England.
Also see Staden Package WWW site at http://www.mrc-lmb.cam.ac.uk/pubseq/
Glenn Jackman
> What are the real benefits to be the true 64bit? A 4G list is not a good
> data structure.
How about:
% clock format [clock scan "35 years"]
Sat Oct 24 1:05:52 PM Eastern Daylight Time 2037
% clock format [clock scan "36 years"]
unable to convert date-time string "36 years"
--
Glenn Jackman
gle...@ncf.ca
Donal K. Fellows
> On my alpha int is 32-bit and long is 64-bit...However, I believe on
> 64-bit SPARC long is 32-bit (and long long is 64 bit).
I used to use 64-bit SPARC, and I can say for sure that long is 64-bit there
(and this is true on 64-bit IRIX as well.)
Donal.
--
Donal K. Fellows http://www.cs.man.ac.uk/~fellowsd/ donal....@man.ac.uk
-- Thanks, but I only sleep with sentient lifeforms. Anything else is merely
a less sanitary form of masturbation.
-- Alistair J. R. Young <avatar...@arkane.demon.co.uk>
Donal K. Fellows
So? Sometimes you need that much anyway. (Actually, at the moment you'll run
into trouble at about 250M elements, due to the pointer-size multiplier.)
lvi...@yahoo.com
According to Chang Li <cha...@neatware.com>:
:What are the real benefits to be the true 64bit? A 4G list is not a good
:data structure.
A 4 Gig image however may be exactly what one has to deal with ...
--
Tcl - The glue of a new generation. <URL: http://wiki.tcl.tk/ >
Even if explicitly stated to the contrary, nothing in this posting
should be construed as representing my employer's opinions.
<URL: mailto:lvi...@yahoo.com > <URL: http://www.purl.org/NET/lvirden/ >
j...@mrc-lmb.cam.ac.uk
> Ray Strode wrote:
> > On my alpha int is 32-bit and long is 64-bit...However, I believe on
> > 64-bit SPARC long is 32-bit (and long long is 64 bit).
>
> I used to use 64-bit SPARC, and I can say for sure that long is 64-bit there
> (and this is true on 64-bit IRIX as well.)
This possibly depends on compiler options. Certain gcc on Irix allows
-mlong64, but by default I get 32-bit longs. I wrote a simple test program:
#include <stdio.h>
int main(void) {
system("uname -sr");
printf("char = %ld\n", (long)sizeof(char));
printf("short = %ld\n", (long)sizeof(short));
printf("int = %ld\n", (long)sizeof(int));
printf("long = %ld\n", (long)sizeof(long));
printf("long long = %ld\n", (long)sizeof(long long));
printf("float = %ld\n", (long)sizeof(float));
printf("double = %ld\n", (long)sizeof(double));
printf("long double = %ld\n", (long)sizeof(long double));
printf("char * = %ld\n", (long)sizeof(char *));
return 0;
}
The results are:
SunOS 5.8 (cc and gcc, Ultra-5_10)
char = 1
short = 2
int = 4
long = 4
long long = 8
float = 4
double = 8
long double = 16
char * = 4
IRIX64 6.5 (gcc)
char = 1
short = 2
int = 4
long = 4
long long = 8
float = 4
double = 8
long double = 8
char * = 4
IRIX64 6.5 (cc)
char = 1
short = 2
int = 4
long = 4
long long = 8
float = 4
double = 8
long double = 16
char * = 4
IRIX 6.5 (gcc on an O2)
char = 1
short = 2
int = 4
long = 4
long long = 8
float = 4
double = 8
long double = 8
char * = 4
Linux 2.4.2-2 (gcc)
char = 1
short = 2
int = 4
long = 4
long long = 8
float = 4
double = 8
long double = 12
char * = 4
OSF1 V4.0 (alpha, cc and gcc)
char = 1
short = 2
int = 4
long = 8
long long = 8
float = 4
double = 8
long double = 8
char * = 8
I imagine on many of these systems there are compile time options to override
these settings.
Chang Li
<lvi...@yahoo.com> wrote in message news:apbeo3$a7m$1...@srv38.cas.org...
>
> According to Chang Li <cha...@neatware.com>:
> :What are the real benefits to be the true 64bit? A 4G list is not a good
> :data structure.
>
> A 4 Gig image however may be exactly what one has to deal with ...
>
It may be better to use an extension to deal with that size of images.
Chang
Chang Li
"Donal K. Fellows" <donal.k...@man.ac.uk> wrote in message
news:3DB9157E...@man.ac.uk...
> Chang Li wrote:
> >> This TIP proposes changes to Tcl to make it operate more effectively
on
> >> 64-bit systems.
> >
> > What are the real benefits to be the true 64bit? A 4G list is not a good
> > data structure.
>
> So? Sometimes you need that much anyway. (Actually, at the moment you'll
run
> into trouble at about 250M elements, due to the pointer-size multiplier.)
>
I'd like to see the improvement of memory allocation of Tcl.
So we can use the 4G memory in C-level for extensions.
Chang
Jacob Levy
* int == 32 bit
* long long == 64 bits
* long == sizeof(void *), i.e. long is whatever the size of a pointer is
on that specific system.
I know because I just spent the last 3 weeks at work rooting out all
32-bit-specific stuff on a Java VM, getting it to work in both 32 bit
and 64 bit systems from one code base.
--JYL
Donal K. Fellows
> On all "reasonable" unixes:
> * int == 32 bit
> * long long == 64 bits
> * long == sizeof(void *), i.e. long is whatever the size of a pointer is
> on that specific system.
Apparently there are systems where this is not the case. And there is at least
a rumour that there is a system existing where sizeof(long)<sizeof(void*) and
that rumour mentioned the words "Windows" and "IA64" and I was not surprised...
> I know because I just spent the last 3 weeks at work rooting out all
> 32-bit-specific stuff on a Java VM, getting it to work in both 32 bit
> and 64 bit systems from one code base.
I wish I could claim I was surprised. C's type size rules suck.
Donal K. Fellows
> IRIX64 6.5 (cc)
> char = 1
> short = 2
> int = 4
> long = 4
> long long = 8
> float = 4
> double = 8
> long double = 16
> char * = 4
That's odd. I'd have expected (from my experience with IRIX64) the 'long' to be
64-bit. Which machine architecture are you building for by default? (Try doing
'file' on the binary to find out.) The problem is that IRIX supports up to 4
different machine architectures (depending on processor) that are set by flags
in the object files.
(In case people are interested, these architectures are, in order, O32, 32, N32
(which is the architecture that Tcl builds for by default) and 64. Only the
last one is a 64-bit architecture. Cunningly, the "O" stands for "old" and then
"N" stands for "new". The other IRIX system you mention (an O2 with gcc) only
supports the first three architectures.)
Donal.
--
"Everyone else is envisioning a future of 32-bit deep bump-mapped lizard-skin
scrollbars that adapt to the lusers mood by imperceptibly alpha-blending
through a range of chromatic combinations not repeating in 3^18 years. Needs
hardware acceleration, anaglyptic glasses, and 2 aspirin." -- Phil Ehrens
Donal K. Fellows
>
> It may be better to use an extension to deal with that size of images.
It depends what you're doing with them. But there's all sorts of problems when
you start to get up to images that large, quite apart from the sheer cost of
rendering. The key one is that a 4G image has 1G of pixels, and that in turn
means that it has to be a square of 32768 by 32768, which in turn is exactly
where you hit a limitation in the X protocol itself.
(IIRC, unrendered images in Tk can potentially get larger than that if you've
got the memory to back them up. Or they will once we've fixed Tcl's memory
allocation routines to handle large blocks.)
David N. Welton
> (IIRC, unrendered images in Tk can potentially get larger than that
> if you've got the memory to back them up. Or they will once we've
> fixed Tcl's memory allocation routines to handle large blocks.)
Are those explained somewhere for the different platforms? What's
wrong with malloc?
--
David N. Welton
Consulting: http://www.dedasys.com/
Personal: http://www.dedasys.com/davidw/
Free Software: http://www.dedasys.com/freesoftware/
Apache Tcl: http://tcl.apache.org/
j...@mrc-lmb.cam.ac.uk
> That's odd. I'd have expected (from my experience with IRIX64) the 'long' to be
> 64-bit. Which machine architecture are you building for by default? (Try doing
> 'file' on the binary to find out.) The problem is that IRIX supports up to 4
> different machine architectures (depending on processor) that are set by flags
> in the object files.
cc defaults to n32. Indeed using cc -64 does give long as 64bit (and 64-bit
pointers too).
> (In case people are interested, these architectures are, in order, O32, 32, N32
O32 and 32 are the same thing by the way, but granted 3 different
architectures is still overly complex. And how come it's N32 but not N64? Both
are in the "new" format. Still, it'd be unlike SGI to make something too
simple :) Afterall this comes from the people who released Irix 6.4 before
Irix 6.3 (both were distinct, but incompatible, upgrades from 6.2).
Getting back to Tcl, all this really points to is that we need to use
something like size_t and let the user control the build environment
themselves. Sometimes there are very good reasons why we'd build an N32 SGI
binary on a SGI 64-bit system.