Bignum Type Question
Andrew Mangogna
the "TclInitObjSubsystem" function, I notice that "tclBignumType" is not a
registered type. I'm curious as to why not, given that every other type
seems to be there.
--
Andrew Mangogna
George Peter Staplin
Based on what some of the Tcl core developers have discussed the
registered Tcl_ObjTypes aren't always a good idea, because they expose
too much of the internals that may change over time.
Some of the languages using Tcl/Tk, such as Python and one version of
Perl that uses Tcl with Tk, depend on such type information.
If you really want the Tcl_ObjType you can get it, but I wouldn't
recommend it in most cases.
mp_int big;
if (TCL_OK != Tcl_GetBignumFromObj (interp, objPtr, &big)) {
/* the object couldn't be converted to a Bignum type */
return TCL_ERROR;
}
/* now objPtr->typePtr has the Bignum type */
And now of course you can invade the Tcl internals in lots of *EVIL*
ways and replace the objPtr->typePtr->...
Maybe someday parts of the Tcl implementation will be protected by
read-only memory on some platforms... That would prevent such internal
Tcl_ObjType manipulation. IMO it's easier and better to just say "DON'T
DO THAT!"
As it is, we have some C extensions for Tcl that are meant to be
portable that use tclInt.h, and that results in distributions like
Debian including tclInt.h and other header files in areas of
/usr/include, when those are really meant to be for Tcl's use only (make
install doesn't install tclInt.h by default). It's somewhat
disappointing, because extensions like Incr Tcl, XOTcl, and others
depend on these internals. Which means when bugs are fixed, or
performance is improved by using better algorithms and data structures
some of those extensions break too.
George
Alexandre Ferrieux
<georgepsSPAMME...@xmission.com> wrote:
> Andrew Mangogna wrote:
> > In looking through the version 8.5.2 "generic/tclObj.c" source file in
> > the "TclInitObjSubsystem" function, I notice that "tclBignumType" is not a
> > registered type. I'm curious as to why not, given that every other type
> > seems to be there.
>
> Based on what some of the Tcl core developers have discussed the
> registered Tcl_ObjTypes aren't always a good idea, because they expose
> too much of the internals that may change over time.
It is true that the boundary between "private" and "public" headers is
currently suboptimal (try to extend the List type without recompiling
all extensions ;-). But Andrew's question is still valid; indeed
registering a type has virtually no effect on the Tcl runtime, it just
allows enumeration and lookup for debugging purposes in tcltest, so
removing bignums from that list doesn't prevent tclBignumType from
being just as reachable as tclListType (MODULE_SCOPE in tclInt.h).
This also means that it's easy to forget to register a type ;-)
-Alex
Andrew Mangogna
Sorry about responding to my own post, but I thought I might expand a little
on why I'm asking this question in the hope that others might have some
insight.
An extension that I wrote, TclRAL, is a relational algebra package. It
introduces two new types "Relation" and "Tuple" and both are registered via
Tcl_RegisterObjType(). In relational data theory, the data type of an
attribute is an important data integrity constraint. As we all know, Tcl
takes a rather different view of data types associating type with an
operation and not with a variable. Nonetheless, attributes of relations in
TclRAL have a declared type and the TclRAL operations insist that the value
assigned to an attribute at least be coercible to the stated attribute
type. This serves as a way for the user to state the types of operations
that are intended to be performed on a attribute and certainly if the
intent is that an attribute be used in an "int" context then
assigning "foo" to it violates some notion of data integrity. And,
since "string" is the universal Tcl type, attributes of type "string" have
no effective type check.
This is all well and good, but TclRAL insists that attribute types be
registered so that that calls to Tcl_GetObjType() return non-NULL values.
This insures that the type exists even if it comes from some other
unforeseen extension and that type conversion is possible. So as things now
stand in 8.5, "bignum" cannot be one of the stated attribute types. The
functions supplied by the Tcl type structure are very few and deal with
conversion between internal and external (i.e. string) form and with
copying and managing any memory associated with the internal
representation. What is not provided in the type functions is any way to
determine if two values are equal or, for those types that provide an
ordering, to compare two values. So, there are occasions when TclRAL needs
to determine if two Tcl_Obj values of a stated type are equal and it needs
to sort relations and therefore needs to compare two Tcl_Obj values, again
of a stated type.
Determining the equality or relative order ends up being a little tricky.
Indeed as I was correcting some errors in this area caused by my
misunderstanding of the way the Tcl_ConvertToType() actually works in
detail that this question came up. Trying very hard to stick to public
functions and not to violate the privacy of the internals, TclRAL examines
the type names and uses the various Tcl_GetXXXFromObj() functions to obtain
values that can be tested for equality or compared for relative order. It
resorts to simple string comparison when there is no other choice.
My concern is that attributes declared as "int" (since TclRAL does not allow
the unregistered "bignum" type) may as a result of arithmetic operations
become "bignum" values and then later when sorting be compared improperly.
Actually, my real concern is that the whole scheme is teetering on the
brink of disaster.
--
Andrew Mangogna
Neil Madden
Tcl's "types" (Tcl_ObjTypes) are an implementation detail designed only
for the purpose for which Tcl uses them. They are not particularly
suitable for being exposed as a type system in the way you are trying to
do things. If you really want this kind of stronger typing then you
would do better to define your own system of types, along with
conversion and comparison functions (which can of course make use of
Tcl_Get<Foo>FromObj etc). I.e., you should define a TclRAL_Type and
associated functions.
-- Neil
Donald G Porter
Among the maintainers most active in that part of the code, there is
a consensus that Tcl_ObjType registration is a mistake. To the extent
we can avoid repeating or growing that mistake, we try to do so.
Several Tcl_ObjTypes that are registered in Tcl 8.4 are no longer
registered in Tcl 8.5, as noted in the Tcl 8.5.0 release notes.
http://sf.net/project/shownotes.php?release_id=562516&group_id=10894
"* The built-in Tcl_ObjTypes "index", "ensembleCommand", "localVarName",
"levelReference", and "nsName" are no longer registered. Programs that
pass any of these strings to Tcl_GetObjType() need to be prepared for
a NULL return."
If you examine the changes and/or ChangeLog files, you will see that we
attempted to remove registration of many more Tcl_ObjTypes as well:
"list", "procbody", and "boolean". These registrations were restored
due to feedback from people testing their legacy code with our alpha and
beta releases.
You may be asking why we think Tcl_RegisterObjType() represents a bad
design. Let me outline the reasoning.
Why would someone want a Tcl_ObjType to be registered? Only so they
might call Tcl_GetObjType on the Tcl_ObjType name to get back a pointer
to the Tcl_ObjType struct.
In turn, why do they want to get a pointer to a Tcl_ObjType struct
defined in some other module? Only two reasons.
1) They want to use that (Tcl_ObjType *) value to compare against the
objPtr->typePtr field of some (Tcl_Obj *) passed to them, and then use
the knowledge that "objPtr" is "of type foo" as the basis for intruding
into objPtr->internalRep and probing the internals based on how you
"know" that Tcl_ObjType is organized internally.
2) They want to pass that (Tcl_ObjType *) to Tcl_ConvertType() to force
conversion of some (Tcl_Obj *) to a Tcl_ObjType of some other module,
based on a justification that you "know" the internal repesentation
of that Tcl_ObjType are most suitable for that value now.
Neither of these uses is a good idea. They're both based on what you
"know" about the internals of another module. It breaks encapsulation,
and will also be fragile to times when that other module changes its
internal representations.
In fact, the removal of Tcl_RegisterObjType() calls for Tcl's own built-in
types during Tcl 8.5 development was originally done as a means to
protect code attempting 1) because significant changes to internal
reps seemed certain to break the assumptions of any such code.
Instead, all uses of a Tcl_ObjType ought to remain within the module
that defines that Tcl_ObjType. For a simple example, if you need the
value from a (Tcl_Obj *) exported as a double, the proper way to get it is
if (Tcl_GetDoubleFromObj(interp, objPtr, &d) != TCL_OK) {
/* Not a double; react as appropriate */
}
This leaves it up to the same module that defines a "foo" Tcl_ObjType
to supply a public routine Tcl_GetFooFromObj() that uses the internal
rep of the "foo" type, as well as closely related Tcl_ObjTypes also
defined by the same module, to most effectively deliver what the caller
needs.
The improper and increasingly unsupported way is
Tcl_ObjType doubleTypePtr = Tcl_GetObjType("double");
if (Tcl_ConvertToType(interp, objPtr, doubleTypePtr) != TCL_OK) {
/* Not a double; react as appropriate */
} else {
d = objPtr->internalRep.doubleValue;
}
In fact, the latter example will not reliably work in Tcl 8.5.
Beyond the reasoning above, which I think is fairly broadly agreed
among the active maintainers, I peronally think the Tcl_RegisterObjType
machinery suffers another significant flaw.
If the same name is passed in two Tcl_RegisterObjType() calls,
the latter overwrites the former. Perhaps it was meant to give
extensions a way to "extend" Tcl's fundamental types, but in
practice it just means if you ask for the "foo" type, you have no
reliable clue just what type from what source you're going to get.
This problem might arise intentionally, or via an accidental name
collision.
Once Tcl_ObjType registration is done away with, the fact that my
module defines a "foo" Tcl_ObjType() and your does too is no longer
a problem needing to be solved.
| Don Porter Mathematical and Computational Sciences Division |
| donald...@nist.gov Information Technology Laboratory |
| http://math.nist.gov/~DPorter/ NIST |
|______________________________________________________________________|
Donald G Porter
> mp_int big;
>
> if (TCL_OK != Tcl_GetBignumFromObj (interp, objPtr, &big)) {
> /* the object couldn't be converted to a Bignum type */
> return TCL_ERROR;
> }
>
> /* now objPtr->typePtr has the Bignum type */
No, that's not true. Try it with
Tcl_Obj *objPtr = Tcl_NewIntObj(1);
Tcl_GetBignumFromObj(... objPtr, ...) will leave objPtr->typePtr as
some Tcl_ObjType which will tend to make future calls of
Tcl_GetBignumFromObj(... objPtr, ...) more efficient. That Tcl_ObjType
could be the "bignum" type, or it could be something else.
Related to this...
Tcl_ConvertToType(interp, objPtr, Tcl_GetObjType("bignum")) likewise
may leave objPtr->typePtr as some Tcl_ObjType different from "bignum",
so long as it is a related type that still fulfills the goal of
efficient transformation to the mp_int type in future uses of the value.
The documentation on this point needs updating. See Tcl Bug 1917650.
--
Donald G Porter
> Sorry about responding to my own post, but I thought I might expand a little
> on why I'm asking this question in the hope that others might have some
> insight.
I don't think I fully digested your use case, but at a quick glance it
appears you might be a good candidate user of the (currently private)
routine TclGetNumberFromObj(). I would be quite happy to make this
routine, or something like it, part of Tcl's public interface. The
main reason that hasn't already happened is there's been a lack of
potential users to try it out and verify that the interface is sensible
and useful. If you like the looks of it, write me and we can start
the machinery to get it exported for Tcl 8.6.
> My concern is that attributes declared as "int" (since TclRAL does not allow
> the unregistered "bignum" type) may as a result of arithmetic operations
> become "bignum" values and then later when sorting be compared improperly.
One property of the numeric Tcl_ObjTypes as defined in Tcl 8.5 is that
each numeric value is associated with exactly one Tcl_ObjType, so that
specific shimmering concern is not something that will happen. I don't
expect this to change, but they are internals, so no guarantees. If
you have code that relies on particular internals details, it's a good
idea to keep an eye on whether ongoing development makes changes you
will care about.
Fredderic
Donald G Porter <d...@nist.gov> wrote:
> Once Tcl_ObjType registration is done away with, the fact that my
> module defines a "foo" Tcl_ObjType() and your does too is no longer
> a problem needing to be solved.
Just a random thought... (They're frightening, I know...)
Instead of dumping type registration, why not remove the machinery that
makes it useful, and have it instead return an error if a type by that
name is already registered (non-fatal, in the case of overriding an
internal structure).
The name of a given type is still obtainable, given a Tcl_Obj of that
type, so a mechanism to check defined types for name collisions might
still be useful, in some evil cases.
As an aside, I think some mechanism to extend Tcl_ObjType needs to be
implemented, both within and without TCL's internals, by abstracting
creation and access to Tcl_ObjType.
Internally, for one thing, I think sooner or later we'll need the
ability to abstract indexing of items within the "container" types
(list, dict, and perhaps arrays also), which will either require a new
parallel mechanism entirely, and/or internal extensions to
Tcl_ObjType. But it's hard to investigate such possibilities at this
time because as far as I can tell, such extensions aren't readily
possible. (This need came out in the search for mutability.)
Externally, I've had need of a "user_data" type field in the
Tcl_ObjType. This came about where I had three pieces of information
to store in a Tcl_Obj's internal representation, of which only any two
were needed to fully describe the value, lending itself perfectly to
fitting within a Tcl_Obj. What I did, was to produce three Tcl_Obj
types representing the three combinations internal values pairs, and
wrapped Tcl_ObjType to provide some extra functions to obtain all three
values given a Tcl_Obj of any of the types (and to cross-convert them).
This type of thing is only made more relevant and useful by removal of
type name registration (I gave all three the same name allowing a simple
test to be done to determine whether the extra data was available,
and from there any internal value could be obtained given any version
of the type).
It may be a fairly bizarre case (most of my ideas are), but getting
such a framework in place BEFORE it's needed (even if it's defined as
experimental and very frightening), is none the less a good idea in my
books.
Fredderic
Andrew Mangogna
> Andrew Mangogna wrote:
>> In looking through the version 8.5.2 "generic/tclObj.c" source file in
>> the "TclInitObjSubsystem" function, I notice that "tclBignumType" is not
>> a registered type. I'm curious as to why not, given that every other type
>> seems to be there.
>
> Among the maintainers most active in that part of the code, there is
> a consensus that Tcl_ObjType registration is a mistake. To the extent
> we can avoid repeating or growing that mistake, we try to do so.
I'm sure I don't have quite the insight into these things, but if the
consensus is that it is design mistake, then I am pleased to know this and
will undertake to rectify my code.
I agree that #1 is not a good idea. I will think hard about #2, but right
now my mind is not quite there.
>
> In fact, the removal of Tcl_RegisterObjType() calls for Tcl's own built-in
> types during Tcl 8.5 development was originally done as a means to
> protect code attempting 1) because significant changes to internal
> reps seemed certain to break the assumptions of any such code.
>
> Instead, all uses of a Tcl_ObjType ought to remain within the module
> that defines that Tcl_ObjType. For a simple example, if you need the
> value from a (Tcl_Obj *) exported as a double, the proper way to get it is
>
> if (Tcl_GetDoubleFromObj(interp, objPtr, &d) != TCL_OK) {
> /* Not a double; react as appropriate */
> }
>
> This leaves it up to the same module that defines a "foo" Tcl_ObjType
> to supply a public routine Tcl_GetFooFromObj() that uses the internal
> rep of the "foo" type, as well as closely related Tcl_ObjTypes also
> defined by the same module, to most effectively deliver what the caller
> needs.
>
> The improper and increasingly unsupported way is
>
> Tcl_ObjType doubleTypePtr = Tcl_GetObjType("double");
> if (Tcl_ConvertToType(interp, objPtr, doubleTypePtr) != TCL_OK) {
> /* Not a double; react as appropriate */
> } else {
> d = objPtr->internalRep.doubleValue;
> }
>
> In fact, the latter example will not reliably work in Tcl 8.5.
As I can personally attest to :-)
>
> Beyond the reasoning above, which I think is fairly broadly agreed
> among the active maintainers, I peronally think the Tcl_RegisterObjType
> machinery suffers another significant flaw.
>
> If the same name is passed in two Tcl_RegisterObjType() calls,
> the latter overwrites the former. Perhaps it was meant to give
> extensions a way to "extend" Tcl's fundamental types, but in
> practice it just means if you ask for the "foo" type, you have no
> reliable clue just what type from what source you're going to get.
> This problem might arise intentionally, or via an accidental name
> collision.
This seems like a misfeature of the implementation of Tcl_RegisterObjType().
It could reject such duplicated names, albeit the current interface has no
way to communicate that rejection.
>
> Once Tcl_ObjType registration is done away with, the fact that my
> module defines a "foo" Tcl_ObjType() and your does too is no longer
> a problem needing to be solved.
>
> | Don Porter Mathematical and Computational Sciences Division |
> | donald...@nist.gov Information Technology Laboratory |
> | http://math.nist.gov/~DPorter/ NIST |
> |______________________________________________________________________|
Thanks for everyone's replies. It's back to the drawing board to find a
better way to do what I need to do. Currently, I have made changes to use
the Tcl_GetXXXFromObj() functions for the various numeric types, but it
will take a bit more work to remove my dependency on registered type names.
The exact manner in which to blend the relational model type integrity with
Tcl's type system is still problematic, especially when determining the
equality of two Tcl_Obj values. But I'm sure there is a way even if I
resort to setting up some "shadow" typing system. After all, "these things
must be done very delicately".
--
Andrew Mangogna
Donald G Porter
> This seems like a misfeature of the implementation of Tcl_RegisterObjType().
> It could reject such duplicated names, albeit the current interface has no
> way to communicate that rejection.
Clearly, I agree completely, but that design choice got made years before
I starting contributing to the Tcl implementation.
--