derived class generators and introspection
Darren Duncan
This email is part of a brain dump from my thoughts over the last
week while I was away from a computer. If anything doesn't make
sense, I will clarify or expand it in the following days.
I believe that Perl 6 already has basically all of the necessary
parts built-in for implementing a true relational database, and that
any needed remainder can be added and integrated in an elegant
fashion, as I will outline and/or ask about.
--------------
At the center of this idea is the thought that the "Tuple" and
"Relation" of the relational data are not each single classes, but
rather are each roles / abstract interfaces that many classes can be
composed of / implement.
This is much the same as how I see the existing "Array" and "Hash" of
Perl 6, where each is a role, and eg, that "Array of Int" and "Array
of Str" are 2 different actual classes (or roles) that .does(Array);
in this context, saying "Array of ..." is acting as a class generator
which defines a new class that composes the Array role. I say that
"Array of Int" and "Array of Str" are 2 different classes because the
Perl 6 type system would treat each as being a repository for
different sets of possible values, and would reject the assignment of
one to the other.
And so, a "Tuple" type is essentially defined using an ordinary class
(or classless object type) definition that .does(Tuple) but that that
it also has certain restrictions. A Tuple value is then simply an
object of that class.
The routines that the Tuple class provides are essentially just
wrappers over certain meta-class and/or class routines, and Tuple
does not add any new attributes nor hide any existing class
functionality. The attributes of the Tuple and the attributes
defined by the class are one and the same.
The design restrictions that a Tuple doing class must obey, or appear
to obey as far as its users can see, are such as these:
1. All significant attributes which together define the Tuple
object's value must be public and/or have accessors with the same
names as the attributes (extra implicit attributes that eg index
those are not significant in this sense); an introspection method
should also exist where one can inquire what the names and types of
the significant attributes are; conceptually, a Tuple class is
transparent.
2. The Tuple class must have a constructor that takes values for
all significant attributes of the new object. Every logically
distinct Capture of that constructor must produce a logically
distinct new object.
3. A Tuple class needs to provide the interface details necessary
that the Perl 6 type system can treat it as a value type. The ===
operator should return True just for 2 Tuple objects that are of the
same class and where all pairs of corresponding significant
attributes using === return True.
Conjecture: As well, 2 Tuple objects that are of different classes,
where at least one is anonymous, should compare like they were of the
same class if their significant attribute definition sets are
identical; this is so a system where most classes are generated from
other classes DWIM correctly, like two distinct "Array of Int" have
the same definition.
4. Conjecture: all Tuple classes should be immutable following
their construction, either actually or in appearance, but this may
not be essential for all uses of it.
5. Each significant Tuple attribute is mutually exclusive from all
of the others, in that the interface to an attribute conceptually
maps 1:1 to the attribute; reading or changing any of them does not
have visible side-effects on any others. This is similar to how
array or hash elements are exclusive.
Now, a "Relation" type is one step removed; it is simply or in
appearances to the user a "Set of <Tuple-doing-class>" or some-such
that also .does(Relation), or perhaps alternately, a "Relation" type
could be declared with "Relation of <Tuple-doing-type>", which looks
more natural.
A "Relation" is a "Set" that is restricted to all of its members
being of the same single class that is specified in the Relation type
definition, and within that constraint, it is useable like any Set,
and the Relation role adds several additional routines that wrap
meta-class or class methods of the Set, but don't hide any existing
features.
Presumably any Tuple used in a Relation has to be immutable, since
changing a Tuple that is a member of a Relation would have the same
issues to contend with as when you change a mutable object that is
used as a Hash key.
Inherited from Set, a Relation class must have the necessary details
that the Perl 6 type system can treat it as a value type, including
that === works.
It goes without saying that any attribute of a Tuple can be either a
class that .does(Tuple) or that .does(Relation), as it can be any
other class.
Note that many of the methods which the Tuple and Relation roles
provide will have the effect of generating new anonymous classes,
along with objects of said new classes; these may very well be in
fact be classless objects as is appropriate; in some cases, the new
objects will have the same classes as the parent objects. Some will
generate one output object from one input object and/or the invocant;
others will generate one output object from 2 or N input objects.
---------------
Now, in order for all the above to work effectively, a few things may
or will be needed in the Perl 6 meta-model and programmatic API to
said:
1. It must be possible to define new classes or classless objects
whose definition can be determined at run time or compile time
depending on what data is known.
2. It must be possible to both read the list of significant
attributes from an existing class definition / meta-class object,
including their names and full type definitions, and to copy these
names and definitions into a new class definition or classless object.
3. Generally speaking, it should be possible to take several
existing meta-class objects and treat their attribute lists as sets
which can then have set operations applied to them, like union,
intersect, subtract, etc; anything that supports this will only
succeed if same-named attributes also have identical type
definitions; non-identical types for the same names result in an
exception. Or additionally, it should be possible to copy one class
definition into another one to use as a basis for further changes.
If this isn't built-in, then the Tuple|Relation Role can implement
this over #1/#2. Note that these such derived definitions are static
and don't change if their derived-from class definitions later
change, as they aren't subclasses.
4. We need to be able to programmatically define accessors and
constructors for new classes/classless-objects too, unless those come
for free when we declare attributes public-ish.
--------------
In closing for now, I imagine that a lot of this stuff is connected
to the meta-model, though doing it well will have clean support in
the language syntax as well.
Feedback is appreciated.
Thank you. -- Darren Duncan
Nigel Hamilton
HI Darren,
Generally I really like the idea of fixing the relational/OO
mismatch problem by swallowing the relational model whole. :-)
But I wonder if we are ready to say goodbye to the tyranny of disk
seek? How will your proposed system use the disk? And if it does use the
disk what about pesky problems like: indexing, locking, seek time etc?
The days of limitless RAM are yet to arrive - until then databases
must rely on the disk - what is the plan for storing the data?
NIge
Darren Duncan
> HI Darren,
> Generally I really like the idea of fixing the relational/OO
> mismatch problem by swallowing the relational model whole. :-)
> But I wonder if we are ready to say goodbye to the tyranny of disk
> seek? How will your proposed system use the disk? And if it does use the
> disk what about pesky problems like: indexing, locking, seek time etc?
> The days of limitless RAM are yet to arrive - until then databases
> must rely on the disk - what is the plan for storing the data?
> NIge
While it is true that the broader design I am addressing should do away
with any relational/OO impedence mismatch, since a complete relational
model would by definition handle data types of arbitrary complexity
(unlike many of today's pseudo-RDBMS products), I am certainly not
proposing doing away with the disk.
Rather, the proposal is focusing on what users of these data structures
would / could see. The idea is that relational structures have the same
ease of use and flexability that things like hashes or arrays or sequences
or sets do now. They can of course just be stored in RAM like the
aforementioned, when the working set of data is appropriately small, but
just as a hash-doing class can have a disk-tied implementation as well,
for scalability and/or persistence so can a relation-doing class. And
this is one main reason why Relation etc is a role rather than a class, so
people can choose how it works.
-- Darren Duncan
Darren Duncan
>>Rather, the proposal is focusing on what users of these data structures
>>would / could see. The idea is that relational structures have the same
>>ease of use and flexability that things like hashes or arrays or sequences
>>or sets do now. They can of course just be stored in RAM like the
>>aforementioned, when the working set of data is appropriately small, but
>>just as a hash-doing class can have a disk-tied implementation as well,
>>for scalability and/or persistence so can a relation-doing class. And
>>this is one main reason why Relation etc is a role rather than a class, so
>>people can choose how it works.
>
>disk problem. But I wonder about some of the other RBMS things on
>offer - locking, indices etc? Some of these features are there to
>assist with getting data efficiently to and from the disk. Although
>they are not artefacts of the relational model they are important
>parts of what makes a database work. Could your relational model be
>"tied" to an existing physical database?
>NIge
Yes it could.
First of all, note in my explanation that I mentioned indices
already, as examples of "non significant" (non value affecting) extra
data that an object could hold or be tied to; these can be stored on
disk and/or in memory as is appropriate.
Next, the Software Transactional Memory concepts / interfaces in Perl
6, such as the concept of atomic code blocks or routines, would be
mapped to ACID features of a disk store; the start of an atomic
operation is like a "start transaction" and the successful completion
of one is a "commit"; a retry or fail involves a "rollback" etc.
Note that a proper system will need to (or will ideally) support
nested transactions, so any given atomic block won't have to know
whether it is inside another one or not to be itself atomic.
As for locks, the Perl 6 interface / functionality for managing
shared data between multiple threads or processes would map to
appropriate locks or other mechanisms on disk where applicable.
Truly, a database was never supposed to be a world separate from an
application; as I recall, Cobol or something introduced the idea of
databases and applications being at arms length, and it stuck, but
that isn't the way things should be; how they should be is
integrated, or at least to the point that a generic reusable module
within an application is integrated; eg, DBM or BDB; also, the
concept of servers is orthogonal to this; anything can be relegated
to a server. Using the DBM analogy, what I propose is that the
relational analogy of an in-memory Hash is built-in to Perl, and any
extensions that make it tied to something external like a disk be
language extensions / CPAN modules; all Perl 6 iteself has to do is
make it easy to attach such extensions, and I think it more or less
does so, save edge cases to hash out.
-- Darren Duncan
Nigel Hamilton
> Rather, the proposal is focusing on what users of these data structures
> would / could see. The idea is that relational structures have the same
> ease of use and flexability that things like hashes or arrays or sequences
> or sets do now. They can of course just be stored in RAM like the
> aforementioned, when the working set of data is appropriately small, but
> just as a hash-doing class can have a disk-tied implementation as well,
> for scalability and/or persistence so can a relation-doing class. And
> this is one main reason why Relation etc is a role rather than a class, so
> people can choose how it works.
>
OK. I can see that a tied-relation could help solve the talking to disk