More object stuff
Dan Sugalski
hackish way) it's time to look forward a bit.
The code definitely needs some looking at and cleanup. It's not
perfect and I expect it's flawed in a number of places. I'm just
happy to get the damn stuff into the repository so the bytecode tests
can start. Implementation can always change as long as the bytecode
interface stays constant.
If someone'd like to take a shot at making a nice OO wrapper for
Postgres, especially if they'd like to upgrade the postgres interface
to 7.4, I would very much appreciate it. It'd be a nice demo, and a
good start on a DBI module for us. (And yeah, there's an element of
"do my job for me" here. Sorry 'bout that)
If we get objects hammered on and functional, we've met a requirement
for a .1 release. Scary, but true.
For the record, the point release criteria are:
*) Objects
*) Objects with multiple inheritance
*) Exceptions we're happy with
*) Events
*) Fully functional IO
*) Working module system we're happy with
There are more, I expect, but we'll deal with that when we get there.
--
Dan
--------------------------------------"it's like this"-------------------
Dan Sugalski even samurai
d...@sidhe.org have teddy bears and even
teddy bears get drunk
Leopold Toetsch
> The code definitely needs some looking at and cleanup. It's not
> perfect and I expect it's flawed in a number of places. I'm just
> happy to get the damn stuff into the repository so the bytecode tests
> can start.
I've put in a first test. Needed some changes to get it running and I'm
not outerly sure if its overall correct. Anyway:
- ParrotObject inherits from ParrotClass now
- move isa to ParrotClass and get it running
- move find_method there too
- IMHO the new class could have a copy of ParrotObjects vtable and not
that of ParrotClass
> *) Objects
Do you have something WRT setting/getting attributes?
> *) Objects with multiple inheritance
Or for 0.2?
> *) Exceptions we're happy with
Create an Exception class hierarchy?
> *) Events
Need an event-handler thread AFAIK.
leo
Dan Sugalski
>Dan Sugalski <d...@sidhe.org> wrote:
>
>> The code definitely needs some looking at and cleanup. It's not
>> perfect and I expect it's flawed in a number of places. I'm just
>> happy to get the damn stuff into the repository so the bytecode tests
>> can start.
>
>I've put in a first test. Needed some changes to get it running and I'm
>not outerly sure if its overall correct. Anyway:
>- ParrotObject inherits from ParrotClass now
>- move isa to ParrotClass and get it running
>- move find_method there too
>- IMHO the new class could have a copy of ParrotObjects vtable and not
> that of ParrotClass
You're right on the last point--while the ParrotClass vtable gets
called to initialize the PMC, the resulting PMC should have a
ParrotObject vtable. The current scheme is, well, wrong in that
regards.
> > *) Objects
>
>Do you have something WRT setting/getting attributes?
I thought we had at least discussed a spec of sorts a while back.
I'll go look and see what we talked about and go from there.
> > *) Objects with multiple inheritance
>
>Or for 0.2?
Yep.
> > *) Exceptions we're happy with
>
>Create an Exception class hierarchy?
I'm not 100% sure I want to go with a real OO style of exceptions,
but that might just be the neo-luddite in me.
> > *) Events
>
>Need an event-handler thread AFAIK.
Yeah, but not for what I'm considering now--events should be handled
in the interpreter that gets them.
Leopold Toetsch
> At 12:17 PM +0100 12/3/03, Leopold Toetsch wrote:
>>Create an Exception class hierarchy?
> I'm not 100% sure I want to go with a real OO style of exceptions,
> but that might just be the neo-luddite in me.
It probably depends what HLL want to have. Anyway, I was thinking of
pre-constructing a bunch of constant exception objects for internal usage.
>> > *) Events
>>
>>Need an event-handler thread AFAIK.
> Yeah, but not for what I'm considering now--events should be handled
> in the interpreter that gets them.
Notifications. I'll rediff / redo my "event handling-2.patch". That one,
that swapped opcode-dispatch tables to run the event-handler code.
leo
Michal Wallace
> Dan Sugalski <d...@sidhe.org> wrote:
> > At 12:17 PM +0100 12/3/03, Leopold Toetsch wrote:
>
> >>Create an Exception class hierarchy?
>
> > I'm not 100% sure I want to go with a real OO style of exceptions,
> > but that might just be the neo-luddite in me.
>
> It probably depends what HLL want to have. Anyway, I was thinking of
> pre-constructing a bunch of constant exception objects for internal
> usage.
I'm trying to understand the question here:
What is the _return_cc attribute on an
exception? Can I use it to resume the
code as if the exception never happened?
Do I have to fill it in manually? Or
could it be automatic?
I'm picturing exceptions as continuations
with an arbitrary message attached... Right
now that message is a string, but eventually
it could be any PMC... Is that about right?
So, if I throw a WarningException, could I
just say "ignore it... go to the next line"?
As for the message, I'm trying to think of
a good reason to standardize that attached
PMC. For example, if I try to open a file
in python and it fails, I get an exception
of class IOError... But if I call a perl
routine that throws the perl equivalent...
What should happen? Should I just catch
PerlException instead?
Each HLL is going to want its own class
hierarchy for exceptions... But it might
be nice to have a predefined hierarchy that
parrot uses internally.
I know perl uses a return value instead of
throwing an exception ("open or die", right?)..
So would parrot's internal file-opener just
throw a ParrotFileException? So perl's
open() command catches it and returns 0,
while python's open() command catches it
and throws a new IOError?
Am I on the right track?
Sincerely,
Michal J Wallace
Sabren Enterprises, Inc.
-------------------------------------
contact: mic...@sabren.com
hosting: http://www.cornerhost.com/
my site: http://www.withoutane.com/
--------------------------------------
Leopold Toetsch
> What is the _return_cc attribute on an
> exception? Can I use it to resume the
> code as if the exception never happened?
When an exception is resumable, you can return by invoking this return
continuation. But details (i.e. is C<_return_cc> put into P1
automatically) will probably change. t/pmc/exception.t has some
examples.
> Do I have to fill it in manually? Or
> could it be automatic?
If the exception originated in the opcode-loop, it can be resumable -
but not all are changed yet. Have a look at C<real_exception()> in
*.ops. For these, setting the return continuation is done automatically.
> I'm picturing exceptions as continuations
> with an arbitrary message attached... Right
> now that message is a string, but eventually
> it could be any PMC... Is that about right?
There will be some default information and user fields.
> So, if I throw a WarningException, could I
> just say "ignore it... go to the next line"?
You can always resume your own exceptions that you C<throw> yourself.
> As for the message, I'm trying to think of
> a good reason to standardize that attached
> PMC. For example, if I try to open a file
> in python and it fails, I get an exception
> of class IOError... But if I call a perl
> routine that throws the perl equivalent...
> What should happen? Should I just catch
> PerlException instead?
I think, that we need classified ParrotExceptions. If HLLs POV differ
they can subclass these to their own needs.
> Each HLL is going to want its own class
> hierarchy for exceptions... But it might
> be nice to have a predefined hierarchy that
> parrot uses internally.
Yep.
> I know perl uses a return value instead of
> throwing an exception ("open or die", right?)..
Where die() is the exception.
> So would parrot's internal file-opener just
> throw a ParrotFileException? So perl's
> open() command catches it and returns 0,
> while python's open() command catches it
> and throws a new IOError?
Very probably yes.
> Michal J Wallace
leo
Jeff Clites
> At 12:17 PM +0100 12/3/03, Leopold Toetsch wrote:
>> Dan Sugalski <d...@sidhe.org> wrote:
>>
>> > *) Exceptions we're happy with
>>
>> Create an Exception class hierarchy?
>
> I'm not 100% sure I want to go with a real OO style of exceptions, but
> that might just be the neo-luddite in me.
I'm of the opinion that it's a conceptual mistake to model exceptions
as objects. My main concrete argument is that it's an abuse of
inheritance--you need a hierarchy of exception reasons, but using
inheritance just to get a hierarchy is silly. (That is, it doesn't make
sense to subclass if you are not changing behavior in some way.) In
addition, I think it's overly restrictive.
Fundamentally, I think raising an exception requires two things: (1)
some indication of "reason" for the exception (used by the exception
infrastructure to select the correct handler), and (2) a way for the
originator of the exception to pass some arbitrary information to the
handler (this information being opaque to the exception
infrastructure).
In C terms, that would mean that the signature is conceptually like:
raise_exception(Reason reason, void* info);
which in Parrot terms would probably shake down to:
raise_exception(Reason reason, PMC* info);
The "Reason" data type could either be a structured string, such as
"parrot.io.staleFileHandle", or some sort of array of strings--just
enough to indicate a hierarchy to be used for selecting a handler.
The key point here is that the "reason" parameter is what is needed by
the exception infrastructure to choose the correct handler (and to
decide what to do if no handler is found), and "info" just has to be of
a form agreed upon by the "thrower" and the "catcher".
This sort of approach would work with languages such as Java which
expect exceptions to be objects--in this case, the Java exception
object would be passed as the "info", and the "reason" would be derived
from the type of that object.
For simpler languages (and possibly even for Perl6), "info" could just
be a hash--no real need for a specialized type of object.
I think the core of my thinking here is that exceptions don't really
have behavior--at most they are a bag of data (which is what hashes are
usually good for), and really they are a process for jumping the flow
of execution, and sending along a bag of data to indicate why it just
happened.
That was a bit long-winded, but basically I'm saying that I'd like to
see the pasm for raising an exception take the form:
raise Px, .parrot.io.StaleFileHandle
where Px may actually be null.
No matter what approach we take, we're going to have issues throwing
exceptions across language boundaries (ie, Perl exception handled by
Python code), since different languages aren't going to agree on how to
classify exceptions--and having exceptions be HLL objects seems to make
this worse. Actually, in a sense this _might_ not be a problem: If we
supply a decent reason hierarchy, then using our supplied parrot.*
exception "reasons" would allow an exception thrown in one language to
be handled in another; if a language-specific reason is used (e.g.,
perl.blah), then it's probably going to only end up being handled by
code in the same language. (And that seems fine--even in Java if I
create my own Exception subclass, then code I haven't written isn't
expected to be catching that type of exception.) But having some
standard types will make it possible to cross language boundaries
meaningfully.
Just some thoughts.
One other thing: Long-term, it would be good to avoid having an
exception handler stack which we push and pop as we do now, and instead
use a variant of the zero-overhead approach used by C++ (and probably
Java), wherein try-catch blocks incur zero execution overhead--CPU time
is only consumed when an exception is actually thrown, and not when a
"try" block is entered or exited.
JEff
Harry Jackson
> If someone'd like to take a shot at making a nice OO wrapper for
> Postgres, especially if they'd like to upgrade the postgres interface to
> 7.4, I would very much appreciate it. It'd be a nice demo, and a good
> start on a DBI module for us. (And yeah, there's an element of "do my
> job for me" here. Sorry 'bout that)
Could you be more specific. If part of what you want is the following
files updated to reflect libpq-7.4 then I am more than willing to do it
for you.
parrot/library/postgres.declarations
parrot/library/postgres.pasm
Although I imagine there is more to it than this ;-)
As for the OO wrapper. Are you after a pasm library that takes advantage
of objects and hides the details of libpq-fe.h? I think I could start
something if no one else has already started anything.
Harry Jackson (no longer surnameless ;-)
Dan Sugalski
>Dan Sugalski wrote:
>>If someone'd like to take a shot at making a nice OO wrapper for
>>Postgres, especially if they'd like to upgrade the postgres
>>interface to 7.4, I would very much appreciate it. It'd be a nice
>>demo, and a good start on a DBI module for us. (And yeah, there's
>>an element of "do my job for me" here. Sorry 'bout that)
>
>Could you be more specific. If part of what you want is the
>following files updated to reflect libpq-7.4 then I am more than
>willing to do it for you.
>
>parrot/library/postgres.declarations
>parrot/library/postgres.pasm
>
>Although I imagine there is more to it than this ;-)
Yep, though that's a big part of it. postgres.pasm is generated from
postgres.declarations, FWIW--there's a script in the library
somewhere.
>As for the OO wrapper. Are you after a pasm library that takes
>advantage of objects and hides the details of libpq-fe.h? I think I
>could start something if no one else has already started anything.
Yep, that's what I was thinking of. If you take a look, there are
parts of the interface that are somewhat awkward to use from a higher
level -- the record fetching code is definitely skewed down to the
bottom end of things. Certainly fine enough, but it'd be nice to be
able to get back an array or hash for a record rather than having to
query individual fields. :)
Harry Jackson
> Yep, though that's a big part of it. postgres.pasm is generated from
> postgres.declarations, FWIW--there's a script in the library somewhere.
Is /parrot/build_tools/build_nativecall.pl the script in question and if
so whats its usage.
I have done postgres.declarations, please see below. I have made
guesses based on /parrot/build_tools/build_nativecall.pl where I was
unsure as to the type to be used. Where I was not sure of the type I
have included the declaration from libpq-fe.h as a comment just above
the entry in the new postgres.declarations. All new entries from the 7.4
file have been included in the file as follows
#### New In 7.4
New function
####
The entries appear in the order they where encountered in libpq-fe.h
which differs considerably from the 7.3 version.
If anyone is interested the following URL will take you to the Postgres
web interface for libpq-fe.h.
http://developer.postgresql.org/cvsweb.cgi/pgsql-server/src/interfaces/libpq/libpq-fe.h
Revision 1.100
>> As for the OO wrapper. Are you after a pasm library that takes
>> advantage of objects and hides the details of libpq-fe.h? I think I
>> could start something if no one else has already started anything.
>
>
> Yep, that's what I was thinking of. If you take a look, there are parts
> of the interface that are somewhat awkward to use from a higher level --
> the record fetching code is definitely skewed down to the bottom end of
> things. Certainly fine enough, but it'd be nice to be able to get back
> an array or hash for a record rather than having to query individual
> fields. :)
I will probably need a bit of help with this. To coin a phrase I am
still very much slipping on the parrot guts. You guys seem to be
producing material at a fair pace :-)
If there is already any material, docs, examples or a wrapper already
written for another library please point me at it.
Harry
########################################
[package]
PostgreSQL
[lib]
libpq
[defs]
# From libpq-fe.h Revision 1.100
p PQconnectStart t
i PQconnectPoll p
p PQconnectdb t
p PQsetdbLogin t t t t t t t
v PQfinish p
p PQconndefaults
v PQconninfoFree p
i PQresetStart p
i PQresetPoll p
#extern void PQreset(PGconn *conn);
# This was set to "c PQreset p" was this correct.
v PQreset p
i PQrequestCancel p
t PQdb p
t PQuser p
t PQpass p
t PQhost p
t PQport p
t PQtty p
t PQoptions p
i PQstatus p
##### New In 7.4
i PQtransactionStatus p
t PQparameterStatus p t
i PQprotocolVersion p
#####
t PQerrorMessage p
i PQsocket p
i PQbackendPID p
i PQclientEncoding p
i PQsetClientEncoding p t
##### New In 7.4
i PQsetErrorVerbosity p i
#####
v PQtrace p p
v PQuntrace p
# Need function pointers. Can't do that yet
##### New IN 7.4
#i PQsetNoticeReceiver p p
#####
# i PQsetNoticeProcessor p p
p PQexec p t
##### New In 7.4
#extern PGresult *PQexecParams(PGconn *conn,
# const char *command,
# int nParams,
# const Oid *paramTypes,
# const char *const * paramValues,
# const int *paramLengths,
# const int *paramFormats,
# int resultFormat);
p PQexecParams p t i i 3 3 i
#extern PGresult *PQexecPrepared(PGconn *conn,
# const char *stmtName,
# int nParams,
# const char *const * paramValues,
# const int *paramLengths,
# const int *paramFormats,
# int resultFormat);
p PQexecPrepared p t i t 3 3 i
#####
i PQsendQuery p t
##### New In 7.4
#extern int PQsendQueryParams(PGconn *conn,
# const char *command,
# int nParams,
# const Oid *paramTypes,
# const char *const * paramValues,
# const int *paramLengths,
# const int *paramFormats,
# int resultFormat);
i PQsendQueryParams p t i i t 3 3 i
#extern int PQsendQueryPrepared(PGconn *conn,
# const char *stmtName,
# int nParams,
# const char *const * paramValues,
# const int *paramLengths,
# const int *paramFormats,
# int resultFormat);
i PQsendQueryPrepared p t i t 3 3 i
#####
p PQgetResult p
i PQisBusy p
i PQconsumeInput p
p PQnotifies p
##### New In 7.4
#extern int PQputCopyData(PGconn *conn, const char *buffer, int
nbytes);
i PQputCopyData p t i
#extern int PQputCopyEnd(PGconn *conn, const char *errormsg);
i PQputCopyEnd p t
#extern int PQgetCopyData(PGconn *conn, char **buffer, int async);
i PQgetCopyData p t i
#####
i PQgetline p t i
i PQputline p t
i PQgetlineAsync p t i
i PQputnbytes p t i
i PQendcopy p
i PQsetnonblocking p i
i PQisnonblocking p
i PQflush p
p PQfn p i 3 3 i p i
i PQresultStatus p
t PQresStatus i
t PQresultErrorMessage p
##### New In 7.4
#extern char *PQresultErrorField(const PGresult *res, int fieldcode);
c PQresultErrorField p i
#####
i PQntuples p
i PQnfields p
i PQbinaryTuples p
t PQfname p i
i PQfnumber p t
##### New In 7.4
#extern Oid PQftable(const PGresult *res, int field_num);
i PQftable p i
#extern int PQftablecol(const PGresult *res, int field_num);
i PQftablecol p i
#extern int PQfformat(const PGresult *res, int field_num);
i PQfformat p i
#####
i PQftype p i
i PQfsize p i
i PQfmod p i
t PQcmdStatus p
t PQoidStatus p
i PQoidValue p
t PQcmdTuples p
t PQgetvalue p i i
i PQgetlength p i i
i PQgetisnull p i i
v PQclear p
##### New In 7.4
#extern void PQfreemem(void *ptr);
v PQfreemem p
#####
# First arg's a neturned char pointer. Can't do that yet
# i PQescapeString t t i
t PQescapeBytea t l 4
t PQunescapeBytea t 4
p PQmakeEmptyPGresult p i
# The following functions are missing from the original 7.3 version
# so I am leaving these as well. I am including the declarations here
# to let people know the ommission was deliberate. Not 100% why
# though ;-)
#extern void PQprint(FILE *fout,
# const PGresult *res,
# const PQprintOpt *ps);
#extern void PQdisplayTuples(const PGresult *res,
# FILE *fp,
# int fillAlign,
# const char *fieldSep,
# int printHeader,
# int quiet);
#extern void PQprintTuples(const PGresult *res,
# FILE *fout,
# int printAttName,
# int terseOutput,
# int width);
i lo_open p i i
i lo_close p i
i lo_read p i t l
i lo_write p i t l
i lo_lseek p i i i
i lo_creat p i
i lo_tell p i
i lo_unlink p i
i lo_import p t
i lo_export p i t
i PQmblen t i
i PQenv2encoding v
Dan Sugalski
>Dan Sugalski wrote:
>>Yep, though that's a big part of it. postgres.pasm is generated
>>from postgres.declarations, FWIW--there's a script in the library
>>somewhere.
>
>Is /parrot/build_tools/build_nativecall.pl the script in question and if
>so whats its usage.
It's util/ncidef2pasm.pl, actually. build_nativecall builds the stub
routines for the interpreter if a JIT isn't available. The
definitions of the characters are the same, but ncidef2pasm's a bit
better documented. (There's embedded POD) Invocation is:
util/ncidef2pasm.pl definition_file output.pasm
>
>I have done postgres.declarations, please see below.
Woo! And, might I add, hoo! :) Thanks.
>Revision 1.100
Gah. Good indication that it's time to get the metadata for library
versioning in place.
>#extern void PQreset(PGconn *conn);
># This was set to "c PQreset p" was this correct.
>v PQreset p
Yours is right. (I expect it was a typo on my part, unless Postgres'
interface changed, which is really unlikely)
># Need function pointers. Can't do that yet
>##### New IN 7.4
>#i PQsetNoticeReceiver p p
>#####
># i PQsetNoticeProcessor p p
I think, oddly enough, that we can actually do these, but I'm not
sure it can be done with the nci interface as it currently stands.
The callback function can be an entry to the interpreter runops loop
and the parameter the pointer to the closure to execute, but I'm not
sure how to express that one properly at the moment.
># The following functions are missing from the original 7.3 version
># so I am leaving these as well. I am including the declarations here
># to let people know the ommission was deliberate. Not 100% why
># though ;-)
Because of the FILE * argument. I didn't have a good way at the time
to get one,so I left the functions out. They're just
debugging/convenience functions, so I was fine with that one.
Looks really good, though. Thanks heaps -- now it's on to
higher-level stuff! :)
Harry Jackson
>
> It's util/ncidef2pasm.pl, actually. build_nativecall builds the stub
> routines for the interpreter if a JIT isn't available. The definitions
> of the characters are the same, but ncidef2pasm's a bit better
> documented. (There's embedded POD) Invocation is:
>
>
>> I have done postgres.declarations, please see below.
>
> Woo! And, might I add, hoo! :) Thanks.
Do you want me to do anything else with it. The script above works fine
on the file.
h
Dan Sugalski
Dan Sugalski
Dunno if I replied, but... Next step is a higher level wrapper, if
you're up for fiddling with Postgres itself. Stuff like a single call
to connect (right now you have to make the connect call and poll over
and over again), getting back a full row as an array, getting back a
full row as a hash, and stuff like that. Nothing fancy, and nothing
that high-level, but enough to work the basics without quite as
manual work as the current libpg requires.
This'll probably be the basis for the DB driver interface for
Parrot's DBI library, so this is your chance to make a mark. :)
Harry Jackson
>
> Dunno if I replied, but... Next step is a higher level wrapper, if
> you're up for fiddling with Postgres itself. Stuff like a single call
> to connect (right now you have to make the connect call and poll over
> and over again),
I did some benchmarks using your original library a few weeks ago and I
did find the polling business a bit odd.
I replied a while back asking asking if anyone had any pointers or
examples on where to start this no matter how simple but I think
"Warnock's Dilema #2" may have kicked in.
Being fairly new to parrot and probably naive I can see a few different
ways of doing this.
A C wrapper ie: parrot/classes/parrotdbi.pmc etc
A C wrapper ie: parrot/dynclasses/parrotdbi.pmc etc
A loadable pasm function library that uses the current libpq/(other
libs) and available available ops.
> getting back a full row as an array, getting back a full
> row as a hash, and stuff like that. Nothing fancy, and nothing that
> high-level, but enough to work the basics without quite as manual work
> as the current libpg requires.
>
> This'll probably be the basis for the DB driver interface for Parrot's
> DBI library, so this is your chance to make a mark. :)
Cheers for the opportunity.
Harry
Tim Bunce
> Dan Sugalski wrote:
> >
> > Dunno if I replied, but... Next step is a higher level wrapper, if
> > you're up for fiddling with Postgres itself. Stuff like a single call
> > to connect (right now you have to make the connect call and poll over
> > and over again),
>
> I did some benchmarks using your original library a few weeks ago and I
> did find the polling business a bit odd.
>
> I replied a while back asking asking if anyone had any pointers or
> examples on where to start this no matter how simple but I think
> "Warnock's Dilema #2" may have kicked in.
>
> Being fairly new to parrot and probably naive I can see a few different
> ways of doing this.
>
> A C wrapper ie: parrot/classes/parrotdbi.pmc etc
> A C wrapper ie: parrot/dynclasses/parrotdbi.pmc etc
> A loadable pasm function library that uses the current libpq/(other
> libs) and available available ops.
Please avoid using dbi and dbd in names for now.
> > getting back a full row as an array, getting back a full
> > row as a hash, and stuff like that. Nothing fancy, and nothing that
> > high-level, but enough to work the basics without quite as manual work
> > as the current libpg requires.
> >
> > This'll probably be the basis for the DB driver interface for Parrot's
> > DBI library, so this is your chance to make a mark. :)
>
> Cheers for the opportunity.
Best wishes.
Tim.
Dan Sugalski
>Dan Sugalski wrote:
>>
>> Dunno if I replied, but... Next step is a higher level wrapper, if
>> you're up for fiddling with Postgres itself. Stuff like a single
>>call > to connect (right now you have to make the connect call and
>>poll over
>> and over again),
>
>I did some benchmarks using your original library a few weeks ago and I
>did find the polling business a bit odd.
>
>I replied a while back asking asking if anyone had any pointers or
>examples on where to start this no matter how simple but I think
>"Warnock's Dilema #2" may have kicked in.
D'oh! I thought I'd replied, but I guess not.
>Being fairly new to parrot and probably naive I can see a few different
>ways of doing this.
>
>A C wrapper ie: parrot/classes/parrotdbi.pmc etc
>A C wrapper ie: parrot/dynclasses/parrotdbi.pmc etc
>A loadable pasm function library that uses the current libpq/(other
>libs) and available available ops.
I'd go with option 3, the loadable pasm library that uses the current
libpq. I think that, right now, there's no need for anything else.
That may turn out to be incorrect, but if so we can try Plan B.
(Which involves either extending Parrot or getting the C extension
stuff working)
Tim's right -- dodge the DBD/DBI names for right now. We're going to
have enough of a headache with those later, what with there being
both Ruby and Perl versions that use it. (Possibly Python too, I'm
not sure)
> > getting back a full row as an array, getting back a full
>> row as a hash, and stuff like that. Nothing fancy, and nothing that
>> high-level, but enough to work the basics without quite as manual work
>> as the current libpg requires.
>>
>> This'll probably be the basis for the DB driver interface for Parrot's
>> DBI library, so this is your chance to make a mark. :)
>
>Cheers for the opportunity.
Always happy to pass on work to other people. :)
Harry Jackson
> At 1:02 PM +0000 12/27/03, Harry Jackson wrote:
>
>> Being fairly new to parrot and probably naive I can see a few different
>> ways of doing this.
>>
>> A C wrapper ie: parrot/classes/parrotdbi.pmc etc
>> A C wrapper ie: parrot/dynclasses/parrotdbi.pmc etc
>> A loadable pasm function library that uses the current libpq/(other
>> libs) and available available ops.
>
>
> I'd go with option 3, the loadable pasm library that uses the current
> libpq. I think that, right now, there's no need for anything else. That
> may turn out to be incorrect, but if so we can try Plan B. (Which
> involves either extending Parrot or getting the C extension stuff working)
OK. I will start on it now. I will probably have lots of questions while
doing this.
> Tim's right -- dodge the DBD/DBI names for right now. We're going to
> have enough of a headache with those later, what with there being both
> Ruby and Perl versions that use it. (Possibly Python too, I'm not sure)
I was only using them as examples. I will leave them pretty generic ie
for the time being.
>> > getting back a full row as an array, getting back a full
>>
>>> row as a hash, and stuff like that. Nothing fancy, and nothing that
>>> high-level, but enough to work the basics without quite as manual work
>>> as the current libpg requires.
>>>
>>> This'll probably be the basis for the DB driver interface for Parrot's
>>> DBI library, so this is your chance to make a mark. :)
>>
>>
>> Cheers for the opportunity.
>
>
> Always happy to pass on work to other people. :)
As soon as I have a something of worth I will post the details.
H
Harry Jackson
>
>> > getting back a full row as an array, getting back a full
>>
>>> row as a hash, and stuff like that. Nothing fancy, and nothing that
>>> high-level, but enough to work the basics without quite as manual work
>>> as the current libpg requires.
OK.
I am at the point now where I need to know what type of format you want
the data to come out in.
We have the following options although some of them will be impractical
in production. I can drop the data into any type of structure currently
available to Parrot at least I am pretty sure I can.
I can create the entire dataset and return this out to the caller as a
hash of arrays or some other structure. For large datasets this will be
completey impractical but I am highlighting it as an option for testing
or possibly avoiding multiple calls between Parrot and Any Old Language
(AOL).
We can call a funtion to return the data in any format you want ie a
single record per call gets passed back. This method will probably be
the closest to most libraries in circulation and is the one that makes
most sense to me. It could be extended to pass back N records depending
on what the caller wants, this might be faster than making lots ot AOL
calls to Parrot but would involve some more work on our part.
For later use would it make it easier for people at a higher abstraction
if some metadata gets passed about the data ie the very first row
returned contains an array of types subsequent calls will return. Perl
is lovely the way it converts types but this might not be very practical
for other languages that are a bit more strict about stuff like this. At
the moment I am using "strings" for all the data coming from the
database but this wont work for everyone. This needs to be decided now
to avoid a re-write later. It would make my life easier if the guys at
the top where to deal with type conversion but I am not sure this is
good choice.
The following is the table that I am testing this against. There are
only very few of the basic types here although for what I have done at
the moment the types have no real affect. This table is loaded with
10000 records (not realistic data).
Table "public.test"
Column | Type | Modifiers
------------+-----------------------------+---------------
_key | integer | not null
_bigint8 | bigint |
_bool | boolean |
_char | character(10) |
_varchar | character varying(100) |
_float8 | double precision |
_int | integer |
_float4 | real |
_int2 | smallint |
_text | text |
_timestamp | timestamp without time zone | default now()
Indexes: parrot_pkey primary key btree (_key)
For the speed freaks doing "select * from test"
real 0m0.997s
user 0m0.630s
sys 0m0.010s
Displaying all 10000 records to screen as follows
9996 9176 t a Varchar here 9176 9176 9176 9176 smallint <- Text
here -> timestamp 2004-01-11 16:45:28.79144
9997 2182 t a Varchar here 2182 2182 2182 2182 smallint <- Text
here -> timestamp 2004-01-11 16:45:28.79379
9998 4521 t a Varchar here 4521 4521 4521 4521 smallint <- Text
here -> timestamp 2004-01-11 16:45:28.79614
9999 4152 t a Varchar here 4152 4152 4152 4152 smallint <- Text
here -> timestamp 2004-01-11 16:45:28.79849
real 0m4.189s
user 0m0.570s
sys 0m0.280s
Any requests, pointers, advice, abuse or general chit chat welcome.
Harry Jackson
Harry Jackson
The following is what I have come up with to date as far as accessing
data in Postgres is concerned. There is very little error handling in
the library at the moment which is something that needs to be addressed
but I can start work on that as soon as the API has been agreed on.
I am fishing for some feedback to see if this is suitable or if it needs
to be changed. The following code is an example of extracting 10,000
rows with field names and types. The types are integers which are local
to Postgres so we probably need to come up with an agreed format for
type identifiers.
1 .pcc_sub _MAIN prototyped
2 .param pmc argv
The first lib is the standard pasm lib that ships with the parrot
source. The second i simply a lib I have created to hold some function
declarations etc.
3 .include "/home/parrot/parrot/library/postgres.pasm"
4 .include "/home/parrot/lib/postgreslib.imc"
5 .local string dbstring
6 dbstring = "host=host dbname=Forum user=u password=pass"
7 .local int answer
8 print "Entering Connect\n"
The call to connect makes whatever calls etc required to get a
connection to the database.
9
10 .pcc_begin prototyped
11 .arg dbstring
12 .pcc_call connect
13 retconnect:
14 .result CONN
15 .result answer
16 .result message
17 .pcc_end
The MetaData hash contains various meta data about the connection ie
filed types and names.
18
19 .local PerlHash MetaData
20 MetaData = new PerlHash
21 MetaData = global "MetaData"
22
23 .PRINT("Connection Message = ", message, " \n")
24 .PRINT("Connection state = ", answer, " \n")
25 eq -1, answer, fail
26 eq 1, answer, go
27 fail:
28 .PRINT("\n", message, "\n")
29 end
30 go:
31
32 .local string query
33 query = "select * from parrot"
34
35 print "Entering Send Query \n"
36 .pcc_begin prototyped
37 .arg CONN
38 .arg query
39 .pcc_call pqsendquery
40 pqsendquery:
41 .result message
42 .pcc_end
The pqgetresult call will populate the MetaData hash with details of the
call.
43
44 .PRINT("Execution = ", message, "\n")
45 .pcc_begin prototyped
46 .arg CONN
47 .pcc_call pqgetresult
48 retrecords:
49 .pcc_end
50 .local int rowcounter
51 rowcounter = MetaData["ROWCOUNT"]
52 eq -1, rowcounter, finished
53
The following bit of code is here to test that fieldnames and types have
been filled correctly.
54
55 .local int Oid_type
56 .local int onfield
57 onfield = 1
58 .local PerlArray TupleData
59 TupleData = new PerlArray
60 TupleData = MetaData["FIELDDATA"]
61 .local int fnum
62
63 fnum = MetaData["NFIELDS"]
64
65 .local string field_name
66 .local PerlArray FieldData
67 FieldData = new PerlArray
68 gettype:
69 FieldData = TupleData[onfield]
70 inc onfield
71 field_name = FieldData[0]
72 Oid_type = FieldData[1]
73 .PRINT("Field Name = ",field_name, "\n")
74 .PRINT("Field Type = ",Oid_type, "\n")
75 if onfield <= fnum goto gettype
76
77 .local PerlString value
78 value = new PerlString
79 rowcounter = 0
Once all the necessary calls and connections have been made we can then
fetch individual rows of data. These come out in a PerlArray at the moment.
80 getnext:
81 onfield = 0
82 .pcc_begin prototyped
83 .pcc_call fetch
84 nextrow:
85 .result record
86 .result answer
87 .pcc_end
88 inc rowcounter
89 nextfield:
90
91 value = record[onfield]
92 .PRINT("", value, "#")
93 inc onfield
94 if onfield <= fnum goto nextfield
95 print "\n"
96 ne 0, answer, getnext
97
98 finished:
99 .PQclear(MetaData["RESULT"])
100 .PQresultErrorMessage(MetaData["RESULT"], message)
101 .PRINT("\n\n Rows",rowcounter , "\n\n")
102 end
103 .end
104 ########################################
105 .include "/home/parrot/lib/pgreslib.imc"
I am not to sure if this is the sort of thing that was wanted or not. If
not let me know what needs to be changed and I will change it.
Harry
Leopold Toetsch
> I am at the point now where I need to know what type of format you want
> the data to come out in.
The first question is: how are these data returned in C. If that's a
defined structure, I'd overlay this structure with an UnManagedStruct
PMC and then access the structure with accessors of that PMC - neede
still some types as well as handling of nested types - but this avoid
creating of new objects and should be rather efficient.
> Harry Jackson
leo
Dan Sugalski
>Harry Jackson <ha...@uklug.co.uk> wrote:
>
>> I am at the point now where I need to know what type of format you want
>> the data to come out in.
>
>The first question is: how are these data returned in C.
For posgres, that's easy (which is part of the problem) -- there *is*
no structure. There's a single function that takes a row/column
coordinate and returns the value. If you make a query that returns,
say, 7 rows with 13 columns each, you have to call this function 91
times to get all the data back...
Dan Sugalski
>Dan Sugalski wrote:
>>
>>> > getting back a full row as an array, getting back a full
>>>
>>>> row as a hash, and stuff like that. Nothing fancy, and nothing that
>>>> high-level, but enough to work the basics without quite as manual work
>>>> as the current libpg requires.
>
>OK.
>
>I am at the point now where I need to know what type of format you want
>the data to come out in.
Well...
What I'd like, I think, is something simple and straightforward.
Right now we've got to fetch each column for each row one by one,
which is a pain in the neck if you want to get a full row back.
Having a fetchrow that returned an Array with the value for column
one in the 0th slot, column 2 in the 1st slot and so on would be
about 80% of the solution.
Having a fetchrow_hash that returned a Hash where the keys are the
column names and the values are the column values would be most of
the rest.
If you wanted to go the rest of the way and add an extra array
version (where you get an array of rows, with each row entry being
either a row array or a row hash) I think we'd be about where we'd
love to be.
Harry Jackson
> At 9:05 PM +0100 1/11/04, Leopold Toetsch wrote:
>
>> Harry Jackson <ha...@uklug.co.uk> wrote:
>>
>>> I am at the point now where I need to know what type of format you want
>>> the data to come out in.
>>
>>
>> The first question is: how are these data returned in C.
>
>
> For posgres, that's easy (which is part of the problem) -- there *is* no
> structure. There's a single function that takes a row/column coordinate
> and returns the value. If you make a query that returns, say, 7 rows
> with 13 columns each, you have to call this function 91 times to get all
> the data back...
Thankyou Dan. I was about to reply honest, you put it much better than
me anyway.
Harry
Harry Jackson
>
> Well...
>
> What I'd like, I think, is something simple and straightforward. Right
> now we've got to fetch each column for each row one by one, which is a
> pain in the neck if you want to get a full row back. Having a fetchrow
> that returned an Array with the value for column one in the 0th slot,
> column 2 in the 1st slot and so on would be about 80% of the solution.
I have done this part.
>
> Having a fetchrow_hash that returned a Hash where the keys are the
> column names and the values are the column values would be most of the
> rest.
I read somewhere that accessing a hash was slightly slower than
accessing and array which is one reason why I never used it. The other
reason is that if I name the fileds in the hash then the user needs to
know the names to access them or perform some magic to get them. With an
array they come out in the order they are aksed for.
Another reason not to use the hash method above is that you are moving
column names around that will not change throughout the transaction (is
this not more bulky than using arrays). Should we not return the names
and types first and then subsequent rows in arrays indexed in order of
retrieval.
I like this method becasue thats how I have already done it ;-) just
being biased.
> If you wanted to go the rest of the way and add an extra array version
> (where you get an array of rows, with each row entry being either a row
> array or a row hash) I think we'd be about where we'd love to be.
It is entirely up to you lot. If you want it on a stamp I will see what
I can come up with ;-)
Harry Jackson.
Dan Sugalski
>Dan Sugalski wrote:
>>
>>Well...
>>
>>What I'd like, I think, is something simple and straightforward.
>>Right now we've got to fetch each column for each row one by one,
>>which is a pain in the neck if you want to get a full row back.
>>Having a fetchrow that returned an Array with the value for column
>>one in the 0th slot, column 2 in the 1st slot and so on would be
>>about 80% of the solution.
>
>I have done this part.
>
>>
>>Having a fetchrow_hash that returned a Hash where the keys are the
>>column names and the values are the column values would be most of
>>the rest.
>
>I read somewhere that accessing a hash was slightly slower than
>accessing and array which is one reason why I never used it. The
>other reason is that if I name the fileds in the hash then the user
>needs to know the names to access them or perform some magic to get
>them. With an array they come out in the order they are aksed for.
Definitely. However... Having seen code in production, generally the
fields aren't changeable and are known at compile-time, or folks are
writing generic code (for better or worse). In the first case people
use hash access because they know there's a "name" and "city" field
in the results, and in the second case they're iterating across the
keys and pulling out values.
Since folks are going to wrap the results in a hash regardless of
whether it's a good idea or not (not going there... :) we might as
well have it in at the very lowest level where we can get the
information most efficiently.
>Another reason not to use the hash method above is that you are
>moving column names around that will not change throughout the
>transaction (is this not more bulky than using arrays). Should we
>not return the names and types first and then subsequent rows in
>arrays indexed in order of retrieval.
>
>I like this method becasue thats how I have already done it ;-) just
>being biased.
That works too. If the information's available someone'll build what
they want. Whichever way you're comfortable with. (Though given my
preferences, I'd add in the hash fetch version as part of the
low-level interface)
Harry Jackson
>
> That works too. If the information's available someone'll build what
> they want. Whichever way you're comfortable with. (Though given my
> preferences, I'd add in the hash fetch version as part of the low-level
> interface)
Roger:
To clarify you want:
1. You want a hash with
key value
"fieldname" "fieldvalue"
done similar to
80 getnext:
81
82 .pcc_begin prototyped
83 .pcc_call fetchrow_hash
84 nextrow:
85 .result rowhash
86 .result answer
87 .pcc_end
where each call to the function will return the hash containg the next
rows data.
Do you have any requests for anything else on, around or near this
before I start. I should be able to ruffle something up pretty quickly.
Harry Jackson
Disclaimer: The lib is not very robust at the moment I am just trying to
get an interface sorted out so I can then get on with coding behind it
rather than chasing goal posts. I am really loking forward to all the
error handling ;-)
Jeff Clites
> Dan Sugalski wrote:
>
>> Having a fetchrow_hash that returned a Hash where the keys are the
>> column names and the values are the column values would be most of
>> the rest.
>
> I read somewhere that accessing a hash was slightly slower than
> accessing and array which is one reason why I never used it. The other
> reason is that if I name the fileds in the hash then the user needs to
> know the names to access them or perform some magic to get them.
The names will be known--they are the column names (or aliases) used in
the query.
> With an array they come out in the order they are aksed for.
The nice thing about a hash (that I've found when using DBI) is that
you don't have a problem if you end up inadvertently changing the order
in which the columns are returned, when modifying a query.
> Another reason not to use the hash method above is that you are moving
> column names around that will not change throughout the transaction
> (is this not more bulky than using arrays).
Hopefully the actual strings used as the keys can be re-used. (That is,
each hash can use the same string instance for a given key. I believe
that Perl5 does this for hash keys.)
JEff
Harry Jackson
> On Jan 12, 2004, at 8:07 AM, Harry Jackson wrote:
>
>> Dan Sugalski wrote:
>>
>>> Having a fetchrow_hash that returned a Hash where the keys are the
>>> column names and the values are the column values would be most of
>>> the rest.
>>
>>
>> I read somewhere that accessing a hash was slightly slower than
>> accessing and array which is one reason why I never used it. The other
>> reason is that if I name the fileds in the hash then the user needs to
>> know the names to access them or perform some magic to get them.
>
>
> The names will be known--they are the column names (or aliases) used in
> the query.
select * from table_name
This does not tell me the names of the tables or anything about the
order they are going to come out in.
>
> The nice thing about a hash (that I've found when using DBI) is that you
> don't have a problem if you end up inadvertently changing the order in
> which the columns are returned, when modifying a query.
That is one advantage I never thought of and a fairly good one.
> Hopefully the actual strings used as the keys can be re-used. (That is,
> each hash can use the same string instance for a given key. I believe
> that Perl5 does this for hash keys.)
Yes inded they can, it was a bit short sighted on my part not noticing
this although the fact the exist is overhead but that is just me being
pedantic ;-) I am currently working on a way to do it at the moment.
Could someone tell me who exactly will be taking advantage of this
library ie at what level are we pitching it at. I seriously doubt that
most developers are going to even see the implimentation of this, rather
its the guys coding drivers etc in which case its those guys that are
going to be worried about the user interface. Or am I off the mark here.
Harry
Dan Sugalski
>Dan Sugalski wrote:
>>
>>That works too. If the information's available someone'll build
>>what they want. Whichever way you're comfortable with. (Though
>>given my preferences, I'd add in the hash fetch version as part of
>>the low-level interface)
>
>Roger:
>
>To clarify you want:
>
>1. You want a hash with
>
> key value
> "fieldname" "fieldvalue"
>
>
>done similar to
>
>
> 80 getnext:
> 81
> 82 .pcc_begin prototyped
> 83 .pcc_call fetchrow_hash
> 84 nextrow:
> 85 .result rowhash
> 86 .result answer
> 87 .pcc_end
>
>where each call to the function will return the hash containg the
>next rows data.
Yep, that's it. "Here's the result handle, gimme a row of data as a
hash". If you want to keep state such that it's a real iterator,
that's fine. If you want to leave it so that the user of the library
has to be explicit with which row they want, that's fine too.
>Do you have any requests for anything else on, around or near this
>before I start. I should be able to ruffle something up pretty
>quickly.
Nope, can't think of anything yet.
>Disclaimer: The lib is not very robust at the moment I am just
>trying to get an interface sorted out so I can then get on with
>coding behind it rather than chasing goal posts. I am really loking
>forward to all the error handling ;-)
Heh. Isn't error handling always fun? I think I need to put
"exceptions" on this week's todo list.
Gordon Henriksen
fetchrow_hashref is definitely a very useful, but my favorite (and also
the most efficient) DBI methodology is bind_columns. DBI maintains a
list of references corresponding to columns in the result set, and when
the result set is advanced, stores the values into the variables
referenced. e.g., Perl 5:
my $sth = $dbh->prepare("select a, b, c from tab");
$sth->execute;
$sth->bind_columns(\my($a, $b, $c));
while ($sth->fetch) {
print "a: $a, b: $b, c: $c\n";
}
Equivalent to:
my $sth = $dbh->prepare("select a, b, c from tab");
$sth->execute;
$sth->bind_col(0, \my $a);
$sth->bind_col(1, \my $b);
$sth->bind_col(2, \my $c);
while ($sth->fetch) {
print "a: $a, b: $b, c: $c\n";
}
So if you're going to basically go all out in emulating DBI's fetch_*
permutations, don't forget this one. :)
—
Gordon Henriksen
mali...@mac.com
Harry Jackson
>
> fetchrow_hashref is definitely a very useful, but my favorite (and also
> the most efficient) DBI methodology is bind_columns. DBI maintains a
> list of references corresponding to columns in the result set, and when
> the result set is advanced, stores the values into the variables
> referenced. e.g., Perl 5:
>
> my $sth = $dbh->prepare("select a, b, c from tab");
> $sth->execute;
> $sth->bind_columns(\my($a, $b, $c));
>
> while ($sth->fetch) {
> print "a: $a, b: $b, c: $c\n";
> }
>
> Equivalent to:
>
> my $sth = $dbh->prepare("select a, b, c from tab");
> $sth->execute;
> $sth->bind_col(0, \my $a);
> $sth->bind_col(1, \my $b);
> $sth->bind_col(2, \my $c);
>
> while ($sth->fetch) {
> print "a: $a, b: $b, c: $c\n";
> }
>
> So if you're going to basically go all out in emulating DBI's fetch_*
> permutations, don't forget this one. :)
I am not really trying to emulate any methods at the moment it just
happens that these ways come naturally to Perl Hackers. In fact the
entire way the data comes out is open for debate and it might be a good
time to add a few nice features to it if anyone can think of any.
I doubt if any of the functionality that currently exists will be
changed but this is not up to me, all that stuff is at a much higher
layer of abstraction than where I am currently digging.
Harry Jackson
Harry Jackson
> At 4:50 PM +0000 1/12/04, Harry Jackson wrote:
>>
>> done similar to
>>
>>
>> 80 getnext:
>> 81
>> 82 .pcc_begin prototyped
>> 83 .pcc_call fetchrow_hash
>> 84 nextrow:
>> 85 .result rowhash
>> 86 .result answer
>> 87 .pcc_end
>>
>> where each call to the function will return the hash containg the next
>> rows data.
>
>
> Yep, that's it. "Here's the result handle, gimme a row of data as a
> hash". If you want to keep state such that it's a real iterator, that's
> fine. If you want to leave it so that the user of the library has to be
> explicit with which row they want, that's fine too.
Ok. I have the following working after a fashion
After connecting and various other contortions we get to the extraction
of the data part which involves the following.
133 getnext:
134 onfield = 0
135 .pcc_begin prototyped
136 .pcc_call fetchrow_hash
137 retfetchhash:
138 .result record_hash
139 .result answer
140 .pcc_end
We have returned a Hash where (key,value) = (field_name, data)
The following shows roughly what I had to do to display it. I am using a
global HASH called MetaData that stores various bits about the current
statement being executed.
141 nextfieldhash:
142
143 FieldData = TupleData[onfield]
144 field_name = FieldData[0]
145
146 value = record_hash[field_name]
147 .PRINT("", value, "#")
148 if onfield == fnum goto nextrowhash
149 inc onfield
150 branch nextfieldhash
151 nextrowhash:
152 print "\n"
153 ne 0, answer, getnext
For the speed freaks displaying 10000 rows to screen takes. This can be
improved as much as my code which could be quite a bit.
Rows 9999
real 0m5.436s
user 0m1.590s
sys 0m0.320s
>> Do you have any requests for anything else on, around or near this
>> before I start. I should be able to ruffle something up pretty quickly.
>
>
> Nope, can't think of anything yet.
Phew.
Harry Jackson
Harry Jackson
> The following shows roughly what I had to do to display it. I am using a
> global HASH called MetaData that stores various bits about the current
> statement being executed.
Sorry: missed a bit
The MetData is where I get the TupleData Array and the associated
FieldData Array from.