Lisp and Persistent Data
Wade Humeniuk
invocations of a system what are merits and problems with the following
approaches? What is the Lisp way, if there is one?
1) Just using the Lisp reader and writer to store persistent data in lisp
files (or fasl files). I have used this method and it seems to be the
easiest and most natural. Just read the file(s) on startup and write them
when exiting. Don't worry about runtime size. How big a database could one
reasonably get this way (Megs, 1 Gig ??). This is probably just a question
of page swapping performance/capabilities of the OS.
2) Use an object database, like PLOB. I have no experience with this
system. For moderately sized databases this approach seems to just add
extra complications (mostly administrative) into an application. What are
the benefits?
3) Use a relational database. The organization of data in the application
is somewhat decoupled from how it is stored in the database. For moderately
sized systems this seems like overkill. A side question, if a client
insisted on an SQL compliant database is there any software to run a SQL
server within a Lisp application to service basic SQL queries? The SQL
queries would have to mapped onto runtime lisp data being used in the
application. Thus approach 1 could be used and still provide a SQL
interface. The simple answer for me would to drop all SQL reporting tools
and just write them in lisp.
I would choose method 1. In a multi-process/multi-machine environment I
would let applications share data using CORBA.
Thoughts?
Wade
Gary Curtis
image. With 1 GB of data the overhead of storing the Lisp is
negligable, and start up will be much faster than any of your
three options.
Gary.
--
Robert Monfera
Wade Humeniuk wrote:
> 1) Just using the Lisp reader and writer to store persistent data in lisp
> files (or fasl files). I have used this method and it seems to be the
> easiest and most natural. Just read the file(s) on startup and write them
> when exiting. Don't worry about runtime size. How big a database could
> one reasonably get this way (Megs, 1 Gig ??). This is probably just a
> question of page swapping performance/capabilities of the OS.
I have run tests of FASL reading and writing of 10s of megs of arrays
(with 256MB physical memory), and these operations took considerably
more time (several minutes) than what I intuitively expected. FASL
operations are fairly general, but more specialized reading or writing
is faster, at the expense of development work (proportional with schema
complexity).
> 3) Use a relational database. The organization of data in the application
> is somewhat decoupled from how it is stored in the database. For
> moderately
> sized systems this seems like overkill.
If (potential) clients (potentially) need SQL access to the database, it
is not an overkill to use an SQL database even for a small and simple
scheme.
> A side question, if a client
> insisted on an SQL compliant database is there any software to run a SQL
> server within a Lisp application to service basic SQL queries? The SQL
> queries would have to mapped onto runtime lisp data being used in the
> application. Thus approach 1 could be used and still provide a SQL
> interface.
If you write a parser for a subset of the SQL language, you may make it
available through a socket or something else. I am not aware of PD
software that does this.
Another option (mentioned by Gary) is that you dump the image. If
scheme is relatively stable but the bulk of the application is still
under development, you can dump an image with only the data and a few
definitions, so you can load any versions of the application logic
(i.e., no need to dump the image for every version of the application).
Robert
Tim Bradshaw
> To store moderate amounts of persistent data (1 Gig) between Lisp
> invocations of a system what are merits and problems with the following
> approaches? What is the Lisp way, if there is one?
I think the standard answer is that you should use a relational
database and some suitably baroque interface to it.
However I've become increasingly convinced that relational databases
are some kind of sinister death cult who want to lure you in and get
you to wear strange stripy clothes with shiny shoes and give all your
money to your superiors in the cult. And if you don't conform, or if
you conform too well, you just know you'll end up in a pit of
dismembered bodies back in the woods somewhere.
You can tell this is true by the way that rdb people (often people you
once knew quite well who mysteriously `went away' for a year or so and
come back looking suspiciously healthy and stripy) believe that the
answer to *all* computing problems is `use a relational database'.
You just have to be suspicious of that kind of thing, it reeks of
brainwashing.
It's also kind of technically strange -- the solution that was really
good in the days when memory cost a thousand dollars a K is somehow
still the right solution, for the same size problem, when memory costs
are a tenth of a cent a K. That just doesn't sound right to me.
So far I have no real conclusive documentary proof, and the North
Leith Advertiser keeps returning my devastating expos\'e (but my
contacts tell me they use sql server, so I know why *they* won't
publish it, and I know who the stripy-suited figures I keep seeing
peering at my house over on the links are now; but I keep my AK47s
loaded and ready in the bay in the study -- they won't get me so
easily).
So I really like some variant of your case (1): Just use Lisp. Don't
let the cult take all your money, and whatever you do, stay vigilant,
drink only pure rainwater, and don't let your rifles get rusty:
they're outside your house, waiting for their moment to drug you and
bury you in the woods.
--tim (if it wasn't just another cult, this article would be MIME type
text/x-green-crayon, if there was such a thing)
Jon Dyte
"Wade Humeniuk" <hume...@cadvision.com> wrote:
> 2) Use an object database, like PLOB. I have no experience with this
> system. For moderately sized databases this approach seems to just
add
> extra complications (mostly administrative) into an application. What
are
> the benefits?
>
I dont think PLob will go to about 1G of data. You get severe
performance problems with a large number of objects, say 100,00O.
This is due to the Postore Persistent Heap that Plob uses, the source
of which is not in Plob's source tree.
Jon
Sent via Deja.com http://www.deja.com/
Before you buy.
David Bakhash
> * Wade Humeniuk wrote:
> > To store moderate amounts of persistent data (1 Gig) between Lisp
> > invocations of a system what are merits and problems with the following
> > approaches? What is the Lisp way, if there is one?
>
> I think the standard answer is that you should use a relational
> database and some suitably baroque interface to it.
Yeah. Consider too the speed/space tradeoff. Each SQL call is slow
and expensive, but will save you from having to suck all the data into
memory, and (if you were willing) write a client/server system
(analogous to how SQL servers work).
> However I've become increasingly convinced that relational databases
> are some kind of sinister death cult who want to lure you in and get
> you to wear strange stripy clothes with shiny shoes and give all your
> money to your superiors in the cult. And if you don't conform, or if
> you conform too well, you just know you'll end up in a pit of
> dismembered bodies back in the woods somewhere.
I disagree. database technology is not simple, and companies strive
to provide state-of-the-art features, speed, and efficiency, and some
charge an are and a leg for it. But consider that MSSQL Server is a
fine db and a few hundred bucks (last time I checked). Consider that
there are a bunch of free RDBMSs out there, namely MySQL and
PostgreSQL.
> You can tell this is true by the way that rdb people (often people you
> once knew quite well who mysteriously `went away' for a year or so and
> come back looking suspiciously healthy and stripy) believe that the
> answer to *all* computing problems is `use a relational database'.
> You just have to be suspicious of that kind of thing, it reeks of
> brainwashing.
while lots of people think that there's only one right solution and
it's the one they're comfortable with, in the RDBMS case, there is
actually a reason why a relation database is the right answer a lot of
the time. Let me explain from my perspective.
First off, let's consider the Web. Lots of web-based applications out
there need to be accessed by many different programs, written in
different languages. I wrote an application a year ago whose data
needed to be accessed by Lisp, Java, and Perl/CGI. When you have to
pick a single, uniform way to store/retreive the data, it's not just
the obvious solution; it's the _only_ solution. We used the MSSQL
Server, and it did okay, though was a pain in the ass too sometimes.
But then again, we didn't have a real dba helping us.
RDBMSs have been evolving for decades, and are now pretty fast. Many
systems (Lisp included) have interfaces into RDBMSs, namely ACL,
LispWorks, CLISP, elisp, and I'm sure more, though that's the extent
of my experience.
I'm not saying that it's the obvious solution for all or most
applications; I'm just saying that this post is inconsiderate of many
of the constraints that programmers have to deal with. I wish there
were an alternative solution that worked across multiple platforms,
but there really isn't any.
One of the reasons people use Lisp is so that they can _avoid_ writing
things like parsers. If there's a solution out there that works, I
say just use it. It's not that bad.
dave
Seth Gordon
> It's also kind of technically strange -- the solution that was really
> good in the days when memory cost a thousand dollars a K is somehow
> still the right solution, for the same size problem, when memory costs
> are a tenth of a cent a K. That just doesn't sound right to me.
As I understand it, the relational database systems on the market today have
their own systems for caching recently- or frequently-requested data in
memory, so the implementations have kept up with changes in memory/disk
costs.
Of course, you should take this information with a grain of salt because:
(1) I'm just a lowly tech writer, not a DBA;
(2) My employer sells software built on top of relational database systems.
I would explain in more detail, but a man in a stripy suit just knocked on my
door and told me it's time for my low-protein lunch.
--
perl -le"for(@w=(q[dm='r 0rJaa,u0cksthe';dc=967150;dz=~s/d/substrdm,\
(di+=dc%2?4:1)%=16,1ordi-2?'no':'Perl h'/e whiledc>>=1;printdz]))\
{s/d/chr(36)/eg;eval;}#In Windows type this all on 1 line w/o '\'s"
== seth gordon == sgo...@kenan.com == standard disclaimer ==
== documentation group, kenan systems corp., cambridge, ma ==
Paolo Amoroso
It's also kind of technically strange -- the solution that was really
good in the days when memory cost a thousand dollars a K is somehow
still the right solution, for the same size problem, when memory costs
are a tenth of a cent a K. That just doesn't sound right to me.
Paraphrasing Sun, "the RAM is the database" :)
Paolo
--
EncyCMUCLopedia * Extensive collection of CMU Common Lisp documentation
http://cvs2.cons.org:8000/cmucl/doc/EncyCMUCLopedia/
Christopher Browne
>Tim Bradshaw wrote:
>
>> It's also kind of technically strange -- the solution that was really
>> good in the days when memory cost a thousand dollars a K is somehow
>> still the right solution, for the same size problem, when memory costs
>> are a tenth of a cent a K. That just doesn't sound right to me.
>
>their own systems for caching recently- or frequently-requested data in
>memory, so the implementations have kept up with changes in memory/disk
>costs.
Indeed. Some newer designs, Caché and TimesTen, for instance, get away
from the tradition of "paging;" they store the whole DBMS in memory,
so that the system touches disk for two reasons:
a) To read in the data at startup time, and
b) To duplicate all updates to disk so they aren't lost if the system
goes down.
I'd think it sensible to use something _like_ paging with the sorts of
data that are only accessed infrequently.
--
"There is no psychiatrist in the world like a puppy licking your
face." -- Ben Williams
cbbr...@ntlug.org - - <http://www.hex.net/~cbbrowne/lsf.html>
Tim Bradshaw
> I'd think it sensible to use something _like_ paging with the sorts of
> data that are only accessed infrequently.
That's what the operating system is for!
--tim
William Deakin
Tim wrote:
> ...I've become increasingly convinced that relational databases
> are some kind of sinister death cult who want to lure you in and get
> you to wear strange stripy clothes with shiny shoes and give all your
> money to your superiors in the cult...
Following my careful investigations: I believe that *you* Tim have worn
stripy clothing that could be (by the unkind at least) described as
`strange'; and I'm sure that have worn *polished shoes*! (or boots anyway) So
I now realise your paranoia is generated by your fear of being got for being
a splitter ;) Deny this if you dare...
> You can tell this is true by the way that rdb people...believe that the
> answer to *all* computing problems is `use a relational database'.
But this is self-evidently true ;) If Solaris was written in SQL just look
how much better it would be; and just think perl written in stored procedures
using the Oracle PL/SQL. Look the name is almost the same!
;) will
ps:
> ...So far I have no real conclusive documentary proof, and the North Leith
> Advertiser keeps returning my devastating expos\'e...
I have high level contacts with the Brecon and Radnor Express and would be
happy to pass on your concerns. (No worries about shenanigans. AFAIK the B&R
is still printed using hand-set type and all related information is stored in
large cardboard boxes containing small pieces of slightly curled yellowing
paper.)