How to store data?
Lars Behrens
For a web project I need a little expert help. I don't have written much
code yet, just been fiddling around a bit, testing and planning.
The web site will have a submission page for attendants of a congress.
In a form the user will submit name, mailadress etc. and I plan to store
the data in a list of dictionaries:
People = [{'name' : 'Sir Galahad', 'address' : 'Camelot', 'mail' :
'Carrie...@england.com'}]
#new attendant:
People.append({'name' : 'Brian', 'address' : 'Palestine', 'mail' :
'fastridin...@rome.com'})
The 'lines' can be accessed by their index.
Now my question: Where do you suggest to store the data? I know that
something sql-like would be quite useful but I don't have any knowledge
of sql. So I thought about putting the data either into a comma
separated file or pickling it.
I should mention that some other people on different operation systems
should be able to import the file (Excel, Open Office) so I tend towards
the csv file.
What would be better?
And another problem: How to lock the file when several users try to
write to it?
As I'm no very experienced programmer I would appreciate your help,
thanks in advance
Cheerz Lars
John Roth
"Lars Behrens" <Spam....@web.de> wrote in message
news:bi8lqg$6qoh4$1...@ID-95936.news.uni-berlin.de...
> Hi there!
>
> For a web project I need a little expert help. I don't have written much
> code yet, just been fiddling around a bit, testing and planning.
>
> The web site will have a submission page for attendants of a congress.
> In a form the user will submit name, mailadress etc. and I plan to store
> the data in a list of dictionaries:
>
> People = [{'name' : 'Sir Galahad', 'address' : 'Camelot', 'mail' :
> 'Carrie...@england.com'}]
>
> #new attendant:
> People.append({'name' : 'Brian', 'address' : 'Palestine', 'mail' :
> 'fastridin...@rome.com'})
>
> The 'lines' can be accessed by their index.
>
> Now my question: Where do you suggest to store the data? I know that
> something sql-like would be quite useful but I don't have any knowledge
> of sql. So I thought about putting the data either into a comma
> separated file or pickling it.
>
> I should mention that some other people on different operation systems
> should be able to import the file (Excel, Open Office) so I tend towards
> the csv file.
>
> What would be better?
Python 2.3 has an excellent CSV module.
> And another problem: How to lock the file when several users try to
> write to it?
>
> As I'm no very experienced programmer I would appreciate your help,
> thanks in advance
John Roth
>
> Cheerz Lars
>
>
>
>
>
>
>
>
>
>
Paul D. Fernhout
I don't want to see you cut on bleeding edge software, but, I just
released an updated version of the Pointrel Data Repository System
http://sourceforge.net/projects/pointrel/
which could in theory handle such a task (subject to bugs etc.), and I
could not resist responding to your post. The Pointrel System can handle
simple locking and simple transactions (although this has only be tested
recently under GNU/Linux and Python2.3). As a caveat, this Pointrel
System database bunny may have long pointy, nasty teeth, etc., and so if
your application is mission critical you should most likely go with a
more proven solution -- like a convetional relational database (and
using SQL to access one is a good skill to learn). And possibly if your
needs are minimal, pickling your dictionaries or using the 'shelve'
module might just be good enough for a simple application.
Mainly just for fun, and also as another example for people who are
interested in playing with the related concepts, rather than to try to
persuade you specifically to try the Pointrel System (since if you are
not an experienced programmer you would have no way to judge what you
were getting into, and so my best advice to you specifically would be to
go with something proven like pickle, shelve, SQL etc.), here is how you
might solve the storage issue you outline using the Pointrel System:
==================================================
File: "congress.py"
==================================================
from pointrel20030812 import *
Pointrel_initialize("archive_congress")
# add a first attendant -- uses built in unique ID function
# but could be any unique ID string
# each change will be implicitely a seperate transaction
attendantID = Pointrel_generateUniqueID()
Pointrel_add("congress", attendantID, 'object type', 'user')
Pointrel_add("congress", attendantID, 'name', 'Sir Galahad')
Pointrel_add("congress", attendantID, 'address', 'Camelot')
Pointrel_add("congress", attendantID, 'mail', 'Carrie...@england.com')
# add a second attendant, this time as an atomic transaction
attendantID = Pointrel_generateUniqueID()
Pointrel_startTransaction()
Pointrel_add("congress", attendantID, 'object type', 'user')
Pointrel_add("congress", attendantID, 'name', 'Brian')
Pointrel_add("congress", attendantID, 'address', 'Palestine')
Pointrel_add("congress", attendantID, 'mail',
'fastridin...@rome.com')
Pointrel_finishTransaction()
# add a third attendant, again as a single transaction
attendantID = Pointrel_generateUniqueID()
Pointrel_startTransaction()
Pointrel_add("congress", attendantID, 'object type', 'user')
Pointrel_add("congress", attendantID, 'name', 'Lars')
Pointrel_add("congress", attendantID, 'address', 'Pythonland')
Pointrel_add("congress", attendantID, 'mail', 'la...@example.com')
Pointrel_finishTransaction()
# update a value, as an implicit transaction
attendantID = Pointrel_lastMatch("congress", WILD, 'name', 'Sir Galahad')
Pointrel_add("congress", attendantID, 'address', "Zoot's castle")
# delete an attendant
attendantID = Pointrel_lastMatch("congress", WILD, 'name', 'Brian')
Pointrel_add("congress", attendantID, 'deleted', 'true')
# get a list of all attendants (deleted or not)
attendants = Pointrel_allMatches("congress", WILD, 'object type', 'user')
print '\nAll attendant IDs (including deleted ones):', attendants
# filter out deleted attendants
currentAttendants = filter(lambda user: Pointrel_lastMatch('congress',
user, 'deleted', WILD) <> 'true', attendants)
print '\nCurrent attendants:', currentAttendants
print "\nAttendant names:"
for attendant in currentAttendants:
print Pointrel_lastMatch('congress', attendant, 'name', WILD)
print "\nAttendant csv info"
for attendant in currentAttendants:
name = Pointrel_lastMatch('congress', attendant, 'name', WILD)
address = Pointrel_lastMatch('congress', attendant, 'address', WILD)
email = Pointrel_lastMatch('congress', attendant, 'mail', WILD)
print '"%s","%s","%s"' %(name, address, email)
# note proper csv output would require escaping embedded quotes etc.
==================================================
Sample output:
==================================================
creating archive file archive_congress.pointrel_database.poi
archive with only header
All attendant IDs (including deleted ones):
['unique://freevolution:23:43207645167115827',
'unique://freevolution:25:37062969616920138',
'unique://freevolution:26:11213192987278475']
Current attendants: ['unique://freevolution:23:43207645167115827',
'unique://freevolution:26:11213192987278475']
Attendant names:
Sir Galahad
Lars
Attendant csv info
"Sir Galahad","Zoot's castle","Carrie...@england.com"
"Lars","Pythonland","la...@example.com"
==================================================
Database file created: 'archive_congress.pointrel_database.xml'
(There is also a binary file which is maintained for efficiency).
==================================================
<?xml version="1.0" encoding="utf-8" ?>
<P:archive xmlns:P="http://www.pointrel.org/Pointrel20030812/NS/"
version="20030812.1" archiveID="unique://freevolution:24:35433055818285208">
<P:transaction>
<P:triad
r="1584L"
s="congress"
a="unique://freevolution:23:43207645167115827"
b="object type"
c="user"
/>
</P:transaction>
<P:transaction>
<P:triad
r="1984L"
s="congress"
a="unique://freevolution:23:43207645167115827"
b="name"
c="Sir Galahad"
/>
</P:transaction>
<P:transaction>
<P:triad
r="2376L"
s="congress"
a="unique://freevolution:23:43207645167115827"
b="address"
c="Camelot"
/>
</P:transaction>
<P:transaction>
<P:triad
r="2792L"
s="congress"
a="unique://freevolution:23:43207645167115827"
b="mail"
c="Carrie...@england.com"
/>
</P:transaction>
<P:transaction>
<P:triad
r="3104L"
s="congress"
a="unique://freevolution:25:37062969616920138"
b="object type"
c="user"
/>
<P:triad
r="3312L"
s="congress"
a="unique://freevolution:25:37062969616920138"
b="name"
c="Brian"
/>
<P:triad
r="3528L"
s="congress"
a="unique://freevolution:25:37062969616920138"
b="address"
c="Palestine"
/>
<P:triad
r="3760L"
s="congress"
a="unique://freevolution:25:37062969616920138"
b="mail"
c="fastridin...@rome.com"
/>
</P:transaction>
<P:transaction>
<P:triad
r="4072L"
s="congress"
a="unique://freevolution:26:11213192987278475"
b="object type"
c="user"
/>
<P:triad
r="4280L"
s="congress"
a="unique://freevolution:26:11213192987278475"
b="name"
c="Lars"
/>
<P:triad
r="4496L"
s="congress"
a="unique://freevolution:26:11213192987278475"
b="address"
c="Pythonland"
/>
<P:triad
r="4712L"
s="congress"
a="unique://freevolution:26:11213192987278475"
b="mail"
c="la...@example.com"
/>
</P:transaction>
<P:transaction>
<P:triad
r="4992L"
s="congress"
a="unique://freevolution:23:43207645167115827"
b="address"
c="Zoot's castle"
/>
</P:transaction>
<P:transaction>
<P:triad
r="5384L"
s="congress"
a="unique://freevolution:25:37062969616920138"
b="deleted"
c="true"
/>
</P:transaction>
</P:archive>
[And, as an example of bleeding edge, from testing this, I found a bug
from a last minute change I made to the unique ID system which had been
working well for a long time; it turns out that because of an error in
how I open a support file when it does not exists, the file
"a_pointrel_uniqueIDCounter.txt" now needs to exist in the local
directory with a number (e.g. '10') in it to get sequence numbers in
addition to a random ID component in unique IDs. The system still works
without this file though -- the unique IDs just aren't as collision
resistant. Something for an incremental release ASAP though.]
The one plug I'd put for the Pointrel System is that the simple (yet, I
hope, elegant) data storage approach is designed to be expandable with
changing needs over time, and usually data storage needs do change, and
in unexpected ways. Whether the Pointrel System can really deliver on
that promise remains to be seen. Again, MySql etc. have a proven track
record handling the sorts of issues you have outlined so far, and are
probably your best choice.
Anyway, if all this sounds like gibberish, feel free to ignore it! :-)
Best of luck with your project.
--Paul Fernhout
-----= Posted via Newsfeeds.Com, Uncensored Usenet News =-----
http://www.newsfeeds.com - The #1 Newsgroup Service in the World!
-----== Over 100,000 Newsgroups - 19 Different Servers! =-----
Paul D. Fernhout
http://sourceforge.net/projects/pointrel/
I also included the example code file previously posted based on your
requirements (and I referenced your email in it through a google news
link). Thanks for the motivation.
--Paul Fernhout
Paul D. Fernhout wrote (in reply to a query by Lars Behrens) :
> [And, as an example of bleeding edge, from testing this, I found a bug
> from a last minute change I made to the unique ID system which had been
> working well for a long time; it turns out that because of an error in
> how I open a support file when it does not exists, the file
> "a_pointrel_uniqueIDCounter.txt" now needs to exist in the local
> directory with a number (e.g. '10') in it to get sequence numbers in
> addition to a random ID component in unique IDs. The system still works
> without this file though -- the unique IDs just aren't as collision
> resistant. Something for an incremental release ASAP though.]
-----= Posted via Newsfeeds.Com, Uncensored Usenet News =-----
Lars Behrens
> Python 2.3 has an excellent CSV module.
Thx, I'll talk to my admin if upgrading would be possible.
Cheerz Lars
Lars Behrens
Thx for your help and hints and examples. But as you assume, my
application *is* mission critical. And as I read Pointres is only tested
under GNU/Linux but the server I have to use runs under Solaris, so I
have to find another solution.
Anyways, what I've read so far about Pointres sounds quite interesting
to me (especially because I'm a real database hater). So I will
certainly take a look at Pointres sometime...
Cheerz Lars
Lars Behrens
> I also included the example code file previously posted based on your
> requirements (and I referenced your email in it through a google news
> link). Thanks for the motivation.
Oh, thx for making me famous :-)
Cheerz Lars