I appear to be corrupting my datastore

[bowman...@gmail.com]

I'm trying to make sure a score field I set for articles on my site in
unique, however, I'm running into an issue where my method is
appearing to corrupt my datastore. After I input stories, I can't view
pages, getting a return size too large error, and when I stop and
start the SDK, it won't restart.

Here's what I'm doing.

I set up a new Score Property.
------------------------------------------------------------------------------------
class ScoreProperty(db.FloatProperty):

def checkScoreValue(self, value):
query = db.GqlQuery('SELECT * FROM Story WHERE score = :1', value)
results = query.fetch(1)
if len(results) > 0:
raise db.BadValueError(
'Property %s must be unique' % self.name)
return value

def __init__(self, verbose_name=None, name=None):
super(ScoreProperty, self).__init__(
verbose_name, name, required=False,
validator=self.checkScoreValue)
------------------------------------------------------------------------------------

Then when I go to add a story, I use this try statement
------------------------------------------------------------------------------------
story_added = False
while story_added == False:
try:
story.put()
story_added = True
except db.BadValueError:
story.score = story.score + 0.001
------------------------------------------------------------------------------------



What should happen is that when a put is attempted, a check is done to
see if a story with that score exists. If it does exist, add 0.001 and
try to put again. After adding several stories in batch mode, I can
using the data view that appears to be working, but strangely. I'll
see a score of ###.0 and then the next would be something like ###.
043, so I'm not sure what happened to .001-.0042. Also, after running
the process the datastore appears to be corrupted as well.

I'm considering swapping to using a DateTimeProperty and keying off of
the miliseconds to see if that is handled better, but I'm confused as
to why this current method is creating problems.

One thing that might be an issue is the development is happening on an
eeepc, which is a single core processor and it's SSD isn't the fast
read/write storage device.

[bowman...@gmail.com]

I didn't have much luck with switching to a datetimeproperty using
milliseconds eithers. I also tried handling the value changing in the
check method directly, since it looked like it was possible there.
Same results, the datastore gets corrupted. The error I get when I try
to start the datastore after stopping it is.

<class 'struct.error'>: unpack requires a string argument of length 8

def checkScoreValue(self, value):
valid = False
while valid == False:

query = db.GqlQuery('SELECT * FROM Story WHERE score = :1', value)
results = query.fetch(1)
if len(results) > 0:

value = value + 0.001
else:
valid = True
return value



I've filed an issue, #922 - http://code.google.com/p/googleappengine/issues/detail?id=922


On Dec 13, 10:01 am, "bowman.jos...@gmail.com"

[Marzia Niccolai]

Hi,

Thanks for filing the issue.

This is related to storing a floatproperty in the SDK datastore, on Macs the datastore file may get occasionally corrupted. Currently you will need to clear the datastore on your local machine to fix this issue.

-Marzia

[bowman...@gmail.com]

This is actually happening on Linux, not sure if that matters or not.

Clearing the datastore is my only solution to getting it back up on
the SDK currently, but once I run the routine again, it's corrupted
again. I did try converting the entire property to a
db.DateTimeProperty and adding milliseconds, and had the same
corruption issues.

On Dec 15, 2:07 pm, Marzia Niccolai <ma...@google.com> wrote:
> Hi,
>
> Thanks for filing the issue.
>
> This is related to storing a floatproperty in the SDK datastore, on Macs the
> datastore file may get occasionally corrupted. Currently you will need to
> clear the datastore on your local machine to fix this issue.
>
> -Marzia
>

> On Sat, Dec 13, 2008 at 7:52 AM, bowman.jos...@gmail.com <

[Marzia Niccolai]

Hi,

I just assumed it was Mac because it's an issue we've seen with Macs that have 2.5.0 installed, and I'm assuming this is your Python installation?  Generally, float values in the datastore just don't work with Python 2.5.0.  Support for buffer objects in struct.unpack was not added until version 54695 (Apr 2007), this is the underlying cause of the error message.

-Marzia

[bowman...@gmail.com]

Ok great. I'll have to check on the python version, it's whatever
xandros makes available (I haven't gotten rid of the default OS on my
eeepc because I'm lazy). If worse comes to worse I'll try compiling
the latest python by hand and see if that works. I should have this
done by the weekend. I'll post a comment to the ticket if this
resolves the issue. Thanks for the quick knowledgeable response as
always.

On Dec 15, 3:06 pm, Marzia Niccolai <ma...@google.com> wrote:
> Hi,
>

> I just assumed it was Mac because it's an issue we've seen with Macs that
> have 2.5.0 installed, and I'm assuming this is your Python installation?
> Generally, float values in the datastore just don't work with Python 2.5.0.
> Support for buffer objects in struct.unpack was not added until version
> 54695 (Apr 2007), this is the underlying cause of the error message.
>
> -Marzia
>

> On Mon, Dec 15, 2008 at 11:54 AM, bowman.jos...@gmail.com <

[bowman...@gmail.com]

compiling 2.5.2 from source appears to have fixed the database
corruption, though I'm still getting the http response too large error
when I try to bring up the stories now that they're scored as floating
points. If it turns out to be an appengine problem, I'll reply to this
thread, otherwise if it is just a problem with my code, I'll let this
topic die. Thanks again.

On Dec 15, 3:33 pm, "bowman.jos...@gmail.com"