Datastore record datetime property stuck

[Daniel Upton]

I have a record which is frozen in the datastore and always shows the
current time. This record is blocking a major time sorted query in my
application. I've had this same problem in June (http://
groups.google.com/group/google-appengine/browse_thread/thread/
b0021ac81b8c24a5/dcf6c71e3e0b3083) and I had to get a tech to fix it.
Please help me correct this record. The app is looxii-beta and the
key is agtsb294aWktYmV0YXIVCxINU2VhcmNoX3VwZGF0ZRi62AUM.

As an aside, it seems ridiculous to me that the database would not
perform as it should. This is a major issue, and actually blocks my
application from properly performing. I accept that AppEngine is in
beta, but at least give your paying customers a contact email address
so that when we have legitimate issues we don't have to post to a
public forum and wait for a google rep to stumble upon our issue.

[Ikai L (Google)]

Hi Daniel,

What is the Entity Kind? 

The best thing I would recommend in the future is the billing support form:

http://code.google.com/support/bin/request.py?contact_type=AppEngineBillingSupport

This is a monitored queue for issues like these, and I expect that we will be emphasizing it more. We're not currently equipped to handle dedicated support, though it is on our roadmap to offer paid support with an SLA for response time.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
Ikai Lan 
Developer Programs Engineer, Google App Engine

Blog: http://googleappengine.blogspot.com 

Twitter: http://twitter.com/app_engine

Reddit: http://www.reddit.com/r/appengine

[Daniel Upton]

Thanks for the fast response. The kind is Search_update.

On Jul 9, 3:16 pm, "Ikai L (Google)" <ika...@google.com> wrote:
> Hi Daniel,
>
> What is the Entity Kind?
>
> The best thing I would recommend in the future is the billing support form:
>

> http://code.google.com/support/bin/request.py?contact_type=AppEngineB...

>
> This is a monitored queue for issues like these, and I expect that we will
> be emphasizing it more. We're not currently equipped to handle dedicated
> support, though it is on our roadmap to offer paid support with an SLA for
> response time.
>
>
>
>
>
> On Fri, Jul 9, 2010 at 12:10 PM, Daniel <danielshaneup...@gmail.com> wrote:
> > I have a record which is frozen in the datastore and always shows the
> > current time.  This record is blocking a major time sorted query in my
> > application.  I've had this same problem in June (http://
> > groups.google.com/group/google-appengine/browse_thread/thread/
> > b0021ac81b8c24a5/dcf6c71e3e0b3083) and I had to get a tech to fix it.
> > Please help me correct this record.  The app is looxii-beta and the
> > key is agtsb294aWktYmV0YXIVCxINU2VhcmNoX3VwZGF0ZRi62AUM.
>
> > As an aside, it seems ridiculous to me that the database would not
> > perform as it should.  This is a major issue, and actually blocks my
> > application from properly performing.  I accept that AppEngine is in
> > beta, but at least give your paying customers a contact email address
> > so that when we have legitimate issues we don't have to post to a
> > public forum and wait for a google rep to stumble upon our issue.
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Google App Engine" group.
> > To post to this group, send email to google-a...@googlegroups.com.
> > To unsubscribe from this group, send email to

> > google-appengi...@googlegroups.com<google-appengine%2Bunsubscrib e...@googlegroups.com>
> > .

[Ikai L (Google)]

Using this GQL query in the admin console:

SELECT * FROM Search_update where __key__=KEY('agtsb294aWktYmV0YXIVCxINU2VhcmNoX3VwZGF0ZRi62AUM')

I was able to set the year to 2011, see the changes reflected, then set them back to 2010 (I didn't want to cause issues with your application). How are you reproducing your issues? Can you see if you can update the timestamp in the datastore viewer?

To unsubscribe from this group, send email to google-appengi...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Daniel Upton]

I'm still waiting on the record repair... which gets back to my point
about having someone to email or contact about the status of the
problem.

In reference to your paid support which is in the pipeline: the only
time that I need help from the appengine team is when you guys break
my database during one of your "planned outages." I would hardly pay
to have you fix problems that you are creating.

[Daniel Upton]

When I run the query Search_update.all().order('last_check').fetch(10)
programtically I consistently get this response:

agtsb294aWktYmV0YXIVCxINU2VhcmNoX3VwZGF0ZRi62AUM 2010-07-09
17:57:37.894714
agtsb294aWktYmV0YXIVCxINU2VhcmNoX3VwZGF0ZRjsm3kM 2010-07-08
00:27:52.385926
agtsb294aWktYmV0YXIVCxINU2VhcmNoX3VwZGF0ZRipkjsM 2010-07-08
00:27:53.689982
agtsb294aWktYmV0YXIWCxINU2VhcmNoX3VwZGF0ZRiqu7ABDA 2010-07-08
00:27:54.616698
agtsb294aWktYmV0YXIVCxINU2VhcmNoX3VwZGF0ZRiTrVwM 2010-07-08
00:27:55.453521
agtsb294aWktYmV0YXIVCxINU2VhcmNoX3VwZGF0ZRjksQUM 2010-07-08
00:27:56.437609
agtsb294aWktYmV0YXIWCxINU2VhcmNoX3VwZGF0ZRjp_KwBDA 2010-07-08
00:28:02.699297
agtsb294aWktYmV0YXIVCxINU2VhcmNoX3VwZGF0ZRiRqAUM 2010-07-08
00:28:04.598150
agtsb294aWktYmV0YXIWCxINU2VhcmNoX3VwZGF0ZRjyzK8BDA 2010-07-08
00:28:05.595406
agtsb294aWktYmV0YXIVCxINU2VhcmNoX3VwZGF0ZRi-2AUM 2010-07-08
00:28:07.950788

Which places the record in question first in the query although it has
a date later than the listed records. You can also see that having
just run the response that the 2011 date reverted. It appears to
change when you save it, but it doesn't actually change.

> > > > google-appengi...@googlegroups.com<google-appengine%2Bunsubscrib e...@googlegroups.com><google-appengine%2Bunsubscrib

[Daniel Upton]

More specifically it appears that the timestamp is stuck at 2010-07-09
17:57:37.894714 and will not commit any change to the record, even if
it seems to work in the admin console. And all queries place this
record first when ordering by this property. This record appears to
have been stuck like this for 2 days, ever since the planned downtime.

[Ikai Lan]

I manually reverted it to 2010 because I didn't want to cause problems with your app by fudging with the data. It didn't revert automatically. I just ran this GQL query:

SELECT * FROM Search_update ORDER BY last_check DESC LIMIT 10

And it displayed the data correctly.

Is there any chance you're setting a different read consistency? 

http://code.google.com/appengine/docs/python/datastore/queriesandindexes.html#Setting_the_Read_Policy_and_Datastore_Call_Deadline

I understand your frustration with regards to paid support - but our pricing structure doesn't bundle in a guarantee for a support response time. We make every effort to address any issues in production either behind the scenes, through proactive monitoring or to reactive reports via the different channels we keep track of, but we simply can't scale up a direct support option for developers who want to be able to pick up the red phone and get someone on the line immediately when there are service issues. In terms of addressing service degradation without additional cost to developers, based on conversations we have had with developers, I believe that we are as good if not much better than the industry standard for platform, managed cloud hosting. There are companies who will always have a resource for you available, but there's a reason any kind of baseline production spend will be in the hundreds of dollars - minimum. We're doing our best to minimize costs, support all developers on our platform, all while not charging developers who don't require a response time SLA. 

To unsubscribe from this group, send email to google-appengi...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Daniel Upton]

Okay, regardless of your ability to offer me any sort of instant
support, I need this problem fixed.

I am not setting any read consistency. The query that you specified
(SELECT * FROM Search_update ORDER BY last_check DESC LIMIT 10) is set
to DESC, so of course it appears correct that the 2011 record is
first, but when you change it to ASC the 2011 record still appears
first. The broken entity
'agtsb294aWktYmV0YXIVCxINU2VhcmNoX3VwZGF0ZRi62AUM' now has changed to
2011 from what you did yesterday but is is showing up as the OLDEST
record. if you look at the list you will see that after the broken
record, the following 9 records are in ascending order and set to
2010. My original query above specified ascending order.


agtsb294aWktYmV0YXIVCxINU2VhcmNoX3VwZGF0ZRi62AUM 2011-07-09
17:57:37.894714
agtsb294aWktYmV0YXIVCxINU2VhcmNoX3VwZGF0ZRiUvi8M 2010-07-10
12:14:52.306180
agtsb294aWktYmV0YXIVCxINU2VhcmNoX3VwZGF0ZRiawS4M 2010-07-10
12:15:18.649276
agtsb294aWktYmV0YXIWCxINU2VhcmNoX3VwZGF0ZRjSkLEBDA 2010-07-10
12:15:40.274290
agtsb294aWktYmV0YXIVCxINU2VhcmNoX3VwZGF0ZRiw_gUM 2010-07-10
12:15:43.913061
agtsb294aWktYmV0YXIVCxINU2VhcmNoX3VwZGF0ZRjw1gUM 2010-07-10
12:15:52.877386
agtsb294aWktYmV0YXIVCxINU2VhcmNoX3VwZGF0ZRif7FkM 2010-07-10
12:15:54.880721
agtsb294aWktYmV0YXIVCxINU2VhcmNoX3VwZGF0ZRipqgUM 2010-07-10
12:15:56.204075
agtsb294aWktYmV0YXIVCxINU2VhcmNoX3VwZGF0ZRiv4AUM 2010-07-10
12:15:59.092526
agtsb294aWktYmV0YXIVCxINU2VhcmNoX3VwZGF0ZRijz20M 2010-07-10
12:16:00.490951

In general I like google app engine, but I am getting frustrated with
it's ability to consistently perform. You have to realize that I have
an app that is coded correctly and was running fine until during the
planned downtime three days ago when this query stopped correctly
returning data, this is a structural flaw and completely disrupts my
ability to provide a service on your system. Also, this is the second
time this has happened and I had to go through quite a bit to fix it
the first time. It seems that I have to repeat my last efforts, and
with no hope that it will not happen again and require me to generate
2 days more worth of posts to get you to fix the problem.

On Jul 9, 5:48 pm, Ikai Lan <i...@google.com> wrote:
> I manually reverted it to 2010 because I didn't want to cause problems with
> your app by fudging with the data. It didn't revert automatically. I just
> ran this GQL query:
>
> SELECT * FROM Search_update ORDER BY last_check DESC LIMIT 10
>
> And it displayed the data correctly.
>
> Is there any chance you're setting a different read consistency?
>

> http://code.google.com/appengine/docs/python/datastore/queriesandinde...

[Daniel Upton]

I would like a reply...

I was able to delete and re-add this record and that fixed that query,
but it turns out I have another record that is not updating and can't
be deleted.

This record key is
agtsb294aWktYmV0YXIaCxIHS2V5d29yZCINa2V5X3N5bnRoZXNpbww the kind is
Keyword.

It appears that the record broke during the same downtime last week.
It always show the current time and cannot be changed in the admin
console or by code. I cannot delete this entity.

It seems my data is going corrupt all over the place...

> > Blog:http://googleappengine.blogspot.com...
>
> read more »

[Ikai L (Google)]

Daniel, I'm going to try running a tool on your data. I'll email this group again when it's finished. I'm curious as to whether or not this does anything for you. My testing via the admin console (I don't have access to your code) seems to indicate that everything is and was working correctly, but I'd like to establish a greater confidence level either way.

To unsubscribe from this group, send email to google-appengi...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Daniel Upton]

Thanks for looking. If you run the query in the admin console:
"SELECT * FROM Keyword WHERE is_active = True ORDER BY last_check ASC"
you will see the entity with key
agtsb294aWktYmV0YXIaCxIHS2V5d29yZCINa2V5X3N5bnRoZXNpbww appear as the
first record although the date is set to 2011. This indicates that
the entity is not returning as it should. This has nothing to do with
my code.

I don't understand how your admin console is returning something
different from mine.

> > > > > > > > > For more options, visit this group at...
>
> read more »

[Ikai L (Google)]

Yeah, I'm checking for index corruption. It could also mean there's a bug in the way we're sorting.

To unsubscribe from this group, send email to google-appengi...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Ikai L (Google)]

I finished running the index tool. This is very strange. I'm seeing exactly what you're seeing: the 2011 entity is the first one returned. Let me chase this down for you.

[Ikai L (Google)]

Hi Daniel,

The sort order should be working correctly again. The issue was a broken index value; we have a short term fix in place, but I wouldn't rule out this appearing again. Is it possible for you to create a new entity with the same values with a different key? That's one way to work around this issue in the future.

[Alfred Fuller]

Just to clarify this is an incredibly rare occurrence (lucky you :-)) and we will fix it permanently (it just takes a little man power). Changing the key name would fix it as well, but we don't expect you to have to do this.

[Daniel Upton]

Thanks guys for fixing this. This is actually the third time that
this has occurred for me. The first was a couple months ago after the
big unexpected downtime, and then these two additional entities.
Unfortunately, this latest record relies upon the key name, so I
couldn't create a record of duplicate values. With my inability to
delete the record, I wouldn't have been able to free up the key name
for use in the new record. Also, this record has several dependencies,
which I could trace down and change, but I would prefer not to if at
all possible.

On Jul 14, 10:57 pm, Alfred Fuller <arfuller+appeng...@google.com>
wrote:

> Just to clarify this is an incredibly rare occurrence (lucky you :-)) and we
> will fix it permanently (it just takes a little man power). Changing the key
> name would fix it as well, but we don't expect you to have to do this.
>
> On Wed, Jul 14, 2010 at 7:44 PM, Ikai L (Google) <ika...@google.com> wrote:
>
>
>
> > Hi Daniel,
>
> > The sort order should be working correctly again. The issue was a broken
> > index value; we have a short term fix in place, but I wouldn't rule out this
> > appearing again. Is it possible for you to create a new entity with the same
> > values with a different key? That's one way to work around this issue in the
> > future.
>
> > On Wed, Jul 14, 2010 at 2:23 PM, Ikai L (Google) <ika...@google.com>wrote:
>
> >> I finished running the index tool. This is very strange. I'm seeing
> >> exactly what you're seeing: the 2011 entity is the first one returned. Let
> >> me chase this down for you.
>
> >> On Wed, Jul 14, 2010 at 12:24 PM, Ikai L (Google) <ika...@google.com>wrote:
>
> >>> Yeah, I'm checking for index corruption. It could also mean there's a bug
> >>> in the way we're sorting.
>

> >>>> > > > > > > back to...
>
> read more »