Appengine timeout..?

[astrid....@googlemail.com]

In general - how much data can I send in an request to the appengine
and be sure it won't time out? Is the amount different between a GET
and POST request?

Another thing is, that I would like to create an iframe and send some
data in the src-URL string:

<iframe src="http://myappspot.appspot.com?somdata=abcdefg" ... >

I fear that in special cases there might be many iframes of this type
on a users site and the src-String could be really really long. Does
this mean it would time out? Does it make a difference if there are
many iframes on a single page, since a users connection will try to
load all at once, connection speed is split by the number iframes.
isn't it? Does this increase the danger of running into timeouts?

[Nick Johnson (Google)]

Hi Astrid,

On Wed, Jun 17, 2009 at 6:20 AM, astrid....@googlemail.com <astrid....@googlemail.com> wrote:

In general - how much data can I send in an request to the appengine
and be sure it won't time out? Is the amount different between a GET
and POST request?

Requests are limited to 10MB, and the counter on your request doesn't start until the entire request has been received.
 

Another thing is, that I would like to create an iframe and send some
data in the src-URL string:

<iframe src="http://myappspot.appspot.com?somdata=abcdefg" ... >

I fear that in special cases there might be many iframes of this type
on a users site and the src-String could be really really long. Does
this mean it would time out?

The most data you can reliably transfer in a URL is on the order of about 1k (different browsers have different URL-length limits), which is so small it will take an insignificant amount of time to transmit.

 

Does it make a difference if there are
many iframes on a single page, since a users connection will try to
load all at once, connection speed is split by the number iframes.
isn't it? Does this increase the danger of running into timeouts?

Browsers will request at most two resources from the same site simultaneously, regardless of how many there are to fetch in total.

-Nick Johnson

[codingGirl]

Ok, I know browsers have URL length limits. But I was talking about
sending data as part of the src="http://myappspot.appspot.com?
somdata=abcdefg" of an iframe. So I was wondering are there limits

1. how much data I can stuff in that src-URL? I guess there is no
other way to do it if one uses an iframe.
2. are there any time limits while the browser fetches the appengine-
webpage for the iframe? Will appengine just time out if the src-URL-
data string is too long so it takes to long to finish the request? How
long can it take?

On Jun 17, 12:41 pm, "Nick Johnson (Google)" <nick.john...@google.com>
wrote:
> Hi Astrid,
>
> On Wed, Jun 17, 2009 at 6:20 AM, astrid.thuec...@googlemail.com <

[Nick Johnson (Google)]

On Wed, Jun 17, 2009 at 5:58 PM, codingGirl <astrid....@googlemail.com> wrote:

Ok, I know browsers have URL length limits. But I was talking about
sending data as part of the src="http://myappspot.appspot.com?
somdata=abcdefg" of an iframe. So I was wondering are there limits

1. how much data I can stuff in that src-URL? I guess there is no
other way to do it if one uses an iframe.

That depends on the browser, rather than App Engine. I'm not sure what the hard limits are on what common browsers will accept.
 

2. are there any time limits while the browser fetches the appengine-
webpage for the iframe? Will appengine just time out if the src-URL-
data string is too long so it takes to long to finish the request? How
long can it take?

The 30 second request limit only starts after App Engine receives the request. At any rate, with a maximum URL length on the order of a few kilobytes, even the slowest user is not likely to take more than a second or two to transmit the URL!

-Nick Johnson

--
Nick Johnson, App Engine Developer Programs Engineer
Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number: 368047

[codingGirl]

I thought the URL lenght limit applies only for URLs typed int he
browsers address bar, doesn't it?

Or does it also apply to the src-URL in an iframe?

On Jun 17, 7:01 pm, "Nick Johnson (Google)" <nick.john...@google.com>
wrote:

> On Wed, Jun 17, 2009 at 5:58 PM, codingGirl

> <astrid.thuec...@googlemail.com>wrote:

[Iap]

2009/6/18 codingGirl <astrid....@googlemail.com>

I thought the URL lenght limit applies only for URLs typed int he
browsers address bar, doesn't it?

Or does it also apply to the src-URL in an iframe?

 

I think that it is depends on the http server for the maximum length

of the request line it can endure. It has to tell from a nomal request

to a over-flow attack. Ifram or not is not under the concern, since the

server does not care about it is iframe or not.

 

 

>
> The 30 second request limit only starts after App Engine receives the
> request. At any rate, with a maximum URL length on the order of a few
> kilobytes, even the slowest user is not likely to take more than a second or
> two to transmit the URL!
>

 

I think the "Timeout problem" of the data store is the most imfamouse problem of GAE.

It was reported in many usage scenarios. We have encoutered this problem

by simply iterate all the members in a small amount of Model.

Some others has the same experience.

By posted mails, It seems to happen in some unbelivable simple case.

I have no idea if the Timeout problem has been improved or not.

Definitely, it is the problem which prohibits my boss from investing

much more man-hour to get involved in the GAE.

It seems that the official answer to this problem is: we do things in wrong way.

(Well, what is the right way?)

Maybe it has been resolved, or never to be taken as a problem.

Who knows the most updated information about that?

 

[Nick Johnson (Google)]

On Thu, Jun 18, 2009 at 10:31 AM, Iap <iap...@gmail.com> wrote:

2009/6/18 codingGirl <astrid....@googlemail.com>

I thought the URL lenght limit applies only for URLs typed int he
browsers address bar, doesn't it?

Or does it also apply to the src-URL in an iframe?

 

I think that it is depends on the http server for the maximum length

of the request line it can endure. It has to tell from a nomal request

to a over-flow attack. Ifram or not is not under the concern, since the

server does not care about it is iframe or not.

Both servers and clients have limits on the length of URL they will accept. For example, Internet Explorer is limited to 2048 characters: http://support.microsoft.com/kb/208427

 

 

>
> The 30 second request limit only starts after App Engine receives the
> request. At any rate, with a maximum URL length on the order of a few
> kilobytes, even the slowest user is not likely to take more than a second or
> two to transmit the URL!
>

 

I think the "Timeout problem" of the data store is the most imfamouse problem of GAE.

It was reported in many usage scenarios. We have encoutered this problem

by simply iterate all the members in a small amount of Model.

Some others has the same experience.

By posted mails, It seems to happen in some unbelivable simple case.

I have no idea if the Timeout problem has been improved or not.

Definitely, it is the problem which prohibits my boss from investing

much more man-hour to get involved in the GAE.

It seems that the official answer to this problem is: we do things in wrong way.

(Well, what is the right way?)

Maybe it has been resolved, or never to be taken as a problem.

Who knows the most updated information about that?

Timeouts can be caused by a number of factors. About 0.1%-0.2% of datastore operations will timeout in normal operation - just due to how Bigtable works, with occasional tablet moves, etcetera. The rate of timeouts can be affected by what you do with the datastore - for example, a single query returning a large number of entities, or for particularly large entities, has a greater chance of timing out than a smaller, but otherwise identical query.

For datastore puts and deletes, we currently automatically retry when timeouts occur, and for datastore transactions, your user code is rerun if a timeout occurs. Gets and queries are currently not retried.

The solution in general is twofold: Retry when you can, and return an error to the user (or have them re-request the page) in the rare instances when you can't.

-Nick Johnson

 

[Iap]

2009/6/18 Nick Johnson (Google) nick.j...@google.com

For datastore puts and deletes, we currently automatically retry when timeouts occur, and for datastore transactions, your user code is rerun if a timeout occurs. Gets and queries are currently not retried.

 

It's exciting that the GAE inherites the "I'm feeling lucky" tradition of Google. >_*

 

Why not retry the Get and Queires ?

If the retry will eventually succeed,

How about provide a decrator for convenience to do the things right.

 

Iap

 

[Nick Johnson (Google)]

Hi Iap,

On Thu, Jun 18, 2009 at 11:01 AM, Iap <iap...@gmail.com> wrote:

2009/6/18 Nick Johnson (Google) nick.j...@google.com

For datastore puts and deletes, we currently automatically retry when timeouts occur, and for datastore transactions, your user code is rerun if a timeout occurs. Gets and queries are currently not retried.

 

It's exciting that the GAE inherites the "I'm feeling lucky" tradition of Google. >_*

 

Why not retry the Get and Queires ?

We are planning to add that in a future release.

If the retry will eventually succeed,

How about provide a decrator for convenience to do the things right.

Retries won't _always_ succeed, but they will avoid transient timeouts, reducing the number of user-visible timeout errors.

-Nick Johnson

 

Iap

 

[Iap]

2009/6/18 Nick Johnson (Google) <nick.j...@google.com>

Hi, Nick

>> Why not retry the Get and Queires ?
>
> We are planning to add that in a future release.

That would be great!
I have good reason to persuade my boss to re-open her mind.

Iap

[Dennis]

I asked about this (datastore timeout exceptions) in yesterday's app
engine chat and just wanted to follow up.

I'm trying to figure out an architecture to deal with the timeout
exceptions.
The "solution" of pushing the exception back to the end user is
actually ok for my app.
This works for read timeouts.

For write timeouts, my code can be in the middle of storing a data
structure when the timeout happens.
Thus, I could have a corrupt data structure instance in my datastore.
So, for fault-tolerant production code, any (later) access to any
datastore data structure should check
the integrity data structure before using it.
This is what Ken Ashcraft's google i/o 2008 talk recommended (without
referring specifically to timeout issues):
http://sites.google.com/site/io/best-practices---building-a-production-quality-application-on-google-app-engine

A bit troublesome but at least we know what to do:
every datastore read needs to do data structure integrity checking.

I just want to get feedback on whether this line of thinking and the
resulting coding style is what the appengine team recommends.
Also, it would be very helpful to know what google engineers do to
avoid this issue (do they use the same style or do they have internal
tools or libraries that hide this issue from them).

Dennis

search term: DatastoreTimeoutException



On Jun 18, 6:01 pm, Iap <iap...@gmail.com> wrote:
> 2009/6/18 Nick Johnson (Google) nick.john...@google.com

[Nick Johnson (Google)]

On Thu, Jun 18, 2009 at 11:45 AM, Dennis <denni...@gmail.com> wrote:

I asked about this (datastore timeout exceptions) in yesterday's app
engine chat and just wanted to follow up.

I'm trying to figure out an architecture to deal with the timeout
exceptions.
The "solution" of pushing the exception back to the end user is
actually ok for my app.
This works for read timeouts.

For write timeouts, my code can be in the middle of storing a data
structure when the timeout happens.
Thus, I could have a corrupt data structure instance in my datastore.

I presume by 'data structure', you mean multiple entities? Because individual entities (and entity groups inside transactions) will always be written atomically - all or nothing - and never 'corrupted' or partially written.

So, for fault-tolerant production code, any (later) access to any
datastore data structure should check
the integrity data structure before using it.
This is what Ken Ashcraft's google i/o 2008 talk recommended (without
referring specifically to timeout issues):
  http://sites.google.com/site/io/best-practices---building-a-production-quality-application-on-google-app-engine

A bit troublesome but at least we know what to do:
every datastore read needs to do data structure integrity checking. 

I just want to get feedback on whether this line of thinking and the
resulting coding style is what the appengine team recommends.
Also, it would be very helpful to know what google engineers do to
avoid this issue (do they use the same style or do they have internal
tools or libraries that hide this issue from them).

More or less. If your write order is predictable, some of your queries will probably not care about incomplete data, or that they may see data that's only partially written. In general, though, the 'roll forward' approach works quite well.

-Nick Johnson
 

Dennis

search term: DatastoreTimeoutException



On Jun 18, 6:01 pm, Iap <iap...@gmail.com> wrote:
> 2009/6/18 Nick Johnson (Google) nick.john...@google.com

>
>
>
> > For datastore puts and deletes, we currently automatically retry when
> > timeouts occur, and for datastore transactions, your user code is rerun if a
> > timeout occurs. Gets and queries are currently not retried.
>
> It's exciting that the GAE inherites the "I'm feeling lucky" tradition of
> Google. >_*
>
> Why not retry the Get and Queires ?
> If the retry will eventually succeed,
> How about provide a decrator for convenience to do the things right.
>
> Iap

[Dennis]

> I presume by 'data structure', you mean multiple entities?

Yes, multiple entity structures need to be integrity checked each time
they are used.
Thanks for the confirmation of how we should code.

> only partially written. In general, though, the 'roll forward' approach
> works quite well.

What is the 'roll forward' approach?



On Jun 18, 6:49 pm, "Nick Johnson (Google)" <nick.john...@google.com>
wrote:

> On Thu, Jun 18, 2009 at 11:45 AM, Dennis <dennisf...@gmail.com> wrote:
>
> > I asked about this (datastore timeout exceptions) in yesterday's app
> > engine chat and just wanted to follow up.
>
> > I'm trying to figure out an architecture to deal with the timeout
> > exceptions.
> > The "solution" of pushing the exception back to the end user is
> > actually ok for my app.
> > This works for read timeouts.
>
> > For write timeouts, my code can be in the middle of storing a data
> > structure when the timeout happens.
> > Thus, I could have a corrupt data structure instance in my datastore.
>
> I presume by 'data structure', you mean multiple entities? Because
> individual entities (and entity groups inside transactions) will always be
> written atomically - all or nothing - and never 'corrupted' or partially
> written.
>
>
>
>
>
> > So, for fault-tolerant production code, any (later) access to any
> > datastore data structure should check
> > the integrity data structure before using it.
> > This is what Ken Ashcraft's google i/o 2008 talk recommended (without
> > referring specifically to timeout issues):
>

> >http://sites.google.com/site/io/best-practices---building-a-productio...

> 368047- Hide quoted text -
>
> - Show quoted text -- Hide quoted text -
>
> - Show quoted text -

[Dennis]

PS: what does google do internally (wrt coding to avoid datastore
timeouts)?

[Nick Johnson (Google)]

On Thu, Jun 18, 2009 at 12:00 PM, Dennis <denni...@gmail.com> wrote:

PS: what does google do internally (wrt coding to avoid datastore
timeouts)?

The same as we recommend to everyone else. :)
 

On Jun 18, 6:58 pm, Dennis <dennisf...@gmail.com> wrote:
> > I presume by 'data structure', you mean multiple entities?
>
> Yes, multiple entity structures need to be integrity checked each time
> they are used.
> Thanks for the confirmation of how we should code.
>
> > only partially written. In general, though, the 'roll forward' approach
> > works quite well.
>
> What is the 'roll forward' approach?

The approach documented in the talk you referenced: First, record your intent. Then, execute it, and if your transaction fails, a later read can detect this and pick up where it left off ('rolling forward' the transaction).

-Nick Johnson