Google AppEngine Indexing
85 views
Skip to first unread message
GeekTantra
Jul 20, 2010, 8:59:33 PM7/20/10
to Google App Engine
I am trying to create an application on appengine. The problem is when
I deploy it to production, even with empty datastore, it takes atleast
2 hours for the indexes to start serving.
The contents of the index.yaml are:
- kind: comments
properties:
- name: content_type_id
- name: is_public
- name: is_removed
- name: object_pk
- name: site_id
- name: created
- kind: comments
properties:
- name: content_type_id
- name: object_pk
- name: user_email
- name: user_name
- name: user_url
- name: created
- kind: content_type
properties:
- name: app_label
- name: name
- kind: pages
properties:
- name: post_status
- name: created
- kind: pages
properties:
- name: post_status
- name: post_title
- kind: posts
properties:
- name: post_status
- name: created
- kind: posts
properties:
- name: post_status
- name: created
direction: desc
No since I want to make this application an opensource product its
important that people can deploy it instantly and see the results but
without the indexes being served instantly I start getting 500
errors(NoIndexErrors) instantly when I deploy. Is there a work around.
Initially the datastore would be completely empty, so I presume the
Indexing shouldn't take so much time.
I deploy it to production, even with empty datastore, it takes atleast
2 hours for the indexes to start serving.
The contents of the index.yaml are:
- kind: comments
properties:
- name: content_type_id
- name: is_public
- name: is_removed
- name: object_pk
- name: site_id
- name: created
- kind: comments
properties:
- name: content_type_id
- name: object_pk
- name: user_email
- name: user_name
- name: user_url
- name: created
- kind: content_type
properties:
- name: app_label
- name: name
- kind: pages
properties:
- name: post_status
- name: created
- kind: pages
properties:
- name: post_status
- name: post_title
- kind: posts
properties:
- name: post_status
- name: created
- kind: posts
properties:
- name: post_status
- name: created
direction: desc
No since I want to make this application an opensource product its
important that people can deploy it instantly and see the results but
without the indexes being served instantly I start getting 500
errors(NoIndexErrors) instantly when I deploy. Is there a work around.
Initially the datastore would be completely empty, so I presume the
Indexing shouldn't take so much time.
Michael DeLaney
Jul 20, 2010, 10:05:14 PM7/20/10
to google-a...@googlegroups.com
I think that it is possible to manually configure your indexes as part of your app.yaml file... or perhaps it is your index.yaml...
--
Michael S. DeLaney
mdel...@delaneylawoffices.com
DeLaney Law Offices, Ltd.
ELDER LAW ATTORNEYS
14524 John Humphrey Drive
Orland Park, IL 60462
(708) 675-7144
(708) 675-7150 - Direct Dial
Fax: (866) 282-6509
NOTICE: The information contained in this electronic mail transmission (including any attachment) is intended by DeLaney Law Offices, Ltd. for the use of the named individual or entity to which it is directed and may contain information that it is privileged or otherwise confidential. It is not intended for transmission to, or receipt by, anyone other than the named addressee (or a person authorized to deliver it to the named addressee). It should not be copied or forwarded to any unauthorized persons. If you have received this electronic mail transmission in error, please delete it from your system without copying or forwarding it, and notify the sender of the error by reply e-mail or by calling DeLaney Law Offices, Ltd. at 708-675-7144 so that our address record may be corrected. Thank you.
IRS CIRCULAR 230 NOTICE. Any advice expressed above as to tax matters was neither written nor intended by the sender or DeLaney Law Offices, Ltd. to be used and cannot be used by any taxpayer for the purpose of avoiding tax penalties that may be imposed under U.S. tax law. If any person uses or refers to any such tax advice in promoting, marketing or recommending a partnership or other entity, investment plan or arrangement to any taxpayer, then (i) the advice was written to support the promotion or marketing (by a person other than DeLaney Law Offices, Ltd.) of that transaction or matter, and (ii) such taxpayer should seek advice based on the taxpayer's particular circumstances from an independent tax advisor.
--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
--
Michael S. DeLaney
mdel...@delaneylawoffices.com
DeLaney Law Offices, Ltd.
ELDER LAW ATTORNEYS
14524 John Humphrey Drive
Orland Park, IL 60462
(708) 675-7144
(708) 675-7150 - Direct Dial
Fax: (866) 282-6509
NOTICE: The information contained in this electronic mail transmission (including any attachment) is intended by DeLaney Law Offices, Ltd. for the use of the named individual or entity to which it is directed and may contain information that it is privileged or otherwise confidential. It is not intended for transmission to, or receipt by, anyone other than the named addressee (or a person authorized to deliver it to the named addressee). It should not be copied or forwarded to any unauthorized persons. If you have received this electronic mail transmission in error, please delete it from your system without copying or forwarding it, and notify the sender of the error by reply e-mail or by calling DeLaney Law Offices, Ltd. at 708-675-7144 so that our address record may be corrected. Thank you.
IRS CIRCULAR 230 NOTICE. Any advice expressed above as to tax matters was neither written nor intended by the sender or DeLaney Law Offices, Ltd. to be used and cannot be used by any taxpayer for the purpose of avoiding tax penalties that may be imposed under U.S. tax law. If any person uses or refers to any such tax advice in promoting, marketing or recommending a partnership or other entity, investment plan or arrangement to any taxpayer, then (i) the advice was written to support the promotion or marketing (by a person other than DeLaney Law Offices, Ltd.) of that transaction or matter, and (ii) such taxpayer should seek advice based on the taxpayer's particular circumstances from an independent tax advisor.
GeekTantra
Jul 21, 2010, 1:03:47 AM7/21/10
to Google App Engine
The above post contains the manually configured indexes from my
index.yaml. But as I said earlier its not configuring the indexes that
I am facing problem with but its the time it takes to start serving
the index even in an empty datastore thats causing the problem in
instantly deploying my application. Any help or work around in this
regard with be appreciable.
On Jul 21, 7:05 am, Michael DeLaney <mdela...@delaneylawoffices.com>
wrote:
> > google-appengi...@googlegroups.com<google-appengine%2Bunsubscrib e...@googlegroups.com>
> > .
index.yaml. But as I said earlier its not configuring the indexes that
I am facing problem with but its the time it takes to start serving
the index even in an empty datastore thats causing the problem in
instantly deploying my application. Any help or work around in this
regard with be appreciable.
On Jul 21, 7:05 am, Michael DeLaney <mdela...@delaneylawoffices.com>
wrote:
> > .
> > For more options, visit this group at
> >http://groups.google.com/group/google-appengine?hl=en.
>
> --
> Michael S. DeLaney
> mdela...@delaneylawoffices.com
> >http://groups.google.com/group/google-appengine?hl=en.
>
> --
> Michael S. DeLaney
Robert Kluin
Jul 21, 2010, 2:59:16 AM7/21/10
to google-a...@googlegroups.com
Sometimes the index servers get bogged down. When that happens, even for an empty datastore, you will have to wait for new indexes.
Slow initial index building is a pretty common complaint.
Robert
> --
> You received this message because you are subscribed to the Google Groups "Google App Engine" group.
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
GeekTantra
Jul 21, 2010, 3:09:08 AM7/21/10
to Google App Engine
Thanks a lot for your response.
I just wanted to know whether this will be a perennial or just a
temporary problem? And any way of monitoring the index servers
something similar to http://code.google.com/status/
I need to know this because I want to launch my Open Source product
and don't want to launch it at a time when this problem persists.
Is there by any chance I can do a work around for this problem till
the indexes are ready to serve?
I have some simple ancestor properties, some filtering and some order
operations which require indexing.
On Jul 21, 11:59 am, Robert Kluin <robert.kl...@gmail.com> wrote:
> Sometimes the index servers get bogged down. When that happens, even for an empty datastore, you will have to wait for new indexes.
>
> Slow initial index building is a pretty common complaint.
>
> Robert
>
I just wanted to know whether this will be a perennial or just a
temporary problem? And any way of monitoring the index servers
something similar to http://code.google.com/status/
I need to know this because I want to launch my Open Source product
and don't want to launch it at a time when this problem persists.
Is there by any chance I can do a work around for this problem till
the indexes are ready to serve?
I have some simple ancestor properties, some filtering and some order
operations which require indexing.
On Jul 21, 11:59 am, Robert Kluin <robert.kl...@gmail.com> wrote:
> Sometimes the index servers get bogged down. When that happens, even for an empty datastore, you will have to wait for new indexes.
>
> Slow initial index building is a pretty common complaint.
>
> Robert
>
Robert Kluin
Jul 21, 2010, 3:35:24 AM7/21/10
to google-a...@googlegroups.com
I do not know of any way to monitor index build times currently.
For me building new indexes is usually fast. Typically under 30 minutes. But I built one today that took 6 hours on a kind with 220 entities. :/
Robert
GeekTantra
Jul 21, 2010, 4:40:51 AM7/21/10
to Google App Engine
I am trying to segregate the queries that require indexes by using a
try except block like following
try:
posts_list =
posts.Post.objects.all().filter(post_status__in=post_status).order_by('-
created')
except:
the_posts_list = posts.Post.objects.all()
posts_list = [ p for p in the_posts_list if p.post_status in
post_status ]
I know its highly inefficient code in the except block but this should
make the application work in case the initial indexes are not being
served for the first deployment, but I still keep getting the
NeedIndexError. Any Idea why this is happening?
On Jul 21, 12:35 pm, Robert Kluin <robert.kl...@gmail.com> wrote:
> I do not know of any way to monitor index build times currently.
>
> For me building new indexes is usually fast. Typically under 30 minutes. But I built one today that took 6 hours on a kind with 220 entities. :/
>
> Robert
>
try except block like following
try:
posts_list =
posts.Post.objects.all().filter(post_status__in=post_status).order_by('-
created')
except:
the_posts_list = posts.Post.objects.all()
posts_list = [ p for p in the_posts_list if p.post_status in
post_status ]
I know its highly inefficient code in the except block but this should
make the application work in case the initial indexes are not being
served for the first deployment, but I still keep getting the
NeedIndexError. Any Idea why this is happening?
On Jul 21, 12:35 pm, Robert Kluin <robert.kl...@gmail.com> wrote:
> I do not know of any way to monitor index build times currently.
>
> For me building new indexes is usually fast. Typically under 30 minutes. But I built one today that took 6 hours on a kind with 220 entities. :/
>
> Robert
>
> On Jul 21, 2010, at 3:09, GeekTantra <geektan...@gmail.com> wrote:
>
>
>
> > Thanks a lot for your response.
>
> > I just wanted to know whether this will be a perennial or just a
> > temporary problem? And any way of monitoring the index servers
> > something similar tohttp://code.google.com/status/
>
>
>
> > Thanks a lot for your response.
>
> > I just wanted to know whether this will be a perennial or just a
> > temporary problem? And any way of monitoring the index servers
P. Petrov
Jul 21, 2010, 6:42:51 AM7/21/10
to Google App Engine
GeekTantra
Jul 21, 2010, 9:16:29 AM7/21/10
to Google App Engine
This still doesn't solve my problem. Its not a one time project I am
trying to build. Had it been a single one time project I am ok with
the delay. I am trying to create generic product which anyone should
be able to use. But delay due to index creation will give a lot of
problem to the end user. So I am trying to work around a solution to
this problem.
> http://ikaisays.com/2010/07/12/google-app-engine-tips-and-tricks-preb...
trying to build. Had it been a single one time project I am ok with
the delay. I am trying to create generic product which anyone should
be able to use. But delay due to index creation will give a lot of
problem to the end user. So I am trying to work around a solution to
this problem.
> http://ikaisays.com/2010/07/12/google-app-engine-tips-and-tricks-preb...
Robert Kluin
Jul 21, 2010, 10:38:38 AM7/21/10
to google-a...@googlegroups.com
Yeah, that does not work because the query is not executed in the try block, hence it does not raise an exception. Try adding a .fetch(500) after order() and see if that stops the exception.
Also check out Andrew Fuller's 2010 next gen query IO talk. Once that gets implemented maybe you'll be able to remove this code.
Robert
Darien Caldwell
Jul 21, 2010, 12:01:32 PM7/21/10
to Google App Engine
It's not a 'problem' per se, it's simply how the system works. There's
a queue for indexes to be built, and depending on if the queue is long
or short when your index is added, the time you wait will be long or
short. Unless Google radically changes their approach to index
building, there will always be a substantial wait.
a queue for indexes to be built, and depending on if the queue is long
or short when your index is added, the time you wait will be long or
short. Unless Google radically changes their approach to index
building, there will always be a substantial wait.
GeekTantra
Jul 21, 2010, 12:15:53 PM7/21/10
to Google App Engine
Thats what I am trying to find, "a work around", otherwise we can
never have instantly deployable opensource applications on Google
AppEngine, which most intermediate level users on the web demand.
never have instantly deployable opensource applications on Google
AppEngine, which most intermediate level users on the web demand.
Reply all
Reply to author
Forward
0 new messages