Bulk inserts in Google App Engine

[Jon Romero]

Is there any way to do bulk inserts in GAE?

Check here number 5 (http://googleappengine.blogspot.com/2009/06/10-
things-you-probably-didnt-know-about.html)

[Jon Romero]

It doesn't seem that's easy to do bulk inserts for AppEngine using
web2py (please correct me if I am wrong).
The insert function for gae does a Table.put(data) (Model.put(data) in
terms of AppEngine) where for bulk we need:
db.put([array_of_Models]).

Is this going to be corrected on the new DAL? Is there a way to have
access to db (gae) so I can do a put?

On Jan 16, 5:08 am, Jon Romero <darks...@gmail.com> wrote:
> Is there any way to dobulkinserts in GAE?

[mdipierro]

We will think about this for the new DAL. It is not difficult to add.
Other than syntactical sugar, is there a speed up? If so for now you
can do

from gluon.contrib.gql import gae

gae.put(...)

and access the low lever gae api.

[Jon Romero]

Yeap the speadup is HUGE (one query to insert 100 rows). Inserting in
gae is slow and if you try to insert 100 one by one you get timeout
errors. I am using the gae_retry but insert queries are bad for
quotas.

I am already using put after importing gae but it accepts gae models
in an array. So I have to declare AGAIN my table with gae.models and
that creates a lot of confusion.

On Jan 31, 2:31 am, mdipierro <mdipie...@cs.depaul.edu> wrote:
> We will think about this for the new DAL. It is not difficult to add.
> Other than syntactical sugar, is there a speed up? If so for now you
> can do
>
> from gluon.contrib.gql import gae
>
> gae.put(...)
>
> and access the low lever gae api.
>
> On Jan 30, 6:21 pm, Jon Romero <darks...@gmail.com> wrote:
>
>
>

> > It doesn't seem that's easy to dobulkinserts for AppEngine using

> > web2py (please correct me if I am wrong).
> > The insert function for gae does a Table.put(data) (Model.put(data) in

> > terms of AppEngine) where forbulkwe need:

[mdipierro]

needs testing but please check in trunk

db.table.bulk_insert([dict(field='value'),dict(field1='value')])

Massimo

[Jon Romero]

I am going to stress test it this week :)

[Jon Romero]

It doesn't seem to work.
This is my code
db.table_name.bulk_insert(
[{
'name': data['name'],
'surname': data['surname'],
'registered': False} for data in some_data])

Problems:
On not GAE, I have this problem:
Expected one of (<class 'google.appengine.ext.db.Model'>,); Got a
dict()

On MySQL, was getting an error that it was expecting a map not a list.
I tried this and it doesn't work:
db.table_name.insert(dict(field='smth'))

Any hints?

On Jan 31, 5:01 pm, Jon Romero <darks...@gmail.com> wrote:
> I am going to stress test it this week :)
>
> On Jan 31, 4:44 pm, mdipierro <mdipie...@cs.depaul.edu> wrote:
>
>
>
> > needs testing but please check in trunk
>
> > db.table.bulk_insert([dict(field='value'),dict(field1='value')])
>
> > Massimo
>
> > On Jan 31, 3:11 am, Jon Romero <darks...@gmail.com> wrote:
>

> > > Yeap the speadup is HUGE (one query toinsert100 rows). Inserting in
> > > gae is slow and if you try toinsert100 one by one you get timeout
> > > errors. I am using the gae_retry butinsertqueries are bad for

> > > quotas.
>
> > > I am already using put after importing gae but it accepts gae models
> > > in an array. So I have to declare AGAIN my table with gae.models and
> > > that creates a lot of confusion.
>
> > > On Jan 31, 2:31 am, mdipierro <mdipie...@cs.depaul.edu> wrote:
>
> > > > We will think about this for the new DAL. It is not difficult to add.
> > > > Other than syntactical sugar, is there a speed up? If so for now you
> > > > can do
>
> > > > from gluon.contrib.gql import gae
>
> > > > gae.put(...)
>
> > > > and access the low lever gae api.
>
> > > > On Jan 30, 6:21 pm, Jon Romero <darks...@gmail.com> wrote:
>
> > > > > It doesn't seem that's easy to dobulkinserts for AppEngine using
> > > > > web2py (please correct me if I am wrong).

> > > > > Theinsertfunction for gae does a Table.put(data) (Model.put(data) in

[mdipierro]

Should be

db.table_name.bulk_insert(
*[{

'name': data['name'],
'surname': data['surname'],
'registered': False} for data in some_data])

The star was missing.

[Jon Romero]

Yeap it works now for MySQL but not for GAE.

Expected one of (<class 'google.appengine.ext.db.Model'>,); Got a
dict()

[mdipierro]

It is a bug. Thanks for checking it.
Please help me test the fix.

In gluon/contrib/gae.py replace line 277

parsed_items.append(fields)
with:
parsed_items.append(self._tableobj(**fields))

Does it work now?

Massimo

[Jon Romero]

It works!!! Thanks!

[Jon Romero]

Shouldn't the bulk insert on Appengine (and on MySQL), return the ids
of the inserted row?
Now, it just returns True, where it should have been something like
(taken from insert in gql.py):

tmp.put()
rid = Reference(tmp.key().id())
(rid._table, rid._record) = (self, None)

[mdipierro]

It should do but the gae bulk insert does not retuns such list. I
would not know how to get it.

[Jon Romero]

Isn't the same thing as simple insert? (http://code.google.com/
appengine/docs/python/datastore/creatinggettinganddeletingdata.html).

On Mar 21, 4:44 pm, mdipierro <mdipie...@cs.depaul.edu> wrote:
> It should do but the gaebulkinsertdoes not retuns such list. I

> would not know how to get it.
>
> On Mar 21, 9:09 am, Jon Romero <darks...@gmail.com> wrote:
>
>
>

> > Shouldn't thebulkinserton Appengine (and on MySQL), return the ids

> > of the inserted row?
> > Now, it just returns True, where it should have been something like

> > (taken frominsertin gql.py):

[Jon Romero]

Ok, then how about, bulk update?

If you retrieve the model, make a small change and then "put" it back,
it should be updated.
I think that would work if I was using the bulk insert you've made but
it's not.
It just adds new rows.

Any ideas for bulk update?

On Mar 21, 6:20 pm, Jon Romero <darks...@gmail.com> wrote:
> Isn't the same thing as simpleinsert? (http://code.google.com/

[Richard]

great! I was going to ask about this.

[Jon Romero]

Any idea/progress?

On Mar 22, 12:45 am, Richard <richar...@gmail.com> wrote:
> great! I was going to ask about this.
>
> On Feb 1, 1:44 am, mdipierro <mdipie...@cs.depaul.edu> wrote:
>
>
>
> > needs testing but please check in trunk
>
> > db.table.bulk_insert([dict(field='value'),dict(field1='value')])
>
> > Massimo
>
> > On Jan 31, 3:11 am, Jon Romero <darks...@gmail.com> wrote:
>

> > > Yeap the speadup is HUGE (one query toinsert100 rows). Inserting in
> > > gae is slow and if you try toinsert100 one by one you get timeout
> > > errors. I am using the gae_retry butinsertqueries are bad for

> > > quotas.
>
> > > I am already using put after importing gae but it accepts gae models
> > > in an array. So I have to declare AGAIN my table with gae.models and
> > > that creates a lot of confusion.
>
> > > On Jan 31, 2:31 am, mdipierro <mdipie...@cs.depaul.edu> wrote:
>
> > > > We will think about this for the new DAL. It is not difficult to add.
> > > > Other than syntactical sugar, is there a speed up? If so for now you
> > > > can do
>
> > > > from gluon.contrib.gql import gae
>
> > > > gae.put(...)
>
> > > > and access the low lever gae api.
>
> > > > On Jan 30, 6:21 pm, Jon Romero <darks...@gmail.com> wrote:
>
> > > > > It doesn't seem that's easy to dobulkinserts for AppEngine using
> > > > > web2py (please correct me if I am wrong).

> > > > > Theinsertfunction for gae does a Table.put(data) (Model.put(data) in