recipy to migrate DB to GAE

[Jurgis Pralgauskis]

Hello,
what's the best way to import my currently sqlite db to GAE?

I think for each table:
1) export CSV on nonGAE,
2) then paste it into some form in GAE environment and adapt
import_from_csv_file... ?

but maybe there are other/better/ready recipies?

[howesc]

take a look at the bulk loader from google:
http://code.google.com/appengine/docs/python/tools/uploadingdata.html

i think it takes a CSV format, so if you can get your data from sqlite
in proper CSV format you should be able to upload via bulk loader.

good luck!

christian

On Aug 15, 8:33 am, Jurgis Pralgauskis <jurgis.pralgaus...@gmail.com>
wrote:

[Jurgis Pralgauskis]

I was experimenting on bulk download/upload.
I have simple table (Languages), which I first fill with a couple of
records.
Then I download them -- seems nice.

The problem I get, is that when Uploaded (actually the same stuff),
id values are stored in some strange column "Key Name", but not in
"id".
and I get TypeError: int() argument must be a string or a number, not
'NoneType'
And I can't see the uploaded records in appadmin.

screenshots, csv and bulkloader.yaml:
http://ftp.akl.lt/users/jurgis/etc/db_migrate_web2py-gae/


By the way GAE has "ID" name (not "id")

if I change "__key__" to "id" in bulkuploader.yaml,
I get extra column "id" -- which is clearly different from GAE "ID",
as GAE "ID" then are inserted independently as if I would not have
them.

===== Another Idea =========
maybe Web2Py could have "Backup"/"Restore" feature per DB,
alongside with "Export"/"Import" per table.

As I need to keep my id's unchanged when migrating (as I have
hierarchical/nested/recursive relation in one table, and also some
foreign keys)

[howesc]

perhaps i have led you astray...i have not done that sort of download
from non-GAE to GAE. references will break for sure, but if you are
lucky enough to not have them, maybe the CSV export/import all tables
at once (see section 6.25.2 of version 3 of the book) will work for
you. when i was first experimenting with the app engine i recall
having the reference problem. included is my method for fixing up
references. this is specific to my model, so it won't just work for
you...but hopefully you can make it work for you. in my model i had
several multi-value fields (strings of id's separated with "|") so
that is what you see going on there.

let me know if you have more questions, and i'll do my best to help!

def replace_db():
"""
Truncate all tables, and replace with data from the uploaded CSV
file.
Note that this is intended to load data from the web2py formatted
csv file
as downloaded from L{export}
"""
id_map = None
form = FORM(INPUT(_type='file', _name='data'),
INPUT(_type='submit'))
if form.accepts(request.vars):
for table in db.tables:
db[table].truncate()
id_map = {}

db.import_from_csv_file(form.vars.data.file,id_map=id_map,unique=False)
#@TODO: fix up song media references
songs = db(db.song.id>0).select()
for song in songs:
if not song.media:
continue
new_media = ''
medias = song.media.split('|')[1:-1]
for m in medias:
new_media += "|" + str(id_map['media'][m])
new_media += "|"
song.update_record(media = new_media)
#@TODO: fix up recording.media references
recordings = db(db.recording.id>0).select()
for r in recordings:
if not r.media:
continue
new_media = ''
medias = r.media.split('|')[1:-1]
for m in medias:
if id_map['media'].has_key(m):
new_media += "|" + str(id_map['media'][m])
new_media += "|"
r.update_record(media = new_media)
#@TODO: fix up product.elements references
products = db(db.product.id>0).select()
for p in products:
if not p.elements:
continue
new_song = ''
songs = p.elements.split('|')[1:-1]
for s in songs:
if id_map['song'].has_key(s):
new_song += "|" + str(id_map['song'][s])
new_song += "|"
p.update_record(elements = new_song)
return dict(form=form, id_map=id_map)



On Aug 17, 8:11 am, Jurgis Pralgauskis <jurgis.pralgaus...@gmail.com>