what is an id map? In the context of import_from_csv_file for pydal (web2py)

[Kevin Huang]

Hi all,

This should be simple, but what is the id_map specified in 

import_from_csv_file(csvfile, id_map=None, null='<NULL>', unique='uuid', id_offset=None, transform=None, validate=False, **kwargs)

function from https://pydal.readthedocs.io/en/latest/#pydal.base.DAL.Table.import_from_csv_file ?

Looking through the source I can see that it's supposed to be a dictionary, but I'm still unsure of how it's supposed to be used. Still it seems like it should be "obvious", but I've yet to find a clear example off google.

Anyone clue me in on how this is supposed to be used?

Thanks,

KH

[Leonel Câmara]

This something used very frequently when you're making code to import something. You can have a map (a dictionary) that maps existing record ids to the ids that are present in the csv. This is used for two reasons, to update records instead of inserting new ones when you have already imported once, and two to "fix" references between tables to the actual ids in the database you imported the data in.

[Kevin Huang]

So let's say I have a CSV with:

id,name

1,Bob

2,Mary

3,John

let's say I update the CSV to the following:

id,name

1,Bob

2,Sue

3,John

How would I use id_map parameter to make this change happen so the table updates appropriately?

[Leonel Câmara]

Ideally you would save the id_map from the first time you imported, you can then supply the same id_map to the import_from_csv_file function so it would update the records instead of inserting new ones. To save the id_map you can do something like this the first time you import

my_map = {}

db.your_table.import_from_csv(csvfile, my_map)

pickle.dump(my_map, pickle_file) # example of how to save id_map

Now my_map will hold the map, you can save it, for instance, using pickle. Next time you want to re-import the file with updates you can load the pickled my_map and use that as id_map. Ideally you would test for the pickle's file existence before doing it, and if you want to be absolutely sure you will use portalocker to avoid having more than one process trying to do imports and changing the pickled id_map.

[Kevin Huang]

Thanks for the response Leonel!