Exporting data from one data.fs to another

[Jo G]

So,

I have two sites using substanced. One has data I imported from a csv file, the other has data that was imported from postgres. There are several references between the objects, these are based on UUID aren't they?

In both cases the data is now 'live' and people are editing it. 

Two things:

I want to have an exportable representation of the important data - ie a backup that isn't a data.fs file. This data could store the UUID of each record. substanced does have something called sd-dump which stores everything in yaml files but I have lots of records and it falls over trying to create everything.

We have also been asked to add to the schema for these objects - to add in extra fields. 

Now if they hadn't changed any of the data I would just reimport everything from postgres or csv and recreate all the links but as I don't want to lose anything they have changed I can't do that.

Is there a way to export the objects and all the references and then reimport into a new data.fs along with the additional data? Can you force a new object to have a particular UUID so I can maintain the relationships? 

I don't have a record of the UUIDs of each of the objects that links it to the original data in the csv file or the postgres database. I do have records of the 'names' (one of the properties) of the new objects and the uniqueID in the postgres database tables

help..

[Tres Seaver]

The 'sd_dump' utility should get you part of the way there:

http://substanced.readthedocs.io/en/latest/dump.html

The dumped data will include the UUIDs, including a 'references.yaml' file
which maps between "paths" and UUIDs.

Re loading: its docs say:

---------------------------- %< --------------------------------
.. note::

You'll note in the following the absence of docs on *loading* data.
This is intentional. The process of loading data into a new,
or semi-new, or newer-than-new site has many policy implications.
Too many to fit into a single loading script. Substance D considers
the particulars of loading data to be in the province of the
application developer.

---------------------------- %< --------------------------------


Tres.
--
===================================================================
Tres Seaver +1 540-429-0999 tse...@palladion.com
Palladion Software "Excellence by Design" http://palladion.com

[Jo G]

Thanks, 

I have tried using sd_dump and it runs out of memory as I have too many objects in my database. It gets part way through and then just falls over. I have already written one upload script for most of the data,looks like another one for the appended parts of the data is needed as well.

I do have some mappings from the old database (postgres) unique keys to the new paths in my data.fs but not sure I got those for the references. I will take a look at references.yaml to see if it helps me to sort those out.

I also got some help on the zodb forum. Basically my idea of exporting all the data from the production database and then reimporting it along with extra data for each record is not sensible. Instead I should write another upload script and use the mappings I have between the old data (postgres or csv) and the new zodb to add in the extra properties for each object. 

Any news on sprints or more support for substanced?

[Steve Piercy]

On 10/4/16 at 1:44 PM, joannegil...@gmail.com (Jo G) pronounced:

>Any news on sprints or more support for substanced?

A few of us will be at ploneconf.
https://2016.ploneconf.org/

Tickets are still available. Maybe sprints are free to participants?

--steve

------------------------
Steve Piercy, Soquel, CA

[Jo G]

ah if only. I'm in the UK so won't make that...