Migrating from Arches v3 to v4

[Dennis Wuthrich]

Migrating from v3 to v4

We've heard from a few people, asking about migrating from Arches v3 to v4.  To help you understand the basic approach, we've outlined the workflow as a series of steps:

Step 1. Export your RDM/Reference data from v3

Step 2. Create graphs in v4

Step 3. Migrate business data from v3 to v4

Of course, the devil is in the details!

Step 1: Export RDM data from v3.

We're working on a patch that will allow you to use the RDM SKOS exporter (Tools -> Export Scheme) to export your existing RDM from v3 into a format that v4 can read.  Once you've exported your existing concepts, use the v4 RDM to import them.

We've introduced a new feature in v4 called "collections" that allow you to manage dropdowns in data entry forms in a manner that adhers to the skos standard.  You'll need to use the RDM to create collections for your dropdowns.

Step 2: Create graphs in v4

In v4, Arches provides a way to create and validate graphs interactively.  The "Arches Designer" in v4 offers visual graph creation tools that you'll need to use to create graphs.

If you're using the default HIP graphs in v3, this step is pretty easy as we'll provide updated versions for v4.  If you've created your own graphs (or modified the existing v3 HIP graphs), you'll need to use the Arches Desiger to create your graphs in v4.  

In either case, migrating from v3 to v4 is a chance for you to re-evaluate your data models and implement improvements based on your experience with v3.  You'll find that v4 comes with a library of re-usable "branches" (or sub-graphs) that make designer a graph much easier than for v3.

One last note: v4 validates graphs to ensure that they are consistent with the CIDOC CRM classes and properties.  This means that graphs in v4 need to be semantically valid (NOT a requirement of v3).

Step 3: Migrate business data from v3 to v4

After configuring the RDM and defining graphs in v4, you're ready to import your v3 data.

To start, export data from your v3 instance of Arches.  V3 comes with a command that will export your business data to JSON.  You'll need to use a JSON to csv converter (or write a script to do it), then use v4's CSV import tools to pull your data into Arches.  V4 comes with mapping tools that allow you to map information in a CSV file to specific nodes in a resource model.

Of course, there are always lots of details when working with a specific dataset.  But the basic process can be broken down to the 3 steps outlined.

We at Farallon will be supporting v3 to v4 migrations, but there's no reason you can't do it yourself.

Hope this helps...  

Cheers,

Dennis

[Joel Aldor]

Thank you Dennis for this. So when will the patch for v3 RDM SKOS exporter be made available?

[Joel Aldor]

Also, will the upgrade won't affect the mapping of images and files uploaded per resource? All our images and files are hosted either on AWS S3 and Azure Storage.

Thanks.

[Ryan Anderson]

Hi Joel,

We have a modified skos exporter complete but we'd like to QA it and determine the best way to deploy it to current users of v3. That being said, if you would like to be a guinea pig I can provide you with the code and instructions on how to deploy it manually.

Ryan

[Joel Aldor]

Hi Ryan,

Yes please. I'll try this one on our old EC2 instance. Kindly provide me the deployment details for the patch.

Thanks.

Joel

[Joel Aldor]

Hi Dennis,

I'm done with Step 1 of the migration. For Step 2, we wish to retain first the default HIP graphs we had for version 3, before we make any changes to our data models using the Arches Designer. Could you provide us the updated graphs for version 4?

Thank you,

Joel

[Vincent Meijer]

Hello Joel,

The graphs that come default with v4 are here: https://github.com/archesproject/arches/tree/master/arches/db/graphs/resource_models

Is that what you were looking for? 

Best,

Vincent

[Annabel Lee Enriquez]

Hi Joel,

To add to both Vincent's and Dennis' posts, I wanted to let you know that the v4 resource models (formerly known as graphs) that are currently available (via the link that Vincent provided) are not exact copies of the v3 graphs because we tried to improve on both the organization and semantics based on our experiences thus far. Having said that, the new v4 graphs should accommodate the same data that the v3 graphs did. I will say that the Arches Designer makes it relatively easy to reevaluate your needs and make changes to the resource models before you load your data, but as Dennis noted, it will enforce semantic rules based on the CIDOC CRM, so do post any CRM related questions you might have as you go through this process.

Thanks!
Annabel

[Joel Aldor]

Thanks Annabel and Vincent for your replies. So if I understand this correctly, the v4 resource models have changes from v3 graphs, but I can revert them back to the default v3 HIP graphs using the Arches Designer, right? We have used only the default v3 HIP graphs and we tried not to change them for the meantime, because changing those graphs can be quite challenging. We would be in a better position to re-evaluate and customize our graphs once we have successfully migrated everything to v4, including all business data and images/files.

I've already imported the v4 resource models on my dev Arches 4 installation. Would you be able to point out to me what were the changes of the resource models on version 4? And how do I make the changes using the Graph Designer (say, I want to change the heritage resource model first)? Once I designed those resource models similar to those of version 3, I should now re-design the cards and menus to look exactly like version 3.

I looked into Version 3 Appendix for the list of resource graphs and nodes as my reference: http://arches-hip.readthedocs.io/en/latest/appendix/. Is there a similar list for version 4?

Your help is very much appreciated. Thanks!

[Adam Cox]

Hi Joel, instead of changing the v4 graphs to match the v3 ones, you should convert your data into an upload file that will match v4 graph nodes.

I have created some thorough documentation on how to make an upload file for v4: 

http://arches.readthedocs.io/en/stable/import-export/

Hope that helps. Hopefully others will have more info on what has changed between the v3 and v4 graphs which should help in your conversion process.

Adam

On May 3, 2017 12:13 AM, "Joel Aldor" <joel....@gmail.com> wrote:

Thanks Annabel and Vincent for your replies. So if I understand this correctly, the v4 resource models have changes from v3 graphs, but I can revert them back to the default v3 HIP graphs using the Arches Designer, right? We have used only the default v3 HIP graphs and we tried not to change them for the meantime, because to be honest, understanding those graphs are quite a challenge. We would be in a better position to re-evaluate and customize our graphs once we have successfully migrated everything to v4, including all business data and images/files.

I've already imported the v4 resource models on my dev Arches 4 installation. Would you be able to point out to me what were the changes of the resource models on version 4? And how do I make the change on Graph Designer? I looked into Version 3 Appendix for the list of resource graphs and nodes as my reference: http://arches-hip.readthedocs.io/en/latest/appendix/

--
-- To post, send email to arches...@googlegroups.com. To unsubscribe, send email to archesproject+unsubscribe@googlegroups.com. For more information, visit https://groups.google.com/d/forum/archesproject?hl=en
---
You received this message because you are subscribed to the Google Groups "Arches Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to archesproject+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Joel Aldor]

Hi Adam,

So that means I have to export first my resources from version 3. If I get it right, I must first create a JSON file that configures all resource export mappings? I'm looking at this document, and I hope I'm looking at the right locaton. http://arches-hip.readthedocs.io/en/latest/system-administration/#exporting-data

I want to first do a test export and import for a particular heritage area (I'm looking into Silay City). If I'm satisfied with the results I will export and import for the rest of the resources from version 3.

Any inputs from others?

Thanks,

Joel

On Thursday, May 4, 2017 at 1:38:07 AM UTC+8, Adam Cox wrote:

Hi Joel, instead of changing the v4 graphs to match the v3 ones, you should convert your data into an upload file that will match v4 graph nodes.

I have created some thorough documentation on how to make an upload file for v4: 

https://arches4.readthedocs.io/en/latest/import-export/#uploading-a-csv-to-arches

-- To post, send email to arches...@googlegroups.com. To unsubscribe, send email to archesprojec...@googlegroups.com. For more information, visit https://groups.google.com/d/forum/archesproject?hl=en

---
You received this message because you are subscribed to the Google Groups "Arches Project" group.

To unsubscribe from this group and stop receiving emails from it, send an email to archesprojec...@googlegroups.com.