Importing Nesstar DDI & data into Dataverse 4x

571 views
Skip to first unread message

Janet McDougall - Australian Data Archive

unread,
Mar 14, 2017, 8:54:53 PM3/14/17
to Dataverse Users Community
Hi All

We are testing existing Dataverse APIs/tools to import our archive from Nesstar 4 (DDI 2.5) into Dataverse 4.6.  I have noticed posts relating to 'creating new datasets using DDI' but many  are from around 2 years ago, and related to 'migration' in version Dataverse v3.x.  

Is or has anyone done any work that might be relevant to us?  If so please share your experiences - we will of course document our work as we go for other to use.  We are looking at native APIs using JSON at the moment, and will post some more questions in the next few days.

thanks

PS hope you're all ok after your winter storms, can't imagine that!

Philip Durbin

unread,
Mar 14, 2017, 9:53:15 PM3/14/17
to dataverse...@googlegroups.com
There was a recent issue at https://github.com/IQSS/dataverse/issues/3632 where we talked about importing DDI files but it's really not very well supported and I figured that issue should remain open until the process is better documented and the code is cleaned up a bit. Like you said, it was written for upgrading from DVN 3.x to Dataverse 4, not from migrating from one platform to another. It would be really nice to make it easy for people to migrate their data to Dataverse!

Phil

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse-community@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/9b4ad8d7-271d-42a6-aecc-585b44bbc21c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--

Janet McDougall - Australian Data Archive

unread,
Mar 15, 2017, 3:36:19 AM3/15/17
to Dataverse Users Community, philip...@harvard.edu
Hi Phil
Well we are working on it...  I was wondering if there is a JSON schema or some doco to assist us in translating the DDI 2.5 to JSON format?  At the moment I'm using the JSON example provided but it would be really helpful to understand what's expected  https://github.com/IQSS/dataverse/blob/v4.3/scripts/api/data/dataset-create-new.json

thanks
Janet
Archivist

Philip Durbin

unread,
Mar 15, 2017, 7:46:28 AM3/15/17
to dataverse...@googlegroups.com
To see the mapping between DDI and Dataverse's native JSON fields, please see the "Dataverse 4.0 Metadata Crosswalk: DDI, DataCite, DC, DCTerms, VO, ISA-Tab" spreadsheet linked from http://guides.dataverse.org/en/4.6.1/user/appendix.html


If you're migrating existing datasets, I assume they already have a DOI or a Handle. This won't "just work" via JSON until this issue is addressed: https://github.com/IQSS/dataverse/issues/3083

That is to say, I believe DDI is a better bet for you, even though it's not well supported yet for your use case of migrating from one platform to another.

You could also play around with harvesting DDI records over OAI-PMH.

Please keep the questions coming!

Phil

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse-community@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Janet McDougall - Australian Data Archive

unread,
Mar 15, 2017, 5:56:16 PM3/15/17
to Dataverse Users Community
Hi Phil, yes I found the mapping document, really great resource. I used it yesterday to writ JSON to set the access conditions. Marina will use native API to test if my json structure valid and if the terms can be set this way. That's what I was more interested in was whether we can use the full suite of DDI 2.5 elements in the mapping. If so then we should be able to set up our datasets this way to import them from Nesstar to data verse. We are working on it this morning.
Janet

Philip Durbin

unread,
Mar 15, 2017, 6:44:55 PM3/15/17
to dataverse...@googlegroups.com
Cool. Again, the main gotcha with JSON is that a new DOI will be created, which may or may not be what you want.

On Mar 15, 2017 5:56 PM, "Janet McDougall - Australian Data Archive" <janet.m...@anu.edu.au> wrote:
Hi Phil, yes I found the mapping document, really great resource. I used it yesterday to writ JSON to set the access conditions.  Marina will use native API to test if my json structure valid and if the terms can be set this way. That's what I was more interested in was whether we can use the full suite of DDI 2.5 elements in the mapping. If so then we should be able to set up our datasets this way to import them from Nesstar to data verse.  We are working on it this morning.
Janet

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse-community@googlegroups.com.

Vyacheslav Tikhonov

unread,
Mar 16, 2017, 11:19:55 AM3/16/17
to Dataverse Users Community
Dear Janet,

We are (DANS, dataverse.nl) also working on the pilot project for the one of EU countries to get all stuff from Nesstar migrated into Dataverse 4. We're not using DDI for this as it has much more fields than Dataverse and it's difficult to map everything, the idea is to convert all metadata to XML and deposit to Dataverse by using native APIs for necessary fields, and attach extra stuff as files.

It should be probably relevant to you.

Best,
Slava

Amber Leahey

unread,
Mar 16, 2017, 1:02:16 PM3/16/17
to Dataverse Users Community
Hi Janet, 

You might also be interested in joining the DDI & Dataverse WG that just started looking at Use Cases, including DDI Import features. 


I think your project would be a great use case and project to highlight to the community. 

Best, 
Amber

Janet McDougall - Australian Data Archive

unread,
Mar 16, 2017, 5:47:06 PM3/16/17
to Dataverse Users Community
Hi all thanks for the responses. We are testing native API now to move our Nesstar studies into DV4 using json to do the translation. We should be able to map all the Nesstar DDI elements into DV as they are already being used or seem to have database fields to hold them. I'm presuming this work was done as part of the 'export DDI' option?

Anyway we are testing the TERMS elements this morning using json. Please see earlier posts for links to docs I've referenced. I'm using my iPad here so I have limited editing skills.


Marina and I have been using the native API to create a new dataset, based on examples in GitHub create_new_dataset3.json.

I'm also using the DV 4 metadata mapping to DDI, DC, data cite etc spreadsheet I found on GitHub to understand where DDI elements are written to fields in the DB tables. Really great doc!

We've also been looking at the jsonpaser.java code to try to understand how it expects to receive the json object but this will require a lot of trial and error.

QUESTION
do you know of where we can access the JSON schema for the the parser? That way I will know exactly how to format the json code to translate from DDI using the API.

thanks
Janet

Philip Durbin

unread,
Mar 16, 2017, 6:19:09 PM3/16/17
to dataverse...@googlegroups.com
There is no JSON schema for the parser. The JSON format Dataverse uses was designed originally for backing up and restoring datasets we were creating on out demo site when we were developing Dataverse 4 and wanted to drop the database and reimport some datasets without manually entering the data again. It worked fine for this purpose and the format is now used in what we document as the "native" API, in contrast to the SWORD API, which is based on an XML standard (dcterms).

Phil

p.s. Raman has done some nice experiments with JSON Schema in this repo but they aren't used by Dataverse: https://github.com/IQSS/json-schema-test

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse-community@googlegroups.com.

Janet McDougall - Australian Data Archive

unread,
Mar 16, 2017, 6:33:28 PM3/16/17
to Dataverse Users Community
Ok thanks Phil, we will work it out and share if successful, will also look at other work you referenced. Our main aim is to be able to populate the relevant db fields, especially the RESTRICTED flag which is set in DDI. If we can do this for most of the Nesstar DDI while creating new datasets in DV4 then our work processes will be much more structured and contained.
Janet

Philip Durbin

unread,
Mar 16, 2017, 8:04:37 PM3/16/17
to dataverse...@googlegroups.com
Ok, I don't know what RESTRICTED means in DDI or why it's ALL CAPS. :)

You may or may not find the following API endpoint helpful. You can pass it a field like "title" and get some information about it: http://phoenix.dataverse.org/api/admin/datasetfield/title

{
  "status": "OK",
  "data": {
    "name": "title",
    "id": 1,
    "title": "Title",
    "metadataBlock": "citation",
    "fieldType": "TEXT",
    "allowsMultiples": false,
    "hasParent": false,
    "parentAllowsMultiples": "N/A (no parent)",
    "solrFieldSearchable": "title",
    "solrFieldFacetable": "title_s",
    "isRequired": true
  }
}

Just http://phoenix.dataverse.org/api/admin/datasetfield with no field at the end will give a list of fields. You could try iterating over all the fields.

Really this is more or less the same information that's in the tsv files, or in their equivalent tabs in the spreadsheet tabs linked from http://guides.dataverse.org/en/4.6.1/user/appendix.html such as "Citation Metadata", "Geospatial Metadata", etc.

I hope this helps,

Phil

On Thu, Mar 16, 2017 at 6:33 PM, Janet McDougall - Australian Data Archive <janet.m...@anu.edu.au> wrote:
Ok thanks Phil, we will work it out and share if successful, will also look at other work you referenced.  Our main aim is to be able to populate the relevant db  fields, especially the RESTRICTED flag which is set in DDI.  If we can do this for most of the Nesstar DDI while creating new datasets in DV4 then our work processes will be much more structured and contained.
Janet

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse-community@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

julian...@g.harvard.edu

unread,
Mar 20, 2017, 4:44:08 PM3/20/17
to Dataverse Users Community
Hi Janet. We also plan on exposing in the UI how Dataverse fields are mapped to fields in other standards, such as DDI. A github issue's been opened for this work here: https://github.com/IQSS/dataverse/issues/3701

Would seeing the mappings this way be helpful for your use case?

Janet McDougall - Australian Data Archive

unread,
Mar 21, 2017, 6:31:16 AM3/21/17
to Dataverse Users Community
Hi Julian & all
Any mappings are useful, I'm not exactly sure what you've suggested really means, but I'm replying now in case you have a response for me in the morning ( 9pm here). what I've been doing is looking at fields in the db tables and mapping them to DDI elements listed in the mapping spreadsheet I referred to in earlier posts (great doc) to try and work out where they're being written to. We are using the jsonparser.java thru API to import the DDI metadata into the db tables. For example, in the TERMS table there are several non DDI fields we can populate usefully plus the DDI elements/fields in the example (create_new_dataset3.json).

Our problem now is testing how far we can use the limited example DDI elements in the jsonAPI (create_new_dataset3.json) without following the code too far to be able to expand to using the full Nesstar DDI suite. I am manually coding ADA DDI metatdata pretty much at amDDI element at a time and Marina (web dev) is running the API to test what the parser will accept among other things. There is no obvious reason as to why some elements are compound - several cols in an aray, or just rows, or primitive, not in an array etc. well this is my interpretation of the json code so we will find out if it's wrong as we go. We hope to build a Nesstar DDI to dataverse 4.6 import tool out of this so any assistance or pointers would be really appreciated. Thanks for following this up.
Janet

Philip Durbin

unread,
Mar 21, 2017, 10:06:26 AM3/21/17
to dataverse...@googlegroups.com
An import tool from Nesstar DDI to Dataverse would be great! I hope you open source it. :)

Also, dataset-create-new3.json is probably a good starting point, but I was recently told that there is a JSON file floating around that exercises as many metadata fields as possible. It's just test data so the title is "title" or something. Would it be helpful for me to try to dig this JSON file up?

I believe that dataverse-client-java is farthest along in terms of allowing users to specify as many metadata fields in JSON as possible. That said, this "Full support for Dataset MetaData objects" issue is still open so I'm not sure of the exact state of things: https://github.com/IQSS/dataverse-client-java/issues/6 . The R and Python clients create datasets using XML (SWORD), I believe.

Phil

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse-community@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Janet McDougall - Australian Data Archive

unread,
Mar 21, 2017, 7:14:59 PM3/21/17
to Dataverse Users Community
hi Phil yes please! re "Would it be helpful for me to try to dig this JSON file up?".  Marina has been testing SWORD to create the dataset container (fish/xxxxxx/ ID etc), then use json java API to populate it - i'm busy on the json side so not sure how that's going at the moment.   We will definitely share our work - it's nice coming to work and hearing from you all...
Thanks
Janet

Philip Durbin

unread,
Mar 21, 2017, 7:23:42 PM3/21/17
to dataverse...@googlegroups.com
Please find "all_metadata_blocks.json" attached. I meant to try it myself but haven't yet. I hope it helps.

Please let me know if it doesn't allow you to create a dataset. I can drop it into my test framework tomorrow.

Thanks,

Phil

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse-community@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.
all_metadata_blocks.json

Janet McDougall - Australian Data Archive

unread,
Mar 21, 2017, 10:42:11 PM3/21/17
to Dataverse Users Community, philip...@harvard.edu
thanks alot Phil! Marina is away today but i will use it to prepare the json code for the API.  will let you know how we go, vice versa if you test it first.  Janet


On Wednesday, 22 March 2017 10:23:42 UTC+11, Philip Durbin wrote:
Please find "all_metadata_blocks.json" attached. I meant to try it myself but haven't yet. I hope it helps.

Please let me know if it doesn't allow you to create a dataset. I can drop it into my test framework tomorrow.

Thanks,

Phil
On Tue, Mar 21, 2017 at 7:14 PM, Janet McDougall - Australian Data Archive <janet.m...@anu.edu.au> wrote:
hi Phil yes please! re "Would it be helpful for me to try to dig this JSON file up?".  Marina has been testing SWORD to create the dataset container (fish/xxxxxx/ ID etc), then use json java API to populate it - i'm busy on the json side so not sure how that's going at the moment.   We will definitely share our work - it's nice coming to work and hearing from you all...
Thanks
Janet


On Wednesday, 15 March 2017 11:54:53 UTC+11, Janet McDougall - Australian Data Archive wrote:
Hi All

We are testing existing Dataverse APIs/tools to import our archive from Nesstar 4 (DDI 2.5) into Dataverse 4.6.  I have noticed posts relating to 'creating new datasets using DDI' but many  are from around 2 years ago, and related to 'migration' in version Dataverse v3.x.  

Is or has anyone done any work that might be relevant to us?  If so please share your experiences - we will of course document our work as we go for other to use.  We are looking at native APIs using JSON at the moment, and will post some more questions in the next few days.

thanks

PS hope you're all ok after your winter storms, can't imagine that!

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse...@googlegroups.com.

Philip Durbin

unread,
Mar 22, 2017, 9:39:24 AM3/22/17
to dataverse...@googlegroups.com
Uh oh, out of the box, Dataverse can't handle the all_metadata_blocks.json file I sent yesterday because it uses metadata blocks that are custom to Harvard's installation*. You should expect to see this error if you try to create a dataset with it:

"message": "Error parsing datasetVersion: Can't find type 'mraCollection'"

I stuck the JSON file in my test suite[2], hacked on the JSON file a bit (removing metadatablocks and making other tweaks) and finally got it to work, to create a dataset. I renamed it default_metadata_blocks-4.6.1.json and will attach it here. I'm also attaching a screenshot of what it looks like in the GUI with all the metadata (I clicked the non-citation blocks to expand them). I also exported the dataset as DDI and Dublin Core and will attach those as well.

It might be that some metadata fields aren't exercised. And for all I know some are used incorrectly. Hopefully this is something we can build on, at least. I hope it helps.

Long story short, please give default_metadata_blocks-4.6.1.json (attached) a try. :)

Thanks,

Phil

1. For example: https://github.com/IQSS/dataverse/blob/v4.6.1/scripts/api/data/metadatablocks/customMRA.tsv
2. getDatasetJson in https://github.com/IQSS/dataverse/blob/v4.6.1/src/test/java/edu/harvard/iq/dataverse/api/UtilIT.java#L210

On Tue, Mar 21, 2017 at 10:42 PM, Janet McDougall - Australian Data Archive <janet.m...@anu.edu.au> wrote:
thanks alot Phil! Marina is away today but i will use it to prepare the json code for the API.  will let you know how we go, vice versa if you test it first.  Janet

On Wednesday, 22 March 2017 10:23:42 UTC+11, Philip Durbin wrote:
Please find "all_metadata_blocks.json" attached. I meant to try it myself but haven't yet. I hope it helps.

Please let me know if it doesn't allow you to create a dataset. I can drop it into my test framework tomorrow.

Thanks,

Phil
On Tue, Mar 21, 2017 at 7:14 PM, Janet McDougall - Australian Data Archive <janet.m...@anu.edu.au> wrote:
hi Phil yes please! re "Would it be helpful for me to try to dig this JSON file up?".  Marina has been testing SWORD to create the dataset container (fish/xxxxxx/ ID etc), then use json java API to populate it - i'm busy on the json side so not sure how that's going at the moment.   We will definitely share our work - it's nice coming to work and hearing from you all...
Thanks
Janet


On Wednesday, 15 March 2017 11:54:53 UTC+11, Janet McDougall - Australian Data Archive wrote:
Hi All

We are testing existing Dataverse APIs/tools to import our archive from Nesstar 4 (DDI 2.5) into Dataverse 4.6.  I have noticed posts relating to 'creating new datasets using DDI' but many  are from around 2 years ago, and related to 'migration' in version Dataverse v3.x.  

Is or has anyone done any work that might be relevant to us?  If so please share your experiences - we will of course document our work as we go for other to use.  We are looking at native APIs using JSON at the moment, and will post some more questions in the next few days.

thanks

PS hope you're all ok after your winter storms, can't imagine that!

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsubscribe...@googlegroups.com.
To post to this group, send email to dataverse...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.
default_metadata_blocks-4.6.1.json
ddi.xml
dc.xml
Replication_Data_for_Title_-_dvd61da9e3_Dataverse_-_2017-03-22_09.27.51.png

Amber Leahey

unread,
Mar 22, 2017, 10:42:03 AM3/22/17
to Dataverse Users Community, philip...@harvard.edu
Hi Janet and Phil, 

It would be interesting for us if you could capture a comparison of supported DDI fields between Nesstar and Dataverse. There are a lot of DDI fields in Nesstar that I would imagine you will lose in the mapping to Dataverse, but I could be wrong in my understanding of this project. 

It would be great to see some kind of comparison list of fields! And thanks for working on this, I imagine it will be useful and interesting for a lot of people. 

Best, 

Amber Leahey

To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse...@googlegroups.com.

julian...@g.harvard.edu

unread,
Mar 22, 2017, 6:05:43 PM3/22/17
to Dataverse Users Community, philip...@harvard.edu
Thanks Phil! That's really helpful!

I'm also attaching a screenshot of what it looks like in the GUI with all the metadata (I clicked the non-citation blocks to expand them)...
It might be that some metadata fields aren't exercised. And for all I know some are used incorrectly. Hopefully this is something we can build on, at least. I hope it helps.

I compared the fields in the screenshot to the fields that exist in Harvard Dataverse now. These fields in the citation block are missing from the screenshot and in the DDI and JSON that you attached:
  • Language
    • It is in the JSON metadata that Harvard Dataverse exports; not in the DDI that Harvard Dataverse exports
  • Topic classification
    • It is in the JSON and DDI metadata that Harvard Dataverse exports
Also, this might be an aside, but Logo URL field (in the producer and distributor fields) just shows the text "Logo URL". I mention it only because I haven't found an issue for this existing behavior. (Problems with both fields were discussed in the closed issue #207.)


Amber, would a spreadsheet with this comparison between Dataverse supported DDI and Nesstar supported work? (Also, Harvard Dataverse has harvested from ICPSR, right? So would it be helpful to also compare ICPSR's DDI, maybe in the same spreadsheet?)

Janet McDougall - Australian Data Archive

unread,
Mar 22, 2017, 9:17:17 PM3/22/17
to Dataverse Users Community, philip...@harvard.edu

 Hi Amber - I will complete this mapping but from what I've looked at it seems the full suite ADA uses is there.  Note that ADA doesn't use the full Nesstar DDI suite which is a mix of DDI 2.1 & 2.5 from my understanding.  

Janet McDougall - Australian Data Archive

unread,
Mar 22, 2017, 9:20:38 PM3/22/17
to Dataverse Users Community, philip...@harvard.edu
Hi All
Thanks Phil for testing & Julian & Amber for your comments.  Marina is away sick so we haven't been able to run the json API but i will go through your material and with comments and get back to you for tomorrow.
Janet

Janet McDougall - Australian Data Archive

unread,
Mar 23, 2017, 2:39:22 AM3/23/17
to Dataverse Users Community, philip...@harvard.edu
Hi All
i've attached an amended version or your Metadata Crosswalk.  I've rushed it a bit so you will have it (this morning...) but it should give you an idea of what we use.  Terms of Use tab is what i've worked on the most.  The ADA/Nesstar DDI info is at the far right cols in each tab - Headings: 'ADA Nesstar Notes' & 'Mandatory'.

Thanks
Janet 
Dataverse 4.0+ADA+Nesstar Metadata Crosswalk- DDI JM 2017-03-23.xlsx

Janet McDougall - Australian Data Archive

unread,
Mar 23, 2017, 4:57:14 AM3/23/17
to Dataverse Users Community
Note, Nesstar DDI template allows extensive use of DDI elements, I've indicated which ones ADA uses.
Message has been deleted

Amber Leahey

unread,
Jun 26, 2017, 12:15:10 PM6/26/17
to Dataverse Users Community
Thanks for this Janet. Apologies for the delay. I've managed to upload this into a Google Spreadsheet for editing. Also added a tab for "Tabular Variable Metadata" as I think this would also need to be migrated from yours and others Nesstar systems at some point.  At the variable-level this would need considerable development since there is no mapping or support yet for some of these fields but I think it will be helpful to improve the workflows for adding metadata at the variable level in Dataverse. For now I've kept them together with your mapping spreadsheet. 

Here is the link to the Cross-walk which includes all the tabs for Study-level migration (as provided by you) as well as the new tab for variable-level metadata using DDI: https://docs.google.com/spreadsheets/d/11Fiy0huoEKzOmJS-hgB-Sq8KOCrsyHY075Yrdkg3U8M/edit#gid=1021946509 

I'll continue adding to this and try to review your mapping to see if this fits with our understanding / metadata as well. 

Thanks again, 
Amber

Andreas Loibl

unread,
Jul 19, 2017, 9:55:34 AM7/19/17
to Dataverse Users Community
Hello, I've got a similar problem: porting nesstar to dataverse.

what's the current state? is there any best-practice round? Maybe some kind of XML translator, etc?
Any help / link / would be very appreciated.

Janet McDougall - Australian Data Archive

unread,
Jul 19, 2017, 11:03:32 PM7/19/17
to Dataverse Users Community
Hi we are working towards running a script developed by our web programmer Marina, mapping the DDI to JSON and using API to import into Dataverse. ADA Director Steve McEacern and Marina will work with the dataverse team on how to integrate and/or make any work we do available. Steve can provide more specifics on this.

Amber, I've made some updates to the DDI to Dataverse mapping so I will update the google spreadsheet tomorrow - I now have a better understanding of dataverse since mapping them earlier in the year.

julian...@g.harvard.edu

unread,
Jul 20, 2017, 4:21:08 PM7/20/17
to Dataverse Users Community
This is great! Thank you Janet. I was going to try answering some questions you had about the crosswalk, but I'll wait until the google spreadsheet is updated.

Janet McDougall - Australian Data Archive

unread,
Jul 21, 2017, 2:08:40 AM7/21/17
to Dataverse Users Community
Hi Julian
sorry, i didn't get a chance to update the spreadsheet today, but basically I'm slightly revising ADA DDI elt mappings to TERMS  -  the non-DDI fields 'terms of use' and 'terms of access', now I'm more familiar with Dataverse.  I am finding both terms very open to interpretation ie the options are license CCO (waiver) or 'terms of use', but it is not clear whether this field is for a license, parts of a license, usage text, links to DUA or other.  Same for 'terms of access', this could incorporate a license agreement, instructions on how to access or apply for access, links to application forms etc.  This may have been the point of creating these fields, so the ADA mappings may not work for others.

I also understand DataTags will be implemented at file level soon (https://groups.google.com/forum/#!topic/dataverse-community/FJaVCuVzkKM) so I am concerned this will increase the variation in use of these terms.

If we want to map these fields to other standards then it may be difficult to identify exactly what is represented ie terms, license, DUA etc.  I may be taking this too far and have moved from the purpose of their design.   Go ahead with your questions, will check on Monday.  Sorry for all my use of etc today.

Thanks
Janet

julian...@g.harvard.edu

unread,
Jul 24, 2017, 8:44:26 AM7/24/17
to Dataverse Users Community
Hi Janet,

From the spreadsheet, it looks like you plan to put the metadata that ADA maps to the DDI element confDecinfo about submitting a "Request for Access to General Datasets"into Dataverse's Terms of Access field. Is that so? I think that works best since the Confidentiality Declaration field, in the collapsed "Addition