Importing Nesstar DDI & data into Dataverse 4x

[Janet McDougall - Australian Data Archive]

Hi All

We are testing existing Dataverse APIs/tools to import our archive from Nesstar 4 (DDI 2.5) into Dataverse 4.6.  I have noticed posts relating to 'creating new datasets using DDI' but many  are from around 2 years ago, and related to 'migration' in version Dataverse v3.x.  

Is or has anyone done any work that might be relevant to us?  If so please share your experiences - we will of course document our work as we go for other to use.  We are looking at native APIs using JSON at the moment, and will post some more questions in the next few days.

thanks

PS hope you're all ok after your winter storms, can't imagine that!

[Philip Durbin]

There was a recent issue at https://github.com/IQSS/dataverse/issues/3632 where we talked about importing DDI files but it's really not very well supported and I figured that issue should remain open until the process is better documented and the code is cleaned up a bit. Like you said, it was written for upgrading from DVN 3.x to Dataverse 4, not from migrating from one platform to another. It would be really nice to make it easy for people to migrate their data to Dataverse!

Phil

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse-community@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/9b4ad8d7-271d-42a6-aecc-585b44bbc21c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Philip Durbin
Software Developer for http://dataverse.org
http://www.iq.harvard.edu/people/philip-durbin

[Janet McDougall - Australian Data Archive]

Hi Phil

Well we are working on it...  I was wondering if there is a JSON schema or some doco to assist us in translating the DDI 2.5 to JSON format?  At the moment I'm using the JSON example provided but it would be really helpful to understand what's expected  https://github.com/IQSS/dataverse/blob/v4.3/scripts/api/data/dataset-create-new.json

thanks

Janet

Archivist

[Philip Durbin]

To see the mapping between DDI and Dataverse's native JSON fields, please see the "Dataverse 4.0 Metadata Crosswalk: DDI, DataCite, DC, DCTerms, VO, ISA-Tab" spreadsheet linked from http://guides.dataverse.org/en/4.6.1/user/appendix.html

Here's a direct link: https://docs.google.com/spreadsheets/d/10Luzti7svVTVKTA-px27oq3RxCUM-QbiTkm8iMd5C54/edit?usp=sharing

If you're migrating existing datasets, I assume they already have a DOI or a Handle. This won't "just work" via JSON until this issue is addressed: https://github.com/IQSS/dataverse/issues/3083

That is to say, I believe DDI is a better bet for you, even though it's not well supported yet for your use case of migrating from one platform to another.

You could also play around with harvesting DDI records over OAI-PMH.

Please keep the questions coming!

Phil

--

You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse-community@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/29b1a03f-56b0-481e-b4ad-5a53944189e8%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Janet McDougall - Australian Data Archive]

Hi Phil, yes I found the mapping document, really great resource. I used it yesterday to writ JSON to set the access conditions. Marina will use native API to test if my json structure valid and if the terms can be set this way. That's what I was more interested in was whether we can use the full suite of DDI 2.5 elements in the mapping. If so then we should be able to set up our datasets this way to import them from Nesstar to data verse. We are working on it this morning.
Janet

[Philip Durbin]

Cool. Again, the main gotcha with JSON is that a new DOI will be created, which may or may not be what you want.

On Mar 15, 2017 5:56 PM, "Janet McDougall - Australian Data Archive" <janet.m...@anu.edu.au> wrote:

Hi Phil, yes I found the mapping document, really great resource. I used it yesterday to writ JSON to set the access conditions.  Marina will use native API to test if my json structure valid and if the terms can be set this way. That's what I was more interested in was whether we can use the full suite of DDI 2.5 elements in the mapping. If so then we should be able to set up our datasets this way to import them from Nesstar to data verse.  We are working on it this morning.
Janet

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse-community@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/a3e48460-29b4-4840-88b7-0a009756fa37%40googlegroups.com.

[Vyacheslav Tikhonov]

Dear Janet,

We are (DANS, dataverse.nl) also working on the pilot project for the one of EU countries to get all stuff from Nesstar migrated into Dataverse 4. We're not using DDI for this as it has much more fields than Dataverse and it's difficult to map everything, the idea is to convert all metadata to XML and deposit to Dataverse by using native APIs for necessary fields, and attach extra stuff as files.

It should be probably relevant to you.

Best,

Slava

[Amber Leahey]

Hi Janet, 

You might also be interested in joining the DDI & Dataverse WG that just started looking at Use Cases, including DDI Import features. 

https://ddi-alliance.atlassian.net/wiki/display/DDI4/DDI+Workflows+for+Dataverse 

I think your project would be a great use case and project to highlight to the community. 

Best, 

Amber

[Janet McDougall - Australian Data Archive]

Hi all thanks for the responses. We are testing native API now to move our Nesstar studies into DV4 using json to do the translation. We should be able to map all the Nesstar DDI elements into DV as they are already being used or seem to have database fields to hold them. I'm presuming this work was done as part of the 'export DDI' option?

Anyway we are testing the TERMS elements this morning using json. Please see earlier posts for links to docs I've referenced. I'm using my iPad here so I have limited editing skills.

Marina and I have been using the native API to create a new dataset, based on examples in GitHub create_new_dataset3.json.

I'm also using the DV 4 metadata mapping to DDI, DC, data cite etc spreadsheet I found on GitHub to understand where DDI elements are written to fields in the DB tables. Really great doc!

We've also been looking at the jsonpaser.java code to try to understand how it expects to receive the json object but this will require a lot of trial and error.

QUESTION
do you know of where we can access the JSON schema for the the parser? That way I will know exactly how to format the json code to translate from DDI using the API.

thanks
Janet

[Philip Durbin]

There is no JSON schema for the parser. The JSON format Dataverse uses was designed originally for backing up and restoring datasets we were creating on out demo site when we were developing Dataverse 4 and wanted to drop the database and reimport some datasets without manually entering the data again. It worked fine for this purpose and the format is now used in what we document as the "native" API, in contrast to the SWORD API, which is based on an XML standard (dcterms).

Phil

p.s. Raman has done some nice experiments with JSON Schema in this repo but they aren't used by Dataverse: https://github.com/IQSS/json-schema-test

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse-community@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/a854dcc2-7375-417c-a9de-cad21a26be81%40googlegroups.com.

[Janet McDougall - Australian Data Archive]

Ok thanks Phil, we will work it out and share if successful, will also look at other work you referenced. Our main aim is to be able to populate the relevant db fields, especially the RESTRICTED flag which is set in DDI. If we can do this for most of the Nesstar DDI while creating new datasets in DV4 then our work processes will be much more structured and contained.
Janet

[Philip Durbin]

Ok, I don't know what RESTRICTED means in DDI or why it's ALL CAPS. :)

You may or may not find the following API endpoint helpful. You can pass it a field like "title" and get some information about it: http://phoenix.dataverse.org/api/admin/datasetfield/title

{
  "status": "OK",
  "data": {
    "name": "title",
    "id": 1,
    "title": "Title",
    "metadataBlock": "citation",
    "fieldType": "TEXT",
    "allowsMultiples": false,
    "hasParent": false,
    "parentAllowsMultiples": "N/A (no parent)",
    "solrFieldSearchable": "title",
    "solrFieldFacetable": "title_s",
    "isRequired": true
  }
}

Just http://phoenix.dataverse.org/api/admin/datasetfield with no field at the end will give a list of fields. You could try iterating over all the fields.

Really this is more or less the same information that's in the tsv files, or in their equivalent tabs in the spreadsheet tabs linked from http://guides.dataverse.org/en/4.6.1/user/appendix.html such as "Citation Metadata", "Geospatial Metadata", etc.

I hope this helps,

Phil

On Thu, Mar 16, 2017 at 6:33 PM, Janet McDougall - Australian Data Archive <janet.m...@anu.edu.au> wrote:

Ok thanks Phil, we will work it out and share if successful, will also look at other work you referenced.  Our main aim is to be able to populate the relevant db  fields, especially the RESTRICTED flag which is set in DDI.  If we can do this for most of the Nesstar DDI while creating new datasets in DV4 then our work processes will be much more structured and contained.

Janet

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse-community@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/bd9eb2a4-71d1-48ce-a554-6d4ab9ab4c59%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[julian...@g.harvard.edu]

Hi Janet. We also plan on exposing in the UI how Dataverse fields are mapped to fields in other standards, such as DDI. A github issue's been opened for this work here: https://github.com/IQSS/dataverse/issues/3701

Would seeing the mappings this way be helpful for your use case?

[Janet McDougall - Australian Data Archive]

Hi Julian & all
Any mappings are useful, I'm not exactly sure what you've suggested really means, but I'm replying now in case you have a response for me in the morning ( 9pm here). what I've been doing is looking at fields in the db tables and mapping them to DDI elements listed in the mapping spreadsheet I referred to in earlier posts (great doc) to try and work out where they're being written to. We are using the jsonparser.java thru API to import the DDI metadata into the db tables. For example, in the TERMS table there are several non DDI fields we can populate usefully plus the DDI elements/fields in the example (create_new_dataset3.json).

Our problem now is testing how far we can use the limited example DDI elements in the jsonAPI (create_new_dataset3.json) without following the code too far to be able to expand to using the full Nesstar DDI suite. I am manually coding ADA DDI metatdata pretty much at amDDI element at a time and Marina (web dev) is running the API to test what the parser will accept among other things. There is no obvious reason as to why some elements are compound - several cols in an aray, or just rows, or primitive, not in an array etc. well this is my interpretation of the json code so we will find out if it's wrong as we go. We hope to build a Nesstar DDI to dataverse 4.6 import tool out of this so any assistance or pointers would be really appreciated. Thanks for following this up.
Janet

[Philip Durbin]

An import tool from Nesstar DDI to Dataverse would be great! I hope you open source it. :)

Also, dataset-create-new3.json is probably a good starting point, but I was recently told that there is a JSON file floating around that exercises as many metadata fields as possible. It's just test data so the title is "title" or something. Would it be helpful for me to try to dig this JSON file up?

I believe that dataverse-client-java is farthest along in terms of allowing users to specify as many metadata fields in JSON as possible. That said, this "Full support for Dataset MetaData objects" issue is still open so I'm not sure of the exact state of things: https://github.com/IQSS/dataverse-client-java/issues/6 . The R and Python clients create datasets using XML (SWORD), I believe.

Phil

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse-community@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/7e1407ae-580d-4619-ba2d-dbe01d87c7bd%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Janet McDougall - Australian Data Archive]

hi Phil yes please! re "Would it be helpful for me to try to dig this JSON file up?".  Marina has been testing SWORD to create the dataset container (fish/xxxxxx/ ID etc), then use json java API to populate it - i'm busy on the json side so not sure how that's going at the moment.   We will definitely share our work - it's nice coming to work and hearing from you all...

Thanks

Janet

[Philip Durbin]

Please find "all_metadata_blocks.json" attached. I meant to try it myself but haven't yet. I hope it helps.

Please let me know if it doesn't allow you to create a dataset. I can drop it into my test framework tomorrow.

Thanks,

Phil

--

You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse-community@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/d75a5a20-21be-4180-a46c-220743726a38%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Janet McDougall - Australian Data Archive]

thanks alot Phil! Marina is away today but i will use it to prepare the json code for the API.  will let you know how we go, vice versa if you test it first.  Janet

On Wednesday, 22 March 2017 10:23:42 UTC+11, Philip Durbin wrote:

Please find "all_metadata_blocks.json" attached. I meant to try it myself but haven't yet. I hope it helps.

Please let me know if it doesn't allow you to create a dataset. I can drop it into my test framework tomorrow.

Thanks,

Phil

On Tue, Mar 21, 2017 at 7:14 PM, Janet McDougall - Australian Data Archive <janet.m...@anu.edu.au> wrote:

hi Phil yes please! re "Would it be helpful for me to try to dig this JSON file up?".  Marina has been testing SWORD to create the dataset container (fish/xxxxxx/ ID etc), then use json java API to populate it - i'm busy on the json side so not sure how that's going at the moment.   We will definitely share our work - it's nice coming to work and hearing from you all...

Thanks

Janet

On Wednesday, 15 March 2017 11:54:53 UTC+11, Janet McDougall - Australian Data Archive wrote:

Hi All

We are testing existing Dataverse APIs/tools to import our archive from Nesstar 4 (DDI 2.5) into Dataverse 4.6.  I have noticed posts relating to 'creating new datasets using DDI' but many  are from around 2 years ago, and related to 'migration' in version Dataverse v3.x.  

Is or has anyone done any work that might be relevant to us?  If so please share your experiences - we will of course document our work as we go for other to use.  We are looking at native APIs using JSON at the moment, and will post some more questions in the next few days.

thanks

PS hope you're all ok after your winter storms, can't imagine that!

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.

To post to this group, send email to dataverse...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/d75a5a20-21be-4180-a46c-220743726a38%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Philip Durbin]

Uh oh, out of the box, Dataverse can't handle the all_metadata_blocks.json file I sent yesterday because it uses metadata blocks that are custom to Harvard's installation*. You should expect to see this error if you try to create a dataset with it:

"message": "Error parsing datasetVersion: Can't find type 'mraCollection'"

I stuck the JSON file in my test suite[2], hacked on the JSON file a bit (removing metadatablocks and making other tweaks) and finally got it to work, to create a dataset. I renamed it default_metadata_blocks-4.6.1.json and will attach it here. I'm also attaching a screenshot of what it looks like in the GUI with all the metadata (I clicked the non-citation blocks to expand them). I also exported the dataset as DDI and Dublin Core and will attach those as well.

It might be that some metadata fields aren't exercised. And for all I know some are used incorrectly. Hopefully this is something we can build on, at least. I hope it helps.

Long story short, please give default_metadata_blocks-4.6.1.json (attached) a try. :)

Thanks,

Phil

1. For example: https://github.com/IQSS/dataverse/blob/v4.6.1/scripts/api/data/metadatablocks/customMRA.tsv
2. getDatasetJson in https://github.com/IQSS/dataverse/blob/v4.6.1/src/test/java/edu/harvard/iq/dataverse/api/UtilIT.java#L210

On Tue, Mar 21, 2017 at 10:42 PM, Janet McDougall - Australian Data Archive <janet.m...@anu.edu.au> wrote:

thanks alot Phil! Marina is away today but i will use it to prepare the json code for the API.  will let you know how we go, vice versa if you test it first.  Janet

On Wednesday, 22 March 2017 10:23:42 UTC+11, Philip Durbin wrote:

Please find "all_metadata_blocks.json" attached. I meant to try it myself but haven't yet. I hope it helps.

Please let me know if it doesn't allow you to create a dataset. I can drop it into my test framework tomorrow.

Thanks,

Phil

On Tue, Mar 21, 2017 at 7:14 PM, Janet McDougall - Australian Data Archive <janet.m...@anu.edu.au> wrote:

hi Phil yes please! re "Would it be helpful for me to try to dig this JSON file up?".  Marina has been testing SWORD to create the dataset container (fish/xxxxxx/ ID etc), then use json java API to populate it - i'm busy on the json side so not sure how that's going at the moment.   We will definitely share our work - it's nice coming to work and hearing from you all...

Thanks

Janet

On Wednesday, 15 March 2017 11:54:53 UTC+11, Janet McDougall - Australian Data Archive wrote:

Hi All

We are testing existing Dataverse APIs/tools to import our archive from Nesstar 4 (DDI 2.5) into Dataverse 4.6.  I have noticed posts relating to 'creating new datasets using DDI' but many  are from around 2 years ago, and related to 'migration' in version Dataverse v3.x.  

Is or has anyone done any work that might be relevant to us?  If so please share your experiences - we will of course document our work as we go for other to use.  We are looking at native APIs using JSON at the moment, and will post some more questions in the next few days.

thanks

PS hope you're all ok after your winter storms, can't imagine that!

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsubscribe...@googlegroups.com.

To post to this group, send email to dataverse...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/d75a5a20-21be-4180-a46c-220743726a38%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Philip Durbin
Software Developer for http://dataverse.org
http://www.iq.harvard.edu/people/philip-durbin

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.

To post to this group, send email to dataverse-community@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/ce30667a-8261-4d9a-99d3-39eceee093c0%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Amber Leahey]

Hi Janet and Phil, 

It would be interesting for us if you could capture a comparison of supported DDI fields between Nesstar and Dataverse. There are a lot of DDI fields in Nesstar that I would imagine you will lose in the mapping to Dataverse, but I could be wrong in my understanding of this project. 

It would be great to see some kind of comparison list of fields! And thanks for working on this, I imagine it will be useful and interesting for a lot of people. 

Best, 

Amber Leahey

To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.

To post to this group, send email to dataverse...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/d75a5a20-21be-4180-a46c-220743726a38%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Philip Durbin
Software Developer for http://dataverse.org
http://www.iq.harvard.edu/people/philip-durbin

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.

To post to this group, send email to dataverse...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/ce30667a-8261-4d9a-99d3-39eceee093c0%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[julian...@g.harvard.edu]

Thanks Phil! That's really helpful!

I'm also attaching a screenshot of what it looks like in the GUI with all the metadata (I clicked the non-citation blocks to expand them)...

It might be that some metadata fields aren't exercised. And for all I know some are used incorrectly. Hopefully this is something we can build on, at least. I hope it helps.

I compared the fields in the screenshot to the fields that exist in Harvard Dataverse now. These fields in the citation block are missing from the screenshot and in the DDI and JSON that you attached:

• Language
• It is in the JSON metadata that Harvard Dataverse exports; not in the DDI that Harvard Dataverse exports
• Topic classification
• It is in the JSON and DDI metadata that Harvard Dataverse exports

Also, this might be an aside, but Logo URL field (in the producer and distributor fields) just shows the text "Logo URL". I mention it only because I haven't found an issue for this existing behavior. (Problems with both fields were discussed in the closed issue #207.)

Amber, would a spreadsheet with this comparison between Dataverse supported DDI and Nesstar supported work? (Also, Harvard Dataverse has harvested from ICPSR, right? So would it be helpful to also compare ICPSR's DDI, maybe in the same spreadsheet?)

[Janet McDougall - Australian Data Archive]

 Hi Amber - I will complete this mapping but from what I've looked at it seems the full suite ADA uses is there.  Note that ADA doesn't use the full Nesstar DDI suite which is a mix of DDI 2.1 & 2.5 from my understanding.  

[Janet McDougall - Australian Data Archive]

Hi All

Thanks Phil for testing & Julian & Amber for your comments.  Marina is away sick so we haven't been able to run the json API but i will go through your material and with comments and get back to you for tomorrow.

Janet

[Janet McDougall - Australian Data Archive]

Hi All

i've attached an amended version or your Metadata Crosswalk.  I've rushed it a bit so you will have it (this morning...) but it should give you an idea of what we use.  Terms of Use tab is what i've worked on the most.  The ADA/Nesstar DDI info is at the far right cols in each tab - Headings: 'ADA Nesstar Notes' & 'Mandatory'.

Thanks

Janet 

[Janet McDougall - Australian Data Archive]

Note, Nesstar DDI template allows extensive use of DDI elements, I've indicated which ones ADA uses.

[Amber Leahey]

Thanks for this Janet. Apologies for the delay. I've managed to upload this into a Google Spreadsheet for editing. Also added a tab for "Tabular Variable Metadata" as I think this would also need to be migrated from yours and others Nesstar systems at some point.  At the variable-level this would need considerable development since there is no mapping or support yet for some of these fields but I think it will be helpful to improve the workflows for adding metadata at the variable level in Dataverse. For now I've kept them together with your mapping spreadsheet. 

Here is the link to the Cross-walk which includes all the tabs for Study-level migration (as provided by you) as well as the new tab for variable-level metadata using DDI: https://docs.google.com/spreadsheets/d/11Fiy0huoEKzOmJS-hgB-Sq8KOCrsyHY075Yrdkg3U8M/edit#gid=1021946509 

I'll continue adding to this and try to review your mapping to see if this fits with our understanding / metadata as well. 

Thanks again, 

Amber

[Andreas Loibl]

Hello, I've got a similar problem: porting nesstar to dataverse.

what's the current state? is there any best-practice round? Maybe some kind of XML translator, etc?
Any help / link / would be very appreciated.

[Janet McDougall - Australian Data Archive]

Hi we are working towards running a script developed by our web programmer Marina, mapping the DDI to JSON and using API to import into Dataverse. ADA Director Steve McEacern and Marina will work with the dataverse team on how to integrate and/or make any work we do available. Steve can provide more specifics on this.

Amber, I've made some updates to the DDI to Dataverse mapping so I will update the google spreadsheet tomorrow - I now have a better understanding of dataverse since mapping them earlier in the year.

[julian...@g.harvard.edu]

This is great! Thank you Janet. I was going to try answering some questions you had about the crosswalk, but I'll wait until the google spreadsheet is updated.

[Janet McDougall - Australian Data Archive]

Hi Julian

sorry, i didn't get a chance to update the spreadsheet today, but basically I'm slightly revising ADA DDI elt mappings to TERMS  -  the non-DDI fields 'terms of use' and 'terms of access', now I'm more familiar with Dataverse.  I am finding both terms very open to interpretation ie the options are license CCO (waiver) or 'terms of use', but it is not clear whether this field is for a license, parts of a license, usage text, links to DUA or other.  Same for 'terms of access', this could incorporate a license agreement, instructions on how to access or apply for access, links to application forms etc.  This may have been the point of creating these fields, so the ADA mappings may not work for others.

I also understand DataTags will be implemented at file level soon (https://groups.google.com/forum/#!topic/dataverse-community/FJaVCuVzkKM) so I am concerned this will increase the variation in use of these terms.

If we want to map these fields to other standards then it may be difficult to identify exactly what is represented ie terms, license, DUA etc.  I may be taking this too far and have moved from the purpose of their design.   Go ahead with your questions, will check on Monday.  Sorry for all my use of etc today.

Thanks

Janet

[julian...@g.harvard.edu]

Hi Janet,

From the spreadsheet, it looks like you plan to put the metadata that ADA maps to the DDI element confDec—info about submitting a "Request for Access to General Datasets"—into Dataverse's Terms of Access field. Is that so? I think that works best since the Confidentiality Declaration field, in the collapsed "Additional Information" section, isn't as prominent as the Terms of Use field is.

It's difficult to map the Terms of Use and Terms of Access fields (which I think were broadly-defined to make it easier for depositors to enter terms metadata) to most of the available DDI Codebook elements: citReq, conditions, confDec, contact, deposReq, disclaimer, restrctn, specPerm. I think only "Conditions" is defined broadly enough for the various things a depositor might put in Terms of Use and Terms of Access, or to map a Creative Commons license to. I didn't think so, but it seems like it would be appropriate to map CC licenses to "Copyright," too (judging from this discussion on the DDI's ticketing system about the use of CC licenses in DDI's Copyright element). Terms of Use and Terms of Access are mapped to DDI's useStmt and dataAccs elements, which the schema doesn't allow, so maybe both should go in Conditions?

The other standards we map or are working on mapping Dataverse metadata to don't seem to have terms metadata as granular as DDIs. Are there any standards with similarly granular elements?

Thank you for your work! I hope this helps,

Julian

[julian...@g.harvard.edu]

Hi Janet. I wrote a doc describing the changes needed, as I understand them, for Dataverse to produce DDI xml that validates against the Codebook 2.5 schema. It proposes changes to how Dataverse Terms metadata is mapped to DDI. I'm hoping you can take a look and share what you think when you get the chance. Thanks!

Julian

[Janet McDougall - Australian Data Archive]

hi Julian

I apologise for the delay in my response(s) - we have so much on!  The proposed changes to copyright are timely as i was planning on  building a custom citation metadata block if needed, but it is preferable to use standard Dataverse metadata blocks.  When do you think copyright elt will be incorporated?

I will work through the spreadsheet & make those updates i've mentioned, as my initial mappings were just for me to get a general feel and I've since found some inconsistencies.

I have been wondering in a general way why avlStatus wasn't used for fileaccessrequest, and as you mention the <usestmnt> elts weren't used for termsofaccess?

<dataAccs>

<setAvail>

<avlStatus>Availability Status</avlStatus>

</setAvail>

Thanks, I appreciate the docs you've put together, v useful.

Janet

[Gautier, Julian]

Hi Janet,

No worries about the delay, and thanks for looking over the docs. We'll have to get back to you about how we might prioritize changes to the DDI mapping. Could you share any deadline(s) your team is working with?

I have been wondering in a general way why...  as you mention, the <usestmnt> elts weren't used for termsofaccess?

Would it be more appropriate for the <setAvail> elements (accsPlac, origArch, avlStatus, collSize, complete) to be used for termsofaccess?

Julian

--
You received this message because you are subscribed to a topic in the Google Groups "Dataverse Users Community" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/dataverse-community/qsY8swD9Hh0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse-community@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/b2823a3c-49c8-4420-b332-507a0d7f518f%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Julian Gautier

Product Research Specialist, IQSS

760-810-9726

[Janet McDougall - Australian Data Archive]

HI Team

Dataverse has not included DDI version  (2.1.6.1-3) elements in the Citation Block as you handle versioning under the Versions tab.  As ADA is migrating to Dataverse from an existing archive, we have used the DDI version elements.  We have been using a NOTES (mapped as DDI 2.6) field to deal with existing Version metadata while migrating data into Dataverse, but i'm wondering if there have been any thoughts on how to deal with data sets that are already several versions along before migrating into Dataverse? If we want to export or migrate any data in the future there is no mapping to identify NOTES as being related to previous DDI versions, as the NOTES field may also hold other 'Notes about a dataset'.  - see  below from https://docs.google.com/spreadsheets/d/10Luzti7svVTVKTA-px27oq3RxCUM-QbiTkm8iMd5C54/edit#gid=0/

DDI 2.5 Dataverse NOTES 2.1.6.1 version Version (Version Text) Versioning is automatic in Dataverse, under Versions tab. WHEN DEALING WITH NOT FIRST VERSION MIGRATION PROCEDURES TO COME 2.1.6.1 version (date attribute) Version (Version Date) Versioning is automatic in Dataverse, under Versions tab. 2.1.6.2 verResp Version Responsibility Contributor to the version is automatically listed under Versions tab in Dataverse. 2.1.6.3 notes Version Notes Version changes are automatically listed in Versions tab in Dataverse.

To post to this group, send email to dataverse...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/b2823a3c-49c8-4420-b332-507a0d7f518f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[julian...@g.harvard.edu]

I'm surprised there isn't a github issue for this, since it was added as an unresolved question in the Dataverse 4.x metadata crosswalk. If there is, someone let me know or feel free to consolidate/close one.

I created an issue, https://github.com/IQSS/dataverse/issues/4570, and added why I think this is important. Please let me know what you think.

[Janet McDougall - Australian Data Archive]

hi Julian

Thanks for creating the issue - I wasn't sure where else the issue was logged.  As I said, for the moment we're using one of the NOTES fields.

Janet

[Philip Durbin]

Hi all, I just wanted to jump in here to say that there is now a GitHub issue to track the desire to import NESSTAR DDI into Dataverse: https://github.com/IQSS/dataverse/issues/4593

This thread is getting quite long so please feel free to comment there if you've had success or failure with this task.

Thanks,

Phil

Janet

HI Team

To unsubscribe from this group and all its topics, send an email to dataverse-community+unsubscribe...@googlegroups.com.

To post to this group, send email to dataverse...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/b2823a3c-49c8-4420-b332-507a0d7f518f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Julian Gautier

Product Research Specialist, IQSS

760-810-9726

--

You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse-community@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/80506f06-74a9-4587-8f70-fbcbe724ff5a%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--