Dataverse configuration

[Youn Noh]

When you install Dataverse, do you need to clone the repository containing the source code separately in order to configure and customize the application? I am specifically interested in metadata customization. I did not run the installation myself, so I don't know exactly how it was done, but I can't find the path to the text files for the metadata blocks on the server that is hosting our application in order to copy and modify them. Thanks in advance for any tips on this. My apologies if this was already covered.

[Philip Durbin]

Hi, no you definitely don't need to clone code and compile it.

Ideally you will not modify the metadata blocks that are shipped with Dataverse, unless you want to create a pull request to improve them.

Rather, you are welcome to create your own custom metadata block for your needs. Harvard Dataverse, for example, has some custom metadata blocks over at https://github.com/IQSS/dataverse.harvard.edu

Hopefully you've found out docs on this: https://guides.dataverse.org/en/6.2/admin/metadatacustomization.html

It's a complicated subject. Please keep the questions coming!

Thanks,

Phil

p.s. On your server you could look for a file call citation.tsv, one of the many metadata blocks we ship with Dataverse: https://guides.dataverse.org/en/6.2/user/appendix.html#supported-metadata

On Fri, Apr 26, 2024 at 1:45 PM Youn Noh <yn...@yale.edu> wrote:

When you install Dataverse, do you need to clone the repository containing the source code separately in order to configure and customize the application? I am specifically interested in metadata customization. I did not run the installation myself, so I don't know exactly how it was done, but I can't find the path to the text files for the metadata blocks on the server that is hosting our application in order to copy and modify them. Thanks in advance for any tips on this. My apologies if this was already covered.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-commu...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/d4e62fb5-c793-4ffd-b733-0e4c37d51990n%40googlegroups.com.

--

Philip Durbin
Software Developer for http://dataverse.org
http://www.iq.harvard.edu/people/philip-durbin

[Bob Rice]

Hi Phil,

The file, citation.tsv, doesn't exist on our servers.  Could that be due to answering "false" to a configuration item during installation?   If so, could you point us to instructions to enable it?

Best,

Bob

[Philip Durbin]

No, answering false to anything shouldn't matter. It sounds like you're using the interactive installer (install.py). Is that right? If so you probably got it from dvinstall.zip, which also contains citation.tsv. When you unzip it, you should see this:

inflating: dvinstall/data/metadatablocks/citation.tsv 

I hope this helps. Please feel free to pop in https://chat.dataverse.org if that's easier! 

To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/b0da6813-fea6-4971-a203-8da1ef767c4en%40googlegroups.com.

[Rice, Robert]

Thank you, Phil. 

 

I’ve found file on our Ansible installation server. 

 

I’ve been looking through the v5.13 documentation, and the link you provided earlier, but it’s still not clear to me how to implement this.

 

Is there a best practice on where to inflate this Zip file and an example of how to make the application aware of it?

 

Thank you,

Bob

--
You received this message because you are subscribed to a topic in the Google Groups "Dataverse Users Community" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/dataverse-community/rGkaHuzA2x4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to dataverse-commu...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/CABbxx8E24FS_En_baj%3Dy%3DUcsOvAJphqnnDr7wCpAbKERukox6w%40mail.gmail.com.

[Philip Durbin]

Well, I might be a little turned around.

The goal is still to do some metadata customization, right? Here are the docs on loading a metadata block you create: https://guides.dataverse.org/en/5.13/admin/metadatacustomization.html#loading-tsv-files-into-a-dataverse-installation

From time to time we do release updates to existing metadata blocks (such as the citation block). We explain how to update them in release notes but we also document it here: https://guides.dataverse.org/en/5.13/admin/metadatacustomization.html#reloading-a-metadata-block

So that's how you make Dataverse aware of these tsv files we use for metadata blocks.

Is this helping? Please keep the questions coming!

Again, I don't recommend editing citation.tsv unless you plan to share your changes (make a pull request). You could try loading one of our experimental metadata blocks: https://guides.dataverse.org/en/5.13/user/appendix.html#experimental-metadata . However, this is better done on a test server than production because it can't easily be removed ( https://github.com/IQSS/dataverse/issues/9628 ). That's why we suggest setting up a dev environment to play around with metadata blocks: https://guides.dataverse.org/en/6.2/admin/metadatacustomization.html#setting-up-a-dev-environment-for-testing

I hope this helps!

Phil

To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/MW4PR08MB81996234629D02408A24250B9E1B2%40MW4PR08MB8199.namprd08.prod.outlook.com.

[Rice, Robert]

Hi,

 

My task would be to make citation.tsv available to someone, Youn Noh, so that they can look into metadata on our system.  The citation.tsv file doesn’t exist on our test server, so my task is to get it there and configured correctly.  If the directions for that are in the “loading-tsv-files*” section, then I can follow those directions.

 

If there’s more to it than that, please let me know.

 

Thanks again,

To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/CABbxx8HvmQe0u9mbpJXRYOJdTbeUBvS7J8X1-OL9vNyYZ2GM7g%40mail.gmail.com.

[Philip Durbin]

Thanks, for the citation.tsv file as of Dataverse 5.13 (which I believe you're running?), I would recommend downloading it from this 5.13 tag: https://github.com/IQSS/dataverse/blob/v5.13/scripts/api/data/metadatablocks/citation.tsv

You can also find citation.tsv linked from https://guides.dataverse.org/en/5.13/user/appendix.html#supported-metadata but the link goes to the latest version in the master branch which I believe has been updated since 5.13.

To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/MW4PR08MB8199B25F9CFF53138F3162899E1B2%40MW4PR08MB8199.namprd08.prod.outlook.com.

[Youn Noh]

I've downloaded and edited citation.tsv by replacing the lines starting from 141 with values from ISO 639-3. I added a value in the identifier column containing the ISO 639-3 language code and added codes from previous versions of ISO 639 after the value in the displayOrder column. I changed the sort order to alphabetize by the Value column. An example is copied below:

ORIG:

language Malayalam 101 mal ml

NEW:

language Malayalam mal 4016 mal mal ml

Given the column definitions provided here, I am not sure about the purpose of the values in columns following displayOrder. Are they variants? Should I eliminate duplicates in cases where previous versions of ISO 639 contained the same code?

For NEW to replace ORIG, would a Dataverse administrator just need to run the following command?

curl http://localhost:8080/api/admin/datasetfield/load -H "Content-type: text/tab-separated-values" -X POST --upload-file citation.tsv

Ideally, I would also like to implement the following and would appreciate any quick tips or warnings.

1. Add a free text language field to be used when the language name is not in the controlled vocabulary list. Index this field separately.

2. Add a tip encouraging the contributor to use a value in the controlled vocabulary field instead if possible.

I am assuming it's fine to make the changes described above locally and reapply if we upgrade after a new release. Please let me know if I am wrong about this.

  

Thanks very much for any help with the above.

[Gautier, Julian]

Hi,

The columns following that displayOrder column are ISO abbreviations. I'm not sure how they're used. Maybe they appear in some dataset metadata that Dataverse exports? I think it's somehow related to OAI-PMH harvesting.

For everyone following this thread, here's a screenshot of those columns that follow the displayOrder column and some of those values:


In the GitHub issue at https://github.com/IQSS/dataverse/issues/8243, there's some recent relevant discussion, and folks there might know more.

About adding a free text language field to be used when the language name is not in the controlled vocabulary list, this sounds similar to what we've had to do in other metadata blocks that ship with Dataverse, like the Life Sciences metadata block.

Here's a screenshot of one of those fields in the Life Sciences metadata block:


You wrote that that field should be indexed separately. Could you write more about why you wrote this?

Relatedly, the external controlled vocabulary functionality can let contributors enter their own values when none of the values in the vocabulary are appropriate. In that GitHub issue I mentioned, there's some discussion about using that functionality. That functionality requires that the vocabulary be hosted externally and in a certain way. There's more info about that at https://guides.dataverse.org/en/latest/admin/metadatacustomization.html#using-external-vocabulary-services.

About adding a tip encouraging the contributor to use a value in the controlled vocabulary field instead if possible, in the Metadata text guidelines, we discourage doing this because for the most part, everyone who hovers or clicks on those blue question mark icons can see the text in those tooltips, and it isn't relevant for everyone, like folks looking for data.

We haven't yet applied these guidelines to all fields that ship with Dataverse, so we still see tooltip text like this, like that "Other Design Type" field in the Life Sciences metadata block.

If you haven't seen it yet, please check the rest of those Metadata text guidelines, too.

The dataset template feature lets us add instructions for each field, and only contributors see these instructions. Could you use dataset templates? That feature is described at https://guides.dataverse.org/en/latest/user/dataverse-management.html#dataset-templates.

I hope this is helpful!

All best,
Julian

Julian Gautier (he/him)

Product Research Specialist, IQSS

Interested in helping test Dataverse? Sign up for usability testing

ᐧ

[Philip Durbin]

Hi Youn Noh,

As Julian mentioned, there's work going on right now with citation.tsv. I just left a comment at https://github.com/IQSS/dataverse/pull/10481#pullrequestreview-2036232676 to let the developer (Leonid) know about it, but I'll ping him in chat as well.

Please take a look at that pull request when you get a chance and please feel free to leave comments on it.

Thanks,

Phil

--

You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-commu...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/CANK3XXwWEUBuX0W3sfGVBmAB_S7VyAvC%2BEpy0g6Gvog4HgwC%2Bw%40mail.gmail.com.

[Youn Noh]

Hi, Julian. Thanks for getting back to me!

It sounds like codes from previous versions of ISO 639 should be listed as separate columns even if duplicated if the purpose is to maintain compatibility with OAI providers and harvesters that use Dublin Core or other metadata standards that refer to previous versions. It's great to learn that Dataverse is considering supporting ISO 639-3, although mapping might require some work. Do the columns for previous versions need to appear in a particular order? Are the mapped versions in the tsv file predetermined, and does the order need to be maintained for compatibility?

I'll take a look at the Life Sciences metadata block and tsv file.

I was mainly concerned about the effect of mixing controlled and uncontrolled values on indexing and searching. When allowControlledVocabulary is True, are uncontrolled entries permitted, or are they captured in a separate text box and indexed with the controlled values? (I still need to examine the Solr schema, so if the answer to this question should be apparent there, please disregard this question.)

If we use the ISO 639-3 language code as an identifier, will that seamlessly replace the value taken from the Value column as an identifier when we reload?

To what extent is the external controlled vocabulary functionality experimental? Are extensive resources required for maintenance? The documentation suggests caution.

I will definitely review the Metadata text guidelines. The templates feature sounds like it has the appropriate functionality, particularly if we can provide a base template for others to use in creating their own templates.

Thanks again!

Phil, will do! Thanks for the heads up.

[Philip Durbin]

There are two types of controlled vocabularies in Dataverse. The older, original type stores the values in the database. They are inserted from citation.tsv and other TSV files for metadata blocks.

The second type is external controlled vocabularies (the link you found). In this case the values are stored in a web service that has an API for Dataverse to call in to. This second kind is flagged as experimental, yes.

--

You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-commu...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/a6285cfe-0d23-4245-9d3e-db4b69edae9dn%40googlegroups.com.

[Youn Noh]

Phil, thanks for getting back to me on that question. It sounds like we should probably hold off on that feature now, but I do hope we explore it in the future. I also had a question about reloading a metadata block. If we reload citation.tsv with different enumerated values for the controlled vocabulary for the language field and use the ISO language code as identifier values rather than a value derived from the name, I am concerned that will create problems based upon the following quote: "Great care must be taken when reloading a metadata block. Matching is done on field names (or identifiers and then names in the case of controlled vocabulary values) so it’s easy to accidentally create duplicate fields." What do we need to do to have our enumerated values replace the existing values? Thanks very much in advance for any help with this question. Youn

[Philip Durbin]

Yes, you should be concerned about modifying citation.tsv because it's one of the metadata blocks we ship with Dataverse.

I would prefer for you to engage with Leonid on the pull request I mentioned adds ISO codes to citation.tsv. (I pinged him about it yesterday but we've been quite busy.) Can you please leave a comment at https://github.com/IQSS/dataverse/pull/10481#pullrequestreview-2036232676 ? I'm hoping we can get your proposed change into that pull request so that the entire community can benefit from it.

I hope this makes sense! Please keep the questions coming!

Phil

To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/b00e72c6-6246-4258-b5df-a83f50f049b2n%40googlegroups.com.