HTML markup in Description field

[Martin Schorcht]

Hi all,

Dataverse allow HTML formatting (as per the supported tag list: <p>, <ul>, <li>, <strong>, etc.) in the dataset Description field, since this is rendered nicely on the dataset landing page.

However, when a dataset is exported to DataCite/Datacite XML (and consequently harvested by OpenAIRE), the HTML markup is not stripped or converted to plain text — it's passed through as-is into the <description> element. As a result, downstream consumers like OpenAIRE display the raw tags literally, e.g.:

<p>Some description text</p>

instead of just:

Some description text

Are there any plans (e.g. as part of the ongoing DataCite/OpenAIRE export alignment work, #12281) to strip or convert HTML to plain text when exporting to Datacite?

thx & best,
martin

[Julian Gautier]

Hi martin,

Thanks for pointing this out! I wasn't aware of this when I was helping scope what's described in https://github.com/IQSS/dataverse/issues/12281, so we weren't planning on addressing this difference between the two exports.

When you see the exports, are you using the latest released version of Dataverse, v6.11? I ask because when I test things on Demo Dataverse (https://demo.dataverse.org), which is using v6.11, the differences are a little different than how I think you've described them.

With HTML tags in the description field of the test dataset at https://demo.dataverse.org/dataset.xhtml?persistentId=doi:10.70122/FK2/LBZENG&version=1.1:

• The DataCite/Datacite export, which is what's sent to DataCite, includes that description text but with HTML entity codes instead, like this:
  • <description descriptionType="Abstract">&lt;p&gt;Description1 &lt;b&gt;This should be bold&lt;/b&gt;&lt;u&gt;This should be underlined&lt;/u&gt;&lt;i&gt;This should be italicized&lt;/i&gt;</description>
  • See https://demo.dataverse.org/api/datasets/export?exporter=Datacite&persistentId=doi:10.70122/FK2/LBZENG
• The OpenAIRE/oai_datacite export, which is what repository managers tell OpenAIRE's systems to harvest, include the HTML tags like this:
  • <description descriptionType="Abstract"><p>Description1 <b>This should be bold</b><u>This should be underlined</u><i>This should be italicized</i></description>
  • See https://demo.dataverse.org/api/datasets/export?exporter=oai_datacite&persistentId=doi:10.70122/FK2/LBZENG

It looks like for some Dataverse software version released in 2024, the HTML tags weren't present in the DataCite exports but were present in the OpenAIRE exports. At least that's what I wrote in 2024 in the GitHub issue comment at https://github.com/IQSS/dataverse/issues/5889#issuecomment-2025510299. So sometime after that, things changed, but I don't know when or why and I haven't found any discussion yet that tells me when or why.

Although I think it's in scope of the overall goal described in https://github.com/IQSS/dataverse/issues/12281, I'd vote for not addressing this as part of that work, since the work has already started, it's already pretty involved work that's planned for an upcoming Dataverse release, and I think addressing this HTML tag issue will involve us doing some investigation.

I think we should learn why the exports handle HTML tags in these ways.

• Did we intend the two exports to handle HTML tags in different ways? Or is it a bug?
• Would the folks working on OpenAIRE's systems prefer that the oai_datacite export convert the HTML tags to HTML entity codes, like the "Datacite" export does? Or should the oai_datacite export remove the HTML tags and not replace them with anything?
• Are we okay with how the HTML tags appear on the work pages on DataCite Commons? In the dataset in Harvard Dataverse at https://doi.org/10.7910/DVN/GLQWAH, there are <br /> tags in the description field, which are converted in the dataset's "Datacite" export (https://dataverse.harvard.edu/api/datasets/export?exporter=Datacite&persistentId=doi:10.7910/DVN/GLQWAH) to &lt;br /&gt;. Then on the DataCite Common's page at https://commons.datacite.org/doi.org/10.7910/dvn/glqwah, they're displayed as <br /> tags.

I'd say we could figure out how it should work as soon as possible, and then make the needed changes as part of a separate effort after the work being tracked in https://github.com/IQSS/dataverse/issues/12281 is done and included in an upcoming Dataverse release.

I was thinking of opening a new GitHub issue about this, but I think this has been discussed recently, maybe because of work on the Modern UI, and there might already be a GitHub issue about this that I just haven't found yet. My developer colleagues might know for sure.

Cheers,

Julian

Julian Gautier (he/him)

Product Research Specialist, IQSS

Interested in helping test Dataverse? Sign up for user experience research

[Martin Schorcht]

Thank you, Julian, for your explanations.

We are using Dataverse 6.10.

By “OpenAire downstream,” I didn't mean the export (from the  Export Metadata button), but rather the content displayed on the OpenAire website, for example:

https://explore.openaire.eu/search/dataset?pid=10.7910%2Fdvn%2Fglqwah

If you click on the “Harvard Dataverse” dropdown at the top, it says “Data sources: Datacite.” That's why I assumed that OpenAire harvests the metadata from Datacite.

And yes, the metadata getting from dataverse over "Export Metadata" button differs. DataCite/Datacite export is xml-escaped and OpenAIRE/oai_datacite is not.
But in my first post, I was referring more to the content that is exported to DataCite.

And now comes the part that's confusing—at least for me  😂

Could it be that Dataverse is escaping “too much”?

For example looking on this dataset:
https://commons.datacite.org/doi.org/10.71830/6phwrq

When I download “DataCite XML” from DataCite UI the descriptions are "double-escaped": 
<description descriptionType="Abstract">&amp;lt;p&amp;gt;
&amp;lt;b&amp;gt;Dataset was published ...

And the "DataCite JSON":
description
'&lt;p&gt;\n&lt;b&gt;Dataset was published ...

In contrast, when I look at the following dataset, bold text is rendered correctly in the DataCite UI:
https://commons.datacite.org/doi.org/10.5281/zenodo.6861793
DataCite XML: 
<description descriptionType="Abstract">&lt;p&gt;&lt;strong&gt;Abstract: &lt;/strong&gt;Feudalism ...
DataCiteJSON:
description
'<strong>Abstract: </strong>Feudalism ...

Also there are no html tags rendered in the corresponding OpenAire:
https://explore.openaire.eu/search/publication?pid=10.5281%2Fzenodo.6861794 

I also found a post where Martin Fenner mentioned that the DataCite UI only processes certain HTML tags:
https://groups.google.com/g/datacite-metadata/c/Di5TSstfafU?utm_source=chatgpt.com
On 2017.08.14 he wrote:
When DataCite processes the DataCite XML, e.g. for display in DataCite Search, we are a little bit more flexible and allow the following tags: "strong em b i code pre sub sup br" 

Perhaps Dataverse could transform the HTML markup it allows in that way? E.g. replace `<li>` elements with indented text in combination with `<br>`, or something like this.

best, martin

[Julian Gautier]

Ah, thanks for clarification martin! I forgot that OpenAIRE also harvests from DataCite. Then it makes sense for us to tell Dataverse to handle HTML markup the same ways when creating its DataCite and OpenAIRE exports, right? If so, I think that makes things a little simpler.

Could you open a GitHub issue about this and about the recommendation you shared today?

[Martin Schorcht]

Hi Julian,

ok, I have opened two issues.

One bug issue for the double-escaping:
https://github.com/IQSS/dataverse/issues/12624

And one feature request that ensures the HTML content exported by Dataverse complies with the HTML tags allowed by DataCite:
https://github.com/IQSS/dataverse/issues/12625

By the way: When I look at the DataCite/Datacite export you showed in your first reply, using cURL also results in the “descriptions” being double escaped:
curl "https://demo.dataverse.org/api/datasets/export?exporter=Datacite&persistentId=doi:10.70122/FK2/LBZENG"
-> response: 
<description descriptionType="Abstract">&amp;lt;p&amp;gt;Description1
&amp;lt;b&amp;gt;This should be bold&amp;lt;/b&amp;gt;&amp;lt;u&amp;gt;This should be underlined&amp;lt;/u&amp;gt;&amp;lt;i&amp;gt;This should be italicized&amp;lt;/i&amp;gt;</description>

However, when I view it in the browser, what you already wrote appears:
description descriptionType="Abstract"> &lt;p&gt;Description1 &lt;b&gt;This should ...
It seems that an unescape happens when viewing it in the browser.

best, martin

[Gautier, Julian]

Ah! That's really helpful to know! Thanks for pointing that out!

I'll leave more comments and questions in the two GitHub issues you made. Thanks for opening them!