SWORD access failed - not authorized

[Michele Dolfi]

Hi everyone,

I just started trying out Dataverse, and I'm currently blocked with the APIs.

My final goal is to download some file with a DOI, but I understand this is not possible because only the Dataset has a DOI, so I can coarse grain the goal and first get the id from the permanent Dataset and then download the files (as I read on some previous post).

It seems that the only way to get the ID is currently via the SWORD API. The problem, is that overtime I try to access my Dataset I get a "not authorized" error. What am I doing wrong?

Note that the system manages to authenticate me, because I see my name in the output.

The API call that I'm trying out is:

https://dataverse-demo.iq.harvard.edu/dvn/api/data-deposit/v1.1/swordv2/edit/study/doi:10.5072/FK2/YUKBHG

Return:

User is not authorized to retrieve entry for doi:10.5072/FK2/YUKBHG

Thanks a lot,

Michele

[Michele Dolfi]

I have the impression this could be a wider problem with my account.

Something similar happens when I link my Dataverse account to the OSF.io account. The authentication is fine, but I get the message that my account is empty "There are no dataverses, datasets, or files associated with the connected account".

Could the two issues be related?

[Condon, Kevin]

Hi Michele,

To use the APIs in general, you need to pass an API token from a valid account wit sufficient permissions along with the request:

curl -u $API_TOKEN: https://$HOSTNAME/dvn/api/data-deposit/v1.1/swordv2/edit/study/doi:TEST/12345

http://guides.dataverse.org/en/4.1/api/sword.html#display-a-dataset-atom-entry

You can find your API token by logging in, click on your name in the upper right, account information, then edit account, API token.

Regards,

Kevin

[Michele Dolfi]

Hi,

As I wrote, the authentication with API key seems to work, in the error message I actually see my name.

But is seems I don't have permissions to download my datasets. Is this normal? I also tried with public datasets from somebody else (that I can get without API key via the native API), and still I cannot access it via SWORD.

--
You received this message because you are subscribed to a topic in the Google Groups "Dataverse Users Community" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/dataverse-community/G9TMVnhab5A/unsubscribe.
To unsubscribe from this group and all its topics, send an email to dataverse-commu...@googlegroups.com.
To post to this group, send email to dataverse...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/8552BDDD3DDE294C995A0182F32F976F0168893EF1%40HARVANDMBX02.fasmail.priv.
For more options, visit https://groups.google.com/d/optout.

[Condon, Kevin]

After some experimentation, it looks like you need admin permissions on the dataverse that contains the dataset for this to work. Being admin on the dataset alone won't do it.

I am not sure why this is a requirement other than when SWORD was first implemented it was dataverse-based to support integration with a journal publishing system and I believe the workflow required the user to be a dataverse admin.

Our SWORD API developer will be back from vacation tomorrow so he should have more information but it looks like for now you would need to create your own dataverse, dataset, and as an admin request that info.

Kevin

[Michele Dolfi]

Thanks a lot, with my own dataverse it works.

Is it common practice to create a dataverse per researcher?

Michele

[Condon, Kevin]

Yes, we have many individual scholar dataverses but also many based on project or organization. Generally, a logical grouping of datasets.

An example scholar's dataverse: https://dataverse.harvard.edu/dataverse/king
An example organization's dataverse: https://dataverse.harvard.edu/dataverse/mra

---

From: dataverse...@googlegroups.com [dataverse...@googlegroups.com] on behalf of Michele Dolfi [dol...@phys.ethz.ch]
Sent: Wednesday, September 02, 2015 3:30 PM
To: Dataverse Users Community
Subject: Re: [Dataverse-Users] Re: SWORD access failed - not authorized

[Philip Durbin]

Hi Michele!

Thank you for reporting this bug with the Dataverse SWORD API. If I had anticipated your problem I probably would have pushed to work on https://github.com/IQSS/dataverse/issues/1070 *before* we shipped Dataverse 4.0.

It was easy for me to replicate your problem so I just created an issue to track it:

SWORD: not authorized when dataset is in root dataverse - https://github.com/IQSS/dataverse/issues/2495

The ability to create datasets in the root dataverse is a new feature of Dataverse 4.0 and it's "on" for https://dataverse.harvard.edu . It's currently "off" for https://apitest.dataverse.org .

I don't see any particular guidance in the User Guide at http://guides.dataverse.org/en/4.1/user if it is recommended to create datasets in the root dataverse or not but I agree with Kevin putting that datasets in dataverses is a good way to organize them. Again, "Who can add to this dataverse?" for the root dataverse is up to the person installing Dataverse to configure. Here's a screenshot (which I'll also attach): http://guides.dataverse.org/en/4.1/_images/dv3.png . For the geeks among us, here's a related script for how the apitest server is set up: https://github.com/IQSS/dataverse/blob/v4.1/scripts/search/tests/grant-authusers-add-on-root

Sorry to hear about your trouble! Thank you for using Dataverse!

Phil

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-commu...@googlegroups.com.

To post to this group, send email to dataverse...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/8552BDDD3DDE294C995A0182F32F976F0168894317%40HARVANDMBX02.fasmail.priv.

For more options, visit https://groups.google.com/d/optout.

--

Philip Durbin
Software Developer for http://dataverse.org
http://www.iq.harvard.edu/people/philip-durbin

[Sherry Lake]

I would like to restart this conversation.

UVa Dataverse has decided to go with the structure that only "dataverseAdmin" can create dataverses. All other UVa users (logged in via Shib) can create datasets in any dataverse. So "dataverseAdmin" is the admin of all dataverses. [NOTE: The decision to structure things this way was based on many conversations with our security and IRB folks, so reversing this decsion wouldn't be easy.]

Because of our structure our users cannot use the COS/OSF-Dataverse connection (users are not admins of any dataverse). I have tested the COS/OSF-UVA Dataverse connection as "dataverseAdmin" and am able to move files from OSF to our UVA Dataverse and publish. So the problem is with permissions. [This is the same observed problem with dataverse.harvard with datasets at the root level.]

So where do we go from here? I have the Center for Open Science willing to update their code, but it looks like until issue: https://github.com/IQSS/dataverse/issues/1070 is resolved, there isn't a solution. Is there any way to get things worked out or investigated on the Harvard Dataverse side?

Probably not something easily done.... BUT could there be an API solution that when using the API-token, the "My Data" (dataRelatedToMe) feature could return a list of datasets belonging to that user (both published and unpublished)?

Thanks for listening.
Sherry Lake

On Thursday, September 3, 2015 at 10:58:46 AM UTC-4, Philip Durbin wrote:

Hi Michele!

Thank you for reporting this bug with the Dataverse SWORD API. If I had anticipated your problem I probably would have pushed to work on https://github.com/IQSS/dataverse/issues/1070 *before* we shipped Dataverse 4.0.

It was easy for me to replicate your problem so I just created an issue to track it:

SWORD: not authorized when dataset is in root dataverse - https://github.com/IQSS/dataverse/issues/2495

The ability to create datasets in the root dataverse is a new feature of Dataverse 4.0 and it's "on" for https://dataverse.harvard.edu . It's currently "off" for https://apitest.dataverse.org .

I don't see any particular guidance in the User Guide at http://guides.dataverse.org/en/4.1/user if it is recommended to create datasets in the root dataverse or not but I agree with Kevin putting that datasets in dataverses is a good way to organize them. Again, "Who can add to this dataverse?" for the root dataverse is up to the person installing Dataverse to configure. Here's a screenshot (which I'll also attach): http://guides.dataverse.org/en/4.1/_images/dv3.png . For the geeks among us, here's a related script for how the apitest server is set up: https://github.com/IQSS/dataverse/blob/v4.1/scripts/search/tests/grant-authusers-add-on-root

Sorry to hear about your trouble! Thank you for using Dataverse!

Phil

On Wed, Sep 2, 2015 at 3:39 PM, Condon, Kevin <kco...@hmdc.harvard.edu> wrote:

Yes, we have many individual scholar dataverses but also many based on project or organization. Generally, a logical grouping of datasets.

An example scholar's dataverse: https://dataverse.harvard.edu/dataverse/king
An example organization's dataverse: https://dataverse.harvard.edu/dataverse/mra

---

From: dataverse...@googlegroups.com [dataverse...@googlegroups.com] on behalf of Michele Dolfi [dol...@phys.ethz.ch]
Sent: Wednesday, September 02, 2015 3:30 PM
To: Dataverse Users Community
Subject: Re: [Dataverse-Users] Re: SWORD access failed - not authorized

Thanks a lot, with my own dataverse it works.

Is it common practice to create a dataverse per researcher?

Michele

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.

To post to this group, send email to dataverse...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/8552BDDD3DDE294C995A0182F32F976F0168894317%40HARVANDMBX02.fasmail.priv.

For more options, visit https://groups.google.com/d/optout.

[Mercè Crosas]

Sherry,

We will review issue 1070 on this Wednesday's issue review. It looks like the best solution is to allow "contributors" (and "curators") be able to use the deposit SWORD API, if I understand well, right?

Merce

Mercè Crosas, Ph.D.

Chief Data Science and Technology Officer, IQSS

Harvard University

http://scholar.harvard.edu/mercecrosas

To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-commu...@googlegroups.com.

To post to this group, send email to dataverse...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/8552BDDD3DDE294C995A0182F32F976F0168894317%40HARVANDMBX02.fasmail.priv.

For more options, visit https://groups.google.com/d/optout.

--

Philip Durbin
Software Developer for http://dataverse.org
http://www.iq.harvard.edu/people/philip-durbin

--

You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-commu...@googlegroups.com.

To post to this group, send email to dataverse...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/e3cd83e4-8165-48bf-82bc-eddc6d485f9e%40googlegroups.com.

[Philip Durbin]

Hi Sherry and others,

I've spent the last few days incorporating SWORD into the Dataverse 4 permission model and am close to making a pull request. I thought I'd share a paragraph I'm planning on putting into the API Guide as part of my pull request. Feedback is quite welcome! Here's what I wrote:

'SWORD operations no longer require “admin” permission. In order to use any SWORD operation in DVN 3.x, you had to be “admin” on a dataverse (the container for your dataset) and similar rules were applied in Dataverse 4.4 and earlier (the EditDataverse permission was required). The SWORD API has now been fully integrated with the Dataverse 4 permission model such that any action you have permission to perform in the GUI or “native” API you are able to perform via SWORD. This means that even a user with a “Contributor” role can operate on datasets via SWORD. Note that users with the “Contributor” role do not have the PublishDataset permission and will not be able publish their datasets via any mechanism, GUI or API.'

Please see also the conversation going on at https://github.com/CenterForOpenScience/osf.io/pull/5344

Sherry, if UVA is interested in test a pre-release war file in a staging environment to make sure the fix meets your needs, please let me know. I'm happy to make one available to you. I've also offered to the OSF folks that I can host a pre-release war file on a server for interop testing, and I plan to extend the offer to OJS and anyone else who wants to do some testing beyond the usual Dataverse release process.

I'm pretty sure my fix won't be ready for the Dataverse 4.4 train ( http://dataverse.org/releases-roadmap ), but I'm hoping my it will land soon after. I'll make some noise once the pull request gets merged. Of course, the pull request needs to exist first. That's what I'm working on now. :)

Thanks,

Phil

To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/CAPAYmDORTP6w27Ea%2BJGzG0hiRKYDD9OJVinBrWM2b2iok2cVzg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Sherry Lake]

Hi Phil,

This all sounds great!

I don’t think we will be able to test a pre-release on our development server. It is behind a firewall and OSF cannot “see” it. I don’t want to test on production.

Could you put the pre-release on either https://dataverse-demo.iq.harvard.edu/ or https://apitest.dataverse.org/ (but is only at 4.2), or your offer on an interop testing server?

I’m pretty sure I can test permissions for “curators” (who are not dataverse owners) in either of those environments.

--
Sherry Lake

To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.

To post to this group, send email to dataverse...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/8552BDDD3DDE294C995A0182F32F976F0168894317%40HARVANDMBX02.fasmail.priv.

For more options, visit https://groups.google.com/d/optout.

--

Philip Durbin
Software Developer for http://dataverse.org
http://www.iq.harvard.edu/people/philip-durbin

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.

To post to this group, send email to dataverse...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/e3cd83e4-8165-48bf-82bc-eddc6d485f9e%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.

To post to this group, send email to dataverse...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/CAPAYmDORTP6w27Ea%2BJGzG0hiRKYDD9OJVinBrWM2b2iok2cVzg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Philip Durbin]

Hi Sherry, I've been testing mostly with the "Contributor" role which can't publish but I'll add some tests for the "Curator" role which can. It sounds like your authors/depositors have the "Curator" role on their datasets and can publish on their own.

I'm curious about how you implement "UVa users (logged in via Shib) can create datasets in any dataverse". Do you use the ":authenticated-users" group? An "institution-wide Shibboleth group" (just UVa)? After you create a new dataverse (using dataverseAdmin) I assume you have a step to go into the permissions for https://dataverse.lib.virginia.edu/dataverse/biology (or wherever) to add one of the other of those two groups as "Dataset Creator". I assume you then (for the new dataverse) change "What should be the default role for someone adding datasets to this dataverse?" from "Contributor" to "Curator". This way your authors will be able to publish.

It sounds like you don't really use the "Contributor" role much, if at all.

If this would be easier to hash this out in real time at http://chat.dataverse.org please feel free to ping me there. :)

I hit a bit of a snag this morning related to permissions on the SWORD "service document" so I'm not as close to creating a pull request as I hoped. Soon, I think. I'll let you and others know which server I use to deploy the code once it's in. It won't be https://demo.dataverse.org since it tracks the production release. https://apitest.dataverse.org is supposed to track the production release too but it's a little behind. I have a new server in mind.

To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-commu...@googlegroups.com.

To post to this group, send email to dataverse...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/8552BDDD3DDE294C995A0182F32F976F0168894317%40HARVANDMBX02.fasmail.priv.

For more options, visit https://groups.google.com/d/optout.

--

Philip Durbin
Software Developer for http://dataverse.org
http://www.iq.harvard.edu/people/philip-durbin

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-commu...@googlegroups.com.

To post to this group, send email to dataverse...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/e3cd83e4-8165-48bf-82bc-eddc6d485f9e%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-commu...@googlegroups.com.

To post to this group, send email to dataverse...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/CAPAYmDORTP6w27Ea%2BJGzG0hiRKYDD9OJVinBrWM2b2iok2cVzg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Philip Durbin
Software Developer for http://dataverse.org
http://www.iq.harvard.edu/people/philip-durbin

--

You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-commu...@googlegroups.com.

To post to this group, send email to dataverse...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/fadfce96-d14e-4ddc-9966-6278d28a16a8%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Sherry Lake]

Hi Phil,

I don't do anything "fancy" to set users up (logged in users, which only happens via shib) to be curators of datasets. Here's a screen shot of our top dataverse. All sub-dataverses (except for Law School, which has contributors, not curators) are also setup like this:

[Philip Durbin]

Thanks, Sherry, this helps a lot. Also, now that I'm in the code again I'm realizing that you can't use Shib groups with SWORD. I just made a pull request at https://github.com/IQSS/dataverse/pull/3137 and this is how I expressed it in my change to the guides:

"Institution-wide Shibboleth groups are based on the "Shib-Identity-Provider" SAML attribute asserted at runtime after successful authentication with the Identity Provider (IdP) and held within the browser session rather than being persisted in the database for any length of time. It is for this reason that roles based on these groups, such as the ability to create a dataset, are not honored by non-browser interactions, such as through the SWORD API."

I also updated the SWORD section of the API Guide to explain what permissions are required for each operation.

I went ahead and took the code from that pull request and put it on https://dev1.dataverse.org and have been encouraging OSF and OJS to help me test at https://github.com/CenterForOpenScience/osf.io/pull/5344 and http://irclog.iq.harvard.edu/dataverse/2016-05-24

The SWORD test suite (implemented using REST Assured) is much improved in that pull request. I can always add more tests, if necessary.

As always, comments and questions are quite welcome! I'm hoping that we can get https://dataverse.lib.virginia.edu integrated with https://osf.io !

Phil

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-commu...@googlegroups.com.
To post to this group, send email to dataverse...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/b0e0cab9-a44c-47df-a61f-4b2c02ab05c9%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.