Github Dataverse integration never deletes existing dataset files

[Dario Basset]

We are trying Dataverse-Github integration. 

We have dataverse 6.10

We configured the Github Action with the dataverse API key, and when we trigger the action, all files belonging to the Github repository are correctly uploaded to the target dataset. 

From the second upload on, all files are again uploaded to the same dataset, even if we specify the Github Action parameter DELETE=True. We cannot understand why. 

The expected behaviour is that the Github Action should delete all files from the dataset, and re-upload the files from the Github repository, but unfortunately the Github Action does not delete the files. This implies that all files are duplicated. 

DELETE=True is the default, and we experienced the same Github behaviour with DELETE=True or even without DELETE parameter. The Github Action  uploads again all files.

Have you ever experienced this? Is there a solution? 

[Philip Durbin]

Hi Dario,

I haven't heard of this but please do feel free to create an issue at https://github.com/IQSS/dataverse-uploader/issues about it,

Thanks,

Phil

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-commu...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/dataverse-community/2fa227ca-a883-4a4c-a3f6-825f779b6b65n%40googlegroups.com.

--

Philip Durbin
Software Developer for http://dataverse.org
http://www.iq.harvard.edu/people/philip-durbin

[Philip Durbin]

Also, Dario, can you reproduce this on https://demo.dataverse.org ? I'm hearing from a developer that it works locally for him in Docker.

[Dario Basset]

Thank you Phil. 

I tried with demo dataverse and it works. 

So I think that the problem is on my dataverse.

Thx a lot again.

[Tony Nguyen]

Hi Dario,

Could you check by following below steps:

1. **Reverse Proxy / Authorization Header**: Is your Dataverse instance behind Apache, Nginx, or an ingress controller? Often reverse proxies strip the standard `Authorization: Basic` header while allowing the custom `X-Dataverse-key` header through (which is why uploads succeed but SWORD deletes fail).
2. **Server URL in GitHub Actions**: Are you using `https://` in `DATAVERSE_SERVER`? (If `http://` redirects to `https://`, Python `requests` drops Basic Auth headers on redirect).
3. **Manual Verification**: If you run this `curl` command using your API token against a test file ID in your draft dataset, what response do you get?
   ```bash
   # Test SWORD delete (what the action currently uses):
   curl -i -X DELETE -u "$YOUR_TOKEN:" "https://your-dataverse-server/dvn/api/data-deposit/v1.1/swordv2/edit-media/file/$FILE_ID"

   # Test Native API delete:
   curl -i -X DELETE -H "X-Dataverse-key: $YOUR_TOKEN" "https://your-dataverse-server/api/files/$FILE_ID"

Tony Nguyen.

[Dario Basset]

Thank you Tony. 

I made the checks. 

1. there's no reverse proxy or auth header

2. I am using https (  DATAVERSE_SERVER: https://dataverse.unimi.it )

3. I've tried the Native test and it works like a stream. I also tried the Sword delete, it works. It seems that the APIs work. 

I've noticed a strange behaviour, which I cannot understand. 

I have this dataset where I upload from github, it was 51 files, I've delete 2 with native and word API, now it's 49 files.

I see 49 files in the dataset, but when I select the files the total is 34! And the system selects only 7 files instead of 10. See pictures. 

[Tony Nguyen]

Thank you Dario,

Have you got any logs from Github Action run?

For the bug relate to dataset counts, we saw in: https://github.com/IQSS/dataverse/issues/10873

[Dario Basset]

Solved. 

It was a security problem.

There was a block on some specific DELETE API coming from Off-Campus (external site out of the University). 

Now it works like a stream.

As usual I thank you all. 

Your help is always precious.

You received this message because you are subscribed to a topic in the Google Groups "Dataverse Users Community" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/dataverse-community/horZTambJHA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to dataverse-commu...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/dataverse-community/d2a85af0-a57b-4117-8748-71eefcba7397n%40googlegroups.com.

[Philip Durbin]

Wonderful news! Thanks for letting us know, Dario!

To view this discussion visit https://groups.google.com/d/msgid/dataverse-community/CAEVPFw8qrbdoM%2Bsttm-Q5ZYhF0ee20YX2XkiP4JGNkyektM0Ow%40mail.gmail.com.