Globus Modern Research Data Portal Sample - Internal Error on Transfer

[Heather Kelly]

Hi, We have based our data portal on the Globus Modern Research data portal.

Since late 2023, perhaps due to the move to Globus Collection v5, our data portal is no longer able to successfully transfer data. Our endpoint was migrated to a v5 Collection and it is working just fine outside of the portal. I went back to this repo (https://github.com/globus/globus-sample-data-portal) hoping to find some direction on what needs to be fixed. I also tried the Globus Sample Data Portal https://mrdp.globus.org/ I see the same error "Internal error" here too no matter what destination collection I use. The collections work just fine outside of the sample data portal. Attached is a screen shot of the error I see when I click transfer.

Is this sample data portal working for everyone else or is there an update that needs to be made? I've been assuming a change needs to happen in declaring the scope.

Thank you,

Heather

[Stephen Rosen]

Sorry all. I sent a direct, instead of to the list where everybody can see. Whoops!
I'm fixing that to provide a better public record of these things.

On 4/15/24 10:39, Stephen Rosen wrote:

Hi Heather,

I think you're looking in the right place.

For the MRDP sample code, the best place to look to track changes is directly at the Pull Request history. [^1]
I see an update to the scope string used as a major part of #66, and there's a significant update (although it may look rather innocuous) in #70 because it updates the globus-sdk version used from 2.x to 3.x .

The scope string update from PR #66 introduces the use of a data_access scope, which is required for a GCS v5 Mapped Collection.[^2]
It's very likely that this is exactly the update you need.

If you run into trouble while trying to do this, please send along a sample of the logs for us to read through. You can send to `sup...@globus.org` if you're uncomfortable sharing log data with the public list, or if you feel uncertain about which parts of the logs might be sensitive.
The server error in the screenshot really just tells us "something went wrong", and we'll need a bit more detail to drive this to resolution.

Best,
-Stephen

---

[1]: We should add a changelog, but I think it's relevant that the MRDP sample doesn't change very frequently. And there aren't versions of the code, so any changelog would be continuous/rolling. Overall, it won't "feel like a product" because it's meant as a starting point/example. But you can read the PRs to see what changes you want to pull in, and I think that's a pretty sensible way to use the project.

[2]: You may see mention of v5.3 vs v5.4. For 99% of users, every GCS v5 is a v5.4 endpoint, so you can usually ignore v5.3 and lower.

[Heather Kelly]

Hi Stephen,

Thank you for the detailed response!  I agree the scope string is likely what needs to be updated.  I don't seem to have the syntax quite right yet, as I continue to receive errors when I attempt to do a transfer.  An example output in the log is at the end of this message, where I note the ConsentRequired, Missing data_access_consent.

I've tried this a couple of different ways and the error remains the same, my most recent attempt was following the documentation here:

https://docs.globus.org/api/transfer/overview/#data_access_consent

urn:globus:auth:scope:transfer.api.globus.org:all[*https://auth.globus.org/scopes/COLLECTION_UUID/data_access]

where I obtained the COLLECTION_UUID by looking at my collection in Globus.

I also tried a variation on what is in the MRD sample:

scopes=['openid', 'urn:globus:auth:scope:demo-resource-server:all', 'urn:globus:auth:scope:demo-resource-server:all[https://auth.globus.org/scopes/' + app.config['GRAPH_ENDPOINT_ID'] + '/https]']):

where I replaced GRAPH_ENDPOINT_ID with my own ENDPOINT ID.

Both fail in the same way.  Any hints would be appreciated!

Take care,

Heather

Log file:

{'name': 'truth_tract2725.parquet', 'path': 'lsstdesc-public/dc2/run2.2i-dr6-v4/truth_match', 'id': '386e2ce3-9840-48f6-a2ef-25b7d12fe642'}
[2024-05-01 17:15:18,551] ERROR in app: Exception on /submit-transfer [POST]
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 2529, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 1825, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 1823, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 1799, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/app/portal/decorators.py", line 20, in decorated_function
    return fn(*args, **kwargs)
  File "/app/portal/views.py", line 403, in submit_transfer
    task_id = transfer.submit_transfer(transfer_data)['task_id']
  File "/usr/local/lib/python3.10/dist-packages/globus_sdk/services/transfer/client.py", line 1270, in submit_transfer
    return self.post("/transfer", data=data)
  File "/usr/local/lib/python3.10/dist-packages/globus_sdk/client.py", line 155, in post
    return self.request(
  File "/usr/local/lib/python3.10/dist-packages/globus_sdk/client.py", line 310, in request
    raise self.error_class(r)
globus_sdk.services.transfer.errors.TransferAPIError: ('POST', 'https://transfer.api.globus.org/v0.10/transfer', 'Bearer', 403, 'ConsentRequired', 'Missing required data_access consent', 'MzBHfrDMu')
216.212.50.9 - - [01/May/2024 17:15:18] " [35m [1mPOST /submit-transfer HTTP/1.1 [0m" 500 -