Trusted Remote Storage Setup

[Cesar INCD]

Hi, I am trying to setup a trusted remote storage on version 6.8 and I am getting some errors. To give you a better context, I have an S3 bucket with Sentinel 2 data downloaded and my idea was to create dataset aggregating that data and use that bucket as a trusted remote storage. The settings I used to configure are as follows:

 

I tried to create a file on the dataset with the following script:
```

export API_TOKEN=XXX export SERVER_URL=https://repositorio.cloud.acnca.pt export PERSISTENT_IDENTIFIER=doi:10.5072/FK2/PKUVLP export JSON_DATA='{"description":"Test file","tabIngest": false,"categories":["Data"], "restrict":"false", "storageIdentifier":"sentinel2://DGT2015/S2A_MSIL1C_20150708T113056_N0500_R080_T29SMA_20231010T003232.SAFE/DATASTRIP/DS_S2RP_20231010T003232_S20150708T113621/MTD_DS.xml", "fileName":"MTD_DS.xml","mimeType":"application/xml","checksum": {"@type": "MD5", "@value": "e0cd675d26aa50e80f6a9d1992955db9"}}' curl -X POST -H "X-Dataverse-key: $API_TOKEN" "$SERVER_URL/api/datasets/:persistentId/add?persistentId=$PERSISTENT_IDENTIFIER" -F "jsonData=$JSON_DATA"
```

but it failed and I got the following messages in server.log:
```

[2025-12-17T15:34:24.641+0000] [Payara 6.2025.3] [INFO] [] [] [tid: _ThreadID=128 _ThreadName=http-thread-pool::http-listener-1(16)] [timeMillis: 1765985664641] [levelValue: 800] [[ -------------------------------]] [2025-12-17T15:34:24.642+0000] [Payara 6.2025.3] [INFO] [] [] [tid: _ThreadID=128 _ThreadName=http-thread-pool::http-listener-1(16)] [timeMillis: 1765985664642] [levelValue: 800] [[ jsonData: {"description":"Test file","tabIngest": false,"categories":["Data"], "restrict":"false", "storageIdentifier":"sentinel2://DGT2015/S2A_MSIL1C_20150708T113056_N0500_R080_T29SMA_20231010T003232.SAFE/DATASTRIP/DS_S2RP_20231010T003232_S20150708T113621/MTD_DS.xml", "fileName":"MTD_DS.xml","mimeType":"application/xml","checksum": {"@type": "MD5", "@value": "e0cd675d26aa50e80f6a9d1992955db9"}}]] [2025-12-17T15:34:24.642+0000] [Payara 6.2025.3] [INFO] [] [] [tid: _ThreadID=128 _ThreadName=http-thread-pool::http-listener-1(16)] [timeMillis: 1765985664642] [levelValue: 800] [[ -------------------------------]] [2025-12-17T15:34:24.880+0000] [Payara 6.2025.3] [WARNING] [] [edu.harvard.iq.dataverse.dataaccess.RemoteOverlayAccessIO] [tid: _ThreadID=128 _ThreadName=http-thread-pool::http-listener-1(16)] [timeMillis: 1765985664880] [levelValue: 900] [[ Not Supported: edu.harvard.iq.dataverse.dataaccess.RemoteOverlayAccessIO store with base store type: null]] [2025-12-17T15:34:24.881+0000] [Payara 6.2025.3] [WARNING] [] [edu.harvard.iq.dataverse.ingest.IngestServiceBean] [tid: _ThreadID=128 _ThreadName=http-thread-pool::http-listener-1(16)] [timeMillis: 1765985664881] [levelValue: 900] [[ Failed to get file size, storage id, or failed to remove the temp tag on the saved S3 objectsentinel2://19b2cf2bef3-4aac2735d32e//DGT2015/S2A_MSIL1C_20150708T113056_N0500_R080_T29SMA_20231010T003232.SAFE/DATASTRIP/DS_S2RP_20231010T003232_S20150708T113621/MTD_DS.xml (Not supported)]] [2025-12-17T15:34:24.881+0000] [Payara 6.2025.3] [WARNING] [] [edu.harvard.iq.dataverse.dataaccess.RemoteOverlayAccessIO] [tid: _ThreadID=128 _ThreadName=http-thread-pool::http-listener-1(16)] [timeMillis: 1765985664881] [levelValue: 900] [[ Not Supported: edu.harvard.iq.dataverse.dataaccess.RemoteOverlayAccessIO store with base store type: null]] [2025-12-17T15:34:24.882+0000] [Payara 6.2025.3] [WARNING] [] [edu.harvard.iq.dataverse.ingest.IngestServiceBean] [tid: _ThreadID=128 _ThreadName=http-thread-pool::http-listener-1(16)] [timeMillis: 1765985664882] [levelValue: 900] [[ Error getting ingest limit for file: null : Not supported]] [2025-12-17T15:34:24.882+0000] [Payara 6.2025.3] [INFO] [] [edu.harvard.iq.dataverse.ingest.IngestServiceBean] [tid: _ThreadID=128 _ThreadName=http-thread-pool::http-listener-1(16)] [timeMillis: 1765985664882] [levelValue: 800] [[ Incrementing recorded storage use by 0 bytes for dataset 4]]

```

[Cesar INCD]

The logs from previous message:

[2025-12-17T15:34:24.641+0000] [Payara 6.2025.3] [INFO] [] [] [tid: _ThreadID=128 _ThreadName=http-thread-pool::http-listener-1(16)] [timeMillis: 1765985664641] [levelValue: 800] [[
 -------------------------------]]

[2025-12-17T15:34:24.642+0000] [Payara 6.2025.3] [INFO] [] [] [tid: _ThreadID=128 _ThreadName=http-thread-pool::http-listener-1(16)] [timeMillis: 1765985664642] [levelValue: 800] [[
 jsonData: {"description":"Test file","tabIngest": false,"categories":["Data"], "restrict":"false", "storageIdentifier":"sentinel2://DGT2015/S2A_MSIL1C_20150708T113056_N0500_R080_T29SMA_20231010T003232.SAFE/DATASTRIP/DS_S2RP_20231010T0
03232_S20150708T113621/MTD_DS.xml", "fileName":"MTD_DS.xml","mimeType":"application/xml","checksum": {"@type": "MD5", "@value": "e0cd675d26aa50e80f6a9d1992955db9"}}]]

[2025-12-17T15:34:24.642+0000] [Payara 6.2025.3] [INFO] [] [] [tid: _ThreadID=128 _ThreadName=http-thread-pool::http-listener-1(16)] [timeMillis: 1765985664642] [levelValue: 800] [[
 -------------------------------]]

[2025-12-17T15:34:24.880+0000] [Payara 6.2025.3] [WARNING] [] [edu.harvard.iq.dataverse.dataaccess.RemoteOverlayAccessIO] [tid: _ThreadID=128 _ThreadName=http-thread-pool::http-listener-1(16)] [timeMillis: 1765985664880] [levelValue: 90
0] [[
 Not Supported: edu.harvard.iq.dataverse.dataaccess.RemoteOverlayAccessIO store with base store type: null]]

[2025-12-17T15:34:24.881+0000] [Payara 6.2025.3] [WARNING] [] [edu.harvard.iq.dataverse.ingest.IngestServiceBean] [tid: _ThreadID=128 _ThreadName=http-thread-pool::http-listener-1(16)] [timeMillis: 1765985664881] [levelValue: 900] [[
 Failed to get file size, storage id, or failed to remove the temp tag on the saved S3 objectsentinel2://19b2cf2bef3-4aac2735d32e//DGT2015/S2A_MSIL1C_20150708T113056_N0500_R080_T29SMA_20231010T003232.SAFE/DATASTRIP/DS_S2RP_20231010T003
232_S20150708T113621/MTD_DS.xml (Not supported)]]

[2025-12-17T15:34:24.881+0000] [Payara 6.2025.3] [WARNING] [] [edu.harvard.iq.dataverse.dataaccess.RemoteOverlayAccessIO] [tid: _ThreadID=128 _ThreadName=http-thread-pool::http-listener-1(16)] [timeMillis: 1765985664881] [levelValue: 90
0] [[
 Not Supported: edu.harvard.iq.dataverse.dataaccess.RemoteOverlayAccessIO store with base store type: null]]

[2025-12-17T15:34:24.882+0000] [Payara 6.2025.3] [WARNING] [] [edu.harvard.iq.dataverse.ingest.IngestServiceBean] [tid: _ThreadID=128 _ThreadName=http-thread-pool::http-listener-1(16)] [timeMillis: 1765985664882] [levelValue: 900] [[
 Error getting ingest limit for file: null : Not supported]]

[2025-12-17T15:34:24.882+0000] [Payara 6.2025.3] [INFO] [] [edu.harvard.iq.dataverse.ingest.IngestServiceBean] [tid: _ThreadID=128 _ThreadName=http-thread-pool::http-listener-1(16)] [timeMillis: 1765985664882] [levelValue: 800] [[
 Incrementing recorded storage use by 0 bytes for dataset 4]]

[James Myers]

Quick thought: Do you have a store with id “s3” fully defined?

 

-- Jim

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-commu...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/dataverse-community/fd696094-a1cd-465b-8eda-694684056937n%40googlegroups.com.

[Cesar INCD]

I have another s3 store, not remote, with id 'minio'.

[James Myers]

That’s probably the issue then – the remote store needs a real, defined backing store. You should be able to point at your ‘minio’ store. (Assuming the remote files are readable, Dataverse will do things like create thumbnails, etc. and use the base-store as a place to put them.).

To view this discussion visit https://groups.google.com/d/msgid/dataverse-community/a5b7409a-f374-4ad9-8e3a-7154373b72dcn%40googlegroups.com.

[Cesar INCD]

So I need to define a new backing store like I did for 'minio', this remote storage is on a different S3 endpoint.The 'sentinel2' settings bellow, that already had configured, are an extra to the S3 backing store base configuration. I was thinking on having this remote storage with just readonly access, in this case it is not possible to setup this way? These new files created by Dataverse are created on the same directory as the original file?

asadmin create-jvm-options '-Ddataverse.files.sentinel2.label=sentinel2'
asadmin create-jvm-options '-Ddataverse.files.sentinel2.base-store=s3'
asadmin create-jvm-options '-Ddataverse.files.sentinel2.base-url="https://stratus.d.incd.pt:8080/sentinel"'
asadmin create-jvm-options '-Ddataverse.files.sentinel2.download-redirect=true'
asadmin create-jvm-options '-Ddataverse.files.sentinel2.upload-redirect=true'
asadmin create-jvm-options '-Ddataverse.files.sentinel2.remote-store-name=Sentinel2'
asadmin create-jvm-options '-Ddataverse.files.sentinel2.upload-out-of-band=true'

asadmin create-jvm-options "-Ddataverse.files.sentinel2.ingestsizelimit=1073741824"

[James Myers]

The remote store, whether that site uses s3 itself or not, is considered read-only by Dataverse. Configuring a remote store requires that you also specify another writable store that can be used with it. The remote store and its base-store don’t have to have any common storage. You could, for example, use a local file store as the base-store for a remote store. In your case, you should be able to use your minio s3 store as the base-store for your sentinel2 remote store. If you reference an image in your sentinel2 store, you should see a thumbnail of that image written to the base-store. (And similarly, any ingested version of tabular files, any cached export files for the dataset, etc. will go in the base-store.)

To view this discussion visit https://groups.google.com/d/msgid/dataverse-community/4b0fa96f-d5f3-44ae-8bb8-f95117c94bc2n%40googlegroups.com.

[Cesar INCD]

Thanks, I changed in my configuration the back-store to an existing storage with write access and I successfully uploaded a remote store file to a dataset.