Parquet files support

[Anna Lungarska]

Dear all, 

There's been a topic here about incorporating parquet files namely partitioned ones. My personal interest is in the generation of an S3 type URI allowing the querying of files without loading them as a whole. My use case is a R Shiny App with the R arrow package and the open_dataset() function working on S3 but not over http(s).

Cheers and great upcoming 2026!

[Philip Durbin]

Hi Anna,

Sounds fancy! The closest thing Dataverse has that comes to mind is the zip file previewer/downloader[1] that allows you to preview the contents of a zip file and selectively download individual files from that preview (rather than having to download the entire zip file).

This is possible due to Dataverse's support for the "Range" HTTP header[2], which the zip previewer/downloader makes use of.

The zip previewer/downloader is implemented as an "external tool", by the community, in this pull request: https://github.com/gdcc/dataverse-previewers/pull/9

Perhaps someone from the community can implement something similar for parquet files?

I hope this helps!

Phil

1. https://guides.dataverse.org/en/6.9/user/dataset-management.html#compressed-files

2. https://guides.dataverse.org/en/6.9/api/dataaccess.html#headers

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-commu...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/dataverse-community/c0a58d27-68ba-404a-b5f6-acc1798a1ee2n%40googlegroups.com.

--

Philip Durbin
Software Developer for http://dataverse.org
http://www.iq.harvard.edu/people/philip-durbin