Office hours notes

[Melissa DeLucchi]

background: Gustavo (undergrad CS student, working with SPLUS) has been working on a really fun, awesome, thing. researchers in brazil have particularly bad network bandwidth, and TAP queries are slow, and they'd like to only move data over the wire that they need (see the intro talk here Passcode: $9.h1ipS). he's been building a rust server to read the parquet files and do some row/column filtering on the server side, and send back a smaller parquet file.

we talked some today about particular issues he's having, other features, and how to move forward.

• lsdb-server

• naming: it provides general parquet slicing, so for now maybe call it "parquet-server" or "parquet-slice-server"

• could still provide some query like &cone=0,5,200,ra,dec,arcs, and it will still be great.

read_hipscat("gaia", search_filter=ConeSearch(0, 5, 200), 

columns=["a", "b", "c"],

filters=[("foo", ">", 2), ("bar", "<>", "baz")])

===>

http://foo/gaia/Norder=1/Dir=0/Npix=34.parquet?columns=a,b,c&filters=...&cone=0,5,200,gaia_ra,gaia_dec,arcs

• filters: 

• see pandas "filter". let's use the same kinda filter syntax that pandas would understand when reading parquet.

• for lsdb-server, if there are errors on fetching, we just read the whole file. so if we accept that query argument on read_hipscat we should:

• pass the query params along to the server

• ALSO attempt to do post-fetch query filtering on the client's side.

• caching: let's not worry too much about this right now. we may need to re-visit this soon anyway, and want to make sure we're doing this consistently.

• response: let's use 404 (or other 4** error) for missing columns.

• where does it live (github organization):

• astronomy-commons

• lincc-frameworks

• splus

• Gustavo will talk to his bosses!!

--

=======

Melissa DeLucchi (duh-LOO-kee)

she/they