HiPSCat meeting 2024-01-05 and LSDB v0.1 release

0 views
Skip to first unread message

Melissa DeLucchi

unread,
Jan 4, 2024, 2:12:48 PM1/4/24
to hipsc...@googlegroups.com, Kiessling, Alina A (3266), Faisst, Andreas, antara.r....@nasa.gov, Bernie Shiao, Brian Hayden, Brian McLean, bsi...@gmail.com, Carlos Adean, ctsl...@uw.edu, David Shupe, Erik Tollerud, fxpi...@gmail.com, gp...@ipac.caltech.edu, Jeremy Kubica, Julia Gschwend, ldacosta, mju...@uw.edu, Max West, Melissa DeLucchi, Rick White, Samuel Dillon Wyatt, Sean McGuire, Sharon Shen, gwyn...@gmail.com, sgr...@ipac.caltech.edu, Steve Lubow, Susan Mullally, Tom Donaldson, Travis Berger, Raen, Troy J., Vandana Desai, msan...@stsci.edu, fri...@slac.stanford.edu, ga...@slac.stanford.edu

Working group meeting


We'll be meeting tomorrow, 1pm eastern, 10am pacific, 2pm Belem.


LINCC Frameworks will give an update on our progress in 2023, and an overview of our engineering goals in 2024.


Zoom link.


Upcoming meetings:

  • 2024-01-12 No meeting (AAS)

  • 2024-01-19 Open for topics

  • 2024-01-26 No meeting (LINCC summit)

  • 2024-02-02 Open for topics


LSDB v0.1 release


We're very pleased to announce the release of LSDB v0.1. We have APIs in place to support several basic end-to-end analysis operations. Included in this release:


  • Catalog-level API, wrapping dask dataframes

  • Create catalog from hipscat-on-disk or in-memory pandas dataframe

  • Crossmatch, using 3D kdtree

  • Filter catalog using cone search or polygon search

  • Wrappers for .query, .assign, and .merge operations on dask dataframes

  • Join catalogs by equijoin column or by association table (pre-computed crossmatch)

  • Read data from cloud buckets

  • Write analysis results to new hipscat catalog

  • Benchmarks of common operations, and some initial performance tuning

  • And LOTS of small polish and bug fixes


We will talk a little tomorrow about our priorities for the coming quarter, as they relate to LSDB as well as hipscat and hipscat-import.


See you then!
-Melissa, on behalf of LINCC Frameworks

--
=======
Melissa DeLucchi (duh-LOO-kee)
she/they

Melissa DeLucchi

unread,
Jan 5, 2024, 2:02:26 PM1/5/24
to hipsc...@googlegroups.com, Kiessling, Alina A (3266), Faisst, Andreas, antara.r....@nasa.gov, Bernie Shiao, Brian Hayden, Brian McLean, bsi...@gmail.com, Carlos Adean, ctsl...@uw.edu, David Shupe, Erik Tollerud, fxpi...@gmail.com, gp...@ipac.caltech.edu, Jeremy Kubica, Julia Gschwend, ldacosta, mju...@uw.edu, Max West, Melissa DeLucchi, Rick White, Samuel Dillon Wyatt, Sean McGuire, Sharon Shen, gwyn...@gmail.com, sgr...@ipac.caltech.edu, Steve Lubow, Susan Mullally, Tom Donaldson, Travis Berger, Raen, Troy J., Vandana Desai, msan...@stsci.edu, fri...@slac.stanford.edu, ga...@slac.stanford.edu
I unfortunately forgot to record today's session, but much of the content is included in these slides, and I put a handful of links on the last slide.

We will try to add sessions in the coming weeks for
- STSci cross-match scientists, discuss association storage in hipscat
- Review Macauff LINCC incubator (used the association storage in hipscat)
Reply all
Reply to author
Forward
0 new messages