Recap of your meeting with Eto Labs

3 views

Skip to first unread message

Fathom

unread,

Jun 18, 2026, 12:34:37 PMJun 18

to Lance Format Devlist

Meeting Purpose Sync on community proposals for governance, roadmap, and technical specs. Key Takeaways - New Format Governance: To prevent flawed designs, format changes will now require 3 PMC GitHub approvals on PRs, replacing the current manual voting process. - Improved Design Reviews: Major design docs will use Google Docs with public comments to enable efficient, inline feedback, replacing the cumbersome GitHub Discussions format. - New Shuffle Format Proposed: A new storage-based shuffle format is proposed to unify GPU and CPU workflows and fix performance bottlenecks in the current two-file shuffler. - Multi-Table Transactions: The community will collaborate on a branch-based, multi-table transaction feature, leveraging existing Lance primitives for a robust solution. Topics Format Change Governance - Problem: Format changes have been merged without the required 3 PMC votes, leading to flawed designs (e.g., a transaction update that caused user issues). - Solution: Implement a stricter approval process. - Rejected: A separate repo for format artifacts was deemed too complex. - Approved: Use GitHub approvals on PRs. - A CI check will block merges until 3 PMC members approve. - Approvals will replace the current manual voting process in GitHub Discussions. - Discussion Location: The community is flexible on whether discussions happen in PRs or Discussions, as long as the approval mechanism is robust. Design Doc Review Process - Problem: Reviewing large Markdown documents in GitHub Discussions is inefficient and discourages participation. - Solution: Use Google Docs with public comment permissions for major design docs. - This enables efficient, inline feedback. - Xuanwo suggested more visual tools (e.g., graphs, SVGs) to improve clarity, especially for non-native English speakers. Public Roadmap - A public roadmap (a locked GitHub issue) will be maintained to: - Increase transparency for potential collaborators. - Help prioritize PR reviews that align with project goals. - This issue is for visibility only; discussion will continue in syncs or GitHub Discussions. New Shuffle Format Proposal - Problem: The current shuffle formats are fragmented and inefficient. - Old GPU format: Very slow with high IOPS. - two-file shuffler (CPU): In-memory only, preventing external tools (like GPUs) from writing shuffle files to storage for the REST core to consume. - Proposal: A new storage-based shuffle format to unify GPU and CPU workflows. - Goal: Define a storage layout for shuffle files and manifests. - Optimization: Designed for efficient per-partition reading, reducing server-side filtering. - Format: Currently JSON; a switch to Lance files is preferred for consistency. - Next Steps: - Xuanwo will update the proposal with a concrete goal for max open files. - Xuanwo will create a benchmark comparing the new format against the two-file shuffler and old GPU format. - Will and Weston will review the code and benchmark results. Multi-Table Transactions - Context: A branch-based, multi-table transaction approach was proposed by Jack. - Rationale: This approach is seen as elegant and efficient, leveraging existing Lance primitives. - Collaboration: Ragnor and Andrew's teams are eager to contribute to the implementation. - Next Steps: - Prashanth will coordinate with Jack to define tasks and divide work. - Ragnor and Andrew will connect on Discord to prioritize and begin work. Next Steps - Will: Create a PR proposing the new GitHub approval process for format changes. - Xuanwo: Update the shuffle format proposal and create a benchmark comparing it to existing methods. - Prashanth: Coordinate with Jack to define tasks for the multi-table transaction feature. - Ragnor & Andrew: Connect on Discord to begin work on multi-table transactions.