Distributed Mathesar
Sean Colsen
Since it’s cooldown, I want to put a bug in people’s ear – a lofty topic to ponder with me. This email has no action items, no requests, no next steps, no response needed – only some nebulous ideas for a possible distant future of Mathesar. I share these thoughts just in case, like me, you might enjoy thinking about them while you’re folding laundry or something.
Here are some difficult problems I’ve encountered with other database software in the past:
-
Offline access: My wife worked with two other midwives in a small home-birth midwifery practice where they used an online medical charting application to manage their client data. The nature of their work required them to visit clients at home, some of whom lived in very rural areas without Internet or access or cell service, meaning they couldn’t use the charting software during clinical visits.
-
Branch, merge: I worked with a non-profit which raised most of its revenue through small donations. In a web-based CRM, they had decades of detailed data on donation history including tens of thousands of recent donors. Paper mailings provided an effective means to regularly solicit donations, but only when sent to the right people. They needed a way to identify the best people to target that would maximize ROI for each appeal. I developed a system to quantitatively prioritize every donor at any given time based on their donation history. It was effective, but very computationally intensive, meaning it could not be run directly on the live data. I built a workflow which, to produce the mailing list, would make a full backup of the live database to a separate machine, then crunch the numbers on that machine, then write back to the live database marking the contacts which where to receive the mailing. It worked, but was kind of a pain.
-
Revert: At this same non-profit, an employee once discovered that the CRM contact records for two highly involved individuals – each having many years of donation history, notes, and program involvement – had been accidentally merged together using the CRM’s “merge duplicate contact records” feature. Could I “un-merge them?” the employee asked me. Hmmm… I had configured backups to be made and retained with a rolling prune algorithm: one per hour for a day, one per day for a month, and so on. Using a rudimentary binary search, I inspected the backups one at a time to identify the most recent backup which contained the two contacts intact. The merge had occurred about a week prior. Obviously I couldn’t roll back to that backup because loads of other important changes had happened in that week. Instead, I painstakingly plucked information out of the backup, placing it into the live database. The whole process took me about a day.
These are just a few of the many interesting scenarios that sometimes lead me to imagine a tool that would allow people to work with data similarly to the way we use git to work with code. Given the UI that we’re building for Mathesar, many of the use-cases that I see it fitting will have very high read/write ratio and data sets well under a GB in size. Within these constraints, an AP system has many benefits. Like git, you sacrifice consistency. But if you don’t necessarily need consistency, then getting availability and partitioning in exchange can open the door to some amazing possibilities.
Imagine being able to install a Mathesar desktop application, use its interface to set up a new local database, and then later synchronize that database with a server or directly with a peer. A workflow like that has the potential to empower non-technical users to use Mathesar autonomously – without having a software engineer son, like Susan Terrell, or necessarily placing trust in a SaaS provider with their private information.
It might not even be that hard to build a desktop Mathesar app using Tauri, which I expect will supplant Electron in the next 10 years. It currently requires a Rust backend, but Python support is on their roadmap. What about the database though? … Dolt is a MySQL-compatible database that stores data in a DAG like git and supports branching and merging. Postgres support is on their roadmap for next year. They appear to be monetizing this product with a model similar to early GitHub – free public repositories, paid private ones. Here’s an example of querying a public data set through their web interface, similar to viewing a source file on GitHub. I played around with it locally and the experience was very smooth! I find this product fascinating. I don’t see them as a competitor, and I even wonder if there may be an opportunity for us to partner with them somehow/someday. Mathesar’s UI has many compelling features that their web UI lacks.
Anyway, take all these thoughts with a big heap of salt. Most likely this is nothing more than a pipe dream. But I’m emailing nonetheless on the off chance that my dreaming-out-loud stirs up some useful brainstorming seeds that eventually germinate into more fully-formed ideas or actions.
