DB connections work

[Sean Colsen]

Team,

I want to give a high-level status update about the DB connections work which Kriti recently assigned me to coordinate. This is an important topic that I expect will be touching nearly everyone’s work in the coming weeks, so I’d appreciate careful attention to this subject.

• How I got here: After being only peripherally involved in the Installation Improvements project (and hence, the DB connections work), I finally previewed Rajat’s DB connection UI PR for the first time late last week. I found the resulting user experience to be very confusing on may levels so I raised some concerns with Kriti in a call, mostly to ask how I ought to go about raising those concerns to the rest of the team since they touched on so many other people’s work. Kriti decided it would be best for her to reply to Mukesh’s email thread, appointing me to coordinate the necessary remaining work. Now I kind of feel like a server at a restaurant, taking a quick taste of a meal that several chefs cooked, and then dumping it on the floor in dissatisfaction. Sorry! 😬 I’m going to help clean it up though!

• Importance: I recognize that many people have been working hard on the new DB connection features for many weeks now, and I’m loathe to hold up the release for the sake of perfection. However, Kriti made it clear to me that the DB connection configuration is crucially important for our next release. She understood the concerns I had about the features as we’ve currently implemented them, and she said it’s worth back-tracking to get it right, even if that work delays our release by several weeks. So that’s what we’ll be doing — stepping back to re-think the DB connections behavior and improve it before release.

• Architectural work and UI work: As I see it, there are two broad categories of work, each with many small outstanding decisions and tasks:

  “Architectural work” is the stuff Mukesh has been designing and Anish has been implementing. Going forward, it will encompass work such as:

  • Removing .env as a source of truth for connection credentials
  • Modifying the necessary data structures to permit editable connection names
  • (Potentially) adding an installation step requiring the administrator create or supply the first user database
  • Figuring out how to strike the right balance between prioritizing a smooth installation and empowering users and administrators with sufficient understanding of Mathesar’s inbuilt Postgres server to manage connections to multiple databases on that server

  “UI work” is the stuff that Ghislaine has been designing, Rajat has been implementing, and Pavish as bee reviewing. Going forward it will encompass work such as:

  • Moving the “DB connection switcher” (in the left pane of the “Database page”) and the “DB connections page” (in the Administration section) into a new root-level “home page” in order to consolidate “DB connection navigation” with “DB connection CRUD” in the same interface (similar to the manner in which those operations are already consolidated within our UI for schemas and tables).
  • Addressing other needs as more architectural decisions are made.

• Additional work: If anyone would like to suggest that we reconsider something else about our DB connection behavior, now is the time to speak up! Brent’s latest email is a great example. He’s highlighted some new questions that deserve thought and discussion. I’ll be making sure that we address any remaining loose ends and reach a point of clarity on them as a team. So please take this time to surface whatever loose ends you may have kicking around in your head!

• Current status and next steps:

  • For the UI work, Ghislaine has already begun making some new designs in Figma. I’ll be coordinating design reviewers and front end implementers.
  • Rajat’s outstanding DB connection UI PR is hanging in limbo. I’ll coordinate the necessary tidying to get it merged so that it doesn’t accumulate conflicts.
  • For the architectural work, we still have some outstanding questions that require further discussion and decisions before we can move forward. I’ll be working with many people to figure this out.
  • I have been having 1/1 conversations with Kriti, Mukesh, and Ghislaine; and the progress we’ve made thus far in those conversations seem to indicate that a series of 1/1 conversations will be more productive than larger group conversations — at least until our plans come into sharper focus.
  • I’ll soon be scheduling 1/1 conversations with Brent, Pavish, and Rajat too. During those conversations you’ll each have the chance to get into the weeds with me about this work and voice any concerns or ideas you have.
  • Dom: as far as I can tell, you haven’t touched the DB connections work or voiced any opinions about it thus far, so I’m not planning to reach out to you about it. Please reach out to me if you want to have a 1/1 call

• “Work” vs “project”: The phrase “DB connections work” encapsulates a number of different tasks, with the scope and goals being somewhat in flux currently. Within our projects framework, we could almost classify this “DB connections work” as a proper project, but I’m using the noun “work” instead because we’re between cycles and trying to move quickly with less overhead. If we think of “projects” as rails to keep us on track, then it feels like we’ve veered off the rails a bit right now. I just want to acknowledge that and state my intention to embrace this tangent as only a temporary divergence from our projects structure. If anyone feels strongly about fitting this work more squarely into the projects framework, then I can take the extra time to write up a project or to modify the Installation Improvements project to include this work.

​

[Kriti Godey]

Thanks for this summary, Sean – this all sounds good to me, and thanks for taking on coordinating this!

[Pavish Kumar Ramani Gopal]

Thanks for the summary Sean. I have some thoughts around this.

UI work

> Moving the “DB connection switcher” (in the left pane of the “Database page”) and the “DB connections page” (in the Administration section) into a new root-level “home page” in order to consolidate “DB connection navigation” with “DB connection CRUD” in the same interface (similar to the manner in which those operations are already consolidated within our UI for schemas and tables).

With our current set of assumptions, we do not expect users to have a large number of connected databases. Having a dedicated CRUD page for it adds more pages and additional steps to the user. Considering that pretty much all our features are encapsulated within schemas, I believe this would probably not the best UX. 

I would lean towards having a single page for both the databases and schemas.

Avoiding similar concerns in the future

I think we need a retrospective on this.

• The UX for this feature was reviewed and approved by the entire frontend team. Both me and Sean reviewed and approved it, after multiple rounds, and Rajat took a look before implementation.
• The confusing aspect of the environment variables and locked databases was not present in the UX design. It was added during implementation.
• I'm not sure if these changes during implementation were discussed with Ghislaine. I noticed them while reviewing the frontend PR.
• FWIW, I understand that it's a tricky problem and required a UX change.
• I think we could have avoided a lot of delay if this had been discussed and addressed when this roadblock was hit, instead of after the work was complete.
• I do think we would benefit from a better process for any UX and product related changes that arise during implementation.