User specific display options

[Anish Umale]

Hey team,

Today, in the 1:1 with Brent I proposed having display options on a per-user basis rather than per DB basis which is the current behavior. Implementing this is fairly easy from a backend perspective and also advantageous as we won't have to manage permissions for it as everyone can have different display options set on their tables. This would also allow us to streamline our permissions UX as we'll have one less choice that a user will need to make while setting up permissions. 

A potential downside to this implementation might be while sharing a table, where the table view that is expected by the user to be shared won't be shown to other users.

So, to summarize, rather than display options being a global setting on a DB/table we would have it such that each mathesar user would be able to manage their own display options.

Curious to know what others think about this proposal.

Thanks,

Anish

[Sean Colsen]

Previous threads

We’ve talked about this before…

https://groups.google.com/a/mathesar.org/g/mathesar-developers/c/uGM_Dwa8UEM/m/5gwrlf3kAwAJ

https://github.com/mathesar-foundation/mathesar/issues/1421#issuecomment-1583812313

In those threads I’ve made the case for storing them globally (as we currently do).

Short term utility

You said:

advantageous as we won’t have to manage permissions for it

Even if we were to make “display options” user-specific, I don’t think that would necessarily allow us to entirely circumvent the challenges associated with permissioning Mathesar metadata. I would argue that we still need to crack that nut. And I don't think there's a ton of intrinsic complexity in that problem.

What exactly is a “display option” though? There is some gray area I think. A “number format” setting fits squarely within the display option classification. Ok. How about the ordering of columns? Column width? Record summary template? That metadata starts to get a little less like “display options”. But in theory we could still make it per-user. What about an entire exploration? Is that a display option? Kind of. But I think we’d all agree there needs to me a mechanism to build shared-workflows around explorations. Sometimes the ability to share display options like number format is also actually important for building shared workflows. As I see it, all of that stuff is “metadata”. And the simplest approach right now is to think of it as globally shared, with a very lightweight permissioning scheme. Don’t get me wrong — I totally agree that there are compelling use cases for having per-user display options, but my main point is that our current approach is simple which fits well with the stage of our product.

Longer term

At some point we start to outgrow the “simple” approach though, and I don’t want to dismiss the importance of eventually building a mechanism for per-user display options. What you’re proposing is taking our current abstractions for display options (and/or metadata) and making some (or all) of it user-specific. What I’d rather do is re-think the abstractions altogether to make them fundamentally more compatible with user-specific settings. This is one thing I’m after with my “worksheets for everything” vision. In that email, I mention display options right under “other cool things” near the bottom. So perhaps I’ve not sufficiently explained or emphasized the benefit I see worksheets having on our approach to display options…

In my “worksheets for everything” vision, there would actually be (almost) no more table-level metadata or column-level metadata. All that stuff would get wrapped up into the “worksheet” abstraction — an entirely new thing in Mathesar. The worksheet would provide a neat, tidy little bundle for storing/sharing/copying/modifying many things including display options.

Here’s an example of how worksheets would let us manage global and per-user display options for the same metadata…

1. When viewing the Library Management schema I click on the “Books” table to navigate to something like our current “Table Page”. But what opens would actually be the “default worksheet” for the Books table.

   • To get pedantic: we would have a (very small) bit of global table-level metadata here because we’d need the ability to associate one or zero worksheet objects with a table oid.

2. Now let’s say I’m looking at that default Books worksheet and I resize the Title column, making it a bit wider.

3. The new column width would initially be persisted only in the browser. And at that point the worksheet would indicate to the user that its definition (which includes the column width) is unsaved.

4. If the user has permission, they could save the worksheet back to its original location (the default Books worksheet), thereby persisting a new global column width for all users.

5. Alternatively the user could perform a sort of “save as” operation, saving a private copy of the worksheet to their user account (or even to a shared team space too). This would allow for per-user display options.

Column width in particular is a great example of something that I think has compelling use cases for being global and compelling use cases for being per-user. Worksheets would allow us to do both! And it would give us a powerful abstraction upon which we could build more elaborate display-options features like hidden columns, joined data, conditional formatting, configurable editor UI, row height, etc. Anything we build into the worksheet would automatically gain the ability to be configured globally or per-user.

​

[Brent Moran]

I re-read through the previous discussions which Sean so helpfully posted. I suppose I'm convinced to defer discussing making the display options per-user for the moment. I'd point out that there are some things currently stored as display options which are weirdly in conflict with other display options  (e.g., display language) which are per-user. For example, number locale. Currently, we have that setting stored globally w.r.t. users, but per column (hmmmmmmmm). Also, we could now be in a situation where the User has set their UI to display in Japanese, but the numbers aren't displayed in the Japanese style. (Although, I think most Japanese folks are comfortable going back-and-forth between numbering styles)

Anyhow, I really think some metadata makes sense per-user, and we should (at some point) figure out which metadata that is, and how to help users manage it. My personal preference is to have "default display settings" globally, and "specific display settings" per-user for the most part. The defaults would apply installation- (or at least organization-) wide, so you wouldn't have to set locale for every number column. As I think about it, "display" may be too constraining of a modifier. Maybe just "default settings".

I'm glad Anish sent this email, however, to make sure we don't forget that this is something to consider after beta.

[Sean Colsen]

we could now be in a situation where the User has set their UI to display in Japanese, but the numbers aren’t displayed in the Japanese style

In the spirit of pragmatism — and to assuage some concern that this could potentially be a high-enough priority to work on — I want to offer two mitigating factors which I think reduce the risk of the above situation:

• As far as I can tell, people in Japan format their numbers the same as in English-speaking countries. You can see this by running the following in your browser:

  (new Intl.NumberFormat('ja-JP')).format(1234567.89)
  

  “1,234,567.89”

  My point here is that both of our supported UI languages conveniently align with the same number formatting scheme.

  Now, were we to add German to our UI translations, then we’d be in more trouble. Change the locale above to ‘de-DE’ and you get “1.234.567,89”. But even in that situation there’s a second mitigating factor…

• By default we format numbers according to the browser locale. So even if we had one US-based user with a UI in English and German-based user with the UI in German, those users could keep a column of numbers set to the default display options and likely be just fine, seeing numbers formatted as they’d expect for their locale. This would only work if those users had their browser locales configured correctly. So if the German person for some reason had their browser locale set to ‘en-US’, then they’d see US-formatted number in Mathesar. But I would venture to guess that in that case they’d probably be fine with seeing US numbers anyway.

To reiterate: I still want to support per-user display options, but I’m not convinced that we actually have a pressing problem right now. Personally, I find things like column width, column order, hidden columns, a minimum fraction digits to be the much more compelling things to make per-user. But the problem is that I also want those to be global in some cases too.

​

[Kriti Godey]

Anish, thanks for proposing this, and it's great that you're thinking about product changes and reducing work for the beta!

Here's some questions I have. I assume people might have differing opinions on these, so please share your opinion even if someone else has answered these.

(1) What are the issues / problems with making all Mathesar metadata (except explorations) be stored per-user, so we only have to manage sharing explorations in the new permissions system?

Sean, you mentioned that we'd still have to figure out sharing Mathesar metadata, could you say more about why?

(2) Would making all Mathesar metadata except explorations be stored per-user actually meaningfully simplify permissions UX and/or implementation?

If there are no significant issues and it simplifies UX or implementation, I think we should do it for beta. We are redoing permissions anyway, and we can always make some things global later.

Otherwise, I think we should put this conversation in the parking lot until after beta.

[Anish Umale]

Initially while writing this email, I only mentioned display options(i.e. Column Metadata) but I meant both Column and Table Metadata, I forgot that we store them separately, sorry for the confusion. 

I agree with you Sean that in the long term, we'd want more sophisticated metadata handling that allows for both the local and global storage of metadata.

To answer Kriti's questions:

(1) is exactly what I am proposing, I'd also like to point out that we already store metadata for explorations in a separate model and we intend to do so after the architectural overhaul as well, the point being that the metadata for exploration is already global and independent of other models, here is the link for the relevant model.

Sean, you mentioned that we'd still have to figure out sharing Mathesar metadata, could you say more about why?

I'd also like to know why.

(2) I'd say so, especially since we won't have to wrangle with metadata permission for each mathesar user either in the service layer or in the UX.

[Sean Colsen]

For the purpose of this topic, I’ll use “shared” to describe the case where multiple users have access to the same state (be it display options or explorations), and “private” to describe the case where each user only has access to their own state.

I support private display options — if it really does look like the quickest path to beta. I’d like to dig into this “quickest path” question by comparing shared vs private display options…

In either case

I think we’re going to need to:

• Keep explorations shared

• Allow all users to read exploration definitions while only allowing some users to write those definitions. We’ll need a mechanism to persist this access level in our internal DB, as well as API and UI to configure it. Plus some design work. This is a non-trivial amount of work that I see us having to do regardless of whether we make display options private.

  ☝️ That’s the work I was referring to when I said “I would argue that we still need to crack that nut”. This is in response to Anish saying “we won’t have to manage permissions for it”. My point here is that we’ll still need a way to manage permissions for explorations (one type of “metadata”, if you ask me). And since our tentative plan seems to be leveraging that same permissions mechanism to manage permissions for display options, there doesn’t seem to be much additional work to “manage permissions” for display options too. But perhaps we are interpreting “manage permissions” in different ways. So this is why I want to dig into the the detail with a comparison…

With shared display options

We’d also need to:

• Validate permissions in the service layer when modifying display options.

  We’d also need to perform this validation when modifying explorations, so I would imagine leveraging some of that code, with the additional code changes here being small.

• Validate permissions in the front end before presenting UI to modify display options.

  There would only be two code locations for this: re-ordering columns and column formatting settings. The front end would already need access to validate these permissions for explorations, so adding this validation to two additional code locations would be just a few lines changed.

With private display options

We’d also need to:

• Join on the user model when reading or writing display options in the service layer, gracefully handling anonymous (read-only) cases where no user exists.

  This is also easy, but seems roughly equivalent in terms of service-layer changes to shared display options.

• (I contend) Make some UX changes to communicate the private nature of display options. Based on the research I did of other products (presented here), I think users are likely to expect display options to be shared state generally. If we don’t communicate such private nature of the state, then I think we’re risking user confusion. The currency symbol for our money type is perhaps the most glaring example. If I set up a money column with a currency symbol of € (stored as a display option, mind you), I would very much want Mathesar to tell me that only I would see that currency symbol and that all other users would see $. Re-ordered columns is another example where I’d argue users would unequivocally expect shared state and become confused if they observed private state. Within the column “Formatting” section communicating shared state could be as simple as some help text. The column ordering is trickier though. How would we communicate the private nature of that state to users? I’m not sure. A toast message the first time they re-order columns? That’s annoying. It’s a tricky problem.

• Be okay sacrificing the ability for users to generate an anonymous link to a shared table without configuring display options for it. Or we’d need to figure out a work-around. This means there would effectively be no way to share a table to show any currency except dollars or to ever modify column order within a shared table.

In comparison

The work needed seems roughly equivalent to me with private vs shared display options. Thus I lean towards shared display options because I think it’s what users will expect. But if other people are seeing this “work needed” balance differently than me, I’m very much open to pursuing private display options for the sake of saving us time.

​

[Anish Umale]

Those are some great points, Sean! 

Before our meeting tomorrow, I wanted to toss out a couple of ideas about the UX for permissions related to Mathesar's metadata.

With private metadata(except explorations):

We won't need to have this option for each DB or for each schema on a DB:  

UX for explorations:

You mentioned that we'll need to:

Allow all users to read exploration definitions while only allowing some users to write those definitions. We’ll need a mechanism to persist this access level in our internal DB, as well as API and UI to configure it. Plus some design work. This is a non-trivial amount of work that I see us having to do regardless of whether we make display options private.

But I think we could potentially avoid that for the Beta, basically, only the user who is creating an exploration would be able to edit it globally and rest everyone will be able to alter that exploration but only temporarily and locally.

Given that everyone editing the exploration temporarily will at least need to have SELECT access on the underlying tables.

 

[Kriti Godey]

Can someone follow up here with the final conclusion?

[Sean Colsen]

A tentative conclusion is that display options will remain shared. Anish, Brent, and I discussed this from 2:18 to 15:00 in our 2024-05-16 technical beta checkin.

I think Anish hadn’t been privy to a meeting (perhaps before tl;dv) where we discussed consolidating the access control for all metadata into one enum field. I vaguely recall a call with me, Pavish, Kriti, and Ghislaine where we discussed this, but it was quite some time ago. In that meeting I made a case for this consolidation for the sake of simplicity. When proposing private metadata, Anish was operating under the assumption that we’d be implementing the more elaborate approach to metadata permissioning that Pavish had originally proposed.

In our 2024-05-16 (at 13:27) Anish said:

as long as we have only have one place where we show the metadata privileges, then I’m okay with like not having private metadata.

I wouldn’t call this decision “final” yet though. At 13:51 Brent said:

we’re probably gonna have to go back go back to this once we have more clarity on the permissions and models.

To add to this…. I recall having some debate at one point about where exactly this “one” metadata enum setting would actually be stored. Would it be per database/role? Or would it be per database/role/user? (I wrote my rationale for scoping this enum setting per database/role/user here.) But it’s hard reach a “final” conclusion here without having the actual models yet since we need to decide where in those models to put the field.

Anish, Brent, Pavish, please correct me if I’m wrong on any of the above!

​

[Kriti Godey]

Has this been finalized now that there's clarity on permissions?

[Sean Colsen]

Yes, my understanding is that all display options will be read/write globally. We discussed this on 2024-06-03 starting at 31:55. I imagine we won't allow writing metadata from anonymous users viewing shared tables and explorations though.

​

[Kriti Godey]

Thanks!