Suggestion: Data Explorer UX changes after the release

[Pavish Kumar Ramani Gopal]

This mail is for future consideration, perhaps, soon after our initial release.

Having worked on the Data Explorer for a while now, and thereby being the one in our team that uses it the most, I find our current implementation severely lacking in capabilities to make it useful on a regular basis. While this is known and we've intentionally limited the features, below are a list of changes I'd like to make to the Data Explorer to make it more useful, thereby showcasing its full potential. Note that this mail focuses mainly on the UX, which I believe is our current bottleneck with Data Explorer.

Core change proposed in this mail: Transformations that truly transform data should be more top level.

•   Operations like joining, filtering, grouping, sorting, and hiding columns do not really transform data, they merely select, filter & change representation of the underlying data.
•   These operations need not be ordered. They are independent of each other.
•   Operations like summarization and visualization do actual transformation of data into a different processed result set. They convert the data entirely.
•   Currently, these are clumped together thereby limiting Data Explorer's UX and potential.

Below is a crude UX flow to make the problem mentioned above more clearer. I'm not proposing this UX flow to be implemented, I'm envisioning that the UX we come up with would solve the problem this flow solves along with all the benefits offered by it:

1. User enters Data Explorer. They can do the following:

•   They choose the base table.
•   There is a top level transformation bar which says 'Build your Exploration', which has disabled buttons for some more operations next to it. We'll ignore this here and come back to it in the next steps.
•   There is a 'Select Columns' panel, the same as what we have now, where they'll add columns, and join tables.
•   There is no 'Transform Results' panel.
•   They can filter, sort, group, hide columns, and reorder columns the same way they would do in tables, mimicking the same UX. This will be part of the results panel.
•   We'd likely have to provide a UX to hide & reorder columns in tables as well, which I think would be useful.
•   This will also enable us to open a table in the Data Explorer with the actual state of the table, copying the filters & other operations the user has applied on the table.
•   Users can edit the rows in the results in this state.
•   We'll have the Exploration Inspector as we have it now.

2. In the top level bar which says 'Build your Exploration', there are buttons that say 'Summarize', and 'Visualize' which become enabled only when there are some results.

•   This top level bar would look like a wizard, which makes it clear that the steps here are ordered.
•   When the user clicks on a transformation here, say, 'Summarize', we add a transformation step.
•   This will add an additional tab in the top level bar, which would look something like: `['1. Build your Exploration' > '2. Summarize Results']`
•   The content of the entire page changes. There is no longer a 'Select Columns' panel.
•   We'd have a left pane that now only has the option to select the 'Base grouped column', 'Grouped columns', and 'Aggregated columns'.
•   This is similar to the UX we have now for summarization within the transform panel.
•   The result pane will show the summarized results.
•   There will be options to add filters, grouping, sorting, and hiding columns here. These will be present above results.
•   Again, these options will have the same UX we have for tables. The columns shown for them will be resultant columns of the summarization.
•   The Exploration Inspector would be present as it did in the previous step.
•   Users cannot edit rows here. This view would be readonly.

3. The user can move between steps by clicking on the top level bar. If they want to join more columns after summarization, they can do so by going to `1. Build your Exploration`, and add a new column. They can then come back to `2. Summarize Results`.

•   We'd automatically figure out how to group or aggregate this newly added column, in the same way we do now.
•   We'll still have to figure out deleting a column after top level transformations are applied. We can show an alert to delete all further top level transformations that depend on the column the user intends to delete.
•   We'd have much better control on the UI states in general, particularly with showing errors.

4. The user can then add more summarizations or visualizations on top of the current state. The top level wizard view bar would keep appending the operations.
  - Eg., `['1. Build your Exploration' > '2. Summarize Results' > '3. Summarize Results' > '4. Visualize Results']`
  - We can limit operations here, saying that users cannot perform any additional transformation after a visualization.

In addition to making the UX more saner, this flow would also make it a lot (I mean a LOT!) easier for the frontend. Right now, state handling in Data Explorer is very complex and a massive pain due to these two completely different categories of operations being clumped together.

Having said that, this may require significant refactoring, UX and some backend work, but I am certain an updated UX is worth it. I do not see the features we plan for Data Explorer in our roadmap becoming possible with the current UX.

[Dominykas Mostauskis]

I like the sound of a lot of this. The observations I find most useful are that we can reorder some transformations without affecting the result and that we have two basic types of transformations (or operations as Pavish called them) and they might warrant fundamentally different handling.

I can't seem to find major problems with Pavish's suggestion. I was trying to imagine where formula columns would go in this workflow, and I think they would be one of the "orderless" transformations, but since their order does matter (if you add two formula columns, one might depend on the other), we can just consider the batch step of applying formula columns as a single transformation.

I've a maybe pedantic note that there is an inaccuracy in saying that you can join, filter, group, sort, hide in any order without affecting the result. If you, for example, [hide, sort, join] (in that order), you'll end up with an invalid query. A more precise (but more verbose) way of saying what Pavish meant, I think, is that these transformations _when batched_ can be reordered within the batches without effect, but the batches themselves have to be applied in a specific order (joining always first, hiding always last, formulas always before hiding): e.g. what a batched transformation pipeline might look like `[[join column 1, join column 2], [filter1, filter2, filter3], [group1, group2], [sort1, sort2], [formula column 3], [hide column 2]]`.

[Pavish Kumar Ramani Gopal]

if you, for example, [hide, sort, join] (in that order), you'll end up with an invalid query. A more precise (but more verbose) way of saying what Pavish meant, I think, is that these transformations _when batched_ can be reordered within the batches without effect, but the batches themselves have to be applied in a specific order (joining always first, hiding always last, formulas always before hiding)

@Dominykas Mostauskis When I mentioned that the order does not matter for these operations, I mentioned that it does not matter to the user, they do not have to be concerned about it. The user can hide, sort, and then join columns, and the ordering among these happens in the backend, since we can deterministically say which order we'd want these operations in. i.e the one you mentioned in the mail: [joins, filters, groups, sort, formula, hide].

For the other transformations (summarization & visualization) however, the order needs to be explicitly mentioned by the user and they have to be aware of it.

[Kriti Godey]

Thanks Pavish! I'll treat this email as a roadmap suggestion and integrate into the larger roadmap.