Criteria for closing use-reported tickets

[Sean Colsen]

I’ve been part of a couple tiny conversations in the past few days which have been insignificant on their own, but together made me want to open a broader conversation.

The question is this: when a Mathesar user opens a ticket, how do we know when it’s appropriate to close that ticket?

I’ll start with an analogy…

Last summer I was having a problem with the air conditioning in my house. The HVAC condenser unit in my basement was accumulating water in its drip pan that was overflowing onto my basement floor. I hired a company to fix it. The first tech did a bunch of troubleshooting and decided to replace the condensate pump. New pump — no more water on floor! Or so he thought… The drip pan took a few days to fill up with water, after which point I had water on my floor again. They sent out a different tech who walked into my house holding second replacement pump, apologizing that they had apparently swapped my defective pump for yet another defective one. Despite my suspicion, I deferred to them because I didn’t have the time to get my head into the project. The second replacement pump did not stop the water on my floor. Neither did a replacement condensate drain pipe installed by the third tech! What finally did the trick was the fourth tech who came out and looked at the problem more holistically. My filter needed to be changed! The filter was blocking airflow. The lack of airflow was causing ice to accumulate in are area of the condenser not designed to catch its drips. The melting ice bypassed the drainage line and condensate pump altogether.

~  ~  ~

How does this relate to us?

We have so few users right now that I think we need to treat them like gold. Every user interaction is an opportunity for us to learn about how people are using Mathesar and to evangelize our product. Every user interaction is also a risk wherein our response might dissuade the user from continuing to interact with us.

The two conversations I’m referring to are:

• In this email thread I suggested that we leave a ticket open in order to continue a conversation with the user and get more feedback about our direction. Kriti suggested we keep the ticket in the v0.1.3 milestone and assign it to me. There’s some nuance here, but to me that approach seems to reduce the user’s problem to making column type inference optional. Yes, the “optional inference” piece of puzzle is assigned to me and in this milestone. But there are potentially many other things to do as well that might also prove to be other important pieces to the puzzle the user is trying to solve.

  With many people working on one project, there is a real danger to getting focused on tasks and forgetting to return to the user’s original problem. This is like the guy who walked into my house holding a new condensate pump. Someone else had already formed the assumption that I needed a new pump. Then the tech’s task was “replace pump”. His task should have been “fix water on floor”, because that was my original problem.

• In this Matrix chat, Pavish said “I’m concerned it wouldn’t actually solve their problem”. That’s the same concern I’m trying to relay with this email. In this case Dom closed a ticket after we added some support for unknown pg types. But that’s not the problem the user reported. The user said:

  When I connect to an existing schema, which was built on the Graphile Starter project, there are no tables.

  I think this is a real gift this user has given us! They’ve provided very clear instructions on how to reproduce the problem. Granted, following those instructions may require some time. But that time would be well-spent I think. I’d be inclined to spend at least a few hours chasing that down before closing this ticket or asking the user to try again. I think it’s really worth it try try hand-holding users like this as much as we can, because I think it’ll help us learn a lot in the process.

Closing user-reported tickets is really a very new process for us! As such, I think it would be healthy to discuss it a little. It makes sense that we all may have slightly different approaches to that process. And I think it would useful to try and get on the same page about it.

My suggestion is that we should generally err on the side of keeping user-reported tickets open until either: (A) the the author confirms we’ve addressed their problem, or (B) the author fails to respond, or (C) we feel very confident that we’ve fixed it (i.e. by fully reproducing the problem ourselves exactly as the user described). This approach would require us to splitter user-reported tickets off into more-specific pieces that we can close more eagerly, just to properly track and organize our work.

If we were drowning in tickets and users and money I would absolutely not want to take the approach above. I would be much more inclined to close user-reported tickets eagerly. But given our early stage, I think it makes sense to hold tickets open for longer to signal that we’re still engaged in a conversation with the user about their problem, and that we still want to hear from them about the our ideas and our approaches to fixing their problem.

​

[Kriti Godey]

Thanks for this well-articulated email, Sean. I especially endorse this sentiment:

We have so few users right now that I think we need to treat them like gold. Every user interaction is an opportunity for us to learn about how people are using Mathesar and to evangelize our product. Every user interaction is also a risk wherein our response might dissuade the user from continuing to interact with us.

As far as the approach outlined for responding to user reported issues, I'm generally in agreement, but I'd like to have a time limit for "(B) the author fails to respond". 

My approach with GitHub issues is that each issue is an action item, and I'd like the associated action item to be very clear for each individual issue (I started a somewhat related discussion about writing better issues a while ago). I think it's fine for the action item to be "we are waiting X amount of time for the author to respond", and I think we should make this clear in a comment (e.g. "hey @author, we think this issue is fixed, please confirm, and if we don't hear from you by X, we'll close this issue"). But since issues are so integral to our workflow, both to get things done as a team and communicate with our community, I want to err on the side of clarity.

[Sean Colsen]

Kriti,

Thanks for your support on this topic. I hope I’m not belaboring it by sending another long email, but this topic seems important right now, and I have some follow-up thoughts.

My approach with GitHub issues is that each issue is an action item

I’d like to suggest a subtle alteration to this approach.

I see the tickets we create as action items — nothing more, nothing less. And the vast majority of the action items described by our tickets are code-related action items. When the the PR is merged, the ticket is closed. Then we put the ticket out of mind and move on. I’d like to avoid this mindset with users’ tickets.

I see the tickets that users create as a mix between action items, support requests, sales leads, and user research. Sure, we could conceivably lump all of these concepts into “action item”, just with different actions. But we are so accustomed to associating one ticket with one code-related action item that I think it may be helpful to adopt a different metal model for user tickets so as not to lose sight of those other goals.

One problem is that we can’t rely on users to clearly communicate code-related action items within their tickets. That leaves the door open for us to inadvertently form suboptimal assumptions about the best action to take. I think it’s worth approaching user-reported tickets with a healthy does of skepticism about our action items, always seeking to reinforce and validate those actions through very explicit user feedback.

Another problem is that “support requests” require more follow-up and hand-holding than pure code-related action items. For example in No tables visible when viewing an existing schema, I get the impression that we really left the user hanging, wondering what to do next. I would argue that the reason we left the user hanging is that we approached this ticket with the same type of action item in-mind as we would with one of our tickets — a code-related action item. Dom wrote a comment explaining why he was closing the ticket. That comment would be very commonplace for tickets we authored. But user tickets like this give us a (currently rare) opportunity to do a lot more to engage our users. Here’s my suggestion for how we might consider handling tickets like this in the future (with that ticket as an example):

@notramo

We’ve merged #3040 which we think might help with this issue you’re having.

• Before that change, Mathesar would crash when it encountered a PostgreSQL type for which we hadn’t provided explicit compatibility. Types like citext and point are examples of such types.
• After that change Mathesar no longer crashes when it encounters such types. However, it only offers limited functionality for such types because much of Mathesar’s behavior is type-specific. We still have more work to do to test Mathesar with implicitly-supported types and identify additional improvements that may be necessary to offer graceful degradation for these types without crashing. For example, sorting, filtering, and editing data of those types may still be problematic. We’d love to have your help testing Mathesar under this scenario if you’re interested! Please comment on this issue with any questions or additional information you can provide.

Our goal is to eventually offer full support for all PostgreSQL types, but this inevitably take some time, as we are building this functionality incrementally. We have tickets to track support for citext and point, but we are prioritizing support for new types on an as-needed basis. Are there any other types that you think would be important for us to prioritize soon? If so, feel free to open new tickets for those types.

I’ve done some rudimentary testing using Mathesar to access the PostgreSQL database from the Graphile Starter Project. Before #3040 was merged, I was able to reproduce the problem exactly as you described it. After merging #3040, I can see all the tables, as shown below.

(insert screenshot here)

Mathesar 0.1.3 will be released in about 3 weeks and will include this fix.

If you want to get this fix sooner, you can use this special docker image and updated the install script in this branch. Install with:

bash <(curl -sfSL https://raw.githubusercontent.com/centerofci/mathesar/2023-06-27/install.sh)

Note that we don’t recommend running the above version of Mathesar in production because it is subject to further testing before our release in order to identify and fix any regressions from our current cycle of development. We’ll also eventually be pushing develop Docker images to our Docker Hub, that work is tracked in #3076.

Would you be interested in testing the pre-release version above and reporting back with your findings to verify that we’ve fixed this problem for you?

If we don’t hear from you in 2 weeks, we’ll close this issue. Thanks for your help!

That comment asks the user questions. It invites the user to follow up through a variety of channels. It gives the user a lot more context about what we’re doing. It gives them clearer instructions for what to do next.

This email is not to say that we should have made a comment like the above. I definitely wouldn’t have thought to write all that before now! It’s more to suggest that, as we’re learning from our users (with that ticket as an example), we may want to consider comments like the above going forward. Those more hand-holding type of comments admittedly require much more effort! But I think it’s worth it for us to prioritize that work right now, especially given the relatively low frequency of user-reported issues. I’d be happy to help with crafting such comments any time someone asks!

Procedurally, we could consider applying a label like “status: waiting” (or a new label) to tickets when the ball is in the user’s court. I think it’s also worth it to avoid placing open user-reported tickets in our current milestone in order to encourage longer, more-nuanced conversations with the user about their needs without blocking our release on those conversations.

​

[Kriti Godey]

I see the tickets we create as action items — nothing more, nothing less. And the vast majority of the action items described by our tickets are code-related action items. When the the PR is merged, the ticket is closed. Then we put the ticket out of mind and move on. I’d like to avoid this mindset with users’ tickets.

I see the tickets that users create as a mix between action items, support requests, sales leads, and user research. Sure, we could conceivably lump all of these concepts into “action item”, just with different actions. But we are so accustomed to associating one ticket with one code-related action item that I think it may be helpful to adopt a different metal model for user tickets so as not to lose sight of those other goals.

I agree with all this, when I said "action item", I was including the concept of different types of actions, not just code. 

I think the example comment you wrote looks great and it makes sense for us to take that approach in the future. I'd like to hear from other members of the team if they feel comfortable with this mental model and authoring comments like the example above.

Procedurally, we could consider applying a label like “status: waiting” (or a new label) to tickets when the ball is in the user’s court. I think it’s also worth it to avoid placing open user-reported tickets in our current milestone in order to encourage longer, more-nuanced conversations with the user about their needs without blocking our release on those conversations.

I'm not sure about putting user-reported issues in later milestones. In my mind, user reported issues are top priority, and should always be in the highest-priority milestone (since we're using milestones to determine priority). We can always move them out to the next milestone if they remain unresolved by the time we cut the release.

[Dominykas Mostauskis]

I like Sean's suggested comment and general model.

The lack of certainty for when a user-submitted issue can be closed does make those issues somewhat different than the rest. I've even started to track them separately in my last project meta ticket (see related user-reported tickets). I then have a more defined issue that's sort of a proxy for the "uncertain" issue. Here's an example of an uncertain and proxy issue pair.

I'm fine with these "uncertain" tickets being in the milestone.

[Pavish Kumar Ramani Gopal]

> I'd like to hear from other members of the team if they feel comfortable with this mental model and authoring comments like the example above

Yes, I'm in agreement with all the points discussed above, and I think we should show extra care with user raised tickets.

> user reported issues are top priority, and should always be in the highest-priority milestone

I think it's okay with keeping user reported tickets in the current milestone. However, I do not think that we should provide exact time estimates to the users, unless we are 100% sure that we are going to make it.

[Dominykas Mostauskis]

So, to put this into practice, here's a post I made regarding closing of a user-reported issue:

@ricnov this issue should be fixed in the last release (0.1.2).

Would you be interested in verifying that your reported problem is indeed fixed?

If not, we'll close this issue in 2 weeks.

That's much more terse than what Sean showed as an example. I wanted to show that the fix is in the release, but I couldn't find a better way to do that than to link the actual code tagged with the release tag (this change was not in the release notes). I wanted to give the user a chance to verify that the problem is fixed, and I wanted to say that we'll close this unless the user reports something new.

PS: does anyone know of a convenient way to check if a given PR made it into a release?

[Kriti Godey]

I'd like to hear from Brent, Rajat, Anish, Ghislaine, and Mukesh also. I think it's especially important to hear from Rajat since he's doing repo admin and often is the first line of response to users.

Also, would it help anyone to write up some of this into a wiki page for "guidelines for responding to users" that could be referenced when you're working on responding to a user?

[Brent Moran]

Generally, I think the idea of waiting for a final 'it fixed my problem' from the reporting user is a good idea, and I support that. 

However, there are possible dangers. In previous projects, I've seen that attitude lead to issues devolving into ongoing conversations about various bits of feedback that may or may not be related to the original issue, since if a user is getting responses on a given channel, they tend to stick to that channel even if it's a comment chain in an unrelated GH issue. In fact, you'll sometimes see other users jump in since that's where the activity is. While it's great to keep getting feedback from the user(s), the feedback may end up being lost or forgotten if the comment chain gets long or complicated. In order to avoid losing or forgetting such feedback, and neglecting any work that might arise from it, I think we need to be mindful of when it's time to nudge a user to open a new issue (or we should open one for them and direct them to it). I don't think I'd be too pushy on that, given how valuable users are at that stage, but we can provide the best possible templates and guidance so pushiness will be less necessary. To help with that, I think we should consider taking a close look at our issue templates in order to improve them if possible to encourage the best-possible issue writeups from users with clearly defined goals and scope. This will help us determine whether the issue should be closed in case the user doesn't respond, and can also actually help guide them to some kind of mental 'checklist' about whether the issue is solved if they do respond. It will also make it clearer when a new issue is warranted.

With that said, to reiterate, even if they ignore the template I think we should wait for a sign-off from the user before closing their issue except in extreme cases.

[Dominykas Mostauskis]

I think an internal resource like a wiki page for how and when to close issues is a good idea. We could make it more general and pave the way for having resources for how to solicit valuable feedback from users (as we were discussing in Pavish's thread on improving user feedback).

[Anish Umale]

I agree with Sean's suggestions and do think a guide for responding to users will be helpful. 

[Sean Colsen]

I'll kick off a "User Communication Guide", as a wiki page. I like Dom's idea for the scope of this page. I'll write it so that it addresses "when/how to close user-reported tickets" as well as the "XY problem" -- but I'll keep details light until we have more agreement and strategies. The page will basically serve as a scaffold for us to flesh out as we continue making smaller decisions moving forward. I'll add this work to my plate for the upcoming cycle, but I'll do it after moving the wiki to mkdocs.

[Kriti Godey]

Sounds good! As mentioned in the meeting, please ask for feedback from a "is this useful as a reference for you?" guide.