About feedback gathering and article types
David Tenser
very relevant to this thread.
Right now we don't seem to have a technical separation between the
"help" and "support" article categories [1]. Is this because we haven't
decided on the best way to do that yet, or because we haven't enforced
it? A quick solution would be to use tags for this, e.g.
"_troubleshooting" (the _ would make this tag hidden if bug 398761 is
fixed). Are there other, better alternatives?
[1] http://wiki.mozilla.org/Support:PRD#Content_Types
In a meeting with Mike Beltzner a couple of weeks ago, we discussed the
need to separate "help" and "support" articles, and that the in-product
help would only show the "help" articles in the search result (with a
link at the bottom saying e.g. "We also found 4 possibly relevant
troubleshooting articles on the web help."). In order to do this, we
would perform search queries based on different tags. One concern I have
here is that we're using Google to search the KB today. Is it possible
to do search queries like "find all articles with the words 'foo' and
'bar' in them, excluding articles with the tag '_support'"?
Now to the point of this post. I had an unplanned meeting with mconnor
today and he had many interesting ideas (and I'm not sure if I remember
everyone of them, but I'm doing my best here!) that I'd like to discuss.
It was originally a follow-up on the discussion about the feedback area
of the kb articles ("Was this article helpful?").
We've had suggestions here about changing that feedback question to "Did
this article solve your problem?", and the main concern I had was that
it still is not good data, because people can vote No because a) they
didn't have a problem in the first place, b) they ended up on the wrong
troubleshooting article, or c) they didn't understand the article
because it was poorly written. Also, I objected to it because it only
applies to "support" articles (see [1] above).
Now, the first concern is still a bit tricky, and mconnor didn't really
have a solution to it (unless I already forgot about it), but he
basically wanted to use the data for a different purpose that my
original intention. I wanted to use the data to measure the quality of
the article, and mconnor wants to use the data as an indicator of what
most people have problems with in Firefox. We discussed having two
independed questions at the end of articles, where the other one would
be strictly about the quality of the article -- not whether or not it
solved a particular problem for the user. Something like "Was this
article easy to understand?"
The second concern (only applies to "support" articles) can be solved by
-- and I mentioned this yesterday -- using different questions for
different types of articles. This goes back to the separation between
"support" and "help" articles again. We need that separation even in the
data gathering. For "support" articles, we need to ask the question "Did
this article solve your problem?", and for "help" articles, we need to
ask something like "Was this the information you were looking for?".
I'll stop here. Have more to post but let's try to limit the scope of
this post. :)
David Tenser
> We've had suggestions here about changing that feedback question to "Did
> this article solve your problem?",
For reference, these discussions started around here:
http://groups.google.com/group/mozilla.support.planning/msg/507564dac4ed0f08
David McRitchie
> "help" and "support" article categories [1]. Is this because we haven't
> decided on the best way to do that yet, or because we haven't enforced
> it? A quick solution would be to use tags for this, e.g.
> "_troubleshooting" (the _ would make this tag hidden if bug 398761 is
> fixed). Are there other, better alternatives?
>
> [1] http://wiki.mozilla.org/Support:PRD#Content_Types
Not clear if you are mentioning this because they should or should
not be hidden.
Is there a real reason to have/need that "_troubleshooting" and
"_support" be hidden tags. Because I'm all for being able to use
any search engine, and not be related to being logged in or out
or being a contributor or not (bug: 398761). Would it be
sufficient if only certain people can assign or change certain
tags but that everyone could see them, and especially these
since they are not related to tracking in any manner.
Mike Connor
> Now, the first concern is still a bit tricky, and mconnor didn't really
> have a solution to it (unless I already forgot about it), but he
> basically wanted to use the data for a different purpose that my
> original intention. I wanted to use the data to measure the quality of
> the article, and mconnor wants to use the data as an indicator of what
> most people have problems with in Firefox.
I think you forgot about something. :)
The biggest reason for this is to sort on relevance, and this is the
best way to integrate real relevance data without manual
intervention. The other bit about closing the support link is a bonus
for the same data.
-- Mike
Jason Barnabe (np)
> We've had suggestions here about changing that feedback question to "Did
> this article solve your problem?", and the main concern I had was that
> it still is not good data, because people can vote No because a) they
> didn't have a problem in the first place, b) they ended up on the wrong
> troubleshooting article, or c) they didn't understand the article
> because it was poorly written.
If people didn't have a problem in the first place, why would they
even be reading that article?
I'd also say that the user ending up at the wrong article is an
indication of a poorly written article. Introductions to articles
should provide a clear description of the problem and possible
alternate articles.
> I wanted to use the data to measure the quality of
> the article, and mconnor wants to use the data as an indicator of what
> most people have problems with in Firefox.
Could this not be determined from other metrics, like article
popularity and search term stats? Using those metrics isn't perfect,
but asking users isn't perfect either.
David Tenser
> "David Tenser"
>> Right now we don't seem to have a technical separation between the
>> "help" and "support" article categories [1]. Is this because we
>> haven't decided on the best way to do that yet, or because we haven't
>> enforced it? A quick solution would be to use tags for this, e.g.
>> "_troubleshooting" (the _ would make this tag hidden if bug 398761 is
>> fixed). Are there other, better alternatives?
>>
>> [1] http://wiki.mozilla.org/Support:PRD#Content_Types
>
> Not clear if you are mentioning this because they should or should
> not be hidden.
>
> Is there a real reason to have/need that "_troubleshooting" and
> "_support" be hidden tags.
These particular tags might not have to be hidden, but we would have to
be very strict about enforcing the use of them.
David Tenser
And to clarify, you mean that if an article gets a high score of the
"Did this article solve your problem?" question, that means it's a
relevant support article because many other users have had that problem
before.
And with "closing the support link", you mean to close the feedback loop
to the Firefox developers, i.e. Sumo providing feedback of what issues
the developers need to prioritize.
Please let me know if I got this right. Thanks!
David Tenser
> On Oct 19, 6:39 pm, David Tenser <djst.mozi...@gmail.com> wrote:
>> We've had suggestions here about changing that feedback question to "Did
>> this article solve your problem?", and the main concern I had was that
>> it still is not good data, because people can vote No because a) they
>> didn't have a problem in the first place, b) they ended up on the wrong
>> troubleshooting article, or c) they didn't understand the article
>> because it was poorly written.
>
> If people didn't have a problem in the first place, why would they
> even be reading that article?
Could be because they searched for a problem, but the wrong article
appeared first (as my example showed in the other thread).
>
> I'd also say that the user ending up at the wrong article is an
> indication of a poorly written article. Introductions to articles
> should provide a clear description of the problem and possible
> alternate articles.
Or an indication of a poor search engine. One thing I think can become
an issue is that there are hundreds of ways to express the same thing.
"lost bookmarks" is one thing, "bookmarks gone" is another, "wiped
favorites" is a third. :) Is it possible to somehow make it catch all
three using keywords or something?
>
>> I wanted to use the data to measure the quality of
>> the article, and mconnor wants to use the data as an indicator of what
>> most people have problems with in Firefox.
>
> Could this not be determined from other metrics, like article
> popularity and search term stats? Using those metrics isn't perfect,
> but asking users isn't perfect either.
>
I guess this is a question directed more to mconnor, but I guess one
benefit of this system is that we get a real proof that someone solved a
problem using the article, which proves that people have problems with
this in Firefox. That an article gets a lot of page hits can probably
have other reasons as well. Using search term stats is something I'd
really want to investigate more into.
And I'd love to find the perfect metrics. :)
David Tenser
the following basic changes (basic meaning "not fully thought through yet"):
* Change start page so we direct people with problems to the "support"
articles, and others to the "help" articles. How can we do this in a way
that makes sense?
* For "support" articles, ask the question "Did this article solve your
problem?"
* For "support" articles, if the user clicks "No" on the question
"Did this article solve your problem?", show 5 alternative
support articles sorted by relevance ("Yes" click score).
* For "help" articles, ask the question "Was this the information you
were looking for?" It would make sense to store the search terms used here.
* Ask two independent questions at the end of each article. One being
the above question that is different depending on the type of article,
and the other one being a question about the _quality_ of the article.
Could be e.g. "Was this article easy to understand?" Any thoughts on that?
My suggestion would be to technically just use the current TikiWiki
solution for the second question, and then implement the other article
type based question as a new feature, since that one also needs to store
other data such as the search terms.
David Tenser
> So to continue with the things mconnor and I talked about, we agreed on
> the following basic changes (basic meaning "not fully thought through
> yet"):
To clarify again, we agreed on the following _suggested_ changes. Really
would appreciate feedback.
Jason Barnabe (np)
> Jason Barnabe (np) wrote:
> > Could this not be determined from other metrics, like article
> > popularity and search term stats? Using those metrics isn't perfect,
> > but asking users isn't perfect either.
>
> I guess this is a question directed more to mconnor, but I guess one
> benefit of this system is that we get a real proof that someone solved a
> problem using the article, which proves that people have problems with
> this in Firefox. That an article gets a lot of page hits can probably
> have other reasons as well. Using search term stats is something I'd
> really want to investigate more into.
>
> And I'd love to find the perfect metrics. :)
Articles often cover one symptom but multiple bugs. For example, the
"Firefox hangs" article covers many different bugs in the product,
which are mostly in their own section. If we could get section-by-
section helpfulness stats, that'd be closer to perfect. We could also
use this info to organize the topics inside the article, putting the
most common first.
Jason Barnabe (np)
> * For "support" articles, if the user clicks "No" on the question
> "Did this article solve your problem?", show 5 alternative
> support articles sorted by relevance ("Yes" click score).
I don't see the point to this as compared to putting the alternative
articles in the intro as we sometimes do now. Putting them in the
intro means the user is less likely to read through the article that
won't help them.
> * Ask two independent questions at the end of each article. One being
> the above question that is different depending on the type of article,
> and the other one being a question about the _quality_ of the article.
> Could be e.g. "Was this article easy to understand?" Any thoughts on that?
I don't think we'd get good stats from this. We'd likely get mostly
the same response as the first question.
David Tenser
A way to work around that would be to link to separate articles for
separate issues. I think we're running into a very complex db design if
we want to support section based feedback, plus a complex page layout.
Jason Barnabe (np)
> A way to work around that would be to link to separate articles for
> separate issues.
That wouldn't be a very good user experience.
> I think we're running into a very complex db design if
> we want to support section based feedback, plus a complex page layout.
I personally think we should just leave the feedback largely as is and
wait to see what problems arise. We have limited resources and I'd
prefer they be directed at things we know to be problems right now
rather than speculating what problems could arise in the future. We
could write Mike's ideas on wiki.m.o and revisit them if we find that
our article feedback isn't doing the job to our satisfaction.
David Tenser
> On Oct 22, 11:46 am, David Tenser <djst.mozi...@gmail.com> wrote:
>> A way to work around that would be to link to separate articles for
>> separate issues.
>
> That wouldn't be a very good user experience.
Not sure if I agree it would be a bad experience. Using sub-articles for
different solutions of the same general topic might help us provide
better searching. The sub-articles would of course link back to the main
article so people could navigate.
>
>> I think we're running into a very complex db design if
>> we want to support section based feedback, plus a complex page layout.
>
> I personally think we should just leave the feedback largely as is and
> wait to see what problems arise.
That's what I think too.
Majken Connor
Jason Barnabe (np) wrote:
>
>> I think we're running into a very complex db design if
>> we want to support section based feedback, plus a complex page layout.
>
> I personally think we should just leave the feedback largely as is and
> wait to see what problems arise.
That's what I think too.
Can we not separate the how to guides vs troubleshooting via categories? We all agree this should be done, so this is one suggestion we should look at implementing now.
Anything we leave to the future becomes more complex as we get more users and more helpers used to the way things work. It's good to wait on things we're not sure of, or things we really don't have time for (and no one in the community has no time for) but for things that we can foresee as being an issue, or gaining us a win we should look into making the time to do it right the first time, or at least close to right so that the difference when we do it the way we really want is negligible to the users.
I disagree with the need for section by section feedback. This is what the comments are for, and soon, the forums. We don't need to collect stats on those bits because those will be the people who still need to ask a question. If we do want to collect more stats, maybe we can consider some sort of survey for now? This would let us implement some of the improvements, but cover the ideas that are more work and might not be worth it.
David Tenser
>
>
> On 10/22/07, *David Tenser* <djst.m...@gmail.com
>
> Jason Barnabe (np) wrote:
>
> >
> >> I think we're running into a very complex db design if
> >> we want to support section based feedback, plus a complex page
> layout.
> >
> > I personally think we should just leave the feedback largely as
> is and
> > wait to see what problems arise.
>
> That's what I think too.
>
>
> We should implement the low energy changes that we can reasonably
> determine will be an improvement. Much of what mconnor is talking about
> is based off of his ideal that he had sketched out before sumo was
> conceived, as it were.
Yeah. That's why I'm suggesting we keep the question we have for the
"article quality" aspect, but reword it to something like "Was this
article easy to understand?" Then we need to implement the article type
specific question in TikiWiki.
>
> Can we not separate the how to guides vs troubleshooting via categories?
> We all agree this should be done, so this is one suggestion we should
> look at implementing now.
One way to do it as I suggested was to use tags for it. Another way
would be to use categories (but we're currently using that for staging
vs not staging).
>
> Anything we leave to the future becomes more complex as we get more
> users and more helpers used to the way things work. It's good to wait on
> things we're not sure of, or things we really don't have time for (and
> no one in the community has no time for) but for things that we can
> foresee as being an issue, or gaining us a win we should look into
> making the time to do it right the first time, or at least close to
> right so that the difference when we do it the way we really want is
> negligible to the users.
Yeah I agree. That's what these discussions are for, I think, as we
should try to find the reasonable solution to our needs now, and then
work out a plan on how and when to implement it (prioritization). The
milestones might need to be revisited.
I would say that these feedback loops are a very nice thing to have, but
milestone 0.1 - 0.3 covers even more important features. To reiterate:
0.1 - Technical stuff needed before we can start linking to Sumo. This
is required to even start promoting Sumo, so this is top priority.
0.2 - Localization. Required so we can attract a bigger community.
0.3 - Forums.
0.4 - Live Chat
Even a basic knowledge base with l10n support is a very good
achievement, but we need to figure out the priority of this.
mconnor mentioned the possibility to get a coder to work on this
full-time for e.g. 3 months. If we could get that, things would be able
to roll out much quicker than currently.
>
> I disagree with the need for section by section feedback. This is what
> the comments are for, and soon, the forums. We don't need to collect
> stats on those bits because those will be the people who still need to
> ask a question.
Right. I agree.
> If we do want to collect more stats, maybe we can
> consider some sort of survey for now? This would let us implement some
> of the improvements, but cover the ideas that are more work and might
> not be worth it.
I don't think this is needed at this point.
Jason Barnabe (np)
> Jason Barnabe (np) wrote:
> > On Oct 22, 11:46 am, David Tenser <djst.mozi...@gmail.com> wrote:
> >> A way to work around that would be to link to separate articles for
> >> separate issues.
>
> > That wouldn't be a very good user experience.
>
> Not sure if I agree it would be a bad experience. Using sub-articles for
> different solutions of the same general topic might help us provide
> better searching. The sub-articles would of course link back to the main
> article so people could navigate.
There are cases where sub-articles could work. For example, in the
"Firefox hangs" article, each of the sections has a different symptom
- "Hang when downloading", "Hang when loading web pages", etc. The
user would understand which they're experiencing. In this case, only
one of the sections applies to each user.
There are cases where this wouldn't work. For example, in "Firefox is
already running but is not responding", there is no unique symptom to
each cause. The user doesn't know which they're experiencing, and any
one of the sections could apply to each user. A user would have to
click back and forth between articles to check each thing.
So in short, if you can create articles that the user can correctly
choose between just by the titles, they then can (not necessarily
"should") be separated. If you can't, they can't.
David Tenser
Exactly. And this is something we can work more on as we actually have
the feedback loop up and have data to gather. Good general thoughts here.
Majken Connor
>
> Can we not separate the how to guides vs troubleshooting via categories?
> We all agree this should be done, so this is one suggestion we should
> look at implementing now.
One way to do it as I suggested was to use tags for it. Another way
would be to use categories (but we're currently using that for staging
vs not staging).
Could we not then have 2 categories for live? "live-support" and "live-help" or are we doing things in an interesting way that makes that difficult?
Jason Barnabe (np)
Alternately, can we put a category inside another category?
Chris Ilias
> Alternately, can we put a category inside another category?
Yup. Tested it last night:
http://support.mozilla.com/kb/Tag+Samples
Chris Ilias
> And to clarify, you mean that if an article gets a high score of the
> "Did this article solve your problem?" question, that means it's a
> relevant support article because many other users have had that problem
> before.
If so, I think that would be a very inaccurate metric. Some articles are
not popular, just very well written. And vice-versa. Plus, some people
just won't bother voting.
I think things like most popular articles and most popular search terms
are better.
http://support.mozilla.com/tiki-wiki_rankings.php?limit=500
http://support.mozilla.com/tiki-search_stats.php
Chris Ilias
> Jason Barnabe (np) wrote:
>
>> I'd also say that the user ending up at the wrong article is an
>> indication of a poorly written article. Introductions to articles
>> should provide a clear description of the problem and possible
>> alternate articles.
>
> Or an indication of a poor search engine. One thing I think can become
> an issue is that there are hundreds of ways to express the same thing.
> "lost bookmarks" is one thing, "bookmarks gone" is another, "wiped
> favorites" is a third. :) Is it possible to somehow make it catch all
> three using keywords or something?
I agree. Plus, some people, I suspect, I are not using search. They are
using links on the home page; which would indicate a need for TOC on the
home page.
David Tenser
> On 10/22/07 1:44 AM, _David Tenser_ spoke thusly:
>> And to clarify, you mean that if an article gets a high score of the
>> "Did this article solve your problem?" question, that means it's a
>> relevant support article because many other users have had that
>> problem before.
>
> If so, I think that would be a very inaccurate metric. Some articles are
> not popular, just very well written. And vice-versa. Plus, some people
> just won't bother voting.
Not sure what you mean here. If someone votes "Yes" on "Did this article
solve your problem?", that means someone actually had that very problem
with Firefox. Do you mean inaccurate in that we would only catch a
subset of the people who actually had the problem?
David Tenser
> Some articles are
> not popular, just very well written. And vice-versa.
Which is why we need to separate the problem solving metrics from the
'article quality' metrics. An article can be popular (page hits) but
still horribly written, and vice-versa.
David Tenser
I sent this privately to Lucy (because Lucy cc's my e-mail on her
replies and I didn't pay attention, grumble grumble):
That could be possible [using categories]. I'm not sure which would make
more sense as far as the implementation of the feedback question feature
goes (since a category is exclusive, it might make more sense to use
that as a separation of the feedback questions).
Using categories seems more logical, since an article is either a
support or a help article.
David Tenser
Yes, that was meant as step two in the layout reorg that we've started
for the article pages. We also need to figure out a good way to slot
users into either a "help" search, or a "support" search. (It doesn't
make sense to use "lost bookmarks" as the #1 search result if the user
is only interested in learning more about bookmarks. Similarly, it makes
little sense to show information about what bookmarks are if someone has
a problem and searches for "bookmarks gone".)
Majken Connor
I asked if sub categories were possible in #tikiwiki, here is the response
<kerrnel22> I haven't tried it, but I assume that unless you specifically set different perms for a sub-category, it'll inherit the perms of the parent. So you can have Cat 1 with a set of perms that get duplicated by 1.1, 1.2, 1.3, but then you want 1.4 to have a different set, so you override it and change them for 1.4
_______________________________________________
support-planning mailing list
support-...@lists.mozilla.org
https://lists.mozilla.org/listinfo/support-planning