Recent changes to comments on Chrome Web Store

[Alexandra Levich]

Hi everyone,

A few days ago we released some changes to the way we sort comments from our users. Instead of sorting these by recency and showing our users a lot of comments which might not be so helpful, we are now sorting these comments by quality and helpfulness to the users. Thank you for your feedback on this change. We are listening and are taking all of your feedback into consideration as we iterate and work more on this feature. Our goal is to make comments be as helpful as possible for our users and our developers.

Thanks,
Chrome Web Store team

[Michael Gundlach]

Thanks for the update, Alexandra.

FYI, the first comment on AdBlock (https://chrome.google.com/webstore/detail/gighmmpiobklfepjocnamgkkbiglidom) is marked "0 out of 4 people found this review helpful" and it's 6 months old, so it looks like the sorting is broken.

Michael

--
You received this message because you are subscribed to the Google Groups "Chromium Apps" group.
To post to this group, send email to chromi...@chromium.org.
To unsubscribe from this group, send email to chromium-app...@chromium.org.
For more options, visit this group at http://groups.google.com/a/chromium.org/group/chromium-apps/?hl=en.

[bill m]

I believe the new comment order is worse than the original
chronological order.

My app has had one negative review. This one negative comment referred
to an out of place apostrophe. I fixed this typo the next day, and
there have been several versions since then. But the original negative
comment is near the top as if it were a valid criticism of the current
version.

If I were reading a review of the Chrome browser and I saw "Warning,
this browser freezes the computer when loading a large PDF file", --
which was a valid criticism of the original version of Chrome a few
years ago -- I would be likely to avoid downloading Chrome based on
this out-of-date review.

[Xan]

I can't believe this is an intended change.

This change is breaking the communication between users and extension
developers.
Comments tend to be organised like this: user feedback -> developer
response -> user feedback -> developer response.
With this new system, this is utterly disrupted.

Furthermore, comments tend to lose validity. There are two most
prevalent types of feedback, in my opinion: "Wow this is awesome" and
"I have a certain bug". The latter tends to get fixed, but with the
new system there's a mess of old bug reports floating to the top,
confusing users (and developers!) as to the real state of things.

And lastly, you are evaluating "quality and helpfulness" by some
heuristic that is bound to be inaccurate. The last system at least was
deterministic and logical.

Please revert this change ASAP.
I will not dare to update my own extension while this is in effect,
since I won't have a chance to collect feedback and correct something.

Regards,
Alexander.

[Xan]

P.S. I raised a bug about this before I learned it was an intentional
change.
http://code.google.com/p/chromium/issues/detail?id=94251

On Aug 26, 10:18 pm, Alexandra Levich <lev...@chromium.org> wrote:

[Michael Gundlach]

On Fri, Aug 26, 2011 at 4:46 PM, Xan <mexmat....@gmail.com> wrote:

I can't believe this is an intended change.

I know Google tends to iterate quickly, so they may improve their ranking quite soon, and their new Web Store layout may make it clear to users that these are reviews, not bug reports.

I agree the current system is crippled, but I expect it'll be improved forthwith rather than reverted.

Michael

[Xan]

Well, look at the Android Market in comparison. Reviews there are
clearly marked as "Reviews", but that doesn't stop people from leaving
1-star reviews with their complaints, even if the description pleads
to contact about issues differently. Most people just use the most
easily accessible method to leave a word - and they expect to be
heard. And this is app/extension's own page.

As for reverting vs improving - this is an interesting idea, but it
would be logical to implement it just like the language filter is
implemented - a prominent link to see an alternative "sort by
importance(?)" view.

But for reasons stated above the default view should be brought back -
this is my personal opinion. Without any implemented feed/
notifications for comments developers have no chance to monitor
feedback.

On Aug 27, 2:12 am, Michael Gundlach <adblockforchr...@gmail.com>
wrote:

[anstosa]

So if someone reports an issue in my comments, I will NEVER SEE IT
unless for some reason everyone thinks it's helpful...

This obviously didn't go through ANY testing because anyone who
actually had an extension in the Web store would have instantly
identified this as a problem.

What were you thinking?

[trustamli]

I thought that was a bug.
But my app's comments aren't sorted by user feedback.
I can't see any mean in this sort order.

My apps are used to be free and the only gain for me is positive user
comments I get every day.

On Aug 27, 8:07 am, anstosa <anst...@gmail.com> wrote:
> So if someone reports an issue in mycomments, I will NEVER SEE IT

> unless for some reason everyone thinks it's helpful...
>
> This obviously didn't go through ANY testing because anyone who
> actually had an extension in the Web store would have instantly
> identified this as a problem.
>
> What were you thinking?
>
> On Aug 26, 1:18 pm, Alexandra Levich <lev...@chromium.org> wrote:
>
>
>
>
>
>
>
> > Hi everyone,
>

> > A few days ago we released some changes to the way we sortcommentsfrom our

> > users. Instead of sorting these by recency and showing our users a lot of

> >commentswhich might not be so helpful, we are now sorting thesecommentsby

> > quality and helpfulness to the users. Thank you for your feedback on this
> > change. We are listening and are taking all of your feedback into
> > consideration as we iterate and work more on this feature. Our goal is to

> > makecommentsbe as helpful as possible for our users and our developers.

[trustamli]

There are actually more problems connected with this issue.

In time-ordered comments I used to response users comments,
And now my responses appear randomly, that make no sense,
because lots of them are about already fixed problems,
and that is really confusing for users.

They even began marking these comments as not useful.

The date of the last comment was kind of indicator of the popularity.

These changes are good only for the apps with 10-15 comments,
but really causes problems for popular apps on the web store.

[Xan]

Web Store Team, please, can we have any kind of reaction from you,
either here or on the bug tracker? It is unanimous that this is viewed
as a serious issue, but we hear nothing from you for 4 days already.
Please comment on the situation. Thanks.

Regards,
Alexander.

[Alexandra Levich]

Hi everyone,

We have continued to listen and hear your feedback on these forums. We realize the importance of this feature for developers and are working on iterating and improving the comments based on your feedback. We ask for your patience while this work goes on and will update soon as changes are rolled out.

Thanks for your patience and understanding,

Chrome Web Store team

[trustamli]

Alexandra, thank you for the response.
I believe we will get a better comments system soon.

I just wanted to suggest one more thing regarding user feedback on the
web store.
It would be great to have some Web Store API to access comments and
other usage/installs/rate details remotely.
Having this API developers would be able to display more information
about their apps on their own websites.
Or alternatively you can just add an option to get that comments via
RSS feeds.

[Michael Gundlach]

Trustamli, there is/was an extension that could show you these details about all extensions you owned.  I think it died when the Extensions Gallery closed down.  But surely you could do the same thing on the Web Store -- just scrape the relevant pages daily to get the stats you want, until a better API comes along.

--

[Phil]

>
> If I were reading a review of the Chrome browser and I saw "Warning,
> this browser freezes the computer when loading a large PDF file", --
> which was a valid criticism of the original version of Chrome a few
> years ago -- I would be likely to avoid downloading Chrome based on
> this out-of-date review.
>

This. There are negative reviews mentioning old already-fixed problems
filling up the first page of reviews of my app. I'm certain this will
put people off downloading.

Phil

[stinkinrich88]

It was obviously a ridiculous change to make. Please just revert back
to the old comment system. I don't want to have to wait while you
develop a new comment system, there was nothing wrong with the
original!

[Hansifer]

There is some form of limited Web Store app/ext stats API available.
It is used by the My Extensions (https://chrome.google.com/webstore/
detail/igejgfmbjjjjplnnlgnbejpkpdajkblm) extension, along with Web
Store screen scrapes, to provide extension stats. It does a pretty
good job and I recommend it for stats tracking.

Anyway, these APIs are also used by the Web Store to display some of
the data on app/ext pages via XHR. There doesn't seem to be any
documentation on these anywhere, so everything I know about them I've
gleaned from an inspection of these pages. If any Google staff reading
this can point us to an official reference for these APIs, it would be
much appreciated.

Anyway, if you check out the XHR calls made by any extension page via
Chrome Dev Tools, you'll find these two:

For extension comments:
https://chrome.google.com/reviews/json/search

For star ratings breakdown:
https://chrome.google.com/reviews/json/lookup

You can play around with the request payload parms to "discover"
different behaviors of these APIs by submitting via Fiddler or similar
tool. Some parms are pretty self-explanatory. For example, you may
notice that the "search" call now has a "sortBy":"quality" parm that,
if omitted, will yield comments in the original desc date order that
we all know and love. Using Fiddler, I was also able to bulk-download
the JSON for all comments to-date submitted for my extension, in desc
date order (if you try this, make sure you fetch them in increments of
100).

Given all the outcry, it would be simple (and wise) for Google to just
remove the "sortBy":"quality" parm to revert comment sorting to the
date desc sort in the interim while this all gets sorted out... that's
just my 2c editorial to add a little fuel to the fire :)

Cheers,
Hans

[trustamli]

Yes, I can also see this sortBy: 'quality' in a javascript on web
store pages:

var commentPanelArgs = {
...
sortBy: "quality",
...
}

Really curious to see it's implementation, what does it mean sort by
'quality'?
How do they decide which comment is good and which one is bad.
As we can see it's not based on user feedback on comments.

I've tried to play with URLs provided by Hans, they really work.
But unfortunately, it seems they are not accessible from other
domains.

--
Turan

[Hansifer]

Turan, that's right. You won't be able to access these APIs via XHR
from your own domain.

If you want to build a web app that uses them, they'd have to be
queried directly from the server.

Hans

[James Schubert]

Here's a wild idea... allow the user to choose a sort order. Default
to relevancy, but provide an option for sorting by date (desc/asc),
rating (positives first, negatives first), or relevancy (how it is
now).

By sorting out of chronological order without an option to change that
order, this is breaking what users and developers both expect from a
list of comments.

[Carl]

I'll add my own coals to this fire by simply noting that dialogues
between developers and users are sometimes carried on through these
comments and disrupting the chronology of those comments makes them
incomprehensible.

I also agree with others who have said that the more recent comments
reflect the most recent state of the app, and so would tend to be most
helpful to place at the top.

[trustamli]

As a workaround I "marked as spam" the comments that doesn't make any
sense in that order. Also deleted my own replies about already fixed
problems.
Now I have only 3 comments on main page, however I'm getting 1-3
comments every day since January 2011.
When I click next I get other comments, but coming back still only 3.

--
Turan

[Xan]

trustamli, I doubt this really works like that.
I'm sure one vote for "mark as spam" is not enough to hide the content
from everybody, just for you. Try viewing your extension's page in
Incognito.
Also, abuse of this feature may be frowned upon by Google, even in
these circumstances.

On the topic, I support James Schubert's having multiple different
views users can select, but I think the date sort should be default.

Also, I don't understand Google's problem in temporarily reverting
this change while they are "iterating and improving", since the
ability to pull comments in that order is still there, as shown by
Hansifer.

Regards,
Alexander.

[trustamli]

You are right Xan, I was fooling myself.

[Sterling]

While you work on a fix for what you broke (sorry, I couldn't help
it), why not just revert back to the was comments were sorted? I'm
having a problem with an extension, but now I won't post a comment on
the developer's Web App Store page 'cause I don't know if he'll see my
bug report. Instead I'm thinking of just uninstalling the extension.

[Akuk]

Why not do what Google should do, and not what Apple would force on
us? Give us the option to sort it as we wish, including by
chronological order, or "helpfulness"? I want to see the latest
reviews, not what some algorithm thinks I might want to see.

[Jan Kleinert]

(cross-posting with chromium-extensions)

As Alex mentioned earlier in this thread, we’ve been listening to your feedback and are taking it into consideration as we continue working on improving the reviews system.

I’d like to make sure we have a clear understanding of your needs as developers, and give you some information about what we’re planning so far to address those needs.  

First, some background information: The reviews component is primarily designed to help users make install decisions by surfacing helpful opinions from other users.  We’ve been getting feedback for a while that our reviews were not very successful in this regard, due to a lot of low-quality reviews showing up on the first page.  The change we rolled out was designed to address that, and we’ve gotten a lot of feedback that it has helped quite a bit.  At the same time, however, the change inadvertently broke some use cases that the reviews component was also helping with.

Here are the main use cases we’ve heard about so far, and how we’re planning to address them:

You used the reviews (in chronological order) as a way of collecting feedback and getting bug reports from your users. With the reviews no longer being in chronological order, it makes it hard for you to see these reports as they come in. To address this, we’re going to be adding a way to view the reviews in chronological order in the Developer Dashboard soon.

Similarly, you were also using the reviews as a way of communicating with your users and replying to their comments and questions. We understand there’s a need for better way to communicate with users, and we’re investigating the best way to handle this, whether it’s via reviews or some other mechanism. In the meantime, we encourage you to prominently place a link to your own support forums in your app description, your app itself, and be sure to fill out the “App support” link in the Developer Dashboard.

You’ve pointed out that the quality rating used for ordering the reviews sometimes brings old, possibly irrelevant reviews to the top of the list. For example, it might be a review referring to an old bug that has since been fixed. We are working on changing the quality rating system to give older reviews less weight.

Please continue to let us know what changes you would like to see to make your experience as a developer for the Chrome Web Store better!  We will continue to make changes and improvements throughout the store in the future.

-Jan

[Xan]

Here are some ideas.

1) Give users an option to see the comment feed sorted by date. Just
like you have a "View reviews in English (United States)" link that
enables a non-standard view of the comment feed, why not have a
similar toggle for the sort order. I understand the rationale behind
the new sort being default, but this CAN be optional.

2) Make a note of the version tied to the comment, displayed next to
it. Either directly poll the user's installed version, if any, or use
the release dates of versions to approximate. This way the concept of
an "old" comment will be more concrete - it's a comment about an older
version and people can see it.

3) Implement highlighting of the developer's own comments to make them
stand out: http://code.google.com/p/chromium/issues/detail?id=70520

4) Implement an RSS feed (so that developers can easily monitor
feedback): http://code.google.com/p/chromium/issues/detail?id=78954 I
think most developers will agree that this will improve things for us
greatly.

5) Perhaps, add an option to mark feedback as "Fixed" - not just by
developers (that can lead to abuse), but as a voting mechanism. It's
not right to call comments about fixed bugs as "unhelpful" or even
spam, but conscious users can down-vote already fixed issues.

6) Add a "What's new" feed similar to Android market, that by default
shows release notes for the last version, but optionally can be
expanded to a history of such notes.

If I have more suggestions, I'll add them later.

> > To post to this group, send email to chromium-a...@chromium.org.

> > To unsubscribe from this group, send email to

> > chromium-apps+unsubscr...@chromium.org.

[divreg]

Thanks for the feedback Jan,
While I don't have very many users for my app, I, as a user, do find
it difficult to read the latest reviews on apps like Netflix or
Offline Google Mail.  It seems that new reviews get absorbed 10 pages
in and will rarely been seen on apps with a lot of users.
Here's a suggestion, why not avoid the complicated algorithm of
sorting reviews, adding more links to the store to view as you wish
and so on, but rather adopt the comment style of YouTube?
* Have one section for "Top Reviews" with two of the most helpful
reviews, which may dynamically change.* Have the rest all in
chronological order. Foo's response to bar being "@bar Lorem ipsum..."
and not necessarily in the thread-like format of needless
indentation.* Have the ability to add urgent "Developer comments".* If
necessary, have a "view all" feature for each category.* Keep the 5-
star rating system though, finding an app useful is not a binary
decision
Or, follow Xan's suggestion which implements an android market style
review system with some additional tools.

Either way, these suggestions avoid the fragmentation between google's
comment base across its products.

> > To post to this group, send email to chromium-a...@chromium.org.

> > To unsubscribe from this group, send email to

> > chromium-apps+unsubscr...@chromium.org.

[stinkinrich88]

How about two sections:

1. Top comments
2. All comments.

This would match the youtube-style comments system and other things
(like product reviews on Amazon).

(The sooner we get chronological back, the better). Also, RSS or email
digest would be great for comments.

[huyz]

Whatever algorithm the Web Store's been using, it's absolutely wrong.
When the algorithm changed, the top comment on
https://chrome.google.com/webstore/detail/oacdcllhgpddmlnhajiacfakhlilbicp
was months old and was nothing special -- it had 0 votes about being
helpful.
Now that it's been sitting at the top, well, people just keep voting
on it. Several people are voting that it's helpful despite the fact
that it's old and irrelevant.

So basically, the algorithm suffers from the "rich get richer"
problem. Most prominent comments will get more votes. It's as simple
as that.
After years of social online voting systems, is this that hard to
understand?
The algorithm needs to mix things up to truly get at the relevant
comments. I believe that Quora has got this figured out. Maybe Web
Store engineers should have a chat with Quora engineers.

> > To post to this group, send email to chromium-a...@chromium.org.

> > To unsubscribe from this group, send email to

> > chromium-apps+unsubscr...@chromium.org.

[Hansifer]

Thanks for this feedback, Jan. It helps to have an opportunity to see
where your guys' heads are with regard to web store user comments and
it's encouraging to see specifics on addressing related developer
concerns.

A couple of additional points I'm sure have crossed your mind:
- have you given any thought toward how to educate/encourage users
with regard to proper commenting?
- as far as giving older reviews less weight, have you considered
including a release history bias in the algorithm? (eg, if 3 releases
have occurred since a comment was posted on a particular date, the
comment might be considered effectively "older" than if no releases
have occurred in the interim). I know it gets tricky because it opens
it up to gaming, but it may still have some value.

Sincerely,
Hans Meyer

> -JanOn Wed, Sep 7, 2011 at 8:19 PM, Akuk <aktu...@gmail.com> wrote:
> > Why not do what Google should do, and not what Apple would force on
> > us? Give us the option to sort it as we wish, including by
> > chronological order, or "helpfulness"? I want to see the latest
> > reviews, not what some algorithm thinks I might want to see.
>
> > On Aug 26, 11:18 am, Alexandra Levich <lev...@chromium.org> wrote:
> > > Hi everyone,
>
> > > A few days ago we released some changes to the way we sort comments from
> > our
> > > users. Instead of sorting these by recency and showing our users a lot of
> > > comments which might not be so helpful, we are now sorting these comments
> > by
> > > quality and helpfulness to the users. Thank you for your feedback on this
> > > change. We are listening and are taking all of your feedback into
> > > consideration as we iterate and work more on this feature. Our goal is to
> > > make comments be as helpful as possible for our users and our developers.
>
> > > Thanks,
> > > Chrome Web Store team
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Chromium Apps" group.

> > To post to this group, send email to chromium-a...@chromium.org.

> > To unsubscribe from this group, send email to

> > chromium-apps+unsubscr...@chromium.org.

[stinkinrich88]

I can't believe this still hasn't been fixed.....

Btw, developers, you can now see your latest (in the normal/useful
order) comments in you "developer dashboard". (But replying to your
users is still useless.)

The users are still stuck, though. For example, I just wanted to check
if an extension had broken for everyone or just for me, but the first
comment I saw was from 5 months ago... Really useful....

[cnanney]

I too am unhappy with the change in sorting order. As others have
said, bug complaints that were fixed over a year ago are now
prominently displayed and my responses are now meaninglessly out of
order.

One solution would be the ability for us to respond directly to
comments from the developer dashboard. No long threads or anything,
just one 'response' allowed per comment, displayed under its parent:

anonymous Feb 13, 2011 Mark as spam
This doesn't work, blah blah...
Developer Response Feb 14, 2011
Sir, your computer must be on first.

Then no matter how you sort the parent comments, responses are always
there with them. I think this is a more elegant solution than us
having to keep polluting the comment stream with explanations, tips,
etc. I have a support and FAQ url in the description, and specifically
say to email me with any issues or questions, but they still end up in
the web store comment section. This is inevitable if you are going to
allow comments, so this gives us a better way to respond to bug
reports or confused users.

[Harm]

In my opinion, it's a very bad decision.

For example, this extension: https://chrome.google.com/webstore/detail/nnancliccjabjjmipbpjkfbijifaainp
was recently updated with spam add-ware inside, there are few recent
comments about this, but all them are hided deep, because they has no
votes. One need leaf through all comments (above 1k) to vote comments
about add-ware in extension.

[Brandon Thomson]

Amazon.com has the same problem with highly-rated products getting
replaced with crappier versions. Their solution is to feature "most
helpful" reviews and put "most recent" in the sidebar.

Although, knowing Amazon, wouldn't surprise me if they applied for a
patent on that idea :/
--
Brandon Thomson
www.bthomson.com

> --
> You received this message because you are subscribed to the Google Groups "Chromium Apps" group.

> To post to this group, send email to chromi...@chromium.org.
> To unsubscribe from this group, send email to chromium-app...@chromium.org.

[Robert Dailey]

Why? Why would you do this? This change is absolutely horrible. As a
developer, I always depend on the latest comments being on top so I
can respond to each one if people report issues. Now shit is all out
of order and there is no way to see if a new comment was posted.

Please revert the sorting back to the way it was. This was a horrible
change.

[Justin]

I found this annoying at first, too. But you can still see them in
chronological order by checking the Developer Dashboard.
https://chrome.google.com/webstore/developer/dashboard If you click
"View user reviews" under your extension, you'll see them in
descending chronological order.

[Justin]

+1