content trustworthiness
2 views
Skip to first unread message
Laurent Savaete
Jul 8, 2012, 11:49:35 AM7/8/12
to wiki...@googlegroups.com
Eric gave us some interesting views on IRC a few days ago, part of
them in French, so I'll sum up my understanding of them here. (Eric,
if I missed your point, please reply here!)
His opinion is essentially that the content we have on the site is of
insufficient quality to be of use to a real language student, which he
suggests turns away potential users because they don't get any "value"
out of it (problems of trust, etc...)
So he suggested we should consider a pre-validation logic for content,
instead of the post-validation logic we currently use. In other words,
we could verify/validate content before it appears on the site, as
opposed to possibly removing/correcting contributions of low quality.
An example of poor quality he mentioned was the russian survival kit I
imported from wikibabel :) Bad audio quality in English (I made them,
my bad!) and lots of errors in Russian (I have to trust him on that
point).
Although I don't agree with everything (switching to a pre-validation
model for instance), I think his opinion is valuable, because he's
both a student of languages, and has been a teacher, and so are many
people around him. That gives him an experience none of us has. But
mostly, he looked at content as a user, not a contributor, and this is
something we don't do enough.
I believe post-validation is the way to go for a wiki. We've discussed
at length the problem of barriers to entrance, and how to make
contributions easier. I don't think checking contributions before they
appear on the site will help on that point.
But as a user, I think it makes a lot of sense to know what to expect
in a lesson, especially if it's full of errors. And right now, we
don't address this issue.
We've talked before of setting up a system like:
- Define a limited set of tags that describe content status (such as:
"may contain errors", "bad audio quality"...). This set of tags is
free to evolve, but it needs to start somewhere.
- we define an action for each of them (like: "bad audio quality" -->
show the page on a list of pages that need new recordings).
- It should be quite prominent when opening a lesson page that the
content is potentially "unsafe". I'm thinking headers like wikipedia
has for certain pages.
- Each language should have an "editor in chief" who masters the
language (preferably a native speaker, ideally a teacher of it). That
person would get notifications of new contributions, so s/he can
validate them (this may be as "simple" as a page watch functionality).
That capacity can obviously be delegated/shared. I don't know how to
deal with languages that have no "master", besides making it clear
that we are looking for someone to hold that "position".
- possibly, the tagging system would work hand-in-hand with a user
badge system, so that if I don't have the "native english speaker"
badge, my edits of english content will be tagged with "non native
edit" (or something similar) automatically, and an english "editor in
chief" will be notified.
So far, I am not bringing up any real new idea, I think we've talked
about all this before. But I think we should prioritise these points a
bit higher. There isn't a huge amount of tech work needed to make all
this happen (I think), and I think it would substantially improve the
user-perceived-value of the site.
A possible (partial) action plan:
- define set of tags rev 0 with corresponding actions, and apply them
manually to existing content
- name "language masters" where we can
- amend page display so that certain tags display a prominent notification
- setup various reports (basically tag searches) to help acting upon tags
- setup page watching system (see https://code.ductus.us/ticket/132)
- setup user badging system
Opinions welcome, in particular from Eric :)
them in French, so I'll sum up my understanding of them here. (Eric,
if I missed your point, please reply here!)
His opinion is essentially that the content we have on the site is of
insufficient quality to be of use to a real language student, which he
suggests turns away potential users because they don't get any "value"
out of it (problems of trust, etc...)
So he suggested we should consider a pre-validation logic for content,
instead of the post-validation logic we currently use. In other words,
we could verify/validate content before it appears on the site, as
opposed to possibly removing/correcting contributions of low quality.
An example of poor quality he mentioned was the russian survival kit I
imported from wikibabel :) Bad audio quality in English (I made them,
my bad!) and lots of errors in Russian (I have to trust him on that
point).
Although I don't agree with everything (switching to a pre-validation
model for instance), I think his opinion is valuable, because he's
both a student of languages, and has been a teacher, and so are many
people around him. That gives him an experience none of us has. But
mostly, he looked at content as a user, not a contributor, and this is
something we don't do enough.
I believe post-validation is the way to go for a wiki. We've discussed
at length the problem of barriers to entrance, and how to make
contributions easier. I don't think checking contributions before they
appear on the site will help on that point.
But as a user, I think it makes a lot of sense to know what to expect
in a lesson, especially if it's full of errors. And right now, we
don't address this issue.
We've talked before of setting up a system like:
- Define a limited set of tags that describe content status (such as:
"may contain errors", "bad audio quality"...). This set of tags is
free to evolve, but it needs to start somewhere.
- we define an action for each of them (like: "bad audio quality" -->
show the page on a list of pages that need new recordings).
- It should be quite prominent when opening a lesson page that the
content is potentially "unsafe". I'm thinking headers like wikipedia
has for certain pages.
- Each language should have an "editor in chief" who masters the
language (preferably a native speaker, ideally a teacher of it). That
person would get notifications of new contributions, so s/he can
validate them (this may be as "simple" as a page watch functionality).
That capacity can obviously be delegated/shared. I don't know how to
deal with languages that have no "master", besides making it clear
that we are looking for someone to hold that "position".
- possibly, the tagging system would work hand-in-hand with a user
badge system, so that if I don't have the "native english speaker"
badge, my edits of english content will be tagged with "non native
edit" (or something similar) automatically, and an english "editor in
chief" will be notified.
So far, I am not bringing up any real new idea, I think we've talked
about all this before. But I think we should prioritise these points a
bit higher. There isn't a huge amount of tech work needed to make all
this happen (I think), and I think it would substantially improve the
user-perceived-value of the site.
A possible (partial) action plan:
- define set of tags rev 0 with corresponding actions, and apply them
manually to existing content
- name "language masters" where we can
- amend page display so that certain tags display a prominent notification
- setup various reports (basically tag searches) to help acting upon tags
- setup page watching system (see https://code.ductus.us/ticket/132)
- setup user badging system
Opinions welcome, in particular from Eric :)
Streit Eric
Jul 8, 2012, 1:06:55 PM7/8/12
to wiki...@googlegroups.com
Hi,
Le 08/07/2012 17:49, Laurent Savaete a �crit :
these ten words/phrases.
> Although I don't agree with everything (switching to a pre-validation
> model for instance), I think his opinion is valuable, because he's
> both a student of languages, and has been a teacher, and so are many
> people around him. That gives him an experience none of us has. But
> mostly, he looked at content as a user, not a contributor, and this is
> something we don't do enough.
>
> I believe post-validation is the way to go for a wiki. We've discussed
> at length the problem of barriers to entrance, and how to make
> contributions easier. I don't think checking contributions before they
> appear on the site will help on that point.
>
> But as a user, I think it makes a lot of sense to know what to expect
> in a lesson, especially if it's full of errors. And right now, we
> don't address this issue.
It's very important to have lessons without errors: they need to be
checked BEFORE publishing; the "work in progress" could be published in
a somehow sandbox.
>
> We've talked before of setting up a system like:
>
> - Define a limited set of tags that describe content status (such as:
> "may contain errors", "bad audio quality"...). This set of tags is
> free to evolve, but it needs to start somewhere.
> - we define an action for each of them (like: "bad audio quality" -->
> show the page on a list of pages that need new recordings).
> - It should be quite prominent when opening a lesson page that the
> content is potentially "unsafe". I'm thinking headers like wikipedia
> has for certain pages.
> - Each language should have an "editor in chief" who masters the
> language (preferably a native speaker, ideally a teacher of it). That
> person would get notifications of new contributions, so s/he can
> validate them (this may be as "simple" as a page watch functionality).
> That capacity can obviously be delegated/shared. I don't know how to
> deal with languages that have no "master", besides making it clear
> that we are looking for someone to hold that "position".
I agree
> - possibly, the tagging system would work hand-in-hand with a user
> badge system, so that if I don't have the "native english speaker"
> badge, my edits of english content will be tagged with "non native
> edit" (or something similar) automatically, and an english "editor in
> chief" will be notified.
>
> So far, I am not bringing up any real new idea, I think we've talked
> about all this before. But I think we should prioritise these points a
> bit higher. There isn't a huge amount of tech work needed to make all
> this happen (I think), and I think it would substantially improve the
> user-perceived-value of the site.
>
> A possible (partial) action plan:
> - define set of tags rev 0 with corresponding actions, and apply them
> manually to existing content
> - name "language masters" where we can
> - amend page display so that certain tags display a prominent notification
> - setup various reports (basically tag searches) to help acting upon tags
> - setup page watching system (see https://code.ductus.us/ticket/132)
> - setup user badging system
>
> Opinions welcome, in particular from Eric :)
>
it's a good r�sum� :)
best regards
Eric
Le 08/07/2012 17:49, Laurent Savaete a �crit :
> Eric gave us some interesting views on IRC a few days ago, part of
> them in French, so I'll sum up my understanding of them here. (Eric,
> if I missed your point, please reply here!)
>
> His opinion is essentially that the content we have on the site is of
> insufficient quality to be of use to a real language student, which he
> suggests turns away potential users because they don't get any "value"
> out of it (problems of trust, etc...)
> So he suggested we should consider a pre-validation logic for content,
> instead of the post-validation logic we currently use. In other words,
> we could verify/validate content before it appears on the site, as
> opposed to possibly removing/correcting contributions of low quality.
> An example of poor quality he mentioned was the russian survival kit I
> imported from wikibabel :) Bad audio quality in English (I made them,
> my bad!) and lots of errors in Russian (I have to trust him on that
> point).
>
My wife is Russian native ... and she was horrified with the errors in
> them in French, so I'll sum up my understanding of them here. (Eric,
> if I missed your point, please reply here!)
>
> His opinion is essentially that the content we have on the site is of
> insufficient quality to be of use to a real language student, which he
> suggests turns away potential users because they don't get any "value"
> out of it (problems of trust, etc...)
> So he suggested we should consider a pre-validation logic for content,
> instead of the post-validation logic we currently use. In other words,
> we could verify/validate content before it appears on the site, as
> opposed to possibly removing/correcting contributions of low quality.
> An example of poor quality he mentioned was the russian survival kit I
> imported from wikibabel :) Bad audio quality in English (I made them,
> my bad!) and lots of errors in Russian (I have to trust him on that
> point).
>
these ten words/phrases.
> Although I don't agree with everything (switching to a pre-validation
> model for instance), I think his opinion is valuable, because he's
> both a student of languages, and has been a teacher, and so are many
> people around him. That gives him an experience none of us has. But
> mostly, he looked at content as a user, not a contributor, and this is
> something we don't do enough.
>
> I believe post-validation is the way to go for a wiki. We've discussed
> at length the problem of barriers to entrance, and how to make
> contributions easier. I don't think checking contributions before they
> appear on the site will help on that point.
>
> But as a user, I think it makes a lot of sense to know what to expect
> in a lesson, especially if it's full of errors. And right now, we
> don't address this issue.
checked BEFORE publishing; the "work in progress" could be published in
a somehow sandbox.
>
> We've talked before of setting up a system like:
>
> - Define a limited set of tags that describe content status (such as:
> "may contain errors", "bad audio quality"...). This set of tags is
> free to evolve, but it needs to start somewhere.
> - we define an action for each of them (like: "bad audio quality" -->
> show the page on a list of pages that need new recordings).
> - It should be quite prominent when opening a lesson page that the
> content is potentially "unsafe". I'm thinking headers like wikipedia
> has for certain pages.
> - Each language should have an "editor in chief" who masters the
> language (preferably a native speaker, ideally a teacher of it). That
> person would get notifications of new contributions, so s/he can
> validate them (this may be as "simple" as a page watch functionality).
> That capacity can obviously be delegated/shared. I don't know how to
> deal with languages that have no "master", besides making it clear
> that we are looking for someone to hold that "position".
> - possibly, the tagging system would work hand-in-hand with a user
> badge system, so that if I don't have the "native english speaker"
> badge, my edits of english content will be tagged with "non native
> edit" (or something similar) automatically, and an english "editor in
> chief" will be notified.
>
> So far, I am not bringing up any real new idea, I think we've talked
> about all this before. But I think we should prioritise these points a
> bit higher. There isn't a huge amount of tech work needed to make all
> this happen (I think), and I think it would substantially improve the
> user-perceived-value of the site.
>
> A possible (partial) action plan:
> - define set of tags rev 0 with corresponding actions, and apply them
> manually to existing content
> - name "language masters" where we can
> - amend page display so that certain tags display a prominent notification
> - setup various reports (basically tag searches) to help acting upon tags
> - setup page watching system (see https://code.ductus.us/ticket/132)
> - setup user badging system
>
> Opinions welcome, in particular from Eric :)
>
best regards
Eric
Jim Garrison
Jul 8, 2012, 3:43:25 PM7/8/12
to wiki...@googlegroups.com
First impressions are indeed crucially important. The real goal here is
to get users to content that is trustworthy. There are many ways to do
this; that is, pre-validation is not the only possible mechanism. As an
alternative, we could just route users to content that we believe to be
correct, and keep the rest around as a staging ground so that other
people (who are fluent in a language) can improve it. This is similar
to the sandbox idea that Eric suggests, but it is a designation (not a
location), so I think that simplifies things a bit.
First we need to decide on some heuristics to determine the quality of a
lesson.
I really like the karma model outlined at
http://blog.stackoverflow.com/2009/05/a-theory-of-moderation/ and I
think we should study it closely. We could even separately consider a
person's karma in each language, so that lots of karma gained from
writing English lessons doesn't automatically imply they are any good at
editing French lessons ;).
Once we have some quantitative way for assessing how reliable a lesson
is (based on tags, who has rated it what, etc), it all comes down to
search. We need users to be directed to lessons that are of high quality.
In addition, we should /only/ list languages on the front page if we
have a healthy moderating person/community for that language.
This indeed is more urgent than meta-lessons, and I don't think we
should work on any other new high-level feature until we have this
figured out.
On 07/08/12 10:06, Streit Eric wrote:
> Hi,
>
> it's a good résumé :)
>
> best regards
>
> Eric
>
to get users to content that is trustworthy. There are many ways to do
this; that is, pre-validation is not the only possible mechanism. As an
alternative, we could just route users to content that we believe to be
correct, and keep the rest around as a staging ground so that other
people (who are fluent in a language) can improve it. This is similar
to the sandbox idea that Eric suggests, but it is a designation (not a
location), so I think that simplifies things a bit.
First we need to decide on some heuristics to determine the quality of a
lesson.
I really like the karma model outlined at
http://blog.stackoverflow.com/2009/05/a-theory-of-moderation/ and I
think we should study it closely. We could even separately consider a
person's karma in each language, so that lots of karma gained from
writing English lessons doesn't automatically imply they are any good at
editing French lessons ;).
Once we have some quantitative way for assessing how reliable a lesson
is (based on tags, who has rated it what, etc), it all comes down to
search. We need users to be directed to lessons that are of high quality.
In addition, we should /only/ list languages on the front page if we
have a healthy moderating person/community for that language.
This indeed is more urgent than meta-lessons, and I don't think we
should work on any other new high-level feature until we have this
figured out.
On 07/08/12 10:06, Streit Eric wrote:
> Hi,
>
>
> best regards
>
> Eric
>
Ian Sullivan
Jul 8, 2012, 10:25:02 PM7/8/12
to wiki...@googlegroups.com
On 07/08/2012 03:43 PM, Jim Garrison wrote:
> This indeed is more urgent than meta-lessons, and I don't think we
> should work on any other new high-level feature until we have this
> figured out.
Agreed.
> This indeed is more urgent than meta-lessons, and I don't think we
> should work on any other new high-level feature until we have this
> figured out.
-Ian
Laurent Savaete
Jul 13, 2012, 10:00:42 AM7/13/12
to wiki...@googlegroups.com
Following up on previous discussions, here's an attempt to describe a
system to promote "good" content to visitors of the site while at the
same time marking/highlighting "poor" content for improvement.
The overarching idea being to welcome any contributions (including
"bad" ones) while giving visitors a better experience by helping them
find what they want (i.e: great lessons) and be informed if there are
known issues with some specific content.
We came up with the following scheme to "rate" content:
- anyone (with a user account?) can +1/like/heart/star a lesson they
like. This should be done with a single click from the lesson viewing
interface, i.e. not require editing content. As an extra suggestion, I
would see this as a lesson bookmarking system, i.e: I should have
direct access to "my favourite lessons" (being the ones I marked)...
- we left the "1 to 5 star" type rating aside, as it requires too much
thinking from the user ("should I give it 4 or 5 stars?"), and would
likely be culturally biased (I wish I could find the reference for
this)
- we don't use a -1 type system to calculate "karma", as it is not
helpful enough to understand why a lesson is "bad", and could also
possibly lead to "-1" wars between users.
- instead we rely on flags to mark specific shortcomings (see proposed
list below).
All this info is stored in mongodb (i.e. it is not part of the lesson
content). The details of which we can discuss on the ductus-developer
list as it a lot more technical. We should also define how to rank
lessons (based on the number of +1/like/heart/star(s))
Here's a list of flags that could be applied to lessons that fall
below "production quality". They would be applied by users (with or
without restrictions) via a simple interface (again, no need to edit
the lesson to flag it) like what stackoverflow.com offers (see
http://blog.stackoverflow.com/2009/05/a-theory-of-moderation/ for
screenshots).
Flags would ideally be applicable to a whole lesson or a specific
section to make it easier to act upon.
Response action will vary depending on the flag chosen (email site
admin for flags that require rapid response, mark urn so it appears in
specific reports for others, show a specific warning for incorrect
content, etc...).
- spam (garbage or commercial spam) (proposed action ==> email site admin)
- inappropriate (adult content, hatred, ...) (in text, images or
audio) (==> email site admin)
- (may) contain(s) errors (language errors, grammar, spelling, ...)
(==> show warning on lesson page for users, mark lesson/section so it
appears in "lessons containing errors" report/search)
- non native audio (==> same as above)
- poor audio quality (no linguistic issue, but bad recording...) (==>
mark + warning?)
- poor image quality (blur, low resolution...) (==> mark + warning?)
- other: free text field (==> ??)
This is a limited list on purpose, as I think too many options will
likely discourage users from flagging content. Suggestions to improve
the above are very welcome.
As a side note, what actions do we want to take regarding
inappropriate content? As the site is a wiki, this content would
technically not disappear. Could this lead to potential problems as we
would be hosting unwanted content?
Special thanks to Eric for sparking this discussion!
system to promote "good" content to visitors of the site while at the
same time marking/highlighting "poor" content for improvement.
The overarching idea being to welcome any contributions (including
"bad" ones) while giving visitors a better experience by helping them
find what they want (i.e: great lessons) and be informed if there are
known issues with some specific content.
We came up with the following scheme to "rate" content:
- anyone (with a user account?) can +1/like/heart/star a lesson they
like. This should be done with a single click from the lesson viewing
interface, i.e. not require editing content. As an extra suggestion, I
would see this as a lesson bookmarking system, i.e: I should have
direct access to "my favourite lessons" (being the ones I marked)...
- we left the "1 to 5 star" type rating aside, as it requires too much
thinking from the user ("should I give it 4 or 5 stars?"), and would
likely be culturally biased (I wish I could find the reference for
this)
- we don't use a -1 type system to calculate "karma", as it is not
helpful enough to understand why a lesson is "bad", and could also
possibly lead to "-1" wars between users.
- instead we rely on flags to mark specific shortcomings (see proposed
list below).
All this info is stored in mongodb (i.e. it is not part of the lesson
content). The details of which we can discuss on the ductus-developer
list as it a lot more technical. We should also define how to rank
lessons (based on the number of +1/like/heart/star(s))
Here's a list of flags that could be applied to lessons that fall
below "production quality". They would be applied by users (with or
without restrictions) via a simple interface (again, no need to edit
the lesson to flag it) like what stackoverflow.com offers (see
http://blog.stackoverflow.com/2009/05/a-theory-of-moderation/ for
screenshots).
Flags would ideally be applicable to a whole lesson or a specific
section to make it easier to act upon.
Response action will vary depending on the flag chosen (email site
admin for flags that require rapid response, mark urn so it appears in
specific reports for others, show a specific warning for incorrect
content, etc...).
- spam (garbage or commercial spam) (proposed action ==> email site admin)
- inappropriate (adult content, hatred, ...) (in text, images or
audio) (==> email site admin)
- (may) contain(s) errors (language errors, grammar, spelling, ...)
(==> show warning on lesson page for users, mark lesson/section so it
appears in "lessons containing errors" report/search)
- non native audio (==> same as above)
- poor audio quality (no linguistic issue, but bad recording...) (==>
mark + warning?)
- poor image quality (blur, low resolution...) (==> mark + warning?)
- other: free text field (==> ??)
This is a limited list on purpose, as I think too many options will
likely discourage users from flagging content. Suggestions to improve
the above are very welcome.
As a side note, what actions do we want to take regarding
inappropriate content? As the site is a wiki, this content would
technically not disappear. Could this lead to potential problems as we
would be hosting unwanted content?
Special thanks to Eric for sparking this discussion!
Jim Garrison
Jul 29, 2012, 8:13:41 PM7/29/12
to wiki...@googlegroups.com
On 07/13/12 07:00, Laurent Savaete wrote:
> As a side note, what actions do we want to take regarding
> inappropriate content? As the site is a wiki, this content would
> technically not disappear. Could this lead to potential problems as we
> would be hosting unwanted content?
I think everything you said sounds great.
> As a side note, what actions do we want to take regarding
> inappropriate content? As the site is a wiki, this content would
> technically not disappear. Could this lead to potential problems as we
> would be hosting unwanted content?
If inappropriate content exists that we would rather not be hosting, we
will simply remove the offending portions entirely from the database.
Reply all
Reply to author
Forward
0 new messages