Wiki Pages rating by users

[Missan]

Dear All,
Can we think of developing an interactive five point/star rating
system for the pages on the WikiEducator. Depending on user hit and
user feedback, the star system should dyanamically change. I am not
sure such a system is possible in WIkiEd or not. But, I think some
rating system is there in Wikipedia, but I am not able to understand.
Can anybody help on this?

with regards,

Sanjaya Mishra
IGNOU

[Wayne Mackintosh]

Hi Sanjaya ---

I'm keen to hear your thoughts how/why you think a start system will help WE achieve its aims?

Do you see a star system as a mechanism to motivate WikiEducators?  or Is this a mechanism to express a collective view on the quality of a resource?

Interestingly -- we host our main list on Google Groups which has a star rating system for posts -- but I've noticed that its not widely used by the community.

That said, I'm sure that there are innovative ways in which a star system could be implemented through a smart template solution.  My personal view is that any system that attempts to make any kind of value judgement on popularity and/or quality must be community driven. I'm also wondering about the impact a start system might have on newbies in the community.

Lets say I'm a new teacher grappling with the wiki syntax, structure, pedagogy etc -- and someone comes along and gives me a poor rating.  Would that motivate me to do better or discourage me from continuing. That said, I do think we can learn from the community incentives in the WMF projects. For example, Wikipedia have a featured article system where the community nominate articles for the status of a feature article.

See: http://en.wikipedia.org/wiki/Wikipedia:Featured_articles

There is a rigorous process regarding the criteria for featured articles as well as community discussion on whether a nomination will qualify.  Once an article has attained the status of featured article -- as star is inserted on the page.

Perhaps we should think about an equivalent or similar process in WE?  I guess the place to start would be to think about the criteria we would use.

On the quality issue -- I've been doing a little thinking about the appropriateness of implementing the Flagged Revisions extension. See: http://www.mediawiki.org/wiki/Extension:FlaggedRevisions

Its a pretty sophisticated tool where assigned reviewers take responsibility for expressing a value judgement on predefined criteria. What I like about the system is that the page view can be set to display the latest reviewed version -- without restricting the ability to draft edit and add value to existing materials.

Perhaps this is something we should think about.

Cheers
Wayne

[Leigh Blackall]

I think the idea has merit. One thing I am concious of in our work is the number of unfinished pages we have let sit. I wish I could nominate (and elevate) what we have 'finished' for peer review - enter the star rating? If the people we have working got a poor rating, I think it would put a bee in their bonnet and they'd jump up and do something about it :) What about we have a range of templates like the WikiMediaFoundation projects have? "This lesson is page is under review", "This page has been nominated for shawcase" etc

--
--
Leigh Blackall
+64(0)21736539
skype - leigh_blackall
SL - Leroy Goalpost
http://learnonline.wordpress.com

[ericdeeson]

I agree that your idea has merit, Sanjaya, but believe that people
should use such a system only for finished products (like published
books and city centre hotels). For an unfinished product (as, almost
by definition, all wiki pages are), we need an information box with
more detail - to guide potential users, editors and other visitors.

Here's what I suspect such a box would best contain ... at least:
* code for type of resource and domain/sub-domain
* if for use with learners, codes for subject/topic and level of main
learners
* page start date (and access to author's name if that is felt
appropriate - maybe authors could have a rating as do sellers on
Amazon to help with Sanjaya's concern)
* last edit date (and access to edit history and all editors' names as
with the initial author)
* summary of further work needed if any (like the Wiki's editorial
notes) (and access to full details)

Users could gain a view of the likely value to them of the page within
a few seconds and also be encouraged to add to it.

Best wishes, all - Eric (ericd...@aol.com)

[Wayne Mackintosh]

Hi Erik,

Thanks --- that's valuable feedback.

See also In text below ...

On Sat, 2008-05-03 at 03:29 -0700, ericdeeson wrote:

I agree that your idea has merit, Sanjaya, but believe that people
should use such a system only for finished products (like published
books and city centre hotels). For an unfinished product (as, almost
by definition, all wiki pages are), we need an information box with
more detail - to guide potential users, editors and other visitors.

Here's what I suspect such a box would best contain ... at least:
* code for type of resource and domain/sub-domain
* if for use with learners, codes for subject/topic and level of main
learners
* page start date (and access to author's name if that is felt
appropriate - maybe authors could have a rating as do sellers on
Amazon to help with Sanjaya's concern)
* last edit date (and access to edit history and all editors' names as
with the initial author)
* summary of further work needed if any (like the Wiki's editorial
notes) (and access to full details)

Brent Simpson has done his usual magic. Brent has created a ContentInfobox which enables us to implement your suggestions

We've not had a chance to publicisze this properly or finish the tutorial on how to use the feature, but you can see an example of our work in progress here:
http://wikieducator.org/Wikieducator_tutorial/Developing_a_teaching_resource/Inserting_an_Infobox

Its a pretty powerful template because it automatically categorises stuff.  There are some instructions on how to use the template here:  See: www.wikieducator.org/template:ContentInfobox

The history page keeps track of all the details relating to editors' names, dates etc. but we've included the option for the originating author to specify their name and the starting date of the project. 

Take a look -- be keen to hear what you think.

Cheers
Wayne

[Missan]

Thanks Wayne, Leigh and Eric for your valuable thoughts. I am deeply
concerned about WE's progress as a resource for quality learning
materials. It should not turn into another Wikipedia. We do not want
that to happen. We want WE to have content that is useful for various
levels of learning leading to some certification. It is possible that
some of the materials can be taken up by reputed insitutions to offer
courses of their own. But, something of such a nature to happen, it is
more important ot have quality resources, and also organized
resources. The ContentInfo template is quite comprehensive and
provides most information that should be attached to all completed
paages. While thinking of the organization of content and its quality,
I thought of having a rating system, and also checked the Wikipedia
ratings. But, these are not suitable, as they are not dynamic like the
one with GoogleGroup. People do not use it, as they are not expected
to. Once, we annouce that the community is responsible for the quality
of the material, and its rating system, users will start rating the
content. However, for new uses it would be de-motivating. Thus, only
completed pages be put on rating. Also rating needs to be dynamic, and
may include number of hits, time a user stay on the page, and actual
user rating on quality as per the specified ContentInfo. The feature
article concept is good. There is also another system on math ratings
(http://en.wikipedia.org/wiki/Template:Maths_rating). These are all
based on select editors. What I intend to is a community based one,
not just select users alone. May be as Eric has pointed out, in the
process of edits, and their familiarity with the system, there also
can be a rating system for the users as well. To start with the
Flagged Revison is also a good idea. Now, we must focus on quality
content development simultaneously with Wiki Skills training. May be
we can come up with an WikiEd Content Excellence Award in PCF 6.

With regards, Sanjaya

Sanjaya Mishra

[mackiwg]

Hi Sanjaya,

The discussions and thoughts on quality of WE materials is extremely
important for the future success and sustainability of our our
project.

At a personal level -- I'm very pleased to see these concerns and
thoughts raised by members of our community -- It shows that we care
about our project <smile>.

Given the importance and substantive nature of these discussions --
I've suggest that we move the planning and discussions on this
important topic into the wiki so that we have a permanent record of
this development with the added advantage of working collaboratively
on the relevant documents.

So I've set up a page for us to begin discussing and drafting a policy
on QA and review in WE:

http://wikieducator.org/WikiEducator:Quality_Assurance_and_Review

I'll post an invitation to the list inviting contributions ...

BIG thanks for taking the initiative in asking the questions and
moving this forward to the next level.

Cheers
Wayne

[Declan]

Perhaps organizational tools such as indices by topic, by educational
level, and by different educational standards could serve the purpose
of rating as well as organization.

If, for example I wanted to construct an index or table of contents
linking US biology educational standards to a set of resources
developed by various WE users. In the process it would be essential
that I evaluated each page before linking from the index. As more and
more such organizational tools are created, link counts could serve as
one indicator of quality. Creators of such indices would tend to
ignore the weaker content.

I think it is inevitable in an open environment that there will be
variability in quality. Two (of many) potential approaches to judging
quality might be benign neglect of poor quality material, or a peer
review system resulting in a rating. Peer review and rating might
please some users and alienate others, but it would likely result in a
decent measure of quality. Of course it would also mean increased
work load for all concerned. One can't expect peers to review unless
one is also willing to serve as reviewer.
Cheers,
Declan

[Robert Kruhlak]

>Of course it would also mean increased
> work load for all concerned. One can't expect peers to review unless
> one is also willing to serve as reviewer.

I recently heard/read about an automatic methods for determining the
quality of a wikipedia article. An example using wikipedia content is
given at the link below

http://gaston.cse.ucsc.edu/index.php/Main_Page

More information can be found at:

http://trust.cse.ucsc.edu/

Cheers
Rob

>
> >
>

--
Robert Kruhlak
Burnaby, BC
CANADA
(M) +1 778 230 1875
(E) kru...@gmail.com

[Wayne Mackintosh]

Hi Rob,

These are powerful tools and we have the benefit of hindsight to learn from the experiences of a humble giant <smile>.

That said -- I think WE is pioneering new ground in the sense that educational materials are different from an encyclopedia article. Education is culturally bounded and its going to be difficult to agree the parameters of a quality educational resource.

Please take a look at the http://wikieducator.org/WikiEducator:Quality_Assurance_and_Review page and see where and how the tools you have linked might assist WE. Automation will help us to scale -- so we need to think very seriously about how to move QA and review processes forward in WE.

Candidly -- I think this is the biggest challenge faced by our community to date.  I see opportunities for us establishing a global leadership position on the challenges of quality in the development of educational materials.

Lets prove the world that we are going to become the global leaders in the field <smile>.

Cheers
Wayne

[mackiwg]

Hi Declan,

That's an excellent point -- and is a good example of the power of the
wiki technology. Essentially the wiki has a flat structure and gives
users the ability to link and remix pages in ways that are not
possible with conventional CMS systems.

So for example, you could create a node page based in US biology
educational standards and someone in New Zealand could create a node
page on the NZ qualifications framework. However, it is conceivable
that both the US and NZ pages may use the same content resource in WE.
So we need to think of smart ways for the US educators to make value
judgement on the quality of materials, while providing the NZ
educators with the same freedom.

In a second scenario, US educators may want to fork or add an extra
section to a resource developed by educators in Ghana. This is where
WE's pdf feature to build customised collections is very powerful. The
US pdf collection may be different from the collection used in Ghana.
(BTW -- one of the reasons I want to be pretty sure that rich text
editors don't break the power of tempates <smile).

Lots for us to think about -- so if you have any thoughts or advice on
how to implement these QA issues in real life -- feel free to add your
ideas on this page:

http://wikieducator.org/WikiEducator:Quality_Assurance_and_Review

Gee -- this is pretty exciting. A big responsibility -- so I hope that
WE gets it right!

Cheers
Wayne

[Declan]

Opt in for peer review.

How about an opt-in template for peer review. By adding the template
to your page you agree to two things:
1. You want peer review and your page is ready for it.
3. You will review three other templated pages in a timely manner.

This would miss some wonderful pages that could still be rated, but it
would even out the work load for a peer review system and not put us
on the months-to-review path of conventional scholarly journals.

My thought for the night.

Cheers,

Declan

[mackiwg]

Hi Declan,

That's a valuable contribution -- thanks. I've added a new section on
the QA and review page for suggestions and considerations, and
included your contribution with attribution.

Just want to make sure that we don't loose any of the thoughts and
suggestions from the community. See:

http://www.wikieducator.org/WikiEducator:Quality_Assurance_and_Review

You're pretty fast hey <smile> --I see you've responded on the page
before I had time to post this reply. That's recent changes for you!

Cheers
Wayne

[mackiwg]

Hi to everyone watching the QA and review thread.

We're looking for volunteers to assist with drafting a WE policy on QA
and review.

If you're interested in helping out --- please add your name to the
list on the QA page on WE.

http://www.wikieducator.org/WikiEducator:Quality_Assurance_and_Review

Hey Peter, you're pretty fast on the draw yourself -- I see you listed
your name before I had a chance to post the invite <smile>. Thanks!

Cheers
Wayne

[Peter]

Wayne, Declan, Everyone...

I like where all this discussion is going. The bit that I want to
throw into all this is the use of Maturity Models. I like how they
encourage continuous improvement, which I believe applies well to Wiki
based OER. The "rating" a page / lesson / resource is based upon a
maturity model that applies to the context the page / lesson /
resource is available. This would then fit within what Declan was
saying earlier about indices, where the maturity model could be
implemented as a rubric (which they could be considered this anyhow)
for the particular grade or subject... I also believe the use of
rubrics would be familiar to the educational population we are working
with.

http://en.wikipedia.org/wiki/Category:Maturity_models

Be Well...

Peter

[Declan]

Peter,

I took a quick look at some of the maturity model pages you linked
to. The approach looks interesting and I agree that framing some
goals in the context of a rubric could have some advantages. I use a
rough rubric when assigning student wiki-writing assignments. I see
flexibility as a key ingredient.

A rubric setting benchmarks for successful content development would
of course serve as the guideline for peer review. However, it would
be important to build in enough flexibility to allow for excellence in
many different categories. In my own limited experience, I ask
students to make a math connection to a science lesson plan for
example. But some projects, like flower dissection, present a better
opportunity for an artistic connection. Why discourage a great
artistic or literary connection just to rigidly stick with math
(although I suppose we could count petals to sort the monocots from
the dicots).

My point is that by building in flexibility to a rubric, we can avoid
fostering a boring bunch of formulaic, clonal pages. One advantage of
this WE format is that by collaborating broadly we can bring our
diverse skill sets to bear on the content of the pages.

Wayne, thanks for getting the discussion page going and updating it.
Do you have a sense for whether we should post here, there, or both?
And this is exciting!
So, if a project is peer reviewed, done, and dusted as they say,
should we retain a wiki version for flexibility and ongoing
collaboration, and also an archived PDF version of acknowledged
quality? And if we keep a PDF version, do we want to go one step
beyond, and get a DOI, volume and page number and call it a journal?
Would that fly in the face of the 'open' piece of OER? Would keeping
a wiki version preserve the 'openness'?

Good night!
Declan

[mackiwg]

Hi Declan -- in text below ...

On May 5, 6:41 pm, Declan <dmcc...@smcvt.edu> wrote:
> Peter,
>
> I took a quick look at some of the maturity model pages you linked
> to.  The approach looks interesting and I agree that framing some
> goals in the context of a rubric could have some advantages.  I use a
> rough rubric when assigning student wiki-writing assignments.  I see
> flexibility as a key ingredient.
>
> A rubric setting benchmarks for successful content development would
> of course serve as the guideline for peer review.  However, it would
> be important to build in enough flexibility to allow for excellence in
> many different categories.  In my own limited experience, I ask
> students to make a math connection to a science lesson plan for
> example.  But some projects, like flower dissection, present a better
> opportunity for an artistic connection.  Why discourage a great
> artistic or literary connection just to rigidly stick with math
> (although I suppose we could count petals to sort the monocots from
> the dicots).

You make a good point -- the criteria for a review process shouldn't
be content driven -- but be able to accommodate the context for which
the materials are intended.

>
> My point is that by building in flexibility to a rubric, we can avoid
> fostering a boring bunch of formulaic, clonal pages.  One advantage of
> this WE format is that by collaborating broadly we can bring our
> diverse skill sets to bear on the content of the pages.

AGREE!

>
> Wayne, thanks for getting the discussion page going and updating it.
> Do you have a sense for whether we should post here, there, or both?
> And this is exciting!

No worries -- This development shows the dynamic and commitment of our
open community. Its also very exciting and to some extent an
indication of the maturity of OUR collective project.

> So, if a project is peer reviewed, done, and dusted as they say,
> should we retain a wiki version for flexibility and ongoing
> collaboration, and also an archived PDF version of acknowledged
> quality? And if we keep a PDF version, do we want to go one step
> beyond, and get a DOI, volume and page number and call it a journal?
> Would that fly in the face of the 'open' piece of OER?  Would keeping
> a wiki version preserve the 'openness'?

Flagged revisions (see the link on the QA page is a technology that
will support both the developmental version as well as the polished
peer reviewed version in the same environment. This will be able to
choose between the cutting edge development or latest peer reviewed
version. The wiki keeps track of every edit -- so at a technical level
this is all doable.

Cheers
Wayne
>
> Good night!
> Declan