Re: scirate coordination

[jono]

Thanks Christian!

scirev.org is currently at http://scirev.org, but Alan (a developer who's interested in the project) has only turned on the quant-ph feed (available at http://scirev.org/taxonomy/term/48).  Alan has also quickly modified the original version so it doesn't look so drupel-esque. It's still in flux, and some of the features are turned off or temporarily broken, so please don't judge it too much by what you see.  Alan figures the main features can be fixed in 10-20 hours. 

As I mentioned to the UDub group, scirev.org combines three ideas:
1) likes (as in scirate)
2) social networking
3) mathoverflow type reputation, and reviewing of papers

I think (3) is needed if we want something which gets us beyond the popularity contest of scirate (although perhaps tagging gets us part of
the way there).  However, I think (3), and the websites which attempt this will fail miserably if (1) is not included.  We know from past
experience that (1) can become popular, but isn't an accurate reflection of quality, while (3) on it's own will fail.  The hope is that by
combining these 3 things, it can work.

Alan is happy to work with Keven, Chris, and the other projects, however, we should first decide if we're able to merge projects.  I
really hope we can, as it really increases the chance that it catches on.  There is an "about" page here: http://scirev.org/node/401 and it
contains links to the roadmap of features, the source code, and the list of the things which we want to fix asap. 

Perhaps as a starting point, people can comment on the current features of version 1, and we modify the list accordingly.  And if we're in
agreement, then we can move forward, decide what code base to use, etc. I think we can pull it together fairly quickly.  I'm happy to skype with whoever wants to -- just drop me a line.

Note that with the switch to a new look, some of the original links are gone.  So, for example, you can create your account, or link it with
facebook here: http://scirev.org/user , and I encourage everyone to give it a try.  Also, liking from the Chronological view is two clicks away at the moment, and a few other things are a bit wonky.

Best,
Jonathan

On 26/06/12 10:02, Christian Schaffner wrote:

Hi all

Apparently, there have been at least three rewrites of scirate so far:

https://scirate3.herokuapp.com/
(by Bill Rosgen, source: https://github.com/rosgen/scirate3/ )

http://scirev.bantalabs.com/
(by Jonathan Oppenheim)

http://quantphr.herokuapp.com/
(by Ben Reichardt, source: https://github.com/breic/quantphr )

I also worked on my own rewrite at
https://github.com/cschaffner/scirate
but it never made it further beyond the source code.

It seems that all three available systems have managed to reproduce the
basic functionality of the original scirate. Some of these systems plan
to expand towards "virtual journals" or want to add functionality that
similar sites have:
http://paperrater.org/
http://www.papercritic.com

In order to coordinate instead of duplicate our efforts, I suggest that
everybody interested in the topic joins the scirate google group at
https://groups.google.com/forum/?fromgroups#!forum/scirate
to discuss how to proceed. Maybe the particular owners could explain to
the group in more details what they want to do with their system?

Best,
Chris

[jono]

Thanks Ben,

I agree with much of what you wrote, but perhaps it will help if I
clarify how it work.

The ratings (1-5) are only used when someone writes a review.  So most
of the immediate feedback, is just through liking, and then looking at
the "most liked" tab.  If someone writes a review, then they are meant
to give it a rating, but it is meant as a rough guide.  I.e. you could
think of it as advice on what journal it could go in.  So, to be a bit
crude:
5 = everyone should know about it (e.g. cover of Nature, Science)
4 = everyone interested in science should know about it (Nature, Science)
3 = ever physicist would want to know about it (PRL, Nature Physics)
2 = experts will want to know about it
1 = its correct and non-trivial
-1 = wrong

Also, it's possible at the moment to vote up/down on reviews.  This is
meant to effect the prominence of the review, and the reputation of the
reviewer (but the latter part hasn't been implemented yet).

I agree that for the purposes of an experienced researcher, who only
cares about a particular subfield, the liking functionality is probably
enough (provided we implement some sort of tagging feature, and
depending on the field they're in).  I think scirate only really worked
for us, because the only people using it were from the qip community. 
If more communities started using it, the noise level would go up
considerably.  Take the hep-th arxiv -- everything that's not a string
theory paper will probably get drowned out by the string community, just
because it is so big in comparison to everyone else.  And it's not clear
to me in such a situation, that tagging alone would help in this case. 
But reviews might.

Also, I want to know when breakthroughs are made in other fields.  E.g.
experimental QIP, thermodynamics, black holes, etc.  Again, I think for
this, reviews become helpful.  I guess I'm worried that with the scirate
model, there's a danger of us becoming overly specialised, and only
reading papers written by researchers in our immediate area.  And not
everyone is in a well-defined research community.  I'm not completely
averse to adding reviews as a rollout feature, although it's pretty easy
to implement.  And not sure to what extent it will catch on -- it does
require having a large number of researchers using the site.

I completely agree that the main goal should be helping people find good
papers.  But that's the main goal of journals as well.

Best,
Jonathan


On 03/07/12 18:56, Ben Reichardt wrote:

Hi Jonathan, 

For me, the main point of Scirate was to ease the discovery of papers.  It gave an easy interface for browsing the daily postings, often with good papers already moved to the top, and sometimes I found papers by other activity.  I don't know that giving "an accurate reflection of quality" should be a main goal of the site.  This site isn't, and shouldn't, be used for evaluating anybody.  We aren't trying to replace journal reviews, either.  The main goal should be helping people find good papers.  

For example, to me star ratings do not seem to be a useful addition.  What is the idea, that I should care more about a 5-star average paper with 2 likes than a 3-star average paper with 20 likes?  This adds complication with little benefit.  While I might rate papers myself, I am mostly not interested in other people's ratings.  All that matters is the binary decision, interesting or not.  Rating colleagues' papers one or two stars seems petty, as bad as a popularity contest.  And adding star ratings to the interface makes using the site more ponderous.  

Let me mention two other issues that I've thought about.  

* A problem with Scirate, and other social sites for scientists, is that people can be reluctant to leave comments.  I think we'd like both serious reviews, addressing technical content, and trivial comments (one-liners like "this papers opens up a whole new area" can be very useful).  It is discouraging to post a serious review and not have any feedback, but on the other hand it can be real work to respond to a review.  One way to encourage both kinds of comments might be to separate them.  That is, have both (length-limited) "comments" and "reviews."  Reviews could themselves be "liked" and would be featured more heavily in the site's interface.  

* Also, there should be a strategy for drawing in more users.  The site is more useful when more researchers use it.  To address this, I think the site should be useful for somebody even before the social effects kick in.  It needs to have a very streamlined interface, to make browsing the arXiv very efficient.  Personalized recommendations, based, e.g., on imported .bib files would also help.  
	A similar point is that there should be a strategy for growth, and it might be best to launch the site deliberately without some features.  For example, a heavy emphasis on reviews just clutters the interface early on with little benefit, and turns me off from using the site, but it's a feature that could be phased in later.  

Ben

[jono]

Hi all,
So, Aram and I met last week in London to try to merge the various
projects (scirev/scirate). Based on our discussion, I've tried to merge
the two design documents.  I've just uploaded to google docs and sent
invitations to you.  The  potential disagreement that I could see was
referee reports.  In response, we suggest:
1) referee reports are now closer to long comments, and the ability to
give ratings is now an optional feature
2) there is no "order by rank" tab, so although one can make referee
reports and respond to them, we do not do much with this at the moment
-- we can just wait and see how and if people use it.

We also suggest a google hangout or skype meeting to finalise this. 
I've made a doodle for Thurs/Friday/Tuesday.  
http://www.doodle.com/tcni6c9tkfiws4pi If you'd like to participate
please fill it in today if possible.  You can instead just email
suggestions or edit the google doc, or annotate it.  I've put yellow
highlights on items which need some fleshing out, or where there is a
question.

Best,
Jonathan

[jono]

Okay,  I've scheduled the meeting for 4 pm (BST) this Tuesday the 24th,
which is the only time everyone can make it.  You should have got an
invitation to the hangout, although I'm not sure Google+ got the
timezone (GMT+1) correct.  If anyone knows how to fix this, please go
ahead, or let me know how to.

There has been some discussion about additional features in v1 (or
quickly after v1).  I've made a place in the google doc, where people
can make proposals, and have made one (feeds), which I think encompasses
some of the goals people have been suggesting.  I called this v1.1 for
the moment, with the understanding that this might be part of v1, or
might be v2, or something inbetween.  Please consider making your own
proposal on the google doc in advance of the meeting, and add footnotes,
ideas, suggestions -- I think it will make the meeting go much smoother.

I'm suggesting the following agenda, with the aim of finishing in 1-1.5
hr(s):
1) consensus on features for v1
2) consensus on highlevel design for v1
3) discussion on proposals for  v1.1 features
4) rough plan for moving forward (development, designers, roles)
5) name? (time permitting)

For future discussion:
6) full design (lowlevel, font, graphics etc.)
8) decision making structure
9 ) rollout

If anyone has particular things they'd like to put/remove from the
agenda, please let me know, and if you want to spend some time on
particular areas of (1), also please let me know.  It's a tight agenda,
so not sure how much we can cover.

Esther's email address by the way is bouncing, which I presume is
because she's left CQT.  I understood her to be travelling, so no idea
if she wants to participate.  If anyone knows how to contact her, and
thinks she wants to be involved, please forward the thread or link to
the googlegroup.

Also, I seem to have moved into the role of giving this discussion a bit
of a push, but only because it seemed to have slowed down.  I'm
certainly happy to relinquish this role, and don't want to be too
managerial, so just let me know if I'm being overly pushy.

Jonathan

On Thursday, 28 June 2012 20:25:04 UTC+1, jono wrote: