Status and Direction

[Ralph Furmaniak]

Hi everybody,
Just wanted to let you know what's been going on, and solicit
comments.

Status
-----

I think I got the main wrinkles out of the system (sorry that there's
been a bit of downtime recently).

In particular, there is no longer a post per (article,category) pair
but rather one post per article, appearing simultaneously in all
arxalivs, sharing the same votes and comments. There may be some
advantage to having separate postings, but the response was
overwhelming that it is a bad idea. The default site shows the
"fullarxiv" category, but you can unsubscribe to it and subscribe to
the specific categories of interest to you.

With this out of the way, I am now downloading the arXiv and should
have all 750,000 articles searchable and browsable on the site in 2
days, at which point I can truly call it version 1.0

The mobile and RSS versions are now working as well. The front-page
could use a lot of tweaking and restyling.

Direction
----
ie: now it's time to figure out how to actually use this :-) and to
figure out what features are missing. I'm using the site to create a
couple of communities around some interests of mine (like
selberg_class.arxaliv.org and riemann_zeta.arxaliv.org) and will be
posting links to papers that I've read and judged, although first I
need to tie the above cross-posting into the interface. Perhaps just
add an extra "cross-post" link to each posting which will allow you to
copy it into other arxalivs.

There's also the question: how democratic should this be. Are votes
important? Is it more important what the moderators choose, and what
comments are left by readers? Since for some important papers there
can be many interested parties but only a few truly qualified and
willing to put in the time to review it, should certain people's votes
have higher value? Or will they just leave their comments and the
community will take care of the rest. Should the default ordering be
top-rated, popular, or new?

[Eric Tramel]

I think I got the main wrinkles out of the system (sorry that there's
been a bit of downtime recently).

In particular, there is no longer a post per (article,category) pair
but rather one post per article, appearing simultaneously in all
arxalivs, sharing the same votes and comments.  There may be some
advantage to having separate postings, but the response was
overwhelming that it is a bad idea.  The default site shows the
"fullarxiv" category, but you can unsubscribe to it and subscribe to
the specific categories of interest to you.

Thanks for making this change. I think that this feature will really help a community coalesce around the articles posted onto Arxaliv. On the topic of posting, will there be any framework in place for posts/links outside of the arXiv repository? Is there a proper way to handle this or is it even desirable? 

With this out of the way, I am now downloading the arXiv and should
have all 750,000 articles searchable and browsable on the site in 2
days, at which point I can truly call it version 1.0

That is some intense scraping.

 

Direction
----
ie: now it's time to figure out how to actually use this :-) and to
figure out what features are missing.  I'm using the site to create a
couple of communities around some interests of mine (like
selberg_class.arxaliv.org and riemann_zeta.arxaliv.org) and will be
posting links to papers that I've read and judged, although first I
need to tie the above cross-posting into the interface.  Perhaps just
add an extra "cross-post" link to each posting which will allow you to
copy it into other arxalivs.

Could you give a little more information on how these sub-groups/communities work? Are these synonymous with subreddits? I tried taking a look at the two you mention here, but the site tells me there is a heavy load at the moment, so I couldn't really get a handle on them. 

 

There's also the question: how democratic should this be.  Are votes
important?  Is it more important what the moderators choose, and what
comments are left by readers?  Since for some important papers there
can be many interested parties but only a few truly qualified and
willing to put in the time to review it, should certain people's votes
have higher value?  Or will they just leave their comments and the
community will take care of the rest.  Should the default ordering be
top-rated, popular, or new?

Democracy seems like a terrific ideal to aspire to, I just

hope that the details don't move the project too far away

from that ideal! 

In terms of voting, one first needs to look at the effects

of the two disperate voting methods that occur within a

Reddit-type system.

1. Link Vote -- votes which have an effect on the visibility

of an article.

Link votes may be entirely democratic, as the number of link

votes is not a direct indicator of the quality of an article.

Even if an article itself might be generally regarded as poor

by experts, the reasons for this could be expounded on in the

comments and can inform non-experts who are otherwise insterested

in the topic. I do not see any problem in allowing uniform voting

weights for link up/down votes. 

Of course, I suppose that this brings into question whether or 

not down votes are useful at all. It seems like the system could

work entirely without them and the negative feelings they bring.

2. Comment Vote -- votes which have an effect on the visibility of

a community member. 

Comment votes may be the most useful and the most problematic

of the two voting systems. How comment votes are structured

depends on the goals for the Arxaliv website. If the goal is

to help make discussion or to share research among colleagues, 

then simply having a system identical to Reddit seems amenable. 

However, if the goal is to allow Arxaliv to go in the direction

of post-publication peer review, then perhaps one might want to 

assign a larger weight

to expert reviewers in certain areas. How this

is enacted will either make or break the entire system. A functioning

system should be: 

A. Verified

/r/AskScience has an interesting verification system whereby Mods

can assign a user to a trusted status within a certain field or 

subfields. A similar system might be employed. here. However, doing

so would incur significant time costs for moderators. By attaching

badges to comments or users that signify their areas and level of

expertise, non-expert users will be more likely to upvote their 

comments.

B. Anonymous (according to user perf.)

There is a reason that pre-publication peer review systems are at 

least single blind, if not double blind, and that is to mainly avoid

the kinds of vendettas that can arise over reviews. Already, one's account

name can mask their real life identity, but it would be an interesting 

experiment to see if blocking any account names at all within the 

comments section could be used. Per-thread ID's would have

to be assigned to keep users from getting confused during discussions.  

C. Democratic

One could make an attempt to change the amount of weight given to 

individual users based upon their credentials and community rapport (comment

karma). However, I feel that

this would not be very effective. If certain individuals have greater weight

in the community, then their opinions alone would be enough to squelch those 

of other participants. If experts are always correct, then there would be no

issue. However, in burgeoning areas of research when many possible avenues 

are open, this kind of vote weighting could have a chilling effect on

novel interpretations and creative expression.

D. Resistant to Gaming  

As the popularity of Reddit has grown, many individuals and businesses

have sought to use its social media platform for advertising, scamming,

trolling, and other self-serving endeavors. It is completely reasonable to

assume that if Arxaliv's user base grows, so will attempts to game it. If the

rewards for gaming are small, then the such events should be, hopefully, low

level and mostly unsophisticated. However, if users are allowed to accrue vote

weight according to comment karma, it is easy to imagine individuals using 

bots or buying votes to increase their rapport within the system. Additionally, and

more nefariously, similar systems could be used to squelch works or comments. 

The removal of downvotes could be one way to avoid this.

Thats just my $0.02. Tl;dr : Verified Users + Democracy = Win. Avoid Oligarchy. Do not make 

incentives for abusing voting.

 

[Bartosz Regula]

A. Verified

/r/AskScience has an interesting verification system whereby Mods

can assign a user to a trusted status within a certain field or 

subfields. A similar system might be employed. here. However, doing

so would incur significant time costs for moderators. By attaching

badges to comments or users that signify their areas and level of

expertise, non-expert users will be more likely to upvote their 

comments.

Are you sure it's a good idea to formally distinguish between expert and non-expert members? I think that, as a simple anti-spam and anti-abuse system, email validation could be implemented (e.g. checking that the email address is at an educational or research institution), then users could be allowed to set their flairs to reflect their field of research/interest. I don't think anything more than that is necessary.

[Ralph Furmaniak]

First of all, the arXiv is now fully loaded! I've also moved to a
bigger server to handle the load. If you have some requests for what
other sources should be added, I've started a thread at
http://arxaliv.org/r/help/comments/h5ah/the_arxiv_is_fully_loaded_what_is_next_to_load/
Once we have different sources it will become more important to figure
out how to organize it, and what notability requirements there will be
for the categories.
Perhaps when you add a source such as the Annals of Math, the postings
would go into /r/annals_math and then the moderators/readers for the
various categories can copy the postings into the appropriate
categories.

> Could you give a little more information on how these
> sub-groups/communities work?

It is the same as subreddits (a little-known feature is that you can
use subdomains for accessing subreddits, so selberg_class.arxaliv.org
is the same as arxaliv.org/r/selberg_class, and it actually exists
now).

And you are right that democracy is better than oligarchy. We could
play games with the votes and with more and less important users, but
that way we can very easily lose trust and lose the community.
I like the idea of adding flair based on academic credentials, either
degree or affiliation (with email as a first proxy). As long as those
not in academia still feel welcome on the site. The exact
implementation might simply come down to whoever is actually willing
to do the legwork for it :)
Another question on the topic of democracy vs authority is: should a
paper from a tier-one journal start with a higher rating (say, 2 or 3,
to bring it a bit above unrefereed papers), or will we just trust the
system to work (even if it means that a few of us vote up the papers).

A lot of this does not matter so much now, and we may just need to
wait until we see what ratio of moderators to users we have, and how
much free rein vs oversight is needed.

[m.tre...@gmail.com]

Hey, I just wanted to say thanks for doing this! I've been wishing someone would do something like this for a while. I'm just a physics undergrad but I'm proud to be a part of the beginning of arxaliv!
Sent on the Sprint® Now Network from my BlackBerry®

[aram harrow]

Hi Ralph et al,

I'm really excited about this site!
I don't know if you guys are aware, but previously there was a site
called "scirate.com" that did a similar thing, and mostly was used
only by the quantum information (i.e. arxiv.org/quant-ph/) community.
This site was in many ways more primitive than this one, but it had a
few features you may want to consider.

1. The interface matched http://arxiv.org/list/quant-ph/new
by listing all the papers since the last update (sorted by number of upvotes).
This I think was really important (at least for the quantum
information community) because many people in our field are used to
going to http://arxiv.org/list/quant-ph/new daily, and scirate let
them substitute a different site with a strictly larger functionality.
Furthermore, there was a "catchup" feature, where the site would list
the last X days, where X is input by the user.

2. No downvotes. They make the site less active in two ways: (a) it
looks like fewer users are present, and (b) if people downvote instead
of commenting, then they get diverted from useful discussions.
Probably the most useful feature of scirate was the way it facilitated
disussions that were started when someone mentioned a suspected flaw
in a paper.

3. This is kind of a minor point, but it seems natural to me to have
the score of a paper be initially 0 instead of initially 1.

4. In response to the democracy question: scirate let anyone vote, and
there wasn't (yet) any collaborative filtering or anything. I think
this doesn't scale arbitrarily, but I can report that it works
decently with a few hundred users.

Another thing: I blog at www.dabacon.org/pontiff and once arxaliv is
in good shape, I'd like to promote it there, as well as over some
quantum-information mailing lists.

Finally, thanks for your work in putting this site up! I think it's
going to be a really great service to the research community.

aram

[Ralph Furmaniak]

Hi Aram,

Thanks for the comments.  It's great to find out how people are exactly using the arXiv and other such websites.

1. The interface matched http://arxiv.org/list/quant-ph/new
by listing all the papers since the last update (sorted by number of upvotes).

Currently where it lists the option of "top today" I can make it instead display "top per day" and allow one to navigate through the different days (or a date range if you want to catch up).  I could have it keep track of when you last viewed or the list and/or which postings you have seen, but showing "top articles since the last time you viewed this page" could be unwieldy since it would reset with each view.  Perhaps everyone can keep track of a bookmark, a date that they feel caught up to, and every now and then move up the bookmark.

I could also make the interface track the arXiv more closely.

2. No downvotes.

 

How about this: now if you try to downvote a link you instead get an alert "To promote healthy and open discourse, you must first comment on an article before you can vote on it".

How about if if this is made necessary for upvotes as well.

 

3. This is kind of a minor point, but it seems natural to me to have
the score of a paper be initially 0 instead of initially 1.

 

Fair enough.  It is now going through the postings and undoing the automatic upvotes by ArxivBot, but this will take a while.  I originally kept the default of upvoting automatically since that made sure that all of the rankings were updated each time.

 

4. In response to the democracy question: scirate let anyone vote, and
there wasn't (yet) any collaborative filtering or anything.  I think
this doesn't scale arbitrarily, but I can report that it works
decently with a few hundred users.

That's good to know.  So far it is is still very much at the point where processes do not need to be automated.

 

Another thing: I blog at www.dabacon.org/pontiff and once arxaliv is
in good shape, I'd like to promote it there, as well as over some
quantum-information mailing lists.

Sounds great! 

[Ralph Furmaniak]

And some further updates.

There is now an RSS feed link on each page.  This works even on the search results pages so you can, for example, get a feed of all papers (1) in some area (2) but not in some other area (3) and matching various keywords.  Currently it uses whatever sorting order is selected on that page, but this doesn't seem to mesh too well with RSS so I may just make it order by date always (or sort primarily by day and secondarily by popularity).

There's been some demand for "anonymous posts".  Just like how reviews are anonymous for a reason, at times one may want to point out various errors or comments, without allowing it to get personal or staking one's career/friendship on it.

I would like to add some concept of correct/incorrect/unrefereed papers.  Papers by default would have a gray question mark to the left, which can be changed to a green checkmark (marking a verifying paper) or a red X (marking a paper that's factually incorrect).  Papers can be verified either by journals (a paper that has been published in a journal of repute will be given a checkmark) or in the discussion by somebody "respectable" posting a comment with "@correct(some explanation)" or "@incorrect(some explanation)".

To push this experiment even further, I have been advised to start an online journal within arxaliv.  The plan is for the editorial board to select and post one paper per week from within the past few years that should be of wide interest (to mathematicians, but I would hope others can start similar ones in their fields).  So this is not (currently) so much a journal in the traditional sense as it does not have exclusive access to the paper, and anybody may recommend anybody else's paper for inclusion, although this may be a model better suited to current technology.  I'll keep you posted, and I hope to have something good to show!

Once this all comes together I'll put together a proper front page instead of just jumping straight into the posts.

Ralph