Retrospective on the Spaced Repetition prize

[John Salvatier]

The spaced repetition prize is now over (final post). Here are some thoughts on how it all worked out (my thoughts at the bottom):

Gwern had some observations about the contest:

With all that said, I do have some feedback for jsalvatier! (Please don't take this as defecting criticism; also, in the interests of honesty - I noticed all of #2 shortly after the contest began, and kept quiet out of plain self-interest, waiting until the judging happened).

1. First, it caused me a little stress that judging took so long. The deadline was 1 August, it didn't have to take 18 days to judge, even if events intervened (as they apparently did) to block the earlier gathering which would judge the contest - it could have been done online by so few judges. This is especially true for a contest with only two entries.

2. The contest was mishandled from the standpoint of publicity. I should not have been so confident I would win; there should have been more than 2 entries with $400 at stake for a task intrinsically worthwhile. There are a number of ways this could have been advertised better:

1. (I actually did this one) This could have been announced on the Mnemosyne and Anki mailing lists and forums. It is quite relevant to their interests and they are the obvious places to announce such a contest, even if they would not link it on their front pages.
2. It could have been advertised better on LW. This was really generous and interesting and potentially a key community tool! (lukeprog won't be this active forever.) Why wasn't the contest an article on the Front Page? (AFAIK, it never appeared there.)
3. Piotr Wozniak, the expert of experts on spaced repetition in practice, could have been contacted to participate or possibly advertise it on Supermemo.com; he may sound intimidating from theWired article, but my email correspondence with him was perfectly normal and fluid, with the ordinary delays.
4. The contest, per #2, could have been submitted to Reddit (the psychology subreddit being only the most obvious of multiple potential targets) or Hacker News; both communities are interested in spaced repetition. This is also true of Imminst.
5. Google AdWords could have helped; $10 buys a lot of clicks when it comes to spaced-repetition ads, and could well be worth it. ($350 is about as impressive as $360 as seed for a prize.)
6. Interested LessWrongers could have been personally contacted. Anyone who has created a deck of flashcards is obvious; and why not contact lukeprog? Reading and summarizing papers and reviews is practically his specialty. He is busy, of course, but it's not like spaced repetition is that worthless compared to his current topics, and the prize may make up for lost time.

During judging we also came up with some observations: 

1. Few people might have worked on the prize because they were intimidated (perhaps correctly) by gwern's prior work on the topic. 
2. Stable, credible prize design is important
3. The SR prize was successful, but I don't know if it is generalizable. There were only two submissions and the good one came from someone with lots of previous experience with the topic.
4. Offering editing and other troubleshooting services might be helpful. 
5. Is there a distribution of people who are more and less interested in different kinds of work (for example, writing, research, editing etc.)? Some people might not mind looking up papers but dread the actual writing or vice versa. If so then it would be highly beneficial to bring those people together.
6. Methodology for coming up with questions to be answered by research: get 4 people to brainstorm, take the union and throw out the bad ones. 
7. Banana's cure hunger! If you ever find yourself saying 'we should go get dinner' but you still have work to do, bananas work miracles. 
8. group judging seemed to work pretty well. i was pretty satisfied with our process (make observations by yourself, then discuss them together).

Other people have thoughts?