8/198

[Yotam Gingold]

So far only 8/198 assignments. Let's hope this picks up!

[Krzysztof Gajos]

Lazy turkers

[Kurt Luther]

If this doesn't pick up, Paul and I were thinking it might be good to
run these in batches over the course of the day tomorrow, say 20 or 30
per hour. That will substantially increase the time that our tasks are
near the top of the list. Yes, the starting times will be different,
but not significantly (all over the course of a U.S. workday).

We might also want to tweak the task phrasing. I'm concerned it seems
like too much work. "Help write a story with 100 words" seems harder
than it is. Maybe we want the task to be "write a short paragraph".
And by the way, it should be 100 words.

--
Kurt Luther
www.kurtluther.com

[Yotam Gingold]

76/197

[Yotam Gingold]

102/197

[Krzysztof Gajos]

Go Turkers!

[Yotam Gingold]

187/198

[Yotam Gingold]

They're done! 197/197! (Actually, due to a mistake there are 2 extra, so 199/197.)

The data is in git at:

    stories-test/data-and-analysis/paragraph-from-snippet-run2/all-results.csv

The columns of interest are "stimulus" and "summary". In python, I would decode them with:

>>>

import csv, json

for line in csv.DictReader( open( "all-results.csv" ) ):

    stimulus = json.loads( line['stimulus'] )[0]

    condition, sentence, factsId = stimulus.split('-')

    

    paragraph = json.loads( line['summary'] )[0]

<<<

The "paragraph" is straight ascii, so put a "<p>" around it if you want HTML.

The stimulus sentences can be found in:

    stories-test/web-app/sentences

and the expansions are in

    stories-test/web-app/facts/<stimulus>.html

The 5 stimuli sentences are:

    arch.txt

    colorado.txt / photo.txt (these two are identical; I apologize)

    surgeon.txt

    widow.txt

    writer.txt

Evaluators: Please let me know if you want anything else. For your convenience, I have packaged all the data as a zip file and attached it to this email.

Yotam

[Yotam Gingold]

== Some analysis ==

Most workers went over the 100-word limit:

Median word-count: 133

Average word-count: 150

We had workers complete multiple tasks. Here is the number of tasks completed by each of our 82 unique workers:

[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 4, 5, 5, 5, 5, 6, 8, 10, 11, 14, 16, 22]

The median is 1, the average is 2.43.

Tasks were completed at a linear rate.

Durations varied. There were four completed in under 30 seconds, which I don't understand. I checked a 7-second one and it does match details of the facts provided, and that worker only did one. I compute duration as SubmitTime - AcceptTime. Anyhow, the minimum duration was 7 seconds, the maximum was 58:49 m:s. The median was 9:10 m:s, and the average was 12:32 m:s.

Over and out,

Yotam

[Kurt Luther]

The 7 sec one could have been a worker copying his paragraph to another task with the same prompt. Might be good to order the paragraphs by character count and scan for duplicates.

--
Kurt Luther

www.kurtluther.com

[Yotam Gingold]

But that worker only completed one HIT!

On May 23, 2013, at 4:48 PM, Kurt Luther <ku...@kurtluther.com> wrote:

> The 7 sec one could have been a worker copying his paragraph to another task with the same prompt. Might be good to order the paragraphs by character count and scan for duplicates.
>
>
> --
> Kurt Luther
> www.kurtluther.com
>
>
> On Thu, May 23, 2013 at 10:10 AM, Yotam Gingold <yo...@yotamgingold.com> wrote:
> == Some analysis ==
>
> Most workers went over the 100-word limit:
> Median word-count: 133
> Average word-count: 150
>
> We had workers complete multiple tasks. Here is the number of tasks completed by each of our 82 unique workers:
> [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 4, 5, 5, 5, 5, 6, 8, 10, 11, 14, 16, 22]
> The median is 1, the average is 2.43.
>
> Tasks were completed at a linear rate.

> <image.png>

>
> Durations varied. There were four completed in under 30 seconds, which I don't understand. I checked a 7-second one and it does match details of the facts provided, and that worker only did one. I compute duration as SubmitTime - AcceptTime. Anyhow, the minimum duration was 7 seconds, the maximum was 58:49 m:s. The median was 9:10 m:s, and the average was 12:32 m:s.

> <image.png>

[Kurt Luther]

A couple of ideas for evaluation: feel free to ignore.

* This seems like a task that would benefit from having fewer workers
do more ratings, to improve consistency. If possible, allow workers to
rate as many stories as they want.

* Because rating is so easy, it can be hard to tell if workers
actually read the story carefully. Consider asking them to provide a
one-sentence summary with their ratings, or some other low-effort way
to prove they read it.


--
Kurt Luther
www.kurtluther.com

[Yotam Gingold]

I binned according to character count, checked all the bins with more than one, and there are no duplicates.

More below:

On May 23, 2013, at 6:45 PM, Kurt Luther <ku...@kurtluther.com> wrote:

> A couple of ideas for evaluation: feel free to ignore.
>
> * This seems like a task that would benefit from having fewer workers
> do more ratings, to improve consistency. If possible, allow workers to
> rate as many stories as they want.

I like!

> * Because rating is so easy, it can be hard to tell if workers
> actually read the story carefully. Consider asking them to provide a
> one-sentence summary with their ratings, or some other low-effort way
> to prove they read it.

Yes! I think the plan was to ask them to summarize it in one sentence right away, then evaluate, and then, at the end, show them the original one-sentence summary and ask them to evaluate how well the paragraph matches it. From the meeting-before-last minutes:

>>>
Evaluation
- Add: Please summarize the story in one sentence. (First just to make sure that they read it)
- Add the original sentences. How well does this sentence accurately summarize the story? 7-point scale.
- Should also ask for the quality of the story.
<<<

Yotam

[Kurt Luther]

Oh right! I forgot all about that conversation. Thank you kindly.

--
Kurt Luther
www.kurtluther.com

[Krzysztof Gajos]

Just FYI, Pao and I will start on it tomorrow morning.