mental model externalization - update

[Kurt Luther]

Hi all, as promised, I wanted to give you a more detailed update on my
progress w/ mental model externalization.

The main point is that yes, people can write short stories/summaries
based on the yes/no clues. However, people can write stories given
just about anything (or nothing), so the question becomes, what is
special about these externalizations? What are they better than, why
are they useful, etc.?

I took two sets of clues generated with our system, the "Jack the
scientist" story and the "Boy and his dog" story. During Crowdcamp, I
asked turkers to write a paragraph-length summary based on just the
clues. I was able to get 5 decent summaries for each story, 10 total.
See for yourself:
https://docs.google.com/document/d/1fi8qT20ugTb-i6DkRBfuE8ehR7kta2m4kWwJHMSDmX4/edit?usp=sharing

The quality and length vary, and I'm guessing the difference mainly
has to do with how good of a writer the turker happened to be, and/or
how enthusiastic they were about the task.

I then deployed a similar task on oDesk, trying to see what
(theoretically) more skilled writers might produce. The minimum
payment on oDesk is $5 per task, so already it was impossible to make
a perfect comparison (turkers were paid only $0.50). Also, I
restricted the mTurk task to US-only, whereas all my oDesk applicants
were outside the US. Ultimately I got 3 writers for the dog story, and
just 1 for Jack the scientist -- this must have seemed much harder,
because hardly anyone applied for it by comparison. The oDesk stories
are longer (I requested <500 words), but not significantly better, in
my opinion. See for yourself:
https://docs.google.com/document/d/1hbx47GlY2QDs4nNOpcRRIw6hGbKEOHT6y115pCr9u4I/edit?usp=sharing

So, now what? As mentioned above, I think we need to make the case
that the clues are useful, that they support creativity somehow. We
know people can use them to write stories, but we don't know how
exactly they use them, or whether they really help. Here are some
possible avenues to explore -- I'd love your feedback, or suggestions
for others:

* How do stories written from the clues compare to stories written
from a simple prompt (e.g. "write a story about Jack the scientist")?
(Hopefully the clues lead to stories that are more creative or more
coherent. Regarding process, the clues might help people write more or
write faster.)

* How do the clues affect a person's self-perceptions of creative
ability? their actual creative ability? (We could give someone pre-
and post-tests measuring creativity and see if writing the story from
clues made them feel/be more creative. That's the original purpose of
the improv game after all.)

* How do stories written by the question-asker compare to stories
written by someone else? (Asker-written stories might be better
because the author feels more ownership, has a clearer mental model --
they got to ask questions, after all. However, strangers-written
stories might be better because it brings a second creative mind to
the task -- a kind of iteration.)

Your thoughts?


Sidenote:

This was my first oDesk experience and it reminded me how annoying it
is to work with real people instead of the crowd (half kidding). The
oDesk workers sent me tons of clarifying questions even though I
provided exactly the same task description to turkers (who completed
the task perfectly w/o questions). It's also much more effort to sort
through applications, hire workers, close the task, and leave
feedback. On the other hand, some workers actually wrote their summary
in the application and just hoped I'd pay them for it--very
surprising.


--
Kurt Luther
http://www.kurtluther.com/

[Kurt Luther]

Yotam reminded me... anyone have thoughts on this?

--
Kurt Luther
http://www.kurtluther.com/

[Yotam Gingold]

One experiment we could run is asking for a large number of people (50?) to "externalize the mental model" of a question-based story, and then evaluating each one's quality (to see how it varies) and performing some kind of textual analysis to measure the similarity (word histogram) of stories written from the same or different stimulus.

Would you like to modify the question-based task to ask questioners to externalize their mental models once they "reach the end"?  I can work together on that.

Yotam

[Robin]

1.This may be because I'm not a Javascript expert but need some help. 

See attached file but if I want to add text prompting users to answer questions based on a new story element after 3 yeses, how would I do so?

2. My suggestions to Kurt's last email

* How do stories written from the clues compare to stories written
from a simple prompt (e.g. "write a story about Jack the scientist")?
(Hopefully the clues lead to stories that are more creative or more
coherent. Regarding process, the clues might help people write more or
write faster.)

We still aren't defining "creative" as we continue. This may pose a problem in evaluation if creativity is part of our criteria. Also, it makes sense that providing more information about the topic may lead to "better" stories especially with workers who don't know the nature of the task. But if we're interested answering if the crowd can generate quality tasks quickly, it may be interesting to study amount of time to complete a task. Question - were the oDesk members given the clues or a simple prompt?

* How do the clues affect a person's self-perceptions of creative
ability? their actual creative ability? (We could give someone pre-
and post-tests measuring creativity and see if writing the story from
clues made them feel/be more creative. That's the original purpose of
the improv game after all.)

Like this idea. Think we should try.

* How do stories written by the question-asker compare to stories
written by someone else? (Asker-written stories might be better
because the author feels more ownership, has a clearer mental model --
they got to ask questions, after all. However, strangers-written
stories might be better because it brings a second creative mind to
the task -- a kind of iteration.)

It'd be hard to get the asker to write the story immediately after writing. I propose issuing pt-2 of the task where we follow up a few days later and ask them to write the stories to receive more payment.

Thanks,
Robin
___________________________________
ROBIN N. BREWER
Human-Centered Computing

University of Maryland, Baltimore County

B.S. Computer Science, 2011
University of Maryland, College Park