Homework

[Yotam Gingold]

Homework: Come up with experiments and associated research questions. Email them on Tuesday (in 7 days). Our next meeting is Friday at 2pm EDT (in 10 days).

[Kanya(Pao) Siangliulue]

I liberally used a self-created late day for this assignment. Hopefully, nobody minds.

Here are my thoughts.

Motivations:

1. Identify different workflows/mechanisms/framework for a medium to large group of people to generate complex creative artifacts together. We are probably not thinking about creating a masterpiece in the domain. It would be exciting to see that with right tools and mechanisms crowd can generate creative artifacts that exceed the best results achievable by individuals in the crowd. 
2. Create a pools of inspiration. Imagine a database of ideas writers/designers/artists can consult when they are stuck or look for interesting constraints.

The two motivations appear quite different from each other to me. I am intrigued by the first motivation, although I find the second one interesting as well. 

I am still pondering on solid research questions. Just pitch my thoughts here to start a conversation. I hope that this will at least give me a /- ? Talk to you all on Friday. :)

Cheers,

Pao

[Robin]

Hi Pao + others,

I like the 2nd idea. It creates an interesting application. The first one still would need us to answer - what is a considered a masterpiece and how should it be evaluated as such?

1. I was thinking something similar but to present different ways for gathering complex creative artifacts from the crowd focusing on the process we had in creating each method we've used (e.g. affordances) and whether those affordances held after evaluation. We could then compare advantages and disadvantages of using the different methods.
   
   
2. We could also link the methods together in some order based on the results of the above experiment and compare which is "best" in some way. For example, if the goal is to create a surprising plot vs. a realistic plot, should we link yes/no structured and snowflake together vs. yes/no unstructured followed by y/n structured followed by blank?
   
   
3. Lastly, it isn't done often but we could present a shortcomings paper (aka a failure paper) on why our original design didn't produce results expected, and challenges of evaluation based on our study and others that have been done.
   

 FYI I may be late on the call since I'm running studies @ work for the next week or so. But I'll try my darndest to make it!

Thanks,
Robin
ROBIN N. BREWER
Human-Centered Computing

[Yotam Gingold]

I like Pao's #1, identifying workflows for generating output better than any individual's inputs. I had an HCOMP paper last year with this motivation for drawing, painting, and singing, as it were ( http://cs.gmu.edu/~ygingold/diamonds/ ), with more unpublished results (in the same domains as well as sculpting aka 3D modeling). For drawing and painting, we obtained good results with a "which is more beautiful" forced-choice survey. That evaluation is harder for stories (Robin's concern), because (a) you can't read them in a gallery and (b) we've been interested in subtler aspects of stories.

I also like the idea of filling out a design space or gallery (Pao's #2). Figuring out a nice way to browse the gallery is interesting, too. Creative artifacts are high-dimensional, not two-dimensional like our screens. This is related to the problem of laying out a museum.

Yotam

P.S. I read a bunch of the 100-word stories. All the ones I read seemed pretty similar in quality.

[Paul André]

Thanks everyone, sorry for being late.

These are some interesting ideas, but we need a more solid approach to determine which are interesting research directions. Niki pointed me to the 'Heilmeier Questions', which are really helpful in motivating and framing a research problem:

1. What is the problem? Why is it hard?

2. How is it done today, what are the limits?

3. What's new/novel in our approach? (What are the challenges?)

4. Who cares?

5. If we're successful, how will it be measured? What difference/impact will it have?

If you look at a lot of abstracts/intros/talks, they're mostly framed around this approach (intentionally or not). We might start with something we think "oh this'd be cool" (which I feel like systems/UIST papers often start with), but to actually make it a convincing research problem, we still need to translate to these questions.

Until we have a few problems set out like this, I'm not sure we can move forward. We also need a clear project leader willing to coordinate and lead the work, and whip others into shape when needed :). This person might only emerge once we have an idea of the direction we want to take. But any collaboration I have been on without such a person inevitably fails.

This isn't a great start, but riffing off of Pao's #2, and some of the previous ideas kicked around:

Creative professions such as writers, designers, artists often keep notebooks or virtual folders of text and images to stimulate creativity or draw inspiration. But such tools still depend on the imagination of an individual, and lack meaningful or directed search abilities. To aid writers in mapping out possible directions for a specific story, we utilize a crowd workflow that (a) takes advantage of the diverse expertise and interests of many people, (b) constrains the ideas in order to be most useful and coherent. Interviews with a number of authors suggest the ideas are useful in sparking new concepts that may not otherwise have been thought of.

cheers,

Paul

[Kathleen Tuite]

Welp, I wrote way more than I intended to...

We've been trying to find a process that leads to a good artifact in the end. That is pretty hard in itself (e.g. what creative process do we each use?) but we are making it extra difficult by looking for a process specifically designed for an inherently uncreative, unimaginative crowd. (Sorry, turkers...) (Yotam's work is one of very few examples where combining a bunch of mediocre artifacts in a reasonable way leads to a good artifact, and I was surprised and intrigued when I saw it.)

I doubt we'll ever find a process that always (or even usually) generates a good artifact. But processes/frameworks exist that allows for high-quality artifacts to be produced/emerge from time to time. My app Sketch-a-bit is one example: (http://sketchabit.adamsmith.as/, http://sketchabit.adamsmith.as/master.php) Many of the sketches are not that great, but some are really phenomenal. And many of the high-quality sketches have a sketch lineage that shows how they evolved over time; these great sketches weren't created from scratch, but emerged from the aimless, wandering, serendipitous creative collaboration of strangers using this app.

Some example sketches
- Most of the sketches are crappy (http://sketchabit.adamsmith.as/view.php?id=127797)
- Some of the sketches are really good (http://sketchabit.adamsmith.as/view.php?id=86305)
- Some of the good/interesting/noteworthy sketches are mutations of other sketches (http://sketchabit.adamsmith.as/view.php?id=99693)

- One user who has drawn some sketches from scratch (the boquet of roses) and has enhanced other sketches (the bright apple): http://sketchabit.adamsmith.as/user.php?uid=6a503954-08be-49cd-9f25-795376d99799

Right now, my analysis of Sketchabit is kind of vague and wandering, too. What I'd really like to do is build a couple similar systems, compare them, and develop a general framework for collaborative exploratory creativity.

The general framework would consist of something like this:

1. Input is the same as the output (e.g. drawing -> person -> drawing)

2. Long chains of mutations (e.g. a game of telephone is interesting because of how it diverges)

3. Randomly selected starter artifacts and the ability to choose *not* to modify something if it is boring or uninspiring (and choose something else instead)

4. Track the lineage of each artifact

Possible add-ons:

1. Rate/star/favorite random artifacts as they cross your path (built-in evaluation!)

2. See what people have added to your artifact

3. Curated gallery of artifacts (so people could seek out the best artifacts to improve them further)

4. etc...

Questions:

- what mediums work/don't work?

- anonymity vs. not?

- curating and finding the cream of the crop?

- do people evolve better when they can curate vs. when its totally random?

- different/complex artifacts and ways of mutating something, e.g. the yes/no questions is an artifact and adding questions, deleting questions, and flipping answers is the mutation

- how do people start to get involved?

- what social things do they need to stay interested?

- how much of the system do pepole need to be exposed to?

- are certain people better at evolving artifacts in certain ways (E.g. cleaning up, adding detail, refining, pivoting)

- are people always better at evolving artifacts than creating them from scratch?

- is evaluating the "goodness" of an idea by how many people choose to work from it a good metric?

Maybe this only works with drawing? Or maybe it would work with short writing, or writing + summary, or certain kinds of music generation, or idea brainstorming... I tried to make a sci-fi short story version, but it's not as compelling as the drawing one for some reason. http://superfiretruck.com/scifi/

I'm now going to try to map this onto Paul's Heilmeier questions:

1. What is the problem? And why is it hard?

How do we get the crowd to be creative? If we had a better idea of how to build an environment for collaborative crowd creativity (especially with novice strangers on the internet) we could use it for large-scale artistic expression, real world problem solving, or invention/design of new ideas... It's hard to take mediocre artifacts from the crowd and combine them into a good artifact. Or develop a process that lets mediocre workers collaborate to produce something greater than the sum of their parts.

2. How is it done today? What are the limits?

There's a focus on creating a very structured process (e.g. our experiments) and focusing on creating a good artifact. Sometimes the process works well (Johnny Cash video) but that seems to rely on people being creative and motivated up front. Arbitrary crowds can't be cranked to reliably produce good artifacts... it seems....

There's also a focus on exploring structure/organization, which is an important (and more realistic) part of collaborative creativity. Kurt's Pipeline system is used by real collaborative teams to create videos and art projects.

The "artifact" is the end game for both of those scenarios, and as we've seen, evaluating those artifacts is hard, let alone always getting artifacts of a good quality.

3. What's new?

Deep chains! Create a system that fosters creativity and lets it emerge naturally, but doesn't force it.

(Ahh, I need to go to dinner and I'm losing steam with these questions)

4. Who cares?

Not sure! I mentioned before that if we had a better understanding of how to foster collaborative creativity, we could use it for large-scale artistic expression, real world problem solving, or invention/design of new ideas...

5. How to measure success? Impact?

Evidence of emergent artifacts that could only have been created through a particular evolutionary path. Applying this general framework to several different domains, especially if one looked like "real world problems". There are some websites/things for "improving your community through crowdsourcing" where people suggest to add a new park at some abandoned lot or whatever, or brainstorm solutions to polluted drinking water in cambodia... and I don't know how successful any of those really are. We could even try it with research paper ideas! 

Okay! That's that. Talk to you folks tomorrow...

- Kathleen

[Kurt Luther]

Lots of interesting ideas. Kathleen, I liked the detail you went into
here. A few reactions:

-Sketch-a-bit is a wonderful example of the power of iteration. Is
there branching too? I can't remember.

-Visuals/drawing does seem to be a special medium because of how
quickly we can grasp visual material. Music, stories, even videos.
take time to experience, and time seems to add significant
complications, like pacing and consistency, or just willingness to sit
through it all.

-"How can we make crowds creative?" is an interesting framing. I might
push back on it a little, though. My opinion is that humans (and
crowds) are already fairly creative -- that's one of the reasons we
want to use them instead of algorithms for certain tasks. I think the
obstacle is figuring out how to channel and direct that creativity in
interesting or useful ways. Consider Johnny Cash Project and Sheep
Market. These projects emphasize the *diversity* of human creativity
and that's what people find so compelling. The craftsmanship of
individual Johnny Cash frames or sheep drawings generally isn't great,
but looking at the variety en masse is the compelling experience.
These projects direct/channel the existing creativity of the crowd in
a compelling way.

Their creative process is really basic, though: "have the crowd do a
lot of the same thing." (I think you've written about these different
patterns of crowd creativity.) It's barely collaborative, if at all. I
am more interested in the projects that attempt real collaboration.
Sketch-a-bit is compelling because it allows for iteration across many
people. Branching is another interesting collaboration pattern.
Multiple workers collaborating in real time is a third pattern. There
are others too.

-If we explore iteration as you suggest, we should consider the
literature on exploration vs. exploitation. How does the quality of
the original seed drawing constrain subsequent iteration, when do we
want to refine vs. explore new ideas, etc. For example, my co-advisor
Steven Dow has shown that making parallel, different prototypes leads
to more creativity than just iteration.



--
Kurt Luther
www.kurtluther.com

[Kurt Luther]

My proposal would be to explore how the crowd can help solve problems
in a fictional narrative. A novel writer is stuck, has writer's block,
etc. and needs ideas for what should happen in a story. The crowd can
offer useful suggestions.

-What's the problem? why is it hard?

Writers have to craft lengthy, complex stories, often without much
help. Brainstorming by yourself is inherently limiting, and
frustrating. They can get feedback from trusted friends, writing
workshops, professional editors, but these require a lot of time,
effort, social capital, money, etc. There's also a limit to how many
times you can ask for help.

-What's our approach, and what's novel?

Use the crowd. Crowds are available quickly, there are always new
workers to hire, and the supply of new ideas is basically
inexhaustible. The idea of using crowds for this is novel. The
challenge is that crowds generally lack the time/interest to read huge
chunks of text. My proposition is a the writer and the crowd engage in
a back-and-forth conversation inspired by yes/no questions, the ESP
Game's "taboo words", and Walter Lasecki's Chorus. The writer provides
a brief summary of the problem, and the crowd offers potential
solutions. The writer marks each suggestion as appropriate or not
appropriate (and explains why not). New crowd members can iterate or
branch from 'appropriate' ideas or suggest new ones. 'Inappropriate'
ideas indicate unproductive directions. Probably the first 10 ideas
will be mundane or obvious, but as workers refine or spin off this
initial set, the solutions may become increasingly
creative/offbeat/clever. One important benefit is the crowd never has
to actually read the novel--they make suggestions and modify those as
the writer corrects their mental model.

-Who cares?

Hopefully writers, but even if this is a contrived problem, the
approach could be valuable for anyone seeking ideas from a crowd which
lacks deep familiarity with the domain.

-How do we measure success?

Did the writer get a good solution from the crowd? Multiple good
solutions? At what point do the good ideas start coming in? Are good
ideas more likely to be iterations, branches, originals? etc.


--
Kurt Luther
www.kurtluther.com