Respondents Wanted: Assess the Creativity of Software Generated Stories

[Lewis Mckeown]

Hello,

I am currently working on my MSc thesis in the domain of computational creativity and have written some software which generates short story outlines.

I would like some feedback about a selection of the stories and to what extent they demonstrate (or do not demonstrate) creativity. 

Due to my project deadlines I have to have collected responses by midnight on the 23rd of August, so any feedback between now and then would be invaluable (although responses the week after will also be useful).

There are a selection of 12 stories to evaluate and I hope it should take no longer than 15 minutes or so. If you could spare the time I would really appreciate the feedback, and any other comments, criticism or discussion would be very welcome. 

Here is a link to the survey: http://co880.lewismckeown.com

Thanks,

Lewis Mckeown

[Leonid Berov]

Dear Lewis,

this looks very promising and (having yet seen only the first story) I like the number of background details the system provides and how they are visualized. Congratulations for your work!

I do have, however, a question regarding the rating task. The definitions of creativity you provide focus strongly on the novelty of the result of the creative task. But sometimes (human or computer) generated artefacts can be of poor quality, or simply so unusual, that they do not seem to solve the creative task that stood at the beginning of the process. In such cases it seems debatable if one should attest the results a high creativity rating (due to their high novelty) or rather a low one (due to their low quality). If you are interested, Ritchie (2007) provides a very thorough discussion of these points.

Coming back to the survey, I am a bit stuck at the first story. I would rate it as very novel. So novel in fact that I would not necessarily place it in the genre of story, which for me implies a low quality. I'd go with a low overall creativity rating, but that would clash with the definitions you provided. Perhaps you could clarify how you would like me to handle this?
I do also, by the way, like it as an aesthetic artefact, its just the label 'story' that makes me feel conflicted here.

Thank you very much, for sharing your work with us!

Leonid

Some ad-hoc references:

- Ritchie, G. 2007. Some empirical criteria for attributing creativity to a computer program. Minds and Machines 17:67–99.

- Colton, S. 2008. Creativity Versus the Perception of Creativity in Computational Systems. In AAAI Spring Symposium: Creative Intelligent Systems (pp. 14–20).

[Lewis Mckeown]

Hi Leonid,

Thanks for the feedback! It's really encouraging and insightful.

I think I can empathise with the problem you're expressing regarding rating the output. 

I struggled later in the project development with how to present the stories. All the components are created as objects and put in order as the software generates the stories, but a selection of JSON objects isn't really a 'story'. So I had some trouble trying to reconcile what I suppose would be the fabula that is generated into a discourse that could be presented in a way that humans could enjoy (or not). 

I settled on the small vignettes using just the shortest descriptions that each object contains so that I could have users read a reasonable selection of them and provide feedback without getting too fatigued. Which may undermine my initial goal of generating 'stories'  so perhaps it's the wrong medium to to present them as here, as they are simply the outlines (the software does generate a little more explanatory text along with each one, but still at a similar high level to the outlines presented). 

I am hesitant however to label the output something other than a story/story outline though, as although they may not fit a conventional conception of a 'story' that's almost the purpose, given the creators whose works or style I am trying to replicate with the software.

As I am primarily interested in generating artefacts on the stranger side, I wanted to try and distinguish people's opinions on the quality (at least somewhat) from the creativity, which is why I gave the option to say if the respondent liked the story separately from how creative they think it is. This is also why I chose definitions of creativity which focus on novelty or unusualness as the artists I am using as inspiration are often praised for the presence of this in their works, which are considered (generally) to be very creative. 

So I suppose if you like it as an aesthetic artefact indicate that you did in the second question on the survey.

With regards to creativity, if it veers too far from what you consider a story (and my above explanation doesn't clarify it's presentation) then maybe you could indicate that it is not demonstrating a high level of creativity? Or if you agree that the definitions given are accurate for creativity and that the output matches then it is demonstrating creativity? The definitions are just a guide that I hoped would put people in the right frame of mind for evaluating output that might be a little unusual, however, if you have another definition for creativity that you would rather evaluate them using, please do and I'd love to know what it is.

I hope this answers your questions at least a little. Apologies if it was a bit rambling, but I wanted to reply as quickly as I could whilst I had time :)

Thanks,

Lewis

P.S I will add both articles to my reading list they both seem really relevant to this part of my project, thank you for the recommendations!

[Nick Montfort]

I just spoke about the topic of evaluating computational creativity
outputs at the KDD Machine Learning for Creativity workshop:

http://nickm.com/post/2017/08/c-creativity-my-talk-at-the-kdd-workshop-on-ml-and-creativity/

My slides by themselves do not explain exactly what I said, but what I
said when on slide 7 is on this topic and I'll try to paraphrase it.

In short, it seems essential to me that evaluation of creative outputs be
done in a cultural context. It matters whether we are generating stories
that are supposed to relate to native cultures (MEXICA) or ones that are
supposed to relate to soap opera plots from the USA (UNIVERSE), even
though Mexico and the USA are neighbors and the work is done in the same
academic community. We should have evaluators who understand and are
somehow situated within the cultural context where we are trying to make a
contribution.

I know this does not respond to the immediate need for evaluation by
August 23, but I hope it contributes to the broader conversation. I
appreciate Lewis's comments and find myself "on the stranger side" with
him when I work in story, poetry, and related sorts of generation.

-Nick

> --
> You received this message because you are subscribed to the Google Groups
> "Computational Creativity Forum" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to computational-creativ...@googlegroups.com.
> To post to this group, send email to
> computational-c...@googlegroups.com.
> Visit this group at
> https://groups.google.com/group/computational-creativity-forum.
> To view this discussion on the web visithttps://groups.google.com/d/msgid/computational-creativity-forum/54671f98-b
> d54-4163-8eb2-64741741c1ee%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
>

[Smith, Gillian M]

Thanks for sharing these slides, Nick. I completely agree – creativity is culturally situated, and we should explicitly acknowledge that culture during both creation and evaluation. Do you have anything written on this, beyond the talk?

Gillian

To view this discussion on the web visit https://groups.google.com/d/msgid/computational-creativity-forum/alpine.DEB.2.20.1708181309480.9061%40nickm.

[Nick Montfort]

Gillian, I'm sorry, but I don't have anything written or published on the
topic, just a recording of my short talk which I hope to put online in the
next week.

If I had done real scholarship in this area I'd probably have cited the
work of Fox Harrell, who spoke on this topic back in 2012:

Harrell, Fox. “Matching Methods: Guiding and Evaluation Interdisciplinary
Projects,” invited talk, NSF, NEA, and NEH Media Systems Workshop at the
University of California, Santa Cruz, Santa Cruz, CA. August 27.

But you will have to check with him to see what was published out of that.

-Nick

> To view this discussion on the web visit https://groups.google.com/d/msgid/computational-creativity-forum/0DFA64C5-15C9-4073-99F9-082B1A23A828%40wpi.edu.

[Rogelio E. Cardona-Rivera]

For folks who aren’t aware, that talk was published online on YouTube: https://www.youtube.com/watch?v=emBFPVjIz1U

I too would be interested to see what else came out of that talk.

— Rogelio

—
Rogelio E. Cardona-Rivera
Assistant Professor
Entertainment Arts and Engineering, School of Computing
University of Utah
http://rogel.io

> To view this discussion on the web visit https://groups.google.com/d/msgid/computational-creativity-forum/alpine.DEB.2.20.1708181917060.31457%40nickm.

[Lewis Mckeown]

Hi all,

Just wanted to thank everyone who responded here, via email and to the survey. It was really useful and encouraging. 

If anyone is interested in reading my dissertation (now it's done!) or discussing anything further, feel free to get in touch.

Thanks again,

Lewis Mckeown