Experimental Design

[Alan Isaac]

After reading the lecture https://subversion.american.edu/aisaac/notes/experimental-design.xhtml,
use this thread to discuss experimental design.

[Alan Isaac]

random-seed

In the “old days”, we got the random numbers for our experiments by taking out a book of random numbers.  We would open the book arbitrarily to pick a starting page, column, and row, and then we would read off the numbers sequentially from that point.

Now we implement the whole process on a computer.

Setting ``random-seed`` is like picking the page column and row.

[Adaora Isichei]

I have a question, ( I apologize in-advance if I am meant to know this terminology)  what is CSS?

[Adaora Isichei]

An experimental design is simply the objective of a research. It conceptualizes what a researcher attempts to study, the variables needed to conduct the study, the structure of the study, the outcomes/results of the experiment and the limitation of the study. Experimental design simply defines what, how and why components of an experiment. It's usefulness is not limited to just ABM, it is applicable to all forms of qualitative research.

[Alan Isaac]

Definition of CSS

CSS = computational social science.

[Amanda Saville]

Adaora,

While I agree with you that experimental design allows a researcher to pursue the main objective of his research using an agent-based model, I think experimental design is even more deeply connected with the planning and decision-making processes that precede experimentation.  Decisions made before experiments shape the outcomes of those experiments, and careful planning is necessary for a research to truly address the questions he had originally sought to answer.  I think for all of us, reviewing the questions Kelton poses in his paper Designing Simulation Experiments will be helpful reminders to keep us on track as we prioritize and plan experiments.  As I think all of us have likely experienced through this course and more specifically while working on our final projects, it is easy to become bogged down with the programming aspect.  Unfortunately the coding obstacle can often leave limited time/energy for ample experimentation once the model is completed.

As we are approaching last steps in this course, I had thought I would have already developed a cohesive plan for my experiments at this point.  Unfortunately, Natalie and I are still working on our core code (which we will hopefully complete this weekend).  More unfortunately, I had several ideas for things I wanted to test or to examine upon implementation of our model that I now wish I had written down as switching gears from coding to experimentation is proving challenging.

When writing our report on Segregation, I found that poor planning made me waste a lot of time and energy, and I was not only unhappy with the end results, but I was unhappy with the way I displayed my results.  Had I planned more thoroughly beforehand, many of my frustrations and shortcomings probably would have been avoided.  So even though I haven't really started the planning of my experiments for the course project, I know that I want to spend a lot of time planning so that I can gather and display data that will be worthwhile and interesting. 

Ultimately, while I recognize the outputs of experiments are unquestionably the main driver of experiments in the first place, I think you were correct in stating:

Experimental design simply defines what, how and why components of an experiment.

 

Experimental design thus does not refer to the results or the outcomes of the experiments, but rather forces us to think about the components of the experiment, and forces us to address questions about expected outcomes, length, variations, and how to display data when it comes time to share our results.

[Adaora Isichei]

Hey Amanda,

I have to agree with on the importance of proper definition of an experimental design for a model. I found that I had to change a lot of perimeters and variable in my model. I believe with ever experiment one will have to constantly redefine a model.

[Alan Isaac]

Experiments vs Models

We create models in order to experiment with them.  We design experiments so that our models can provide us with useful information.  In the process of designing our experiments, we may discover the need to change a model.  This often happens.  But ideally we plan ahead, so that the model we develop is suitable to the experiments we wish to run.  We then run many experiments with the same model.