Master Thesis usability study

[Robin Faber]

I am currently doing my master thesis in CS at the TU Delft with the algorithmics department. The subject is data mining in agent-based simulation models. The goal of this master thesis project is to develop a tool that allows model designers to easily execute and analyse model experiments for NetLogo. Currently the tools for this purpose lack functionality or usability, so we attempt to develop a tool that achieves both of these and is available to everyone. The tool consists of a GUI made in Python that allows users to input run configurations, run models, visualize results and save/import results.

In order to evaluate whether the tool has successfully achieved its goals, I plan to do a usability study with the actual target group for the tool (model designers). The overall plan for this user study is to be in a voice call with me (Teams, Skype etc) while you share your screen with the GUI on it. You will then execute the tasks with the GUI on screen and the results will be recorded by me. Afterwards, there will be a small questionnaire to get your opinion on the tool and its functionalities.

I estimate that the entire user study will not take more than an hour of your time. My plan is to perform this user study somewhere between the end of April 2021 and halfway May 2021. My schedule is extremely flexible, so I’ll (usually) be able to plan it whenever it’s convenient for you.

Would anyone here be interested in participating in this user study to help me out with my master thesis? It would be very much appreciated. If you are not interested personally, do you have any recommendations for people that have experience with NetLogo and may be interested in participating?

If you have any questions, feel free to contact me at r.j....@student.tudelft.nl.

Thanks in advance, 

Robin Faber

[Wade Schuette]

Robin, yes, I'd be glad to participate in your study.

While you have specific requirements for your Masters Degree that I'm sure limit the scope and time of your work, I'd hope that you intend to put your software project up on GitHub where the rest of us can survey it,  debate it, and fork various variations of it to make an ever improving tool over time!  I suspect again that your degree committee wants to see work that is uniquely attributable to you, but everyone else would prefer to see work that we've all participated in making the best it can be, so the sooner the better.

Yes, I'm frustrated that once a model is tentatively built,  and one has run ten thousand variants via Behavior Space, that cleanly pulling together the results and analysis in clean, if not pretty publication-ready charts and graphs and interactive pivot tables, etc., would be lovely.   Here one has to go outside NetLogo to Python or R or Tableau or something and it is frustrating if not hard work and a total roadblock to many people who are not good at such things.

In the meantime, short of sharing the software, I'd be interested in seeing your analysis of what is right and wrong with current solutions and how you derived what you think is a fix for some of the problems.   For one thing, I don't see "model designers" as the ultimate target audience / users / customers in the public policy arena, and would be more interested in what interface ( GUI? ) would make models more comprehensible and useful to, say, policy makers -- with the idea of models of Covid-19 as an exemplar.   In 2019, before the pandemic arose,  there was much grieving the fact that policy makers don't turn to epidemiology models soon enough by the epi community,  led by Caitlin Rivers at Johns Hopkins.  ( see my discussion in my weblog here: https://newbricks.blogspot.com/2020/02/systems-thinking-about-outbreak-science.html ).      In that sense "usability" of a "GUI" is the key factor, and I suspect these days that interactivity ( versus passive "output" charts and graphs ) is crucial to making data approachable, accessible, and useful and allows some assessment as to whether to believe it or not.

But stepping back a level, or up another notch in "systems thinking",  the whole concept of the Epi models the USA was exposed to were, in my mind, seriously lacking, because they showed biologically accurate portrayals of a socially embedded disease process,  totally missing the social response aspect,  totally not showing in any way shape or form the feedback loop of social response on the disease progression in the USA -- probably the single most important factor, and one that allowed Public Health to be tarred and feathered as public enemy number one, totally unconcerned about job loss or even death due to health insurance loss due to job loss due to shutdowns.    In other words, while biologically accurate, socially the models were a total and complete disaster,  setting the country into partisan war and setting public health back a decade.

So -- the larger question arises of where the whole basis of the model is represented and discussed in a collaborative learning way, and I don't see any trace of that in your Master's project.   Which makes sense for you, but in the long run, not for us.   I am far more concerned that the underlying assumptions of the model are wrong than that the data-processing-algorithms are incorrect.  And I certainly would envision a software environment in which user experience and stories can be stored along with the software to see who has tried to use it where with what success. Again to me that matters more than the algorithms.

Useful papers on the modeling process, the first one using NetLogo examples

http://jasss.soc.surrey.ac.uk/21/1/14.html

And one should not miss Bruce Edmond's Center for Policy Modeling for wonderful insights

http://cfpm.org/

and his great

Using agent-based simulation to inform policy – what could possibly go wrong?

http://cfpm.org/discussionpapers/236/discussion-paper-using-agent-based-simulation-to-inform-policy-what-could-possibly-go-wrong

But even sticking inside the box, to the algorithms,   the process of TESTING a model remains a semi black-box to many people, as with all software, and anything which allowed gathering together of suitable regression tests for new variants of a model would be helpful as would something stronger than "diff" that allowed comparison of the outcome of that test plan on a new version of code with prior runs on prior versions of the software.

Similarly, the question of how to easily check all the boxes of identifying the key factors of one's model in a standard way suitable for publishing the model and the results and letting readers see what's going on under the covers would be of interest.

So there are an ever larger set of lenses and system frameworks in which to examine a particular instance of a particular NetLogo model species, make sense of it,  and know how much to rely on it or trust it, in situ.  

I look forward to seeing your GUI / interface project, and I hope that the communities at large can join in on ripping it to shreds and rebuilding it once you get it published !! :) 

R. Wade Schuette, MBA, MPH

--
You received this message because you are subscribed to the Google Groups "netlogo-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to netlogo-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/netlogo-users/411d029d-4003-46c5-96cb-e4acc8a55bc7n%40googlegroups.com.

[Michael Tamillow]

So what you are saying Wade is that there is a serious doubt on whether we should enable policy makers with the arsenal of models and data manipulation tooling that justifies whatever action they were already intending to take?

The standard for what qualifies as predictions based on causal intervention in a lot of modeler's minds is pretty low and unscientific. Many modelers simply develop regressions and believe changing the values of independent variables, to get alternative predictions, will justify a change in outcomes based on interventionist policy.

Clearly the models that justified this policy intervention were so wide in scope that it would be impossible to evaluate causally. That's probably why the policy interventions had been so critically advised against for over a decade. No legitimate medical practitioner or scientist would advise an approach which destroys the ability to introspect on the causal relationships in the world.

There are more people flooding into industries associated with data and information, which results in more cool tools, more interesting technologies, and also more noise. It takes time and effort and perhaps even some unlearnable traits to get a sense of what can actually be derived from data. It does not take nearly the same amount of these attributes to talk about the topic, push agendas, or make a profit. Differentiating is often hard for the general population - sometimes it is even hard for the individuals themselves!

To view this discussion on the web visit https://groups.google.com/d/msgid/netlogo-users/CACv-4rBnyHCFCBK0od4frdjXRQwNCEmbN0aoVr450CeVzMFHSA%40mail.gmail.com.

[Wade Schuette]

No Michael that is not what I said. I said there’s many larger contexts to take into account if we don’t want to be blindsided as model builders. 

I think that’s part of a model builders moral responsibility, but others disagree. 

Any tool can be misused.  The conversation a model generates is important and probably should somehow be attached to the model.

We had a saying where I used to work - the table giveth and the footnote taketh away.  

Perhaps models for policy makers might come with chauffeurs .  Anything can be cherry picked or misunderstood or misused.  

And the public has little capacity for nuance.  The press will take something out of context and headline it. 

Still - if you model a pandemic and don’t include the impact of job loss, I don’t see how that is helpful or that it correctly predicts what will actually happen.  Only in some very narrow academic sense is such a model “ correct” .   In my mind the GUI should remind one to look outside the box. 

--

R. Wade Schuette, MBA MPH

2401 Parker St. APT 3

Columbia MO 65202

mobile: 1 (734) 635-0508

wade.s...@gmail.com

[Klaus G. Troitzsch]

I agree with Michael that much caution is necessary when models are developed and evaluated without carefully including the expertise of all specialists necessary to build such a model (and regression of time series as dependent on time would be the worst case, as one of President Trump's advisers found out that on May 15, 2020, Covid would stop according to a third order polynomial based on the infections numbers of the past three months (https://www.businessinsider.com/trump-advisor-kevin-hassett-excel-function-cubic-model-coronavirus-2020-5?r=DE&IR=T).

My advice at building such models — apart from the papers cited by Michael — is: have a look at ocopomo.eu, a project that developed open collaboration modelling.

Best

Klaus

Prof. (i.R.)i Dr. Klaus G. Troitzsch
Schillerstraße 26
79713 Bad Säckingen
oder
Aeussere Dorfstrasse 39
3718 Kandersteg, Schweiz
Telefon +49 7761 9982074/5 bzw. +41 33 6750166
klaus.g....@bluewin.ch
https://www.uni-koblenz-landau.de/en/campus-koblenz/fb4/iwvi/former-rg/rgtroitzsch/klaus-troitzsch
https://klausgtroitzsch.academia.edu/

To view this discussion on the web visit https://groups.google.com/d/msgid/netlogo-users/CAFkCkGceKKgbRc2aoD%3D%2BKSwYtnmMjt1EEhKQNYYZADQYnugwQQ%40mail.gmail.com.

[Wade Schuette]

Actually if you conceive of one instance of a series or species of 

model as a catalyst for  uncovering hidden assumptions and a productive social conversation about what is known and what isn’t and what’s worth digging into ,  that might suggest a different GUI - perhaps more collaborative multi user even simultaneous multiple user- than a lone model developer.  

On Wed, Apr 7, 2021 at 07:16 Michael Tamillow <mikaelta...@gmail.com> wrote:

[wade.s...@gmail.com]

Longer term , and getting much deeper into software engineering than this master's thesis,   one place that the model building  and iterative improvement over time process could use help is connecting assumptions or intentions to make a particular change in behavior to the code one writes to implement that assumption or change.

Then you could answer with confidence questions like:  Why is this line of NetLogo code here?  What does it do?  or What changes in the code were made between September 1st and September 15th in order to implement this change in assumptions about agent behavior?  

As an aside, i love color-coded-syntax editors but one that I wish I had somedays could flip a switch and color-code the lines of code by how solid and reliable and tested they are -- so truly solid code we've used for 3 years would maybe be gold,  and changes put in yesterday would maybe show up in green  or yellow, depending on how confident one is that the line implements the change you intended.   Or maybe the code would be in red if you know you're having a terrible day and you're stretching your limit of understanding here and have the sense that your fingers are working faster than your brain or good sense would indicate, possibly writing lines of code you are sort of, you know,  HOPING do what you want, but you can't quite recall if that's true and it's too much trouble to go look it up right now.

[Robin Faber]

Yes I understand where you are coming from. However, I have studied Computer Science and thus this is also a CS thesis. We decided at the beginning to focus only on the algorithms and visualisations and to leave the model designing and the explanation/justification of these models to the people developing them. I can see how this could be a problem indeed, but sadly it won't be something that will be addressed in this thesis. 

And I indeed plan to publish the software such that everyone has access to it with a public GitHub repo. 

Thank you for participating in the user study Wade

Robin Faber