Magnum Prediction Statistics
Josefa Palsgrove
How might AI change the way scientific progress happens? In particular, how might it help us make progress in areas of science where progress has historically been slow, like psychology or other fields of social science?
There's the person system, so to speak, with your thoughts and behaviors and feelings, and your genetic setup and so forth. But there's your partner who influences you and your family history and your folks and life events and stressors, and all of that is part of what I think to be the mental health system of a person and your current state.
DS: That makes sense. It's so interesting because I think everyone sort of agrees with that story, or not everyone, but a lot of people would say they're emergent and it's sort of bio-psychosocial. It's a combination of all these things, and the combination is probably different for different people.
If you ask me what the orbits of the moon is, I have an equation. Do you think we'll ever get to a place where we're going to get down to that level, or that there's this very high-level story you can tell and then the details for each individual person are so complicated that having an explanation is going to be hard? An explanation that's compressible is going to be hard to find, or are you looking for that explanation?
We worked on panic disorder first because if you draw 50 random researchers on panic disorder from around the world to the table, most of them will actually agree on the etiology and phenomenology of panic disorder, which is not the case for some other mental problems. So we started there, and the model is basically a formal theory, a formal model, and their equations. And then you can simulate data from the model. And then you can see if the data you get for a person with panic attacks, for example, corresponds to data we observe in the real world. You can see, [what] does the phenomenology of panic attacks look like? OK, they're brief, check that they should be pretty brief. Panic attacks don't last for half an hour or three hours.
And I think that work is promising, although it is far away from being a Einstein's theory of relativity. I think it is a model to begin with. And it was indeed quite tricky to decide what's in the model, what's not, what is just important enough to warrant modeling. That's my first answer.
The second answer is, there's work on dynamic properties of systems. This work argues that it actually doesn't matter too much what particular nodes you assess in your system, as long as all of these nodes tap into the dynamics of the system, because it is measuring the dynamics that give you information about the system and not necessarily all the rest.
A researcher in our field has a really cool paper talking about the two worlds of psychopathology. In it, he shows that he has a couple dozen people undergoing psychotherapy. They use a system where they ask people once or maybe multiple times a day about their moods, feelings, thoughts, behaviors, and ecological momentary assessment. They track them for multiple weeks. And the cool thing is that every person gets different variables assessed. Everybody's different. They all agree with their own clinician on what is most central to their psychopathology, even if the diagnosis is the same. Some folks sleep too little, some sleep too much, even if they have the same diagnosis. Some people are sad, others are suicidal and so forth. The analysis in the paper shows that you can, independent of the content of the network or the system, use these dynamical principles to see if people are going to get better or not.
DS: That's really interesting. I hear you on, rather than looking at the content of the nodes, so rather than looking at, for me, maybe I sleep too little. I know that I sleep too little. And if I sleep too little, that increases my symptoms. You're actually looking at, it sounds like, the relationship between nodes. What are some examples of nodes? And then what are some examples of relationships? How would you look at the relationships independent of the nodes as a way to assess things?
The way critical slowing down works, without being super-technical about it, is that when a system transitions from one stable state into another stable state, and when this transition is abrupt, this is important. We'll talk about this later, perhaps because there's also slow transitions, and it doesn't really work that well then. But if the transition is abrupt, like a catastrophic shift, then there's evidence in ecology and cancer biology and economics and other climate science, that the elements of the system change their autocorrelations over time. The system becomes more predictable, and the system moves slower, so to speak.
That's why you say critical slowing down. So translating this to my mental health example, if I know your current mood or sleepiness or concentration or suicidal state right now, and I see that your state tomorrow will become more and more predictable from your current state, we're talking about critical slowing down, which is an early warning sign for an upcoming transition. This has been shown a couple of times in data with depression, for example, in usually just one particular person. There's other dynamic principles, connectivity, and so forth. But this early warning, critical slowing down, is one of the ones that has been discussed the most. If you think of a system like a river, and you can measure the speed of the river using different types of thermometers, this ideographic argument where the content doesn't really matter, the dynamic principles matter. [That] translates into, well, as long as you put your thermometer somewhere in the river, and you pick up some part of the system, that will give you enough information to pick up on changes, and for example, autocorrelations to tap into critical slowing down. If that works or not, we don't know.
DS: That makes a lot of sense. So it sounds like what you're saying is you have a system of interconnected parts. And what you've observed is that there's an abrupt or catastrophic change from one regime to another in the system. Thereafter, that system will slow down, or it will not change as quickly.
EF: Right. So indeed, the cool thing is that this is actually independent of severity, right? You can have lower variation in sleep problems in two ways. Maybe you sleep well every night or you sleep badly every night, but the lack of variability translates into higher auto correlations or lower standard deviation over time. So the system becomes more predictable. And that might signal an upcoming transition. People in my field say forecast rather than predict, because predict has this in 10 days, whereas a forecast test is soonish, like a weather forecast, which tend to be pretty bad in the Netherlands. Still, it's going to rain perhaps at some point in the next three days. So yeah, we use forecast at the moment.
When I talk to journalists about the warranty system we're building, I always say that measuring wind is probably a bad early warning sign for a thunderstorm or a hurricane, because when the wind starts, it's probably too late already. In the same way, measuring symptoms is probably a bad early warning for depression because when the symptoms start, you're probably already in the onset phase of depression. So they cannot forecast based on the severity of symptoms or the symptoms at the mean level, but based on the autocorrelations of the symptom relations or the affect relations over time.
EF: The lag one coefficient of one node in the system to the same node in the system over time. Linear regression of one is univariate. Just one has nothing to do with the system itself per se, just your sleep on your sleep on your sleep over 100 days. If the autocorrelation is extremely high, it means your sleep tomorrow is extremely predictable by your sleep today.
DS: I see. Meaning that you don't fluctuate as you should with the environment. Like I'm not stressed every day, but there's stuff that happens to me sometimes. And if you don't respond to that stuff with stress, that's not normal.
EF: Yeah. In our data, initially we see that a sign for depression might be that people have low mood, independent of the context. We have lots of context data in our data set. Are you with friends, with family at school, at work, traveling in nature? We see that some folks have context-independent, really low mood. That might be a marker for depression, for example.
EF: Right. This is one of many early warning signals. This all only works if shifts are catastrophic, like for this particular person I talked about before they really relapsed. Some patients talk about it like a black wave falling over them, but it's very open to question whether depression onset looks like this in most people.
There's very little data because we've only now been able to collect these daily data for months and months in folks. One of my graduate students is actually working on the nature of onset at the moment, just phenomenologically to see how people onset depression.
DS: One of the things that strikes me about this approach is requires the moment-by-moment data. That seems dramatically easier to gather now. Tell me about that. Everyone's got a smartwatch. It's funny that you're mentioning this because I literally built a little text bot that texts me everyday, every hour with a bunch of different questions about me, and then has a readout of it. I haven't been doing any statistics on it, but the overall idea is maybe at some point, I don't know that it generalizes in a scientific way, but it might be helpful for me. I'm curious about it.
They also fill out questionnaires four times a day. It's a lot actually. They're only two minutes, very short questionnaires. They have like, I don't know, 15, 20 questions. They're very short. How happy are you right now? One, two, three, four, five, six, seven, stuff like this. At the end of the day, there are a couple more questions about how was your day? What was the worst thing that happened to you today?
c80f0f1006