LMS Data and Predictive Analysis paper

[Bob Edmison]

Saw this on the Educause learning analytics slack channel this morning. 

https://www.edworkingpapers.com/sites/default/files/ai22-647.pdf

Authors from UVA. TL;DR: 

"Among students with prior academic history in college, administrative data-only models substantially outperform LMS data-only models and are quite accurate at predicting whether students will struggle in a course. Among first-time students, LMS data-only models outperform administrative data-only models. We achieve the highest performance for first-time students with models that include data from both sources. We also show that models achieve similar performance with a small and judiciously selected set of predictors; models trained on system-wide data achieve similar performance as models trained on individual courses.” 

So, LMS data usefulness over time decreases, and hybrid models work best. Might this also apply to predictive models from non-LMS data sources?

Worth a read.

Cheers,

Bob

 

-- 
Bob Edmison, Ph.D.
Collegiate Assistant Professor
Coordinator for Online Teaching and Learning
Department of Computer Science 
Virginia Tech (0106) 
1220A Torgersen Hall
Blacksburg, VA 24060, United States (ET)
bedm...@vt.edu

[Stephen H Edwards]

This looks interesting. In a brief scan, I was caught by this summary of the LMS data:

total number of click actions, total time spent online, percent of on-time assignment submissions ...

There's more to it than that (session length, discussion posts started, etc), but once you start digging into the details, it becomes clear that there can be very wide (and expected) variations across the LMS data across teachers, delivery modes, etc., and it isn't so easy to see how the LMS data in this study is doing much more than giving an indication of the student's level of engagement/involvement with the course's website, and the percent of assignments turned in.

It also isn't surprising that when the "admin" data includes gpa, total credits earned, past academic history with the current course (have they attempted it before and past grade in it), average grades/completion rates for each course, etc., that this data provides more predictive power as long as they have a substantial history with the student, but fails for "first-time students" where the only admin data available is average performance info from past students in the course.

The paper is interesting, but I think there are deeper issues regarding which predictor variable(s) are a good fit for the actual problem being solved, and the nature of the prediction problem. Here, the stated goal was predicting "successful course completion". It is interesting to look at the practical considerations in framing the predictor variables to address the intended problem, and it is also useful that they find different value from different parts of the data, including identifying meaningful contributions from both admin and LMS data (for perhaps complementary insights) in some of the most common top predictor variables:

the number of total credits attempted in the target term, the two measures of historic performance in the course, and the two LMS measures of student engagement

I'm not entirely sure the overall conclusions hinted at in the abstract are the meaningful parts of the paper, but there's definitely some value and learning that can be drawn from this study.

-- Steve

---------------------------------------------------------------------------
Virginia Tech, CS Dept.     Web-CAT: Web-based Center for Software Testing
2202 Kraft Drive            Automatic grading using student-written tests
Blacksburg, VA 24060 USA    http://web-cat.org/
(540)-231-5723              http://people.cs.vt.edu/~edwards/

--
You received this message because you are subscribed to the Google Groups "Digital Education Research at VT" group.
To unsubscribe from this group and stop receiving emails from it, send an email to digital-education-rese...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/digital-education-research-at-vt/55790197-E14E-4FB5-9CDF-6FB2C9F2D2E4%40vt.edu.

[Cliff Shaffer]

I read the paper a bit quickly, so I could have missed something. But my takeaway from this is that they are essentially advocating for the view that LMS data are pointless and not worth the effort to gather. But actually, my interpretation of their work is "We think that the effort of collecting LMS data is not worthwhile because we couldn't improve our pointless models used for a pointless use case."

Seriously, what would you do with the result of such a study, regardless of the outcome? Consider:

* We've known for decades that we could predict student performance "pretty well" from simple indicators like their past grades.

* Given how well we could do that, it's a tall order to do better by another method.

* For decades, educators have tried to take advantage of that information with some form of intervention for "high risk" individuals as identified by the models.

* Obviously that hasn't worked... as evidenced by the fact that the models give us pretty much the same exact results even with the interventions in place!! (After all, it's not like this group can still collect "clean" data that don't include ongoing interventions of this nature.)

* I found it hard to suss out the probability of predicting a given case accurately from their data. A lot of their measures are more abstract, like "c-statistics". But it looks to me, to greatly oversimplify, that one can generally predict accurately about 85% of the time when someone will "struggle" (meaning not succeed in the next class).

* OK, so now what do you do with that? Well, one example is to impose a policy like a requirement to get a certain grade to continue. Which generally, academic programs do. One might look at the data and decide to tweak the threshold grade. Perhaps an 85% failure rate for people who got a C is too high, and we should require a C+? (Not an arbitrary example -- we require a grade better than C- to continue past a prereq course.) Keep in mind that anything you do is probabilistic. We only know "mostly" what will happen, not perfectly.

None of this strikes me as being on the right track. Ultimately, static predictive models don't get us anywhere. If we really want to effect change, then we have to dynamically, when-it-happens, identify that someone IS JUST STARTING to struggle, and we have to come up with an effective intervention RIGHT THEN that changes their trajectory.

I am not necessarily saying that LMS data is the solution to that (though I have difficulty imagining success in a large class for any sort of monitoring that doesn't include LMS data). But I am saying that the work in this paper doesn't appear to me to have anything to do with that.

To view this discussion on the web visit https://groups.google.com/d/msgid/digital-education-research-at-vt/CAAJPYVBXSM%3DVQL5FZmk4QHKQXLA%3Djj31tnZdvAC7hG0rmZe9OA%40mail.gmail.com.

--

Dr. Cliff Shaffer                                               

Professor and Associate Department Head for Graduate Studies
Department of Computer Science              Phone: (540) 231-4354
Virginia Tech, Blacksburg, VA 24061          WWW: www.cs.vt.edu/~shaffer

[Cliff Shaffer]

I did notice that there is a fallacy in my argument. The rate at which we can successfully predict struggling students is probably independent of how many struggling students there are. So an intervention on struggling students might succeed and therefore reduce their number. But then the model might continue to predict the remainder at the same rate. (I think they won't be completely independent, but none of these numbers are exact anyway.)

That might not change my basic objection, I need to think more about it.

[Stephen H Edwards]

Cliff, I agree with a lot of your reactions to this paper.

my takeaway from this is that they are essentially advocating for the view that LMS data are pointless and not worth the effort to gather. But actually, my interpretation of their work is "We think that the effort of collecting LMS data is not worthwhile because we couldn't improve our pointless models used for a pointless use case."

This is reasonable. I'm not entirely sure the actual details they report entirely support the broader view promoted in the paper's intro/abstract, but I also agree that there are clear issues in how to frame the "use case".

In a more pessimistic light, one can also interpret the paper as basically saying interventions can't/won't matter. By framing past academic/course performance as one of the strongest predictors, the paper is implicitly supporting the belief that "generally, strong students do better and weaker students struggle more". But that type of model really suggests that struggle is primarily pre-determined by factors outside/before the course ... which completely undermines the strategy of providing interventions to help struggle (sigh).

If we really want to effect change, then we have to dynamically, when-it-happens, identify that someone IS JUST STARTING to struggle, and we have to come up with an effective intervention RIGHT THEN that changes their trajectory.

I'd argue that this view is oversimplified (and perhaps misleading). The problem is definitely harder than that.

It turns out that "just starting to struggle" is actually when students learn. Students who never struggle arguably aren't learning as much. The whole notion of the "zone of proximal development" is centered around the idea that students have to be working at the edge of, or just outside, their area of mastery for the best learning to occur. In other words, successful learning involves a certain amount of struggle when operating right at the edge of one's current capabilities.

So, when someone is "just starting to struggle", that's actually too soon to intervene. What you really want to know is whether the student will "get stuck" because that struggle is too much for the individual to overcome (i.e., they'll need assistance/coaching in order to overcome it), or whether they'll actually be able to overcome the struggle without too much frustration/negative experience (and learn through the process, as well as feel the accomplishment of doing it on their own).

I think the prediction problem is really about identifying excessive/impeding/demotivating struggle rather than "any" struggle, because some struggle is necessary/beneficial. Intervening too early could actually have negative consequences on learning, depending on the situation. But little work makes any meaningful distinctions in this regard.

Things get even worse when you think about the fact that different students react to light/medium/heavy struggle in different ways. Mindset and other self-theories can lead some students to "give up" rather than persevere when they encounter even light struggle ... even if that light struggle is an important part of their learning. In that case, should the intervention be about the "struggle" itself, or about the student's reaction to it?

This paper definitely does not reflect a deeper view of the prediction problem or what they would do if they could accurately predict struggle early enough.

[Cliff Shaffer]

OK, I will accept that the phrasing "just starting to struggle" is not right. But under some other wording, I think that there is an appropriate intervention point. And like I indicated in my followup, there probably is a place for what I am calling a "static intervention" -- something where we address all students or a group of students who performed badly in the past under some definition. But fundamentally, I think we should be aiming for interventions that catch the problem (defined in some way) well before they get a bad grade in a class. That is what I am referring to as a "dynamic intervention".

A lot of what we are doing in our research is of that nature. Pretty much all of the feedback from Web-CAT, Code Workout, or OpenDSA is immediate, dynamic intervention.

                       -- Cliff

[Stephen H Edwards]

I agree with everything you've said here. Most importantly, if one is shooting for what you're calling a "static intervention" (which is a very good idea), then that minimizes the need for an accurate predictive model for individual students. Instead, you just need to identify the best time to apply that static intervention to all (or a group) of students, and that can probably be more easily planned out than individual just-in-time interventions. Which gets back to your point that the original use case in this paper may not be the best choice to start with.

-- Steve

---------------------------------------------------------------------------
Virginia Tech, CS Dept.     Web-CAT: Web-based Center for Software Testing
2202 Kraft Drive            Automatic grading using student-written tests
Blacksburg, VA 24060 USA    http://web-cat.org/
(540)-231-5723              http://people.cs.vt.edu/~edwards/