Parsons Problems Systems Catalog: Getting Started on Standards

[Cliff Shaffer]

Hello, SPLICE Parsons Problems Interoperability Working Group members! I hope that you have had a good summer so far!

I apologize for how little traffic there has been in this Google group forum, and lack of progress overall with getting the group to set working group goals. But as a number of you know, I have been working this summer on developing a document that I consider a good first step toward goals that I have. This is the Parsons Problems Systems Catalog, see https://docs.google.com/document/d/11OOJ1p7rNFKN6Ltdko9rAyKhuQMPaq9D1KguTDQUtjI/edit?usp=sharing

I keep hoping to do more, but at this point I find that I have not touched this in literally a month, and I am about to go into an email void for a couple of weeks (vacation). So I at least want to get this out the door, so that people can start to give feedback!

To back up a bit: First let's talk about goals. Perhaps others will suggest different approaches, but what I have in mind as the products for this group are two standards documents:

* An input file format for a Parsons Problem exercise specification that can be used by any PP system that chooses to support the standard.

* An output file format for the event trace generated by a user working a Parsons Problem, that can be created by any system that chooses to support the standard.

Let's talk first about the input format. This is where the Parsons Problems Systems Catalog is intended to be of help. My primary goal in creating this document was to discover as many variations on Parsons Problems in use as I could. You can see a summary of the results on the first page of the document. My idea is that the file format standard be strong enough to describe problems that use any of the variant features. Of course, the format will need to flag the features used in a way that a system adopting the standard can determine easily whether it can support a given problem or not (if the system can't support a variant required by the problem, it probably can't do much other than reject the file).

The document captures most of the systems that I know about, and hopefully captures all of the variations expressible by those systems. If you know of other systems, especially if they add something useful like either a new variation or a good example of an input format or output format, please let me know. I believe that Zybooks has PP support, but I don't list them here. If you have information, please let me know.

So, how do we design/define the input format? I believe that the place to start is with an existing standard that gives us probably 90% of what we need: the Programming Exercise Markup Language (PEML). See: https://cssplice.github.io/peml/

My idea is that we should only need to make small tweaks to the existing standard (add or modify a field here or there) to be able to capture all the PP variation information. Then we can hopefully negotiate with the PEML Powers That Be (some of whom are hopefully reading this) to get the changes added to the PEML standard.

The main thing that is likely to be needed is to add annotation to the input problem lines to support things like distractors, pairs/grouped lines, "pseudocode" (beware the slightly odd way that this term is used in the document), etc. If I sit down and do this myself, I would likely start with some combination of the input formats from CodeCheck and Runestone for these annotations. But since I am about to go away for a bit, perhaps someone else will get enthusiastic and take care of this before I get back. ;) Absent other input, I do plan to work on this most likely in September.

So, what about the output? My litmus test for whether the output format is "strong enough" is whether I could use it to write a system that can replay the student's work. So it would not be enough (in my opinion) to just capture a "snapshot" of the submitted solution. I would want an event trace of the steps that the student used as well.

Again, there already exists a standard that I hope can solve 90% of our problem. This is ProgSnap (see https://cssplice.github.io/progsnap2/). In theory it already should be able to capture everything, since it both stores program snapshots (ultimately, that is what the student solution is), and also has a data model for event traces with enough "generic event" features. But that is rather unsatisfactory to me. Again, I hope with a few small additions (primarily to the event trace definition so that it explicitly recognizes key event types used by PP systems) that we could submit changes to the Powers That Be to update the ProgSnap standard to support our needs.

I know this is a lot to absorb. I am hoping for your feedback!

                             -- Cliff

--

Dr. Cliff Shaffer                                               

Professor and Associate Department Head for Graduate Studies
Department of Computer Science              Phone: (540) 231-4354
Virginia Tech, Blacksburg, VA 24061          WWW: www.cs.vt.edu/~shaffer

[Thomas Price]

Cliff,

Thanks for putting this document together. I'd support both goals you've outlined here, and I also agree that starting with PEML (input) and ProgSnap2 (output) is a good starting point.

I know @Dr. Carl C. Haynes-Magyar has done a fair bit of work on representing Parsons problems in a generalizable format. Perhaps he could comment on what extensions would be needed to make PEML 

Related question: Is there a bank of parsons problems that could serve as a test case for how well the format can represent things?

As for output, I'm happy to support any conversation of how ProgSnap could be extended to better support Parsons problems. In general, I imagine the main steps are:

• Identifying events (if any) that happen in ParsonsProblems that aren't already represented in the ProgSnap spec.
• Identifying any attributes that should be present for those events, or broadening the definition of existing attributes to work for PPs as well as coding. 
• My guess is you might need new values for the EditType attribute (beyond Insert, Delete, etc.), and maybe some new columns to describe the edits that occur.
• Note that we already support block-based programming, so for example most edits have a "SourceLocation" attribute that describes where the edit occurred. It can either be a row/column format for text, or a path in an AST. That's probably not quite right for PPs but it could be extended for them.
• Defining a CodeState representation. That could just be a textual representation (i.e., convert the PP to source code). But you might want something more structured, such as JSON.that includes that block bank as well as the constructed code.
• Note that I've experimented similarly with how to represent block-based programs, and use a JSON representation of the AST. Something similar might work for PPs.

I'm guessing Carl and @Barbara Ericson have the most experience with logging PPs, and maybe have some good "lessons learned' we could benefit from.

Best,

Thomas

--
You received this message because you are subscribed to the Google Groups "SPLICE Parson's Problems Working Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to splice-parsons-problems-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/splice-parsons-problems-working-group/CAF3%2B%2B94_RhkXsciZog5DH%2BSiOBTf5cO6i6BGwr1vnRffpsP7ug%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

Thomas Price

Assistant Professor

Department of Computer Science

North Carolina State University

[ Website | Email | Calendar ]

(he / him / his)

[Carl Haynes-Magyar]

Hi Cliff and Thomas,

I'll take a look at PEML and let you know. I'm ironing out some things based on my think-aloud sessions.