Re: DB interactive exercises

[Sef Kloninger]

Hi David -

You asked a good question about how the interactive autograder worked.  Good question.  I hope you don't mind me cc'ing the forum on my.  I'm not sure I've explained much there about the interactive trader -- what it does and more importantly, what it doesn't -- so I figured it was worth sending to the forum.  Hopefully this will serve as a bit of a spec/sketch for someone looking to extend or reuse the interactive autograder for their own purposes.

There are three main pieces: the frontend, backend, and external grader:

1. Frontend

Javascript on the UI to implement the "check answers button."  This hits the exam/feedback endpoint with the student input and gets back an HTML fragment to display.  We expect the autograder to pass back the HTML to show the answer to the student, including making the "Incorrect" text red and such.  

2. Backend

We have a special autograder type that knows how to call out to the external graders.  This is very specific to the db-class graders.  They didn't implement some general grading protocol, but something that Got The Job Done For Them (tm) and we wrote something to interface with that.

The best description of how this works are in the docstrings for the autograder interfaces in main/courses/exams/autograder.py:

• _parse_interactive - how the XML works to specify graders, questions, etc.
  
• _INTERACTIVE_grader_factory - the grader interface, how we post student responses, what parsing what we get back
  
• _external_grader_request - how we call the external grader (caching, retry logic)
  

We expect the graders to be homogeneous (any grader can grade any question), so I just put them behind a load balancer.  The endpoint is stored in GRADER_ENDPOINT.  I have anywhere between 3 and 10 m1.large graders behind that load balancer depending on load.  It's easy to scale these out since they're stateless.

3. External Grader

This code was developed by Jennifer Widom's DB Class TA's over a few generations of the course.  It can grade several different types of questions: it uses sqlite to answer SQL questions, saxon to handle XML, XQuery, and XPath questions, and so on.  

Generally these are done in chroot jails, so those machines are (surprisingly) IO bound as they set up and tear down instances.  Hence the why they're running on large instances.  The servers are PHP behind apache, with some bash setup scripts.  

One important thing we added were test scripts to ensure that the grader had come up successfully, and then wired those into a healthcheck endpoint.  That way the load balancer will only send requests to graders that are healthy.  We had an incident with a sick grader that caused us to do some regrading that was a bummer.

To support other kinds of external graders, eg. LTI, one could create a new interactive autograder that extends the existing interactive autograder.  If someone does this then please contact me so we can refactor.  The existing autograder is unfortunately hardcoded to the DB Class protocol, and could be generalized.   

For questions, contact me.

- Sef

(s...@stanford.edu)

On Sat, Mar 16, 2013 at 8:02 AM, David Glance <david....@csp.uwa.edu.au> wrote:

Hi Sef,

I was just wondering how the interactive side of the exercises works - is that all done on on the grader machine on AWS or is it something Jennifer had already set up?

david

Assoc Prof David Glance
Director, UWA Centre for Software Practice
http://www.csp.uwa.edu.au
t: 0408 916 579
e: david....@csp.uwa.edu.au

[Wei Liu]

Dear Sef,

I am working with David on one of the UWA units on Class2Go. David has extended the DB autograder to grade Octave submissions, which has been working quite nicely.

Within the code, we would like to access the username who is currently logged in - to generate some personalised datasets for each student, just wondering if you can point us to any existing modules or the right direction on this.

Many thanks,

Wei