Proposed Feature - Timed Problems

[Steven Burch]

The following is a draft proposal for a new platform feature.
Please feel free to reply with feedback.

Abstract
========
A previous Stanford class required a timed exam, so we built enough platform
support to be able to run timed exams in the course, but we didn't build a
general solution.  In particular, the functionality provided was the ability
for an instructor to set a time limit (in minutes) for a single problem
component in the "Settings" of the component.  The functionality of having more
than one question on the exam was provided by creating more than one response
in the problem component.  The system then rejects any submissions made more
than that number of minutes after the student started the component.

But additional support is needed to make this feature robust and usable for
students.  In particular, students need an interstitial with a chance to back
out before they load the exam,  they need a visible countdown timer on their
timed item, and if a student hasn't submitted by the time the timer reaches 0,
their last saved version should be submitted.

We'd like students to lose work as infrequently as is feasible, and we'd also
like the system to be transparent to students and easy to understand, with
buttons performing expected features (e.g. "save" button doing what a student
expects that it will do).


Terminology
===========
Instructor
    a course creator

Student
    a course enrollee

Client
    a user's web browser

Server
    an edX instance

Problem Component
    A single problem (advanced or basic) that has one or more responses.


Proposed Functionality
======================

Setting a time limit
--------------------

Functionality
~~~~~~~~~~~~~
An instructor should be able specify a time limit (number of minutes) for
completing a problem component.  A student must not be able to submit work
after the deadline expires.

Implementation
~~~~~~~~~~~~~~
A setting (``minutes_allowed``?) can be stored per problem component.

Rationale
~~~~~~~~~
An instructor should be able to assign a desired time limit per problem
component.


Interstitial Warning
--------------------

Functionality
~~~~~~~~~~~~~
A student will be presented with an interstitial confirmation dialog before
viewing a timed problem, so that they have a chance to back out if they didn't
intend to start the timer.

Implementation
~~~~~~~~~~~~~~
After accepting the confirmation, the server will log the time at which the
student began the problem.  This will be used to prevent a student from
circumventing the deadline by editing JavaScript, HTML or any local code.

Rationale
~~~~~~~~~
A student could load a problem component, not knowing that it is timed.  This
dialog will explain how much time will be allotted to complete the problem, so
that the student can begin once they are prepared.


Countdown Timer
---------------

Functionality
~~~~~~~~~~~~~
A student should see a countdown timer indicating the time remaining.

Implementation
~~~~~~~~~~~~~~
Once a student begins a timed problem, the client is responsible for displaying
the amount of time remaining; i.e., the client should not constantly poll the
server to determine time remaining (though, the server does have a record of
this).  The client will synchronize its timer with the server each time the
problem is opened, refreshed, or saved.

Rationale
~~~~~~~~~
A student must know how much time they have remaining, in order to complete and
submit their work.


Manual Saving
-------------

Functionality
~~~~~~~~~~~~~
A student must be able to manually save the current state of their work.

Implementation
~~~~~~~~~~~~~~
Clicking a save button must save the current problem state in the browser to
the server.

Rationale
~~~~~~~~~
While automatic saving could be convenient, a student should always have the
final say on which version of their work is submitted.  A student will feel
more comfortable if they can save their work when they want to, knowing that if
they were to lose connectivity before the end of the exam, their last saved
version would be graded.


Automatic Submission
--------------------

Functionality
~~~~~~~~~~~~~
The client should automatically submit work when the timer expires.

Implementation
~~~~~~~~~~~~~~
If it exists, the most recent manual save should be used.

Note: As a client may lose connectivity after saving (manually or
automatically) but before time expires, the server must be able to properly
handle these case.  At grading or re-rendering time, if a student has saved
state in an expired problem, that saved state will be submitted and graded.

Rationale
~~~~~~~~~
If a student has saved work but has not submitted before the timer expires, we
don’t want them to lose all of their work.


Manual Submission
--------------------

Functionality
~~~~~~~~~~~~~~
A student must be able to submit their work before time expires.

Implementation
~~~~~~~~~~~~~~
Multiple submissions and submissions past the timer expiration must be
rejected.

Rationale
~~~~~~~~~
Students should complete and submit their work before the timer expires.


Desired Analytics
-----------------
- Distribution of time taken
- Number/percentage of automatic submissions (time expired)
- Number/percentage of non-submissions (exam started, never saved, no answers
  attempted at time expiration)


Deferred
========
- Warnings:  It'd be nice to warn students (flashing? beeping?) shortly before
  the time is up.  This could either be at a fixed interval (e.g. 1 minute
  before the end), or configurable per item by course staff.
- Per-student time limit adjustments (e.g. for a student with a disability
  requiring extra time).


Implementation Approach
=======================
Most, if not all of the functionality will be implemented by adding
functionality to the ``CapaModule`` class and associated templates.  For
possible reuse in other module class, the functionality may be isolated in a
mixin class.

[Yarko Tymciurak]

MOOC context, and what I personally experienced this week suggests this is a hard problem.

Construction on my floor cut the internet cable affecting several offices.  How do you handle that?

Worldwide, internet throughput and reliability is uneven, making this a bigger concern.

If you do this in a "on page" app way, time-stamping questions, and security comes up (if it's that important), how will you handle that?

If it's critical, an on-site monitored test (not online) may be the best option.

If it's not that critical, the second question which comes to mind is why even talk about cutting a test off? If learning is the goal, why not mark the time, remaining questions will not count towards certificate/grade, but "you may finish the test" (that seems motivating, and a reasonable & important option to offer the course creator).

[Ned Batchelder]

Steven, thanks, I'm very pleased to see people starting to discuss development work here before it gets started.

I have a few questions:

1) Does the course author have any control over the text on the interstitial warning?

2) Do you have a design for the interstitial and the countdown timer? These seem like the new UI components that will affect the student experience.

3) You said "most, if not all" of the implementation would be in CapaModule.  Are there parts that might be implemented elsewhere?

--Ned.

[Steven Burch]

Ned, thanks for the questions. I’ve gone ahead and replied inline.

— Steven

On Wednesday, April 30, 2014 2:43:58 PM UTC-7, Ned Batchelder wrote:

1) Does the course author have any control over the text on the interstitial warning?

 We hadn’t originally planned that, though it would make sense to add an option allowing an instructor to specify an accompanying message.

2) Do you have a design for the interstitial and the countdown timer? These seem like the new UI components that will affect the student experience.

This hasn’t yet been designed; we’re certainly open to input.
 

3) You said "most, if not all" of the implementation would be in CapaModule.  Are there parts that might be implemented elsewhere?

I believe the “elsewhere” is in reference to templates, etc. that are not explicitly “in” CapaModule. We intend to only extend CapaModule functionality.

[Piotr Mitros]

I had feedback on the proposal: 

On Wednesday, April 30, 2014 12:58:44 PM UTC-4, Steven Burch wrote:

Rationale
~~~~~~~~~
An instructor should be able to assign a desired time limit per problem
component.

It would be helpful to have a more complete rationale in this section. How does the time limit feed into the objectives of either what the instructor is trying to teach or what the instructor is trying to assess? Why is this the best approach to those objectives? The proposal should address why this is a good idea (beyond mirroring traditional residential).

In terms of implementation, I would consider whether this makes sense to add to CapaModule, or whether this makes sense as a stand-alone XBlock which would then contain a single child which would be the problem. 

Upsides: (1) Capa is very complex. As more things get added, it becomes less and less maintainable. (2) An external solution could be applied to other types of elements. For example, an interactive XBlock or a whole problem set could have a time limit. (3) A new XBlock may be easier to develop from a process point of view. It would undergo much less scrutiny, since the code would not have the potential to break unrelated courses or effect future development, so development may be faster. 

Downsides: (1) This would make it more difficult to do automatic submission. (2) XBlocks+Studio don't work very well yet with children, although this is an area of rapid development. 

Piotr

[Xavier Antoviaque]

If you go through the XBlock path, this would be welcomed as a contribution on the mentoring block, which already provides an interface of a parent node (mentoring) adding functionality around child problems (MCQ, MRQ, Free Text Answer):

 

https://github.com/edx-solutions/xblock-mentoring#mentoring-xblock

 

The timer could be added to the <mentoring> block level, by adding the option on the <mentoring> https://github.com/edx-solutions/xblock-mentoring/blob/master/mentoring/mentoring.py

 

and then adding a confirmation dialog, getting the problem content through an XBlock JSON handler like https://github.com/edx-solutions/xblock-mentoring/blob/master/mentoring/mentoring.py#L126 to track when the problem is displayed https://github.com/edx-solutions/xblock-mentoring/blob/86c53315d74e7ff8843f4a61deca9b4513b9390f/mentoring/public/js/mentoring.js#L99

 

--

Xavier.

 

[Jason Bau]

(Reposting via Web UI, as email reply to edx-...@googlegroups.com didn't work.  Is that intentional?)

Piotr and Xavier and Ned,

Implementing timing as a xblock wrapping a set of "children" seems like a promising alternative that I'm willing to consider.  This has the nice property of making arbitrary component types timeable, and if the wrapping could be extended to "descendants" rather than just children, we might have a nice mixin-like way to extend different blocks along functionality dimensions.

However, are there currently examples / is there support for this sort of wrapping?  I see mentoring is using "lightweight" blocks as its children instead of full xblocks (though I'm not yet informed as to the difference).  Are A/B and Conditional XModules a good example for handling children xblock/xmodules?  Realistically, the thing we want to wrap most is CapaModule, so what are the barriers (if any) to getting wrapping of CapaModule/Arbitrary XModules/Arbitrary XBlocks to work?

Thanks,

Jason

[Xavier Antoviaque]

Hi Jason,

 

Yes, the idea of wrapping blocks this way is very tempting - currently it only works with children which have been designed to work with the mentoring block though. You won't easily be able to wrap the Capa XModule.

 

One reason for this is the lack of proper support for XBlock chidren in the LMS at the moment, which makes it necessary to use a workaround to emulate  the behvior (cf https://github.com/edx-solutions/xblock-mentoring/blob/master/mentoring/light_children.py#L59  for details) -- this should disappear once the LMS supports XBlock children, and then it would be possible to embed arbitrary XBlock.

 

There will still be a need to interface with the containing mentoring block though - it delegates processing to the children, but expects the children to offer certain methods if they want to use the mentoring block features, like handling of form submission (cf https://github.com/edx-solutions/xblock-mentoring/blob/master/mentoring/answer.py#L100 https://github.com/edx-solutions/xblock-mentoring/blob/master/mentoring/public/js/answer.js )

 

Btw, if you want to install the mentoring XBlock against the latest master, note that there are two changes pending merging which you will likely need:

 

- XBlock fix to enforce the type of attributes: https://github.com/edx/XBlock/pull/200

- Studio template fix (the expected DOM format changed recently on master): https://github.com/edx-solutions/xblock-mentoring/pull/27

 

Life on the bleeding edge : )

 

Please feel free to ask more questions if you have some.

[Calen Pennington]

I would advise against building on top of the LightChildren framework used by the Mentoring block for a Timing wrapper block (as LightChildren were, as far as I understood it, just a temporary measure to work around the lack of children support in edx-platform). I would in fact advise building the TimingBlock as an independent entity, separate from either Mentoring or Capa. I suggest, for the moment, building it within the xblock-sdk, and using the children facilities that work there, as that's the standard we're moving edx-platform towards. You won't be able to test in edx-platform (yet), but you have a better chance of not having to do additional work in the future.

-Cale

[Xavier Antoviaque]

Right, if you can wait for proper support, that's probably an easier way - I was answering with the assumption that you wanted that feature now.

 

Also, building it separately can be good, but it also means that if you want to have several similar features for your children XBlock, you'll have to have several layers of containing XBlocks embedding your real content, which could make it more cumbersome to manage in Studio, depending on how XBlock children studio editing will work.

 

--

Xavier.

 

 

 

On Fri, May 2, 2014, at 15:44, Calen Pennington wrote:

I would advise against building on top of the LightChildren framework used by the Mentoring block for a Timing wrapper block (as LightChildren were, as far as I understood it, just a temporary measure to work around the lack of children support in edx-platform). I would in fact advise building the TimingBlock as an independent entity, separate from either Mentoring or Capa. I suggest, for the moment, building it within the xblock-sdk, and using the children facilities that work there, as that's the standard we're moving edx-platform towards. You won't be able to test in edx-platform (yet), but you have a better chance of not having to do additional work in the future.

 

-Cale

On Fri, May 2, 2014 at 8:52 AM, Xavier Antoviaque <xav...@antoviaque.org> wrote:

Hi Jason,

 

Yes, the idea of wrapping blocks this way is very tempting - currently it only works with children which have been designed to work with the mentoring block though. You won't easily be able to wrap the Capa XModule.

 

One reason for this is the lack of proper support for XBlock chidren in the LMS at the moment, which makes it necessary to use a workaround to emulate  the behvior (cf https://github.com/edx-solutions/xblock-mentoring/blob/master/mentoring/light_children.py#L59  for details) -- this should disappear once the LMS supports XBlock children, and then it would be possible to embed arbitrary XBlock.

 

There will still be a need to interface with the containing mentoring block though - it delegates processing to the children, but expects the children to offer certain methods if they want to use the mentoring block features, like handling of form submission (cf https://github.com/edx-solutions/xblock-mentoring/blob/master/mentoring/answer.py#L100https://github.com/edx-solutions/xblock-mentoring/blob/master/mentoring/public/js/answer.js )

[Piotr Mitros]

I'll mention that the xblock-sdk has a couple of examplars of
children. Specifically:

https://github.com/edx/xblock-sdk/blob/master/sample_xblocks/basic/structure.py

Cale or Ned might be better able to advise on timing of when this will
work in edx-platform.

Piotr

[Steven Burch]

Thanks to everyone for the feedback!

We have taken some time to weigh our original implementation ideas with
those presented here and we have come to the following conclusions:

1. Our original idea was to implement this feature as an extension to
the Capa module.
Pro: could work with current infrastructure
Con: Capa complexity

2. We considered Xavier's suggestion of implementing it via the
mentoring block.
Pro: some existing infrastructure
Con: incomplete support

3. We also considered Piotr's idea of implementing it as a XBlock
wrapper.
Pro: Solution could be applied to other elements.
Con: incomplete support

Based on this, #3 looks to be the ideal approach, though it currently
lacks full platform support (namely, nested XBlocks).

That said, while we have received requests for this functionality in
the past, we don't have a pressing requirement for it, at the moment.
Given that, we'll be shelving the idea, for now.  We will reevaluate
down the line, when we have increased interest from instructors and/or
the platform matures to more easily implement an ideal solution.

Thanks again for all of the input!
-- Steven

[shubham rawat]

Any chances of this feature being added in the near future?

[Chris Dodge]

Hello,

Our team has embarked on a new feature to do proctored timed exams (we're hoping to support timed exams without proctoring, should be possible but not our primary use case).

It's on a short development cycle as it has a hard delivery date (late July/early August) for a customer's course launch, so we don't have a lot written up on this right now. I'll see if I can write up a public Wiki page. I'll try to remember to swing back to this posting and give a link to the page.

Best wishes,

- Chris

[Amish Gandhi]

Hi All,

We just open sourced a proctoring solution for Open edX

https://github.com/perpetualny/proctoru-xblock 

Thanks,

-Amish