Issues with WeBWorK essay grader

[Sean Fitzpatrick]

For Alex, I guess, since he wrote the problems ;-)

APEX has a lot of problems where there is a pop-up, or answer blank, followed by an "explain why" essay answer portion.

If there is just one part, this seems to be OK.

But there are several problems where students are asked about continuity at multiple points. For example, it might say:

(a) Is f(x) continuous at 0? (Pop-up: yes/no)

If not, explain why not. (Essay answer)

(b) Is f(x) continuous at 1? (Pop-up: yes/no)

If not, explain why not. (Essay answer)

The second pop-up is getting replaced by the first answer blank, and getting marked wrong as a result.

Here is an example of such a problem in PTX source (note that WW seems to not like the PiecewiseFunction context; I had to comment it out when testing the PG code):

        <exercise label="ex-limit-continuity-point-1">

          <webwork xml:id="webwork-ex-limit-continuity-point-1">

            <pg-code>

              Context("PiecewiseFunction");

              @choices=('?',

                'Yes.',

                'No.'

              );

              @answer=(PopUp(~~@choices,1),PopUp(~~@choices,1));

              $showwork = '[@ explanation_box(message => "If not, explain why not.") @]*';

            </pg-code>

            <introduction>

              <p>

                <m>f(x)=\begin{cases}1\amp x=0\\\frac{\sin(x)}{x}\amp x\neq0\end{cases}</m>

              </p>

            </introduction>

            <task label="ex-limit-continuity-point-1a">

              <statement>

                <p>

                  Is <m>f</m> is continuous at <m>0</m>?

                </p>

                <p>

                  <var name="$answer[0]" form="popup"/>

                </p>

                <p>

                  <var name="$showwork" data="pgml"/>

                </p>

              </statement>

            </task>

            <task label="ex-limit-continuity-point-1b">

              <statement>

                <p>

                  Is <m>f</m> is continuous at <m>\pi</m>?

                </p>

                <p>

                  <var name="$answer[1]" form="popup"/>

                </p>

                <p>

                  <var name="$showwork" data="pgml"/>

                </p>

              </statement>

            </task>

          </webwork>

        </exercise>

It produces the following PG code:

#############################################
###       Generated from PreTeXt source      
###       on 2024-05-14T12:04:02-06:00      
###   A recent stable commit (2022-07-01):  
### 6c761d3dba23af92cba35001c852aac04ae99a5f
###                                          
###         https://pretextbook.org          
###                                          
#############################################
## DBsubject()
## DBchapter()
## DBsection()
## Level()
## KEYWORDS()
## TitleText1()
## EditionText1()
## AuthorText1()
## Section1(not reported)
## Problem1(1.5.19)
## Author()
## Institution()
## Language(en-US)

DOCUMENT();

############################################################
# Load Macros
############################################################
loadMacros(
  "PGstandard.pl",
  "PGML.pl",
  "scaffold.pl",
  "AnswerFormatHelp.pl",
  "PGessaymacros.pl",
  "parserPopUp.pl",
  "PGcourse.pl",
);
COMMENT('Authored in PreTeXt');
COMMENT('This problem is scaffolded with multiple parts');

############################################################
# Header
############################################################
TEXT(beginproblem());

############################################################
# PG Setup Code
############################################################
#Context("PiecewiseFunction");
@choices=('?',
'Yes.',
'No.'
);
@answer=(PopUp(~~@choices,1),PopUp(~~@choices,1));
$showwork = '[@ explanation_box(message => "If not, explain why not.") @]*';

############################################################
# Body
############################################################

BEGIN_PGML
Determine if [`f`] is continuous at the indicated values.

[`\newcommand{\amp}{&}f(x)=\begin{cases}1\amp x=0\\\frac{\sin(x)}{x}\amp x\neq0\end{cases}`]

END_PGML

############################################################
# Scaffold
############################################################
Scaffold::Begin(numbered => 1,is_open => "correct_or_first_incorrect",can_open => always,);

############################################################
# Section
############################################################
Section::Begin("");

BEGIN_PGML
Is [`f`] is continuous at [`0\text{?}`]

[_]{$answer[0]}{5}

[$showwork]**

END_PGML

Section::End();

############################################################
# Section
############################################################
Section::Begin("");

BEGIN_PGML
Is [`f`] is continuous at [`\pi\text{?}`]

[_]{$answer[1]}{5}

[$showwork]**

END_PGML

Section::End();

Scaffold::End();

############################################################
# End Problem
############################################################

ENDDOCUMENT();

In the PG editor on my WW server, I can answer "Yes" for part (a). If I answer "Yes" for part (b), I get the "this answer will be graded at a later time" message, and then an error that no answer blank is provided for AnSwEr003, which is the second pop-up.

When I first load the problem, I get the following warning messages:

Warning messages

• Non fatal warnings. These are only displayed for users with permission to view problem debugging info.
• Use of uninitialized value in concatenation (.) or string at line 184 of [PG]/macros/core/PGessaymacros.pl
• from within main::explanation_box called at line 2 of (eval 4497)
• from within PGML::Eval called at line 902 of [PG]/macros/core/PGML.pl
• from within PGML::Parse::replaceCommand called at line 1339 of [PG]/macros/core/PGML.pl
• from within PGML::Format::Command called at line 1199 of [PG]/macros/core/PGML.pl
• from within PGML::Format::string called at line 1392 of [PG]/macros/core/PGML.pl
• from within PGML::Format::html::Indent called at line 1192 of [PG]/macros/core/PGML.pl
• from within PGML::Format::string called at line 1179 of [PG]/macros/core/PGML.pl
• from within PGML::Format::format called at line 1837 of [PG]/macros/core/PGML.pl
• from within PGML::Format called at line 1847 of [PG]/macros/core/PGML.pl
• from within PGML::Format2 called at line 79 of undefined
• Use of uninitialized value in concatenation (.) or string at line 184 of [PG]/macros/core/PGessaymacros.pl
• from within main::explanation_box called at line 2 of (eval 4507)
• from within PGML::Eval called at line 902 of [PG]/macros/core/PGML.pl
• from within PGML::Parse::replaceCommand called at line 1339 of [PG]/macros/core/PGML.pl
• from within PGML::Format::Command called at line 1199 of [PG]/macros/core/PGML.pl
• from within PGML::Format::string called at line 1392 of [PG]/macros/core/PGML.pl
• from within PGML::Format::html::Indent called at line 1192 of [PG]/macros/core/PGML.pl
• from within PGML::Format::string called at line 1179 of [PG]/macros/core/PGML.pl
• from within PGML::Format::format called at line 1837 of [PG]/macros/core/PGML.pl
• from within PGML::Format called at line 1847 of [PG]/macros/core/PGML.pl
• from within PGML::Format2 called at line 95 of undefined

[Sean Fitzpatrick]

Related question: can a PreTeXt-WeBworK question with an  "essay answer" part be answered if the book is on Runestone?

Or will the essay answer only ever work if the problem is exported to a WeBWorK server?

If the latter, it seems like I would be better off replacing all WeBWorK problems of this type with something using <choices> and <response/>.

[Alex Jordan]

A PG problem can have an "explanation essay". This is a place to explain something, typically putting into words why some multiple choice answer was selected.

Instructors may or may not want to deal with manually grading those explanation essay responses. Also, maintaining two versions of each such question (one with an essay box and one without) would bring up issues. So the specially flagged "explanation essays" (as opposed to regular essay answer boxes) are controlled with a coursewide Config setting you can toggle in a Course Config page. Either never show these explanation essay answer boxes, or always show them.

For a PTX host course, you probably want this setting to never show them. Then they should not ever appear in any part of problem processing, and the answer input labeling should straighten out.

What is the host course here? And can you tell if explanation essay answers are turned off there?

--
You received this message because you are subscribed to the Google Groups "PreTeXt development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pretext-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pretext-dev/a6c22b36-d6e0-4239-9223-f3677b0b71acn%40googlegroups.com.

[Alex Jordan]

> Related question: can a PreTeXt-WeBworK question with an  "essay answer" part be answered if the book is on Runestone?

Most likely no one has looked at what happens in Runestone when there is an essay answer field. Also most likely, that situation does not behave like you'd want it to.

> If the latter, it seems like I would be better off replacing all WeBWorK problems of this type with something using <choices> and <response/>.

Instead of replacing APEX problems, and instead of leaving scores of OPL problems unusable in Runestone, I think ideally you would want to make it so Runestone does the right thing with a PG essay answer problem. What the right thing is is not clear to me. But I imagine it would need work on the PG side as well as on the Runestone side.

[Sean Fitzpatrick]

I agree that it would be great if a webwork essay answer would work in Runestone.

But I don't think I have the toolset necessary to make that happen.

The host course had the "Skip essay answer fields" option set to false. Changing that to "true" fixes things.

But let's say this problem was on a WeBWorK server, with essay answer fields turned on. Testing suggests that if a student is doing this question, and they correctly answer "Yes" for both of the points where they are asked about continuity, the second one is marked wrong, because the first essay answer field is taking the place of the second pop-up answer field. Maybe that's just because I am viewing it in the problem editor, and it needs to be assigned.

[Alex Jordan]

If the problem structure is:

Answer blank

Essay field

Answer blank

Essay field

Then the answer inputs should be labeled

AnSwEr0001  Answer blank

AnSwEr0002  Essay field

AnSwEr0003  Answer blank

AnSwEr0004  Essay field

What you are describing is like if there is some mixup with the labeling, like:

AnSwEr0001  Answer blank

AnSwEr0003  Essay field

AnSwEr0002  Answer blank

AnSwEr0004  Essay field

But if this is happening, now it's a PG issue, not a PTX/Runestone issue. If you can post a minimal example to the WW forum, we can look into it there.

To view this discussion on the web visit https://groups.google.com/d/msgid/pretext-dev/ab02a19d-5668-4830-8c0c-701557439e28n%40googlegroups.com.

[Sean Fitzpatrick]

I'm pretty sure that's the problem structure you can see in the example I posted above, and yeah, I think the parts are being mislabeled.

To view this discussion on the web visit https://groups.google.com/d/msgid/pretext-dev/CA%2BR-jrfd2sjh4-v-EUZAAjVO7HEd6jGMyZGOfF_xaArZ5CQ7qQ%40mail.gmail.com.

[Sean Fitzpatrick]

@Alex: the problem is with the PGcourse.pl file that we add as part of the PreTeXt configuration.

So I am posting here before moving things over to the WW forum.

Below is my minimal example. It just has two "choose yes" pop-ups, with an essay answer field after each one.

As-is, with the skip essay answer set to false, I get the message from PGcourse.pl:

"If you were logged into a WeBWorK course and this problem were assigned to you, you would be able to submit an essay answer that would be graded later by a human being."

I'm supposed to get that. But I also get several warnings when the question is loaded (these are the same as the warnings I posted above), and the following results:

The warnings after submitting answers are about the lack of answer blanks for AnSwEr003 and 004.

If I delete PGcourse.pl from the macros, everything works fine, with or without the essay answer fields.

#############################################

DOCUMENT();

############################################################
# Load Macros
############################################################
loadMacros(
  "PGstandard.pl",
  "PGML.pl",

  "AnswerFormatHelp.pl",
  "PGessaymacros.pl",
  "parserPopUp.pl",

  "contextForm.pl",
  "PGcourse.pl",

);
############################################################
# Header
############################################################
TEXT(beginproblem());

############################################################
# PG Setup Code
############################################################

$answer1=PopUp(['?','Yes','No'],1);
$answer2=PopUp(['?','Yes','No'],1);
$showwork = '[@ explanation_box(message => "If not, state why it is not.") @]*';

############################################################
# Body
############################################################

BEGIN_PGML

Choose yes: [_]{$answer1}

[$showwork]**

Also choose yes: [_]{$answer2}

[$showwork]**

END_PGML

############################################################
# End Problem
############################################################

ENDDOCUMENT();

[Alex Jordan]

I am exploring this, not ghosting the issue :) Things are a little different when I am looking at this in 2.19, but I see evidence of the same underlying issue.

To view this discussion on the web visit https://groups.google.com/d/msgid/pretext-dev/a25cf132-0273-461b-8d0e-86f87cfa5deen%40googlegroups.com.

[Sean Fitzpatrick]

Thanks. For now, I have kicked the can down the road by turning off essay answer blanks on the host course.

(Which I should have done before, but I didn't realize it was an option.)

It should only be an issue if someone wants to export the problems to webwork.

[Alex Jordan]

The issue turns out to be a PTX issue (not a WeBWorK/PG issue).

This PR changes what is suggested for the PGcourse.pl file so that it now covers this:

https://github.com/PreTeXtBook/pretext/pull/2163

You could change your PGcourse.pl file accordingly and not have to worry about this now.

To view this discussion on the web visit https://groups.google.com/d/msgid/pretext-dev/b49d8f6f-a81d-4de8-9403-5ca786ee7111n%40googlegroups.com.

[Sean Fitzpatrick]

Great, thanks! 

I guess that leaves two WeBWorK things on my wish list, but I think both are hard. Not sure if either is something we could start next month: 

1. Make these essay boxes talk to runestone

2. Get WeBWorK to process long image descriptions

To view this discussion on the web visit https://groups.google.com/d/msgid/pretext-dev/CA%2BR-jrchugTLgLhrko237j8ZZ9jWY30x%3DEpq08ejmQMkwEC72Q%40mail.gmail.com.

[Alex Jordan]

I think I can get #2 going for WeBWorK/PG 2.19. Stay tuned.

To view this discussion on the web visit https://groups.google.com/d/msgid/pretext-dev/CAH%2BNcPajkxDHZGHC%3Dygu%3D5GLor8r5OVzsAO3wBdYs90qM3zvfg%40mail.gmail.com.

[Brad Miller]

Sean,

There exists:  pretext/examples/webwork/sample-chapter

Rob and I created this so that we could build a sample book with WW problems in it. It would be very helpful to have one of these explanation essays in that book, then I can actually look and try to verify what is going on in Runestone and on the assignment page.

Brad

Bradley Miller, PhD
Professor Emeritus Computer Science
Luther College
Founder, Runestone Academy LTD
Blog: A Reputable Journal
Schedule a Meeting

To view this discussion on the web visit https://groups.google.com/d/msgid/pretext-dev/CA%2BR-jrfOQjYKvZHATxzR784oHiVBN3uubGr177Z7HBWjAJU3Gw%40mail.gmail.com.

[Sean Fitzpatrick]

Brad: I just checked, and there is one in there already.

In the HTML: https://pretextbook.org/examples/webwork/sample-chapter/html/sample-ww-chapter-4-5.html#ww-use-the-definition-of-the-derivative

In source, it begins at line 763 of sample-chapter.ptx

[Brad Miller]

Thanks for pointing that out Sean.

Here is what I see.

It seems like I should see a message about showing my work.  Not sure what work I would show for this simple example.

What should I see??

Bradley Miller, PhD
Professor Emeritus Computer Science
Luther College
Founder, Runestone Academy LTD
Blog: A Reputable Journal
Schedule a Meeting

To view this discussion on the web visit https://groups.google.com/d/msgid/pretext-dev/a9e8be0c-9a1c-46a2-9892-a66f01d8e889n%40googlegroups.com.

[Brad Miller]

Doh!!  I guess I would see that paragraph replaced with a textinput area for me to type in my reasoning.

Bradley Miller, PhD
Professor Emeritus Computer Science
Luther College
Founder, Runestone Academy LTD
Blog: A Reputable Journal
Schedule a Meeting

On May 16, 2024 at 12:33 PM -0500, Brad Miller <br...@runestone.academy>, wrote:

Thanks for pointing that out Sean.

Here is what I see.

<Screenshot 2024-05-16 at 12.31.26 PM.png>

[Sean Fitzpatrick]

Yes exactly! 

There's a macro PGcourse.pl on the WeBWorK server that replaces the text input box with that message, since in a regular PreTeXt book, answers are not stored. 

If you wanted to get the text box, you would remove that macro, or replace/edit it. 

But instead of student input getting stored on a WeBWorK server, you would want to somehow capture that and pass it to Runestone as if it came from a <response/>. Or something like that?

To view this discussion on the web visit https://groups.google.com/d/msgid/pretext-dev/34201269-a699-4df2-ac2e-92cc36fb8d9c%40Spark.

[Brad Miller]

Once Alex's PR is merged Runestone should have access to whatever essay type response the student enters.  

The problem then becomes recognizing that this particular webwork question is partly autograded and partly requires manual grading. So far the understanding has been that all webwork problems are autograded.

As a starting point it could just be that the instructor has to review these questions and override the partial credit that will already have been assigned by WW.

Bradley Miller, PhD
Professor Emeritus Computer Science
Luther College
Founder, Runestone Academy LTD
Blog: A Reputable Journal
Schedule a Meeting

To view this discussion on the web visit https://groups.google.com/d/msgid/pretext-dev/CAH%2BNcPZOxJQG0Na-Rpwdz6Nn5zQ2UGEkwJJfspBC0MN6XasZfQ%40mail.gmail.com.

[Alex Jordan]

That PR is only changing documentation for setting up a host course. I just visited the "anonymous" host course at

webwork.runestone.academy

webwork-ptx.aimath.org

webwork-dev.aimath.org

and in each one, I updated the PGcourse.pl file as is recommended in the PR.

But there is a misunderstanding. The current status of things still leaves all users who would be viewing an essay seeing the "If you were logged in to a WeBWorK course..." message.

To experiment with how to handle essay answers in WeBWorK in Runestone, I suggest that on this server:

webwork.runestone.academy

I make a new host course, maybe calling it "develop". I could leave out the configuration block that is covering over essay input fields from that course. Then Brad could build the sample chapter, but would want to change the publication file to use this other course. Does this sound like what you would want to do?

To view this discussion on the web visit https://groups.google.com/d/msgid/pretext-dev/507cfcb0-a133-4b04-8cb9-ceb8390961ce%40Spark.

[Sean Fitzpatrick]

I could also try this experiment next week, since I have a WeBWorK server and a Runestone server (which I need to update, now that the semester is over...)

But I would only get as far as seeing what a WeBWorK essay answer looks like for the instructor.

Changing how things work on the Runestone side is above my pay grade ;-)

[Sean Fitzpatrick]

I have a test course set up on my Runestone server, with a small subset of APEX that has some essay answer questions.

I changed the PGcourse.pl file this course uses so that the essay answer boxes don't get covered.

As a student, the essay box shows up just fine.

As an instructor, I can't figure out any way to access what the student entered.

[Sean Fitzpatrick]

OK, I can see things on the Student Progress page. But this won't work well for more substantial answers.

I also note that there is a broken button (this is on the WeBWorK side): that first grey button beside Preview opens the equation editor.

[Brad Miller]

I can only say that I didn't anticipate that any webwork questions would require manual grading in Runestone.

My glib answer is why not just use the shortanswer question type that PreTeXt already provides and are fully supported for grading?

I'd be happy to help someone get started who could take over fully supporting webwork in Runestone.

Brad

Bradley Miller, PhD
Professor Emeritus Computer Science
Luther College
Founder, Runestone Academy LTD
Blog: A Reputable Journal
Schedule a Meeting

On Jun 12, 2024 at 12:36 PM -0500, Sean Fitzpatrick <dsfitz...@gmail.com>, wrote:

OK, I can see things on the Student Progress page. But this won't work well for more substantial answers.

I also note that there is a broken button (this is on the WeBWorK side): that first grey button beside Preview opens the equation editor.

<Screenshot 2024-06-12 113306.png>

On Wednesday, June 12, 2024 at 11:17:25 AM UTC-6 Sean Fitzpatrick wrote:

I have a test course set up on my Runestone server, with a small subset of APEX that has some essay answer questions.

I changed the PGcourse.pl file this course uses so that the essay answer boxes don't get covered.

As a student, the essay box shows up just fine.

As an instructor, I can't figure out any way to access what the student entered.

On Saturday, May 18, 2024 at 8:14:40 AM UTC-6 Sean Fitzpatrick wrote:

I could also try this experiment next week, since I have a WeBWorK server and a Runestone server (which I need to update, now that the semester is over...)

But I would only get as far as seeing what a WeBWorK essay answer looks like for the instructor.

Changing how things work on the Runestone side is above my pay grade ;-)

On Fri, May 17, 2024, 9:53 a.m. Alex Jordan wrote:

That PR is only changing documentation for setting up a host course. I just visited the "anonymous" host course at

webwork.runestone.academy

webwork-ptx.aimath.org

webwork-dev.aimath.org

and in each one, I updated the PGcourse.pl file as is recommended in the PR.

But there is a misunderstanding. The current status of things still leaves all users who would be viewing an essay seeing the "If you were logged in to a WeBWorK course..." message.

To experiment with how to handle essay answers in WeBWorK in Runestone, I suggest that on this server:

webwork.runestone.academy

I make a new host course, maybe calling it "develop". I could leave out the configuration block that is covering over essay input fields from that course. Then Brad could build the sample chapter, but would want to change the publication file to use this other course. Does this sound like what you would want to do?

--
You received this message because you are subscribed to the Google Groups "PreTeXt development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pretext-dev...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/pretext-dev/62010ef8-a2b1-41af-9e69-34d0e0a01f7bn%40googlegroups.com.

[Sean Fitzpatrick]

My long term solution for APEX is to go through the book, and convert and problems that are T/F, multiple choice, or essay answer to use the built in question types for this purpose. 

(These did not exist when Alex wrote the WeBWorK code for them.)

This is fine when there is no other part to the question.

But there are definitely some problems where we use WeBWorK to randomize (e.g. so each student gets a different function), and there is structure like: 

(a) Evaluate the limit of f(x) at the indicated point

[auto-graded by WeBWorK] 

(b) Explain how you found the answer to part (a)

[WeBWorK essay answer] 

We can't use WeBWorK for only one of the two parts (it is all or none), so that's where I would anticipate an instructor wanting to access a WeBWorK essay answer on Runestone.

To view this discussion on the web visit https://groups.google.com/d/msgid/pretext-dev/096cebcc-6538-457f-88e0-5a07e0b5da5f%40Spark.