Answers in a scaffolded WeBWorK problem are getting mixed up

[Sean Fitzpatrick]

Sorry if this problem is better suited for the WeBWorK forum, but I can only produce this issue once the question is in PreTeXt.

One of the users of APEX Calculus reported that this question is not being graded correctly:

https://opentext.uleth.ca/apex-calculus/sec_limits_one_sided.html#ex-limit-onesided-use-graph-4

I can confirm that sometimes it works, but clicking the randomize button produces versions where the answers to parts (a) and (b) are swapped, or parts (b) and (d), etc.

In other words, between parts (a), (b), and (d), the answers that are marked correct come from some permutation of the correct answers.

Below is both the PreTeXt source, and the WeBWorK code that PreTeXt generates for this problem. I've put the WeBWorK code on the server being used for this problem, and I've tried about 30 different random seeds, and they all worked correctly.

In summary: I can't reproduce the bug by running the generated .pg file on WeBWorK. But when I activate the problem in PreTeXt, the answers are mixed up about 80% of the time.

This makes me think it's a PreTeXt problem and not a WeBWorK problem.

PreTeXt souce:

<exercise label="ex-limit-onesided-use-graph-4">
          <webwork xml:id="webwork-ex-limit-onesided-use-graph-4">
            <pg-code>
              Context()-&gt;strings-&gt;add('does not exist'=&gt;{alias=&gt;'DNE'});
              @x=num_sort((0,random_subset(2,1..5)));
              @y=random_subset(3,1..5);
              ($w,$z)=random_subset(2,1..$y[1]-1,$y[1]+1..5);
              if($envir{problemSeed}==1){@x=(0,1,2);@y=(1,0,2);$z=2;$w=1;};
              $a=$x[1];
              $f=Formula("($z-$y[0])/($x[1]-$x[0])*(x-$x[0])+$y[0]");
              $g=Formula("($y[2]-$y[1])/($x[2]-$x[1])^2*(x-$x[1])^2+$y[1]");
              $xmin=min(0,@x)-1;$xmax=max(0,@x)+1;$ymin=min(0,@y,$z,$w)-1;$ymax=max(0,@y,$z,$w)+1;
              @L=($z,$y[1],Compute("DNE"),$w);
            </pg-code>
            <introduction>
              <image>
                <description>
                  <p>
                  Graph that shows the domain <m><var name="$x[0]]"/></m> to <m><var name="$x[2]"/></m>. There
                  are two lines on the graph. The first is defined by the solid
                  point <m>(<var name="$x[0]"/>, <var name="$y[0]"/>)</m> and hollow
                  point <m>(<var name="$x[1]"/>, <var name="$y[1]"/>)</m>. The second
                  is defined by the hollow point <m>(<var name="$x[1]"/>, <var name="$z"/>)</m>
                  and solid point <m>(<var name="$x[2]"/>, <var name="$y[2]"/>)</m>. There
                  is also a solid point at <m>(<var name="$x[1]"/>, <var name="$w"/>)</m>.
                  The first line is straight while the seconds line is curved.
                 </p>
                </description>
                <shortdescription>
                  Graph for exercise problem 8.
                </shortdescription>
                <latex-image>
                  \begin{tikzpicture}
                    \begin{axis}[
                      xmin = $xmin,
                      xmax = $xmax,
                      ymin = $ymin,
                      ymax = $ymax,
                    ]
                      \addplot[firstcurvestyle, domain=$x[0]:$x[1]] {$f};
                      \addplot[firstcurvestyle, domain=$x[1]:$x[2]] {$g};
                      \addplot[soliddot] coordinates {($x[0],$y[0]) ($x[1],$w) ($x[2],$y[2])};
                      \addplot[hollowdot] coordinates {($x[1],$y[1]) ($x[1],$z)};
                    \end{axis}
                  \end{tikzpicture}
                </latex-image>
              </image>
              <instruction>
                If you need to enter <m>\infty</m>, you may type <c>infinity</c>, or just <c>INF</c>.
                If the limit does not exist, you may type <c>does not exist</c>, or just <c>DNE</c>.
              </instruction>
            </introduction>

            <task label="ex-limit-onesided-use-graph-4a">
              <statement>
                <p>
                  <m>\lim\limits_{x\to <var name="$a"/>^-} f(x)</m>
                </p>
                <p>
                  <var name="$L[0]" width="10"/>
                </p>
              </statement>
            </task>

            <task label="ex-limit-onesided-use-graph-4b">
              <statement>
                <p>
                  <m>\lim\limits_{x\to <var name="$a"/>^+} f(x)</m>
                </p>
                <p>
                  <var name="$L[1]" width="10"/>
                </p>
              </statement>
            </task>

            <task label="ex-limit-onesided-use-graph-4c">
              <statement>
                <p>
                  <m>\lim\limits_{x\to <var name="$a"/>} f(x)</m>
                </p>
                <p>
                  <var name="$L[2]" width="10"/>
                </p>
              </statement>
            </task>

            <task label="ex-limit-onesided-use-graph-4d">
              <statement>
                <p>
                  <m>f(<var name="$a"/>)</m>
                </p>
                <p>
                  <var name="$L[3]" width="10"/>
                </p>
              </statement>
            </task>
          </webwork>
        </exercise>

WeBWorK source:

#############################################
###       Generated from PreTeXt source      
###       on 2025-12-12T19:03:05Z       
###   A recent stable commit (2022-07-01):   
### 6c761d3dba23af92cba35001c852aac04ae99a5f 
###                                          
###         https://pretextbook.org          
###                                          
#############################################
## DBsubject()
## DBchapter()
## DBsection()
## Level()
## KEYWORDS()
## TitleText1()
## EditionText1()
## AuthorText1()
## Section1(not reported)
## Problem1(1.4.8)
## Author()
## Institution()
## Language(en-US)

DOCUMENT();

############################################################
# Load Macros
############################################################
loadMacros(
  "PGstandard.pl",
  "PGML.pl",
  "scaffold.pl",
  "AnswerFormatHelp.pl",
  "PGlateximage.pl",
  "PCCmacros.pl",
  "contextForm.pl",
  "APEX.pl",
  "PGcourse.pl",
);
COMMENT('Authored in PreTeXt');
COMMENT('This problem is scaffolded with multiple parts');

############################################################
# Header
############################################################
TEXT(beginproblem());

############################################################
# PG Setup Code
############################################################
Context()->strings->add('does not exist'=>{alias=>'DNE'});
@x=num_sort((0,random_subset(2,1..5)));
@y=random_subset(3,1..5);
($w,$z)=random_subset(2,1..$y[1]-1,$y[1]+1..5);
if($envir{problemSeed}==1){@x=(0,1,2);@y=(1,0,2);$z=2;$w=1;};
$a=$x[1];
$f=Formula("($z-$y[0])/($x[1]-$x[0])*(x-$x[0])+$y[0]");
$g=Formula("($y[2]-$y[1])/($x[2]-$x[1])^2*(x-$x[1])^2+$y[1]");
$xmin=min(0,@x)-1;$xmax=max(0,@x)+1;$ymin=min(0,@y,$z,$w)-1;$ymax=max(0,@y,$z,$w)+1;
@L=($z,$y[1],Compute("DNE"),$w);

$image_1 = createLaTeXImage();
$image_1->addToPreamble(latexImagePreamble());
$image_1->BEGIN_LATEX_IMAGE
\begin{tikzpicture}
  \begin{axis}[
    xmin = $xmin,
    xmax = $xmax,
    ymin = $ymin,
    ymax = $ymax,
  ]
    \addplot[firstcurvestyle, domain=$x[0]:$x[1]] {$f};
    \addplot[firstcurvestyle, domain=$x[1]:$x[2]] {$g};
    \addplot[soliddot] coordinates {($x[0],$y[0]) ($x[1],$w) ($x[2],$y[2])};
    \addplot[hollowdot] coordinates {($x[1],$y[1]) ($x[1],$z)};
  \end{axis}
\end{tikzpicture}

END_LATEX_IMAGE

############################################################
# Body
############################################################

BEGIN_PGML
Evaluate each expression using the given graph of [`f\text{.}`]

>> [@image(insertGraph($image_1), width=>600, alt=>qq"Graph for exercise problem 8.")@]*  <<

[@KeyboardInstructions('If you need to enter [`\infty\text{,}`] you may type [|infinity|]*, or just [|INF|]*. If the limit does not exist, you may type [|does not exist|]*, or just [|DNE|]*.')@]**

END_PGML

############################################################
# Scaffold
############################################################
Scaffold::Begin(numbered => 1,is_open => "correct_or_first_incorrect",can_open => always,);

############################################################
# Section
############################################################
Section::Begin("");

BEGIN_PGML
[`\lim\limits_{x\to [$a]^-} f(x)`]

[_]{$L[0]}{10}

END_PGML

Section::End();

############################################################
# Section
############################################################
Section::Begin("");

BEGIN_PGML
[`\lim\limits_{x\to [$a]^+} f(x)`]

[_]{$L[1]}{10}

END_PGML

Section::End();

############################################################
# Section
############################################################
Section::Begin("");

BEGIN_PGML
[`\lim\limits_{x\to [$a]} f(x)`]

[_]{$L[2]}{10}

END_PGML

Section::End();

############################################################
# Section
############################################################
Section::Begin("");

BEGIN_PGML
[`f([$a])`]

[_]{$L[3]}{10}

END_PGML

Section::End();

Scaffold::End();

############################################################
# End Problem
############################################################

ENDDOCUMENT();

[Sean Fitzpatrick]

OK, I am only seeing this issue with one version of the book: answers are mixed up in the version of APEX without videos, but these questions work fine in the version with videos.

My guess is that I've hit some sort of cross-pollination with webwork-representations files.

I will rebuild and see if that helps.

[Sean Fitzpatrick]

Rebuilt, and I still see the same problem.

Rebuilding again, with debugging. Maybe this is caused by the CLI's cache?

I will try nuking everything and rebuilding from scratch as a next step.

[Alex Jordan]

Is it just this one exercise? I think the code for the exercise is just buggy, and that is responsible for some seeds leading to wrong answers. Sorry for probably contributing the buggy PG code. Here is a revamp that makes sense to me:

Context()->strings->add('does not exist'=>{alias=>'DNE'});
@x=num_sort((0,random_subset(2,1..5)));

# actual y-values the function takes
@y=random_subset(3,1..5);
# y-values for limit points ($w from left; $z from right)

($w,$z)=random_subset(2,1..$y[1]-1,$y[1]+1..5);

$a=$x[1];
$f=Formula("($w-$y[0])/($x[1]-$x[0])*(x-$x[0])+$y[0]");
$g=Formula("($y[2]-$z)/($x[2]-$x[1])^2*(x-$x[1])^2+$z");
$xmin=min(0,@x)-1;$xmax=max(0,@x)+1;$ymin=min(0,@y,$z,$w)-1;$ymax=max(0,@y,$z,$w)+1;
@L=($w,$z,Compute("DNE"),$y[1]);

And then in the latex-image:

\begin{tikzpicture}
  \begin{axis}[
    xmin = $xmin,
    xmax = $xmax,
    ymin = $ymin,
    ymax = $ymax,
  ]
    \addplot[firstcurvestyle, domain=$x[0]:$x[1]] {$f};
    \addplot[firstcurvestyle, domain=$x[1]:$x[2]] {$g};

    \addplot[soliddot] coordinates {($x[0],$y[0]) ($x[1],$y[1]) ($x[2],$y[2])};
    \addplot[hollowdot] coordinates {($x[1],$w) ($x[1],$z)};
  \end{axis}
\end{tikzpicture}

--
You received this message because you are subscribed to the Google Groups "PreTeXt support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pretext-suppo...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/pretext-support/5afb77c2-fd95-445f-a958-fae138580d91n%40googlegroups.com.

[Sean Fitzpatrick]

There are a couple exercises in this set with similar issues, and similar code, so probably the same bug in each.

Thanks for the revamp. I will try this tomorrow.

But the confusing part for me is that the buggyness only shows up in the PreTeXt HTML, and I've only been able to produce it in one version of the book (out of 5, including the one on Runestone)

If I pull the PG code out of webwork-representations.xml and run it on a WeBWorK server, it seems to work every time.

Maybe the problem is OK for seeds used by the problem editor and bad for seeds used when it's in the book.

And I guess maybe the other versions of the book are using different seeds...

[Alex Jordan]

What I did was based on your link

https://opentext.uleth.ca/apex-calculus/sec_limits_one_sided.html#ex-limit-onesided-use-graph-4

and peeking in the inspector, I grabbed the .pg file:

https://opentext.uleth.ca/apex-calculus/generated/webwork/pg/APEX_Calculus/1_4_8.pg

And running that on my WW server, with seed 62 (as in the HTML page) I get the bad behavior.

Then I also grabbed the .pg file here:

https://opentext.uleth.ca/apex-video/generated/webwork/pg/APEX_Calculus/1_4_8.pg

and did a diff with the first file, and aside from the date stamp it was an empty diff.

So to me it seems like the issue is just with my bad PG code for the exercise, not anything systemic with PTX or PG.

To view this discussion visit https://groups.google.com/d/msgid/pretext-support/62813078-5c05-4fab-a6c4-3f3b28316726n%40googlegroups.com.

[Sean Fitzpatrick]

OK, this whole set is buggy in really confusing ways. If you can explain what is going wrong with the first problem, I can probably work out how to debug.

The first problem in the set: https://opentext.uleth.ca/apex-calculus/sec_limits_one_sided.html#ex-limit-onesided-use-graph-1

The pg-code is as follows:

              Context()-&gt;strings-&gt;add('does not exist'=&gt;{alias=&gt;'DNE'});
              @x=num_sort((0,random_subset(2,1..5)));
              @y=random_subset(3,1..5);

              $z=list_random(1..$y[1]-1,$y[1]+1..5);
              $b=list_random($x[0],$x[2]);
              $a=$x[1];
              $f=Formula("($y[1]-$y[0])/($x[1]-$x[0])*(x-$x[0])+$y[0]");
              $g=Formula("($y[1]-$y[2])/($x[1]-$x[2])**2*(x-$x[2])**2+$y[2]");
              $xmin=min(@x)-1;$xmax=max(0,@x)+1;
              $ymin=min(0,@y,$z)-1;$ymax=max(@y,$z)+1;
              @L=($y[1],$y[1],$y[1],$z);
              ($L[4],$L[5])=($b==0)?(Compute("DNE"),$y[2]):($y[2],Compute("DNE"));

The graph uses:

                    \begin{axis}[
                      xmin = $xmin,
                      xmax = $xmax,
                      ymin = $ymin,
                      ymax = $ymax,
                    ]
                      \addplot[firstcurvestyle, domain=$x[0]:$x[1]] {$f};
                      \addplot[firstcurvestyle, domain=$x[1]:$x[2]] {$g};

                      \addplot[soliddot] coordinates {($x[0],$y[0]) ($x[1],$z) ($x[2],$y[2])};
                      \addplot[hollowdot] coordinates {($x[1],$y[1])};
                    \end{axis}

When I activate, I get the following graph:

The first limit is the limit as x approaches 3 from the left, which should be 4, but the value that is marked correct is 5!

That is the value of $ymax, not the value of $y[1], but the answer blank for this part is <var name="$L[0]" width="10"/>, and $L[0] is set to $y[1].

[Sean Fitzpatrick]

For this problem: https://opentext.uleth.ca/apex-calculus/sec_limits_one_sided.html#ex-limit-onesided-use-graph-5

When I activate I see this graph:

The inspector says the problem was generated with seed 63.

If I take the problem source and run it on WeBWorK, I get a graph where the peak is at (1,4) instead of (1,3).

Using the value 4 in the textbook is marked correct! This matches the graph I get on WeBWorK, but not the graph that is produced when I activate the question in HTML.

[Sean Fitzpatrick]

I tried randomizing a few more times.

In every case, if I take the value provided by data-current_seed in the inspector, and use it to generate the PG version in WeBWorK, I get a graph that is shifted up by one unit compared to the one I see in HTML!

[Alex Jordan]

We should rule out cached images. In the webwork server, look in:

/opt/webwork/webwork2/htdocs/tmp/apex-main/images/

and delete everything. Then try what you've been trying.

To view this discussion visit https://groups.google.com/d/msgid/pretext-support/9679a143-54c4-4b1c-af08-2a58e8ae8ee2n%40googlegroups.com.

[Alex Jordan]

When I grabbed the PG for that exercise and ran it on my server with the right seed, I get the graph that is at height 4 (just like the static version). So I am feeling like your WW server may be sending cached images for the rendered problems.

[Sean Fitzpatrick]

OK -- can confirm that its a caching issue.

But my cache is in /var/www/wwtmp/apex-main/images

In localOverrides.conf I've uncommented lines 412-417, which set the location of the wwtmp files.

Maybe this was a bad decision. Do they need to be in webwork2/htdocs/tmp for the pretext script to clean them?

[Alex Jordan]

So there's no way for pretext to tell a webwork2 server to clear its image cache. The closest thing is that perhaps pretext can insert a line of PG into the problem code it sends, that sets $refreshCachedImages = 1. But with this situation, where it's using the PG file at:

https://opentext.uleth.ca/apex-calculus/generated/webwork/pg/APEX_Calculus/1_4_8.pg

the server is being told the location of the PG code, it's not being sent the PG code. So there's not really an opportunity to set $refreshCachedImages = 1.

Instead, there are ways to set up the webwork2 server to regularly clear its image cache. Maybe once a day would be good? Perhaps we should document somewhere that this is a recommended step each time a book is rebuilt.

Note that in general, once the image is stable, it's good that image caching happens. Because asking your webwork2 server to rebuild those images over and over again for each student loading or reloading the page would be a huge waste.

To view this discussion visit https://groups.google.com/d/msgid/pretext-support/e990c68d-3b97-4f61-95e9-b7a8808875c6n%40googlegroups.com.

[Sean Fitzpatrick]

I think I've got this sorted out now. There was indeed a bug in the one problem, but for the rest it seems it was just the image caching.

It would be a good idea to set up my server to regularly clean images. There are no courses running on the one I use for PreTeXt, so the load from having to rebuild images occasionally shouldn't be a problem.