2.13 -> 3.0 LTI provider upgrade issue

[Tom Salyers]

Hi, all. 

I've just done an upgrade of our staging instance of the LTI provider to 3.0. It was a bit rocky (I had to update from Ubuntu 18 to 20, which got me the required level of Redis and Python, but broke a couple of other things), but I've got a mostly-working installation now. 

I'm running into a problem, though, when I go into a resource on the dashboard and either attempt to download scores as a CSV or report them back to the VLE. I get a 500 Internal Server Error in my browser, and looking in /var/log/supervisor/numbas_lti_daphne_stderr.log, it looks like there's an exception being thrown:

redis.exceptions.RedisError: ZADD requires an equal number of values and scores

It's done this across three different assignments/tests so far. Do I have something misconfigured, or did I miss an upgrade step? (I'm doing this while also training a new member of our team, so it's entirely possible.) I've included the full stack trace below in case that'll help. Thanks in advance for any advice!

2022-01-19 14:34:35,735 ERROR /opt/numbas_lti_python/lib/python3.8/site-packages/django/utils/log.py: Internal Server Error: /resource/1/report_scores
Traceback (most recent call last):
  File "/opt/numbas_lti_python/lib/python3.8/site-packages/asgiref/sync.py", line 482, in thread_handler
    raise exc_info[1]
  File "/opt/numbas_lti_python/lib/python3.8/site-packages/django/core/handlers/exception.py", line 38, in inner
    response = await get_response(request)
  File "/opt/numbas_lti_python/lib/python3.8/site-packages/django/core/handlers/base.py", line 233, in _get_response_async
    response = await wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/opt/numbas_lti_python/lib/python3.8/site-packages/asgiref/sync.py", line 444, in __call__
    ret = await asyncio.wait_for(future, timeout=None)
  File "/usr/lib/python3.8/asyncio/tasks.py", line 455, in wait_for
    return await fut
  File "/opt/numbas_lti_python/lib/python3.8/site-packages/asgiref/current_thread_executor.py", line 22, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/opt/numbas_lti_python/lib/python3.8/site-packages/asgiref/sync.py", line 486, in thread_handler
    return func(*args, **kwargs)
  File "/opt/numbas_lti_python/lib/python3.8/site-packages/django/views/generic/base.py", line 70, in view
    return self.dispatch(request, *args, **kwargs)
  File "/srv/numbas-lti-provider/./numbas_lti/views/mixins.py", line 94, in dispatch
    return super(MustHaveExamMixin,self).dispatch(*args,**kwargs)
  File "/opt/numbas_lti_python/lib/python3.8/site-packages/django_auth_lti/mixins.py", line 31, in dispatch
    return super(LTIRoleRestrictionMixin, self).dispatch(request, *args, **kwargs)
  File "/srv/numbas-lti-provider/./numbas_lti/views/mixins.py", line 86, in dispatch
    return super(ResourceManagementViewMixin,self).dispatch(*args,**kwargs)
  File "/opt/numbas_lti_python/lib/python3.8/site-packages/django/views/generic/base.py", line 98, in dispatch
    return handler(request, *args, **kwargs)
  File "/srv/numbas-lti-provider/./numbas_lti/views/resource.py", line 334, in get
    resource.task_report_scores()
  File "/srv/numbas-lti-provider/./numbas_lti/models.py", line 514, in task_report_scores
    tasks.resource_report_scores(self)
  File "/opt/numbas_lti_python/lib/python3.8/site-packages/huey/api.py", line 823, in __call__
    return self.huey.enqueue(self.s(*args, **kwargs))
  File "/opt/numbas_lti_python/lib/python3.8/site-packages/huey/api.py", line 298, in enqueue
    self.storage.enqueue(self.serialize_task(task), task.priority)
  File "/opt/numbas_lti_python/lib/python3.8/site-packages/huey/storage.py", line 571, in enqueue
    self.conn.zadd(self.queue_key, {prefix + data: priority})
  File "/opt/numbas_lti_python/lib/python3.8/site-packages/redis/client.py", line 1691, in zadd
    raise RedisError("ZADD requires an equal number of "
redis.exceptions.RedisError: ZADD requires an equal number of values and scores

--

Tom Salyers

[Tom Salyers]

Actually, cancel that--it looks like the virtual environment had a conflicting library installed. I removed it and the internal server error's gone away. I'm getting another error, though, when I try to download CSV reports--the browser's telling me that the server can't find the file. I don't see any immediately obvious errors in the logs...does anyone have any ideas? 

--

Tom Salyers, MBCS
Senior Education Developer/System Administrator

The University of Sheffield
IT Services

10-12 Brunswick Street

Sheffield, S10 2FN

Tel:  0114 222 1141

t.sa...@sheffield.ac.uk

--
You received this message because you are subscribed to a topic in the Google Groups "Numbas Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/numbas-users/dyeP_z-58VQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to numbas-users...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/numbas-users/4dc84fb6-3308-4949-b656-9886f32351a5n%40googlegroups.com.

[Christian Lawson-Perfect]

Does the folder /srv/numbas-lti-media/reports exist, and is the file you're expecting to see in there?

You received this message because you are subscribed to the Google Groups "Numbas Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to numbas-users...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/numbas-users/CAAVE4MwLHpSqCJPOCZiE3xjGPAqA0vJrhDYdGAS3xh4h-Gwhug%40mail.gmail.com.

[Tom Salyers]

The directory exists now--it didn't before, for some reason--but it looks like no files are being generated in there. 

--

Tom Salyers, MBCS
Senior Education Developer/System Administrator

The University of Sheffield
IT Services

10-12 Brunswick Street

Sheffield, S10 2FN

Tel:  0114 222 1141

t.sa...@sheffield.ac.uk

To view this discussion on the web, visit https://groups.google.com/d/msgid/numbas-users/CAEMHSOi69C-pvc3kWdpdzU0e%2BUMHkrm_q6hGbMYJKoPi7f6pJA%40mail.gmail.com.

[Tom Salyers]

Hi, Christian. 

Just to follow up on this...the reports directory exists, and it looks like its permissions are all right, but it looks like the CSV files aren't actually being generated. The system says it generates them, and there's a link on the reports page, but actually clicking the link gives me this error in the server logs: 

2022/01/24 15:34:52 [error] 664#664: *31 open() "/srv/numbas-lti-media/reports/capybaras-exam-scores-2022-24-01-15_34_46-2c7e634e.csv" failed (2: No such file or directory), client: 172.16.208.103, server: , request: "GET /media/reports/capybaras-exam-scores-2022-24-01-15_34_46-2c7e634e.csv HTTP/1.1", host: [redacted], referrer: "https://[redacted]/resource/1/reports"

I'm sure I've missed something obvious...any ideas? 

--

Tom Salyers, MBCS
Senior Education Developer/System Administrator

The University of Sheffield
IT Services

10-12 Brunswick Street

Sheffield, S10 2FN

Tel:  0114 222 1141

t.sa...@sheffield.ac.uk

[Christian Lawson-Perfect]

Sorry, I'm not sure I can think of anything that would cause that.

You could try creating a report in the command shell, and see if you get any error messages:

from numbas_lti.models import *

from numbas_lti.tasks import *

r = Resource.objects.first()

fr = FileReport.objects.create(name='testing',created_by=User.objects.first(), resource=r)

fr.outfile.save('testing.csv', ContentFile(''))

tasks.resource_scores_csv_report(fr)

To view this discussion on the web, visit https://groups.google.com/d/msgid/numbas-users/CAAVE4My2XPZpLCjJAEj1H1fV2xF2fBUwsR_vU1n%3DJOkuqn3ZOg%40mail.gmail.com.

[Tom Salyers]

Hi, Christian. 

Thanks for that--I initially tried it and was getting a permissions error, so I re-checked the permissions on that directory and fixed that particular error. I'm getting a different kind of weirdness now, though, where it looks like it generates the files, but I'm still unable to download. Running the Python you provided works fine up until the last step: 

>>> from numbas_lti.models import *
>>> from numbas_lti.tasks import *
>>> r = Resource.objects.first()
>>> fr = FileReport.objects.create(name='testing',created_by=User.objects.first(), resource=r)
>>> fr.outfile.save('testing.csv', ContentFile(''))
>>> tasks.resource_scores_csv_report(fr)

Traceback (most recent call last):

  File "<console>", line 1, in <module>
NameError: name 'tasks' is not defined

....but that might just be an import issue.  The Nginx logs show some oddness with one of the CSS files (although I'm fairly sure that won't affect finding the file--I'm just mentioning it in passing). Meanwhile, the file's still not being found: 

2022/01/28 09:46:15 [error] 76690#76690: *18 open() "/srv/numbas-lti-static/numbas_lti.760362260019.css" failed (2: No such file or directory), client: 143.167.16.2, server: , request: "GET /static/numbas_lti.760362260019.css HTTP/1.1", host: "numbas-staging.shef.ac.uk", referrer: "https://numbas-staging.shef.ac.uk/resource/15/reports"
2022/01/28 09:46:46 [error] 76690#76690: *14 open() "/srv/numbas-lti-media/reports/chickens-exam-scores-2022-28-01-09_46_05-65a209be.csv" failed (2: No such file or directory), client: 143.167.16.2, server: , request: "GET /media/reports/chickens-exam-scores-2022-28-01-09_46_05-65a209be.csv HTTP/1.1", host: "numbas-staging.shef.ac.uk", referrer: "https://numbas-staging.shef.ac.uk/resource/15/reports"

The file exists, though, and should be readable: 

-rw-r--r-- 1 www-data www-data 146 Jan 28 09:46 media/reports/chickens-exam-scores-2022-28-01-09_46_05-65a209be.csv

I'm kind of stumped....any ideas? 

--

Tom Salyers, MBCS
Senior Education Developer/System Administrator

The University of Sheffield
IT Services

10-12 Brunswick Street

Sheffield, S10 2FN

Tel:  0114 222 1141

t.sa...@sheffield.ac.uk

To view this discussion on the web, visit https://groups.google.com/d/msgid/numbas-users/CAEMHSOhGaW4G7bg2mUd8xLiAQoc7aN7kSL53Lf6EKjidzhcBGA%40mail.gmail.com.

[Christian Lawson-Perfect]

Ah sorry, the line 'from numbas_lti.tasks import *' should be 'from numbas_lti import tasks'

To view this discussion on the web, visit https://groups.google.com/d/msgid/numbas-users/CAAVE4MxQrmxCrWxr5gE8C3kPUmrVMucgXizDQ-cpF5UQEiSTOw%40mail.gmail.com.

[Tom Salyers]

Hi, Christian. 

Sorry for the delay...I finally managed to catch COVID, and I've been out for a while. (Fortunately with all the vaccinations, it's basically been a bad cold...but still not fun.) 

I corrected the line you mentioned previously, and it worked fine--it created the test file just as you'd expect. From what I can tell from looking in the media/reports directory, the CSV files are getting created just fine. They're just not getting served up when I request a download and I get a "Server could not find the file" message. 

Might there be some kind of additional permissions issue? That directory and its files *should* be readable, but I'm not ruling anything out at this point. 

Oh, and almost certainly completely unrelated, but I'm getting a CSS error. The exact message is: "Refused to apply style from 'https://[our hostname]/static/numbas_lti.760362260019.css' because its MIME type ('text/html') is not a supported stylesheet MIME type, and strict MIME checking is enabled." I'm pretty sure it's not the source of the problem, but I thought I'd mention it. 

--

Tom Salyers, MBCS
Senior Education Developer/System Administrator

The University of Sheffield
IT Services

10-12 Brunswick Street

Sheffield, S10 2FN

Tel:  0114 222 1141

t.sa...@sheffield.ac.uk

To view this discussion on the web, visit https://groups.google.com/d/msgid/numbas-users/CAEMHSOhsV%2BBeh2fLeeiW7dzB8yTKan1VeMp4Vj5ihbXGZHDnTA%40mail.gmail.com.

[Christian Lawson-Perfect]

It sounds like your static files aren't being served properly. Can you check your nginx configuration? 

To view this discussion on the web, visit https://groups.google.com/d/msgid/numbas-users/CAAVE4MwR3X11Tb83W%3Dr0AASHn9cJb4NqOA-kAiCFh04TvvNCNQ%40mail.gmail.com.

[Tom Salyers]

Sure. What am I looking for, mainly? I don't think I've changed the config from the original installation instructions all that much beyond adding some security headers requested by our infosec people. This is still in sites-available from the original installation, for instance: 

    location /static {
        alias /srv/numbas-lti-static;
    }

--

Tom Salyers, MBCS
Senior Education Developer/System Administrator

The University of Sheffield
IT Services

10-12 Brunswick Street

Sheffield, S10 2FN

Tel:  0114 222 1141

t.sa...@sheffield.ac.uk

To view this discussion on the web, visit https://groups.google.com/d/msgid/numbas-users/CAEMHSOg8QbSoW2oD76dirRdGWa9V8XxtS3YE4JCdm77HakJvwA%40mail.gmail.com.