Multiple students missing their score and attempt record

67 views
Skip to first unread message

Dev Shah

unread,
Aug 26, 2025, 8:57:08 PMAug 26
to Numbas Users
We had an incident yesterday when more than 30 students attempted a Quiz but there are no records in Numas to reflect that they attempted neither any scores. Has anyone came across this situation? we had no outage etc when these attempts were made. How do we resolve this? Appreciate quick help as there are large number of students impacted.

Dev Shah

unread,
Aug 27, 2025, 12:58:53 AMAug 27
to Numbas Users
Hi Christian

Would you please let us know what the issue is? I can see below errors in Postgres container logs. Could this be related? appreciate your quick help.

2025-08-27 04:54:00.065 UTC [43253] DETAIL:  Key (diff_of_id)=(50340069) already exists.
2025-08-27 04:54:00.065 UTC [43253] STATEMENT:  INSERT INTO "numbas_lti_scormelementdiff" ("element_id", "diff_of_id") VALUES (50340063, 50340069) RETURNING "numbas_lti_scormelementdiff"."id"
2025-08-27 04:54:00.065 UTC [43254] ERROR:  duplicate key value violates unique constraint "numbas_lti_scormelementdiff_diff_of_id_key"
2025-08-27 04:54:00.065 UTC [43254] DETAIL:  Key (diff_of_id)=(50340069) already exists.
2025-08-27 04:54:00.065 UTC [43254] STATEMENT:  INSERT INTO "numbas_lti_scormelementdiff" ("element_id", "diff_of_id") VALUES (50340063, 50340069) RETURNING "numbas_lti_scormelementdiff"."id"
2025-08-27 04:55:00.082 UTC [43290] ERROR:  duplicate key value violates unique constraint "numbas_lti_scormelementdiff_diff_of_id_key"
2025-08-27 04:55:00.082 UTC [43290] DETAIL:  Key (diff_of_id)=(50340101) already exists.
2025-08-27 04:55:00.082 UTC [43290] STATEMENT:  INSERT INTO "numbas_lti_scormelementdiff" ("element_id", "diff_of_id") VALUES (50340098, 50340101) RETURNING "numbas_lti_scormelementdiff"."id"
2025-08-27 04:55:00.082 UTC [43287] ERROR:  duplicate key value violates unique constraint "numbas_lti_scormelementdiff_diff_of_id_key"
2025-08-27 04:55:00.082 UTC [43287] DETAIL:  Key (diff_of_id)=(50340101) already exists.
2025-08-27 04:55:00.082 UTC [43287] STATEMENT:  INSERT INTO "numbas_lti_scormelementdiff" ("element_id", "diff_of_id") VALUES (50340098, 50340101) RETURNING "numbas_lti_scormelementdiff"."id"
2025-08-27 04:55:00.082 UTC [43286] ERROR:  duplicate key value violates unique constraint "numbas_lti_scormelementdiff_diff_of_id_key"
2025-08-27 04:55:00.082 UTC [43286] DETAIL:  Key (diff_of_id)=(50340101) already exists.
2025-08-27 04:55:00.082 UTC [43286] STATEMENT:  INSERT INTO "numbas_lti_scormelementdiff" ("element_id", "diff_of_id") VALUES (50340098, 50340101) RETURNING "numbas_lti_scormelementdiff"."id"


Christian Lawson-Perfect

unread,
Aug 27, 2025, 4:35:57 AMAug 27
to numbas...@googlegroups.com
Looks like the job to diff attempt suspend data has gone awry. Do you have anything in the logs for the huey process?
It sounds like something went wrong with your database upgrade. Are you able to make any records in the database? Could you try launching an exam as a student, and see if its data is saved?

--
You received this message because you are subscribed to the Google Groups "Numbas Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to numbas-users...@googlegroups.com.
To view this discussion, visit https://groups.google.com/d/msgid/numbas-users/9ac7ed01-1b02-4c23-9577-7837c73e297en%40googlegroups.com.

Dev Shah

unread,
Aug 28, 2025, 12:11:51 AMAug 28
to numbas...@googlegroups.com
Hi Christian

No, we didn't upgrade the database yet. Basically nothing has been changed. We have 4 Huey containers running and they all look normal except in huey_3, I saw below exception but this log is from today so, this may not be relevant to the issue which was reported on Aug. 26, 2025, 5:54 p.m

[2025-08-28 13:16:47,721] ERROR:huey:Worker-1:Unhandled exception in task 16579b5c-7036-40d4-9e47-aeae335eaae5.
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/huey/api.py", line 378, in _execute
    task_value = task.execute()
  File "/usr/local/lib/python3.9/site-packages/huey/api.py", line 764, in execute
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/huey/contrib/djhuey/__init__.py", line 133, in inner
    return fn(*args, **kwargs)
  File "/opt/numbas-lti-provider/numbas_lti/tasks.py", line 28, in send_attempt_completion_receipt
    attempt.send_completion_receipt()
  File "/opt/numbas-lti-provider/numbas_lti/models.py", line 1104, in send_completion_receipt
    send_mail(
  File "/usr/local/lib/python3.9/site-packages/django/core/mail/__init__.py", line 61, in send_mail
    return mail.send()
  File "/usr/local/lib/python3.9/site-packages/django/core/mail/message.py", line 284, in send
    return self.get_connection(fail_silently).send_messages([self])
  File "/usr/local/lib/python3.9/site-packages/django/core/mail/backends/smtp.py", line 102, in send_messages
    new_conn_created = self.open()
  File "/usr/local/lib/python3.9/site-packages/django/core/mail/backends/smtp.py", line 62, in open
    self.connection = self.connection_class(self.host, self.port, **connection_params)
  File "/usr/local/lib/python3.9/smtplib.py", line 255, in __init__
    (code, msg) = self.connect(host, port)
  File "/usr/local/lib/python3.9/smtplib.py", line 341, in connect
    self.sock = self._get_socket(host, port, self.timeout)
  File "/usr/local/lib/python3.9/smtplib.py", line 312, in _get_socket
    return socket.create_connection((host, port), timeout,
  File "/usr/local/lib/python3.9/socket.py", line 844, in create_connection
    raise err
  File "/usr/local/lib/python3.9/socket.py", line 832, in create_connection
    sock.connect(sa)
OSError: [Errno 99] Cannot assign requested address
Unhandled exception in task 16579b5c-7036-40d4-9e47-aeae335eaae5.
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/huey/api.py", line 378, in _execute
    task_value = task.execute()
  File "/usr/local/lib/python3.9/site-packages/huey/api.py", line 764, in execute
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/huey/contrib/djhuey/__init__.py", line 133, in inner
    return fn(*args, **kwargs)
  File "/opt/numbas-lti-provider/numbas_lti/tasks.py", line 28, in send_attempt_completion_receipt
    attempt.send_completion_receipt()
  File "/opt/numbas-lti-provider/numbas_lti/models.py", line 1104, in send_completion_receipt
    send_mail(
  File "/usr/local/lib/python3.9/site-packages/django/core/mail/__init__.py", line 61, in send_mail
    return mail.send()
  File "/usr/local/lib/python3.9/site-packages/django/core/mail/message.py", line 284, in send
    return self.get_connection(fail_silently).send_messages([self])
  File "/usr/local/lib/python3.9/site-packages/django/core/mail/backends/smtp.py", line 102, in send_messages
    new_conn_created = self.open()
  File "/usr/local/lib/python3.9/site-packages/django/core/mail/backends/smtp.py", line 62, in open
    self.connection = self.connection_class(self.host, self.port, **connection_params)

You received this message because you are subscribed to a topic in the Google Groups "Numbas Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/numbas-users/uADztIofMfM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to numbas-users...@googlegroups.com.
To view this discussion, visit https://groups.google.com/d/msgid/numbas-users/CAEMHSOjSQEyC4icu5wmGG8opvgb3bknoApTu2kAt75wDSe_imQ%40mail.gmail.com.

Dev Shah

unread,
Aug 28, 2025, 1:39:24 AMAug 28
to numbas...@googlegroups.com
On the other note, I am not sure if I can play around with the DB inserts in prod as it may cause other issues but please advise if there is a safer way to do this without impacting the DB integrity. also, I don't have a student account in live env to test this issue.

Christian Lawson-Perfect

unread,
Aug 28, 2025, 2:20:05 AMAug 28
to numbas...@googlegroups.com
You should only have one Huey container running, shouldn't you?

Dev Shah

unread,
Aug 28, 2025, 3:21:07 AMAug 28
to numbas...@googlegroups.com
docker-compose up -d --scale daphne=6 --scale huey=4

This is the command we have used since past few years to start containers. I wasn't aware that only one Huey container should be running. Not sure what Huey process does but if it needs to be only one instance, I'll change it. Hope it won't affect the performance of the app.

Dev Shah

unread,
Aug 28, 2025, 5:55:38 AMAug 28
to numbas...@googlegroups.com
One more thing reported by our academic staff is Past quizzes (Computer Practical Quiz 1, 2, and 3) are still working fine, it is just when he loads new ones, no matter what it doesn't work.So, it seems like any new quiz is having these issues.
Message has been deleted

Dev Shah

unread,
Aug 29, 2025, 4:30:58 AMAug 29
to Numbas Users
This is now impacting all academics as we have more people reporting that neither scores or attempts are being saved. We really appreciate some help. What surprises me is, there has been no changes to the app since it was implemented. Could this be Postgres DB issue? if so, is there any way to check this? e.g. DB corruption etc?

Christian Lawson-Perfect

unread,
Aug 29, 2025, 4:36:33 AMAug 29
to Numbas Users
I'm sorry, but it sounds like a problem with your infrastructure. I'm not sure what help I could offer.
I'm at a conference today and on annual leave next week, so I wouldn't be able to do a video call until the week after. 

Do you have a snapshot that you could try reverting to? 
Do you have free disk space?
Does it affect all exam packages, or only recent ones? 

Getting a student account that you can use to test would speed up your troubleshooting process. 

Dev Shah

unread,
Aug 30, 2025, 12:38:43 AMAug 30
to numbas...@googlegroups.com
Thanks Christian for your response. We started noticing this issue since 26th of this month. Our VMs are getting backed up so, I can ask our infra team to revert to a previous backup though this would result in some loss of data (tests, quizzes etc). There is plenty of free Disk space ..around 37GB. 

The issue seems to have started recently around 26th August. It only affects newly created packages. 

Could this be an issue with Postgres sequence? based on SQL errors it looks like code is trying to insert a non unique value in one of the table. Without knowing the architecture of the app, it is difficult to say where it gets those duplicate values from.

Dev Shah

unread,
Aug 31, 2025, 7:50:22 PMAug 31
to numbas...@googlegroups.com
Now, after restarting the app with just one Huey container, I am seeing below errors... this looks like some sort of database corruption where unique keys are violated and foreign keys are missing.

2025-08-30 02:35:45.514 UTC [2802] ERROR:  insert or update on table "numbas_lti_attempt" violates foreign key constraint "numbas_lti_attempt_completion_status_el_2dc29def_fk_numbas_lt"
2025-08-30 02:35:45.514 UTC [2802] DETAIL:  Key (completion_status_element_id)=(50370416) is not present in table "numbas_lti_scormelement".
2025-08-30 02:35:45.514 UTC [2802] STATEMENT:  UPDATE "numbas_lti_attempt" SET "end_time" = NULL, "completion_status" = 'incomplete', "completion_status_element_id" = 50370416 WHERE "numbas_lti_atte
mpt"."id" = 100804
2025-08-30 04:52:45.702 UTC [5191] ERROR:  insert or update on table "numbas_lti_attempt" violates foreign key constraint "numbas_lti_attempt_completion_status_el_2dc29def_fk_numbas_lt"
2025-08-30 04:52:45.702 UTC [5191] DETAIL:  Key (completion_status_element_id)=(50371479) is not present in table "numbas_lti_scormelement".
2025-08-30 04:52:45.702 UTC [5191] STATEMENT:  UPDATE "numbas_lti_attempt" SET "end_time" = NULL, "completion_status" = 'incomplete', "completion_status_element_id" = 50371479 WHERE "numbas_lti_atte
mpt"."id" = 100811
2025-08-30 05:25:11.514 UTC [7405] ERROR:  insert or update on table "numbas_lti_attempt" violates foreign key constraint "numbas_lti_attempt_completion_status_el_2dc29def_fk_numbas_lt"
2025-08-30 05:25:11.514 UTC [7405] DETAIL:  Key (completion_status_element_id)=(50372542) is not present in table "numbas_lti_scormelement".
2025-08-30 05:25:11.514 UTC [7405] STATEMENT:  UPDATE "numbas_lti_attempt" SET "end_time" = NULL, "completion_status" = 'incomplete', "completion_status_element_id" = 50372542 WHERE "numbas_lti_atte
mpt"."id" = 100815
2025-08-30 16:26:22.512 UTC [11787] ERROR:  insert or update on table "numbas_lti_attempt" violates foreign key constraint "numbas_lti_attempt_completion_status_el_2dc29def_fk_numbas_lt"
2025-08-30 16:26:22.512 UTC [11787] DETAIL:  Key (completion_status_element_id)=(50376350) is not present in table "numbas_lti_scormelement".
2025-08-30 16:26:22.512 UTC [11787] STATEMENT:  UPDATE "numbas_lti_attempt" SET "end_time" = NULL, "completion_status" = 'incomplete', "completion_status_element_id" = 50376350 WHERE "numbas_lti_att
empt"."id" = 100828
2025-08-31 12:18:28.512 UTC [25491] ERROR:  insert or update on table "numbas_lti_attempt" violates foreign key constraint "numbas_lti_attempt_completion_status_el_2dc29def_fk_numbas_lt"
2025-08-31 12:18:28.512 UTC [25491] DETAIL:  Key (completion_status_element_id)=(50388278) is not present in table "numbas_lti_scormelement".
2025-08-31 12:18:28.512 UTC [25491] STATEMENT:  UPDATE "numbas_lti_attempt" SET "end_time" = NULL, "completion_status" = 'incomplete', "completion_status_element_id" = 50388278 WHERE "numbas_lti_att
empt"."id" = 100868
2025-08-31 13:07:28.511 UTC [25988] ERROR:  insert or update on table "numbas_lti_attempt" violates foreign key constraint "numbas_lti_attempt_completion_status_el_2dc29def_fk_numbas_lt"
2025-08-31 13:07:28.511 UTC [25988] DETAIL:  Key (completion_status_element_id)=(50388724) is not present in table "numbas_lti_scormelement".
2025-08-31 13:07:28.511 UTC [25988] STATEMENT:  UPDATE "numbas_lti_attempt" SET "end_time" = NULL, "completion_status" = 'incomplete', "completion_status_element_id" = 50388724 WHERE "numbas_lti_att
empt"."id" = 100871
2025-08-31 23:12:06.529 UTC [30427] ERROR:  insert or update on table "numbas_lti_attempt" violates foreign key constraint "numbas_lti_attempt_completion_status_el_2dc29def_fk_numbas_lt"
2025-08-31 23:12:06.529 UTC [30427] DETAIL:  Key (completion_status_element_id)=(50390953) is not present in table "numbas_lti_scormelement".
2025-08-31 23:12:06.529 UTC [30427] STATEMENT:  UPDATE "numbas_lti_attempt" SET "end_time" = NULL, "completion_status" = 'incomplete', "completion_status_element_id" = 50390953 WHERE "numbas_lti_att
empt"."id" = 100879
2025-08-31 23:14:24.512 UTC [30503] ERROR:  insert or update on table "numbas_lti_attempt" violates foreign key constraint "numbas_lti_attempt_completion_status_el_2dc29def_fk_numbas_lt"
2025-08-31 23:14:24.512 UTC [30503] DETAIL:  Key (completion_status_element_id)=(50391257) is not present in table "numbas_lti_scormelement".
2025-08-31 23:14:24.512 UTC [30503] STATEMENT:  UPDATE "numbas_lti_attempt" SET "end_time" = NULL, "completion_status" = 'incomplete', "completion_status_element_id" = 50391257 WHERE "numbas_lti_att
empt"."id" = 100880

Christian Lawson-Perfect

unread,
Sep 9, 2025, 11:50:05 AMSep 9
to numbas...@googlegroups.com
Hi Dev,
Thanks for this. Looking at another server's logs, I've seen a lot of the same error. Clearly there's a new timing issue, somehow. I'll see if I can pin this down.

Martin Jones

unread,
Sep 15, 2025, 10:56:24 AMSep 15
to Numbas Users
I think we are suffering from the same issue. Students can access the quizzes but no results are being received by the LTI server. We are still using version 3.5. Could this be the reason?

Dev Shah

unread,
Sep 18, 2025, 1:14:39 AMSep 18
to numbas...@googlegroups.com
Hi Martin

Most likely, you are in the same boat i.e. breaking changes in v9.0. We were asked to use the 'Standard' theme while creating tests\quizzes etc and it seemed to work for us. We are still at v3.1.

Martin Jones

unread,
Sep 18, 2025, 2:56:33 AMSep 18
to Numbas Users
Hi Dev,
Thanks for the tip. I will try reverting to the standard theme to see if that fixes it for me.

Christian Lawson-Perfect

unread,
Sep 19, 2025, 8:34:08 AMSep 19
to numbas...@googlegroups.com
This is now fixed.
I'm really sorry this took so long - I had no idea what the problem could be until Dewang and Laura worked out that it was to do with the old theme. 
The cause of the problem was that I hadn't updated the old theme after I changed the attribute specifying which storage mechanism to use. I've now done that, and checked that a package using the old theme correctly saves data.
Sorry again.

Reply all
Reply to author
Forward
0 new messages