Chemical Equation problem error: Couldn't execute jailed code
178 views
Skip to first unread message
Jessica Reel
Jul 7, 2016, 6:43:59 AM7/7/16
to Open edX operations
Hi all,
I'm trying to set up an Open edX site, running on the Google Cloud platform. New VM instance, custom machine (1 vCPU, 4 GB memory), Ubuntu 12.04 on 50 GB standard persistent disk.
I have followed the instructions here (https://openedx.atlassian.net/wiki/display/OpenOPS/Native+Open+edX+Ubuntu+12.04+64+bit+Installation) for Manual Installation using the latest master. After restarting various services, the website is working, I can create a new superuser and log in, access/create courses.
However, there is an error that I am experiencing with a particular problem type, the "Chemical Equation" problem. This error occurs in both the pre-existing "edX Demonstration Course" and in a new course with a newly-created problem of this type.
This is the error from the Chemical Equation problem in the Example Week 1 section of the edX Demo course:
And this is the error for a newly-created Chemical Equation problem in a different (newly-created) test course:
...
Ok. As I was typing this, suddenly I can't reproduce the error. I've been trying to research fix for this all day...
...
I feel like I am taking crazy pills. Any thoughts?
I'm trying to set up an Open edX site, running on the Google Cloud platform. New VM instance, custom machine (1 vCPU, 4 GB memory), Ubuntu 12.04 on 50 GB standard persistent disk.
I have followed the instructions here (https://openedx.atlassian.net/wiki/display/OpenOPS/Native+Open+edX+Ubuntu+12.04+64+bit+Installation) for Manual Installation using the latest master. After restarting various services, the website is working, I can create a new superuser and log in, access/create courses.
However, there is an error that I am experiencing with a particular problem type, the "Chemical Equation" problem. This error occurs in both the pre-existing "edX Demonstration Course" and in a new course with a newly-created problem of this type.
This is the error from the Chemical Equation problem in the Example Week 1 section of the edX Demo course:
Staff debug info: Traceback (most recent call last): File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/capa_base.py", line 1063, in check_problem correct_map = self.lcp.grade_answers(answers) File "/edx/app/edxapp/edx-platform/common/lib/capa/capa/capa_problem.py", line 378, in grade_answers return self._grade_answers(answers) File "/edx/app/edxapp/edx-platform/common/lib/capa/capa/capa_problem.py", line 440, in _grade_answers results = responder.evaluate_answers(self.student_answers, oldcmap) File "/edx/app/edxapp/edx-platform/common/lib/capa/capa/responsetypes.py", line 280, in evaluate_answers new_cmap = self.get_score(student_answers) File "/edx/app/edxapp/edx-platform/common/lib/capa/capa/responsetypes.py", line 2375, in get_score self.execute_check_function(idset, submission) File "/edx/app/edxapp/edx-platform/common/lib/capa/capa/responsetypes.py", line 2416, in execute_check_function self._handle_exec_exception(err) File "/edx/app/edxapp/edx-platform/common/lib/capa/capa/responsetypes.py", line 2632, in _handle_exec_exception raise ResponseError(err.message, traceback_obj) ResponseError: ("Couldn't execute jailed code: stdout: '', stderr: '' with status code: -9", <traceback object at 0x7f73578b4bd8>)
And this is the error for a newly-created Chemical Equation problem in a different (newly-created) test course:
...
Ok. As I was typing this, suddenly I can't reproduce the error. I've been trying to research fix for this all day...
...
I feel like I am taking crazy pills. Any thoughts?
Jessica Reel
Jul 7, 2016, 9:35:18 AM7/7/16
to Open edX operations
I added another newly-created Chemical Equation problem to my test course (in the same unit), and the error came back.
Adding a third Chemical Equation problem to the test course, in a new unit, I cannot reproduce the error.
This is a tricky one.
Staff debug info: Traceback (most recent call last): File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/capa_base.py", line 1063, in check_problem correct_map = self.lcp.grade_answers(answers) File "/edx/app/edxapp/edx-platform/common/lib/capa/capa/capa_problem.py", line 378, in grade_answers return self._grade_answers(answers) File "/edx/app/edxapp/edx-platform/common/lib/capa/capa/capa_problem.py", line 440, in _grade_answers results = responder.evaluate_answers(self.student_answers, oldcmap) File "/edx/app/edxapp/edx-platform/common/lib/capa/capa/responsetypes.py", line 280, in evaluate_answers new_cmap = self.get_score(student_answers) File "/edx/app/edxapp/edx-platform/common/lib/capa/capa/responsetypes.py", line 2375, in get_score self.execute_check_function(idset, submission) File "/edx/app/edxapp/edx-platform/common/lib/capa/capa/responsetypes.py", line 2416, in execute_check_function self._handle_exec_exception(err) File "/edx/app/edxapp/edx-platform/common/lib/capa/capa/responsetypes.py", line 2632, in _handle_exec_exception raise ResponseError(err.message, traceback_obj) ResponseError: ("Couldn't execute jailed code: stdout: '', stderr: '' with status code: -9", <traceback object at 0x7f735c4bcf80>)
Adding a third Chemical Equation problem to the test course, in a new unit, I cannot reproduce the error.
This is a tricky one.
Jessica Reel
Jul 7, 2016, 9:46:15 AM7/7/16
to Open edX operations
I found this (https://openedx.atlassian.net/browse/TNL-835), in particular a response by Ned B.:
"I've been doing testing on my devstack instance. If the problem uses too much real time (runs too long), it ends with a status of -9, and there is a clear log message indicating that we are killing it. If it runs out of memory, it ends with a status of 1 or -9, depending on how quickly the process hits the limit. In that case, AppArmor kills the process."
All of the above errors have a status code -9, and the intermittent nature of the issue also points to a resource problem.
Can this be solved via config (AppArmor or CodeJail config?) somehow, or do I need to deploy on a more powerful VM?
Thanks all.
"I've been doing testing on my devstack instance. If the problem uses too much real time (runs too long), it ends with a status of -9, and there is a clear log message indicating that we are killing it. If it runs out of memory, it ends with a status of 1 or -9, depending on how quickly the process hits the limit. In that case, AppArmor kills the process."
All of the above errors have a status code -9, and the intermittent nature of the issue also points to a resource problem.
Can this be solved via config (AppArmor or CodeJail config?) somehow, or do I need to deploy on a more powerful VM?
Thanks all.
LIubov Fomicheva
Jul 8, 2016, 7:17:51 AM7/8/16
to Open edX operations
Hello Jessica,
четверг, 7 июля 2016 г., 16:46:15 UTC+3 пользователь Jessica Reel написал:
We solved such an issue by changing environment setting CODE_JAIL_LIMITS in lms.env.json and cms.env.json (corresponding server-vars.yml variable is EDXAPP_CODE_JAIL_LIMITS); we've set CPU to 2 (seconds) and REALTIME to 3, that was enough for our dev and production.
I'll be glad to hear any comments from community on this issue (maybe some deep optimization would help better then just changing the limits).
Kind regards,
Liubov
четверг, 7 июля 2016 г., 16:46:15 UTC+3 пользователь Jessica Reel написал:
Reply all
Reply to author
Forward
0 new messages