How to configure Xqueue and External grader?

[Naresh Makwana]

I want to add code response problem so the students can submit their program which are evaluated by the external grader.

I have followed the steps mentioned in the edx documentation and other forums but still i am not able to do this.

I have production server in use and sudo /edx/bin/supervisorctl -c /edx/etc/supervisord.conf status gives

certs                                                         FATAL      Exited too quickly (process log may have details)

edxapp:cms                                             RUNNING    pid 1745, uptime 7 days, 1:25:41

edxapp:lms                                             RUNNING    pid 1744, uptime 7 days, 1:25:41

edxapp_worker:cms_default_4               RUNNING    pid 2938, uptime 7 days, 1:24:13

edxapp_worker:cms_high_1                   RUNNING    pid 1776, uptime 7 days, 1:25:41

edxapp_worker:cms_low_3                    RUNNING    pid 2902, uptime 7 days, 1:24:13

edxapp_worker:lms_default_3                RUNNING    pid 1775, uptime 7 days, 1:25:41

edxapp_worker:lms_high_4                   RUNNING    pid 1782, uptime 7 days, 1:25:41

edxapp_worker:lms_high_mem_2        RUNNING    pid 1774, uptime 7 days, 1:25:41

edxapp_worker:lms_low_1                   RUNNING    pid 1777, uptime 7 days, 1:25:41

forum                                                     RUNNING    pid 6106, uptime 7 days, 1:22:03

ora                                                         RUNNING    pid 1783, uptime 7 days, 1:25:41

ora_celery                                             RUNNING    pid 1735, uptime 7 days, 1:25:41

xqueue                                                  RUNNING    pid 6337, uptime 1:11:33

xqueue_consumer                                RUNNING    pid 6410, uptime 1:11:18

I have xqueue configuration as given below json files

/edx/app/xqueue/xqueue.env.json

{

    "LOCAL_LOGLEVEL": "INFO",

    "LOGGING_ENV": "sandbox",

    "LOG_DIR": "/edx/var/logs/xqueue",

    "RABBIT_HOST": "localhost",

    "S3_BUCKET": "sandbox-bucket",

    "S3_PATH_PREFIX": "sandbox-xqueue",

    "SYSLOG_SERVER": "localhost",

    "XQUEUES": {

        "certificates": null,

        "edX-Open_DemoX": "http://192.168.0.250:18050",

        "open-ended": null,

        "test-pull": null

    },

    "XQUEUE_WORKERS_PER_QUEUE": 12

}

/edx/app/xqueue/xqueue.auth.json

{

    "AWS_ACCESS_KEY_ID": "",

    "AWS_SECRET_ACCESS_KEY": "",

    "DATABASES": {

        "default": {

            "ENGINE": "django.db.backends.mysql",

            "HOST": "localhost",

            "NAME": "xqueue",

            "PASSWORD": "password",

            "PORT": "3306",

            "USER": "xqueue001"

        }

    },

    "RABBITMQ_PASS": "edx",

    "RABBITMQ_USER": "edx",

    "REQUESTS_BASIC_AUTH": [

        "edx",

        "edx"

    ],

    "USERS": {

        "lms": "password"

    }

}

And I am just adding a advanced blank problem which contains:

<problem display_name="Problem 6">
<text>
<p>Write a program that prints "hello world".</p>
</text>
<coderesponse queuename="edX-Open_DemoX">
<textbox rows="10" cols="80" mode="python" tabsize="4"/>
<codeparam>
<initial_display>
# students please write your program here
print ""
</initial_display>
<answer_display>
print "hello world"
</answer_display>
<grader_payload>
{"output": "hello world", "max_length": 2}
</grader_payload>
</codeparam>
</coderesponse>
</problem>

When I press on check button at studio it gives following error

Error checking problem: no external queueing server is configured.

Can anybody help me what steps should I follow to overcome this error?

[Huy Quang]

Ok, you need to start an external grader listen at port 18050 to handle request.
You can take a look at: https://github.com/huynq55/java-grader

Vào 19:28:24 UTC+7 Thứ tư, ngày 10 tháng mười hai năm 2014, Naresh Makwana đã viết:

[Naresh Makwana]

Thank you Huy Quang for the quick reply

I have followed the steps you have suggested but still I am not able to do this.

I have added a new queue named "java-queue": "http://localhost:1710" in xqueue.env.json

Then I copied the grader code you have given in /edx/java-grader

Then I also imported the course you have given and started the grader by sudo python JavaGrader.py

I am accessing the course at LMS. It gives me the following message when I am pressing the check button to submit my java code.

Your answer has been submitted. As soon as your submission is graded, this message will be replaced with the grader's feedback.

And shows "processing" circular progress continuously.

Need I mention the authentication details in any configuration?

How can I come to know my xqueue is listening or nor?

Also want to clarify that my production repository has not updated last since 3-4 months.

[Naresh Makwana]

Finally I am now able to submit the code and can get the response from the external grader.

The mistake that I made after following the steps suggested by Huy Quang was I forgot to restart the xqueue.

Thanks for the great support.

[Huy Quang]

Nice to hear this from you. Hope you can write your own grader soon, it's not hard :D

Vào 14:50:18 UTC+7 Thứ sáu, ngày 12 tháng mười hai năm 2014, Naresh Makwana đã viết:

[melvin bautista]

Hi can i ask some help. Im currently installed the edx and trying the code grader module, but when i try to paste my answer it tays:

Your submission could not be graded. Please recheck your submission and try again. If the problem persists, please notify the course staff.

what should i do?

thanks.

[shubham rawat]

Hi Huy,

I was looking at your java-grader, and it has helped me understand XQueues.Thank you for that.
I wanted to know if you were able to enable java syntax highlighting in the editor. syntax highlighting for python is working in my case, but it seems java,cpp are not supported (by which i mean the appropriate codemirror files not added).

[Sivakumar]

Hi Naresh,

Hope you are doing well. 

I followed the steps mentioned by you to setup the external grader. But still i am getting "Error: No grader has been set up for this problem." message in my CMS. I am not able to debug this. Can you help me to resolve this?

Looking Forward. 

Thanks.

[Darwin Marcelo Poveda Oyos]

Hi Naresha,
I have same problem: Stay the circle procesing, My question es How I must restrat the xqueue??

[dimitris agelakis]

I have come to a dead end with this thing, can somebody give me a hint on how to overcome my problem with the external grader?
Here is what I have done so far:

I have imported the sample course and also cloned the java grader from here:
https://github.com/huynq55/java-grader

I have edited the /edx/app/xqueue/xqueue.env.json to look like this:

{
    "LOCAL_LOGLEVEL": "INFO",
    "LOGGING_ENV": "sandbox",

    "LOG_DIR": "/edx/var/log/xqueue",

    "RABBIT_HOST": "localhost",
    "S3_BUCKET": "sandbox-bucket",
    "S3_PATH_PREFIX": "sandbox-xqueue",
    "SYSLOG_SERVER": "localhost",
    "XQUEUES": {
        "certificates": null,

        "java-queue": "http://localhost:1710",
        "open-ended": null,
        "open-ended-message": null,

        "test-pull": null
    },
    "XQUEUE_WORKERS_PER_QUEUE": 12
}

I have restarted the xqueue with the following command:

sudo /edx/bin/supervisorctl restart xqueue:

Seems that all the services are running normally:

$ sudo /edx/bin/supervisorctl -c /edx/etc/supervisord.conf status
certs                            RUNNING   pid 5412, uptime 0:10:20
edxapp:cms                       RUNNING   pid 1506, uptime 1 day, 19:59:30
edxapp:lms                       RUNNING   pid 1504, uptime 1 day, 19:59:30
edxapp_worker:cms_default_4      RUNNING   pid 1526, uptime 1 day, 19:59:29
edxapp_worker:cms_high_1         RUNNING   pid 1521, uptime 1 day, 19:59:29
edxapp_worker:cms_low_3          RUNNING   pid 1512, uptime 1 day, 19:59:30
edxapp_worker:lms_default_3      RUNNING   pid 1517, uptime 1 day, 19:59:29
edxapp_worker:lms_high_4         RUNNING   pid 1531, uptime 1 day, 19:59:29
edxapp_worker:lms_high_mem_2     RUNNING   pid 1513, uptime 1 day, 19:59:30
edxapp_worker:lms_low_1          RUNNING   pid 1523, uptime 1 day, 19:59:29
forum                            RUNNING   pid 1500, uptime 1 day, 19:59:30
xqueue                           RUNNING   pid 5409, uptime 0:10:20
xqueue_consumer                  RUNNING   pid 1499, uptime 1 day, 19:59:30

I run the JavaGrader.py:

sudo python /edx/java-grader/JavaGrader.py

And then I go to the course and try to test the grader...

When I run the one of the 3 problems from the sample course, i see that the grader seems to work correctly, it puts the data to the "xqueue" database in MySQL for example the following query:

select grader_reply from queue_submission;

returns something like this:

+------------------------------------------------------------------------------------------+
| grader_reply                                                                             |
+------------------------------------------------------------------------------------------+
| {"msg": "Good job!", "score": 1, "correct": true}                                        |
| {"msg": "Good job!", "score": 1, "correct": true}                                        |
| {"msg": "Good job!", "score": 1, "correct": true}                                        |
| {"msg": "Good job!", "score": 1, "correct": true}                                        |
| {"msg": "Good job!", "score": 1, "correct": true}                                        |
.....

(the rows increase every time I check the program from the sample course)...

But although the grader passes, and it puts the data into database, it NEVER returns the answer to the LMS.

The logs from:

/edx/var/log/xqueue/edx.log

are the following (after each check):

Mar 17 16:33:09 Mathesis1 [service_variant=xqueue][queue.consumer][env:sandbox] INFO [Mathesis1  1896] [consumer.py:321] - Processing submission from queue_name: java-queue, submission_id: 31
Mar 17 16:33:09 Mathesis1 [service_variant=xqueue][requests.packages.urllib3.connectionpool][env:sandbox] INFO [Mathesis1  1896] [connectionpool.py:191] - Starting new HTTP connection (1): localhost
Mar 17 16:33:09 Mathesis1 [service_variant=xqueue][queue.consumer][env:sandbox] ERROR [Mathesis1  1896] [consumer.py:185] - Could not connect to server at http://localhost:1710 in timeout=30.000000
Mar 17 16:33:09 Mathesis1 [service_variant=xqueue][queue.consumer][env:sandbox] ERROR [Mathesis1  1896] [consumer.py:403] - Submission 31 to grader http://localhost:1710 failure: Reply: cannot connect to server,
Mar 17 16:33:09 Mathesis1 [service_variant=xqueue][requests.packages.urllib3.connectionpool][env:sandbox] INFO [Mathesis1  1896] [connectionpool.py:191] - Starting new HTTP connection (1): localhost
Mar 17 16:33:09 Mathesis1 [service_variant=xqueue][queue.consumer][env:sandbox] ERROR [Mathesis1  1896] [consumer.py:185] - Could not connect to server at http://localhost:3180/courses/CUP/j101/2014_T12/xqueue/9/i4x://CUP/j101/problem/a2ed4ae1e2e94c278d92c494b2b738b6/score_update in timeout=5.000000
Mar 17 16:33:09 Mathesis1 [service_variant=xqueue][requests.packages.urllib3.connectionpool][env:sandbox] INFO [Mathesis1  1896] [connectionpool.py:191] - Starting new HTTP connection (1): localhost
Mar 17 16:33:09 Mathesis1 [service_variant=xqueue][queue.consumer][env:sandbox] ERROR [Mathesis1  1896] [consumer.py:185] - Could not connect to server at http://localhost:3180/courses/CUP/j101/2014_T12/xqueue/9/i4x://CUP/j101/problem/a2ed4ae1e2e94c278d92c494b2b738b6/score_update in timeout=5.000000
Mar 17 16:33:09 Mathesis1 [service_variant=xqueue][requests.packages.urllib3.connectionpool][env:sandbox] INFO [Mathesis1  1896] [connectionpool.py:191] - Starting new HTTP connection (1): localhost
Mar 17 16:33:09 Mathesis1 [service_variant=xqueue][queue.consumer][env:sandbox] ERROR [Mathesis1  1896] [consumer.py:185] - Could not connect to server at http://localhost:3180/courses/CUP/j101/2014_T12/xqueue/9/i4x://CUP/j101/problem/a2ed4ae1e2e94c278d92c494b2b738b6/score_update in timeout=5.000000
Mar 17 16:33:09 Mathesis1 [service_variant=xqueue][requests.packages.urllib3.connectionpool][env:sandbox] INFO [Mathesis1  1896] [connectionpool.py:191] - Starting new HTTP connection (1): localhost
Mar 17 16:33:09 Mathesis1 [service_variant=xqueue][queue.consumer][env:sandbox] ERROR [Mathesis1  1896] [consumer.py:185] - Could not connect to server at http://localhost:3180/courses/CUP/j101/2014_T12/xqueue/9/i4x://CUP/j101/problem/a2ed4ae1e2e94c278d92c494b2b738b6/score_update in timeout=5.000000
Mar 17 16:33:09 Mathesis1 [service_variant=xqueue][requests.packages.urllib3.connectionpool][env:sandbox] INFO [Mathesis1  1896] [connectionpool.py:191] - Starting new HTTP connection (1): localhost
Mar 17 16:33:09 Mathesis1 [service_variant=xqueue][queue.consumer][env:sandbox] ERROR [Mathesis1  1896] [consumer.py:185] - Could not connect to server at http://localhost:3180/courses/CUP/j101/2014_T12/xqueue/9/i4x://CUP/j101/problem/a2ed4ae1e2e94c278d92c494b2b738b6/score_update in timeout=5.000000
Mar 17 16:33:09 Mathesis1 [service_variant=xqueue][queue.consumer][env:sandbox] ERROR [Mathesis1  1896] [consumer.py:163] - Unable to return to LMS: lms_callback_url: http://localhost:3180/courses/CUP/j101/2014_T12/xqueue/9/i4x://CUP/j101/problem/a2ed4ae1e2e94c278d92c494b2b738b6/score_update, payload: {'xqueue_header': u'{"lms_key": "eb1915fccfad4b664377bf8e47e27d9e", "lms_callback_url": "http://localhost:3180/courses/CUP/j101/2014_T12/xqueue/9/i4x://CUP/j101/problem/a2ed4ae1e2e94c278d92c494b2b738b6/score_update", "queue_name": "java-queue"}', 'xqueue_body': '{"msg": "<div class=\\"capa_alert\\">Your submission could not be graded. Please recheck your submission and try again. If the problem persists, please notify the course staff.</div>", "score": 0, "correct": null}'}, lms_reply: cannot connect to server

Anyone knows what this "lms_reply: cannot connect to server" means and why it is happening?
And what is this "/score_update" at the end of the url shown in the logs?
How can I make the LMS accept the reply?

Any help would be appreciated as I am totally lost on this one!
And a thorough tutorial by the creators of the external grader on how to make it work would also be appreciated !

Thank you in advance guys, I hope someone has an answer :)

[dimitris agelakis]

I managed to find a solution for the newest OpenEDX named release (Dogwood) but NOT for Birch which is my current running platform.

I have created a simple problem and test using some of the https://github.com/huynq55/java-grader code but with some essential differences.

If anyone is interested, my solution can be found here:

https://github.com/dagg/OpenEdxExternalGrader

I will be glad to see some ideas on how I could make this work a bit more efficiently!

Thank you all.

[ambie]

hi Dimitris! How do you manage to resolve this issue? my problem is exactly the same as you have. 

I would really appreciate your reply. thanks

[dimitris agelakis]

Hi Ambie,

I have uploaded a detailed description of the steps I followed here:
https://github.com/dagg/OpenEdxExternalGrader
Follow these steps to create a simple external grader, and if it works you can use this method for more advanced problems as well.
If you have any questions about these steps please ask me again :)

Dimitris