Database update issue?

[Madhavan Mukund]

I am trying to migrate a contest to a server with more CPU/memory
resources. I exported a dump on the old server with the -G option

-G, --no-generated don't export data and files that can be automatically
generated

I have successfully imported this dump on the new server. However,
every time I start cms, the server starts to compile all the
submissions from scratch and then stops after one round of
compilation.

I checked the sandbox files and there are valid executables being
generated, so actual compilation is happening. However, i don't see
any update to the status of the evaluations in the database, and the
admin interface continues to show "Compiling ..." against all
submissions.

There are no errors/worning in the CMS logs. I see a few messages of
the following form in the postgres log,

2018-04-08 17:09:58 UTC [17924-1] cmsuser@ioitcdb LOG: could not receive data from client: Connection reset by peer

but this is not a continuous problem and the number of such lines is
far less than the number of submissions that are being compiled.

I am unable to find any explanation why this is happening. It seems
like the database is not being updated, but I'm not sure why.

--Madhavan

[Stefano Maggiolo]

Hello Madhavan!

I tried exporting & reimporting with -G and couldn't reproduce (as I expected, at first sight this doesn't seem to be related to the dumping process). 

Some random questions:

- is this CMS 1.3.x?

- do you see anything interesting in the logs, especially ES's?

- can you link to a screenshot of (or copypaste) a submission's page in AWS?

[Luca Wehrstedt]

Another possibility is that there's some programming error in ES, like an unexpected exception being raised, that causes the database commit to be aborted. This is exactly what was happening when the compilation output contained NULL characters. The code doesn't fully take this possibility into account thus logging is poor, but something should still be shown...

--
You received this message because you are subscribed to the Google Groups "Contest Management System (support)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to contestms-supp...@googlegroups.com.
To post to this group, send email to contestm...@googlegroups.com.
Visit this group at https://groups.google.com/group/contestms-support.
To view this discussion on the web visit https://groups.google.com/d/msgid/contestms-support/9a8d0f14-b4f2-48bb-b4df-8edc6652a8d2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Madhavan Mukund]

I am continuing to get this problem when running cms on a vm under
xen. I have tried out cms v1.3.1 on two separate xen host servers: on
each xen server I have tried it out on xen clients running both Ubuntu
16.04 and Debian 9.3, so four independent test installations.

What happen is that at some point a db write fails due to an illegal
character (UnicodeEncodeError). After that the whole evaluation
system stops updating the database and everything comes to a halt
after the current cycle of jobs finishss, without proceeding to the
next step (becuase the db status is not updated). I've attached the
relevant part of the log.

I also set up v1.3.1 on a regular desktop running Ubuntu 16.04 on the
same contest data and this ran without any problems or errors.

I don't understand why this problem is occurring on the xen vms and
not on the desktop. Any suggestions?

--Madhavan

[Madhavan Mukund]

Forgot to add another clue:

This happens on a contest that I dumped using -G on another cms
instance and imported here. If I export the same contest without -G,
I get a similar Unicode error in DumpImporter.py and the import fails.

--Madhavan

[Madhavan Mukund]

I isolated one of the offending jobs. It was a user test where the
source code appears to be encoded in an invalid ascii format.

madhavan@ioicms:~/db-error$ file RajasAndHisVisaIssues.cpp
RajasAndHisVisaIssues.cpp: C++ source, Little-endian UTF-16 Unicode text, with CRLF line terminators

cms is able to display the source code and download it, but when it
comes to storing the output of this job in the database, it generates
a UnicodeEncodeError as described in my previous mail.

I've attached a tar.gz wih the job (source code and test input)

--Madhavan

[Luca Wehrstedt]

It appears you are hitting on a problem similar to:

https://github.com/cms-dev/cms/issues/888

For which we added a workaround in:

https://github.com/cms-dev/cms/commit/a0701c4a9df588356bb9b5ba9dddfd13df0d57e1

And then backported to v1.3.

https://github.com/cms-dev/cms/commit/3ef129252ba4e898009ed57f2634469816acdec9

In that case we had a source file which contained a NULL character (unicode codepoint 0). That file was correctly stored but upon compilation (which failed) portions of the data were printed by the compiler to stdout, including the one with the NULL character. When ES tried storing that compilation text into the DB an exception was raised. The "workaround" consists in the exception only affecting the single submission it comes from, not the whole batch.

Here, maybe, a similar situation is occurring. Perhaps the exception is being raised at a different stage (when parsing the files, not when committing the transaction) and thus it isn't caught by the fix we introduced.

Unfortunately I am currently unable to investigate further.

--
You received this message because you are subscribed to the Google Groups "Contest Management System (support)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to contestms-supp...@googlegroups.com.
To post to this group, send email to contestm...@googlegroups.com.
Visit this group at https://groups.google.com/group/contestms-support.

To view this discussion on the web visit https://groups.google.com/d/msgid/contestms-support/20180415120738.2n3sjgdvhip64eqv%40cmi.ac.in.

[Luca Wehrstedt]

I guess a traceback, if you have any, would help in figuring out where that exception comes from.

[Madhavan Mukund]

Thanks for the replies. I sent another mail with one of the offending
files. The situation indeed seems to match the one you mention. What
further traceback could I provide that would be helpful?

--Madhavan

[Stefano Maggiolo]

I'll try to reproduce soon on my side, I think it should happen here too. Thanks Madhavan, this is an interesting bug and very useful to have discovered now and not during a competition! :)

If I need more info I'll ask you, thanks.

[Madhavan Mukund]

Something that occurred to me --- this error is on a user test, not a
regular submission. Is it possible that a separate fix is needed for
user tests vs regular submissions? I am retrying the contest after
deleting user tests from the database. If this goes through smoothly,
I will also try to submit this code as a regular submission and see
what happens.

--Madhavan

[Luca Wehrstedt]

Sorry, I missed that in your previous email you attached a log excerpt. I looked at it and it appears that Stefano's commit (the one I linked before) does actually play a role (as is shown by the second line of that excerpt). Maybe it isn't working as intended? Is there anything interesting printed after the excerpt that you sent? A fuller log may be useful (send it privately if it contains confidential information).

It is also weird that the error is about decoding using the ASCII codec, as we try to use UTF-8 everywhere (this may not make any difference in your case as the source file is UTF-16). Are you using Python 2 or 3?

[Madhavan Mukund]

I've attached the full log of that session of cms. (Nothing
confidential here!)

I'm using python2. (I just followed the default install instructions
at https://cms.readthedocs.io/en/v1.3/.)

--Madhavan

[Madhavan Mukund]

I'm running it "non-installed" using virtualenv (Method 2). But I get
the same error when I install it globally (Method 1).

--Madhavan

[Madhavan Mukund]

I deleted all the user tests and restarted cms. The same error crops
up with regular submissions that are not in valid format (e.g one user
has submitted a pdf file for compilation). So I get this problem
regardless of whether it is a user test or a reglar submission.

--Madhavan

[Stefano Maggiolo]

I tried to reproduce, but only get good responses from CMS. I tested on 1.3.1 with Python 2, using both submissions and user tests. All the time I got compilation failed, which is the correct response. My compiler's output was in UTF-8, as far as I can tell, and without null characters (safe to say, otherwise I would get an error when storing in Postgres). I also tried with -finput-charset=UTF-16LE, but I didn't manage to get it to compile.

Interesting: if I inject a null character in the input, g++ says "batchstdio.cpp:1:1: warning: null character(s) ignored".

During all of this I manage to find an unrelated bug in CWS' UI, so at least it wasn't time wasted :) I'll take a look at the full log now, as a last attempt to understand what was going on.

[Stefano Maggiolo]

So looking at the logs it seems that at least the fix in 1.3.1 to avoid blocking the evaluation when something like this happens works, so there's that.

I tried a few more things, like explicitly outputting a non-ascii char from compilation, but that was also handled correctly. 

Madhavan, if you want to spend some more time on this, it is still interesting because it seems different from the "null character in postgres". I'm pretty sure the problem is the compilation stout or street, it would be nice if you can take those from a failed sandbox and attach them here.

I start to suspect this has to do with some locale differences between my system and yours...

[Madhavan Mukund]

Dear Stefano,

I'm pretty sure the error comes when writing the result to postgres,
not from the compiler. I did save the sandbox a few times and it
showed sensible responses from the compiler. I'll do it again and
send you the sandbox. Also, on the 8889 admin interface I get
messages saying, approximately, "Unexpected error when writing to
database". I will run this again and send you the sandbox contents.

Also, I get this error on installations running on vm client machines
under Xen (I tried both Ubuntu 16.04 and Debian 9.4 clients, on two
different xen host servers). I also installed 1.3.1 on a standalone
desktop running Ubuntu 16.04 and it did not have this problem. So I
am quite mystified as well.

--Madhavan

[Stefano Maggiolo]

Thanks, the sandbox content will be helpful (I wasn't very clear: yes, the error comes when writing to the db, but I think the field giving problem is either compilation_stdout or compilation_stderr).

[Madhavan Mukund]

I haven't had time to check the sandbox and send it to you, but I have
CMS running on regular desktops for our selection+training camp and
I'm not facing this issue. So it seems to be peculiar to the
configuration I was using (Debian 9 under Xen).

I have another issue that I will report in a separate mail to not mix
up threads.

--Madhavan

[Stefano Maggiolo]

Thanks, if that's the case it's really weird. If you stumble upon it again let us know.