Submission error. non int values row xxx ...

[김현우]

Hi, I recevied submission error code "non-int value in row x.." . 

but i check the file, it looks well. (don't have a non-int value) 

Our team both failed 2 submissions today, could you please confirm it? (It is a strange situation because the previously successful submission code does not work today.)

- case1. d37926a9-5ae6-4cd2-872c-61054072ac5d

- case2. 3cfda59f-c679-4e3b-a1dc-26ba67d560b8

Thank you. 

[TrueRank]

We've seen multiple submissions throw errors (always the "non-int value in row x... " pattern)

They work fine on re-submission; is is possible to stop counting these against the daily total?

(We're not aware of any ways this can really be exploited--either the csv uploads correctly or not, with the not seemingly happening to dozens of teams)

[Noelia Leung]

the same issue

1. 1b1360c0-3c1b-488a-be14-22c2de425eda

2.  6d956e53-ce07-42a3-a1f7-9d1113da91ad

[Benedikt S]

Yes, I got the same error right now

[Nick Landia]

Hi,

we have some developer time on Friday to make it so that it retries the same file on error without requiring a resubmission. The issue is somewhere in the evaluation infrastructure, probably something around compute cluster machines having network issues, not in the initial file upload (the files I can see on our system are fine). As a side note, there are 5 tries allowed per day for the upload of the "final" predictions. We can also retry those runs manually after the deadline in the unlikely event that your last upload has an error and you don't have time or any remaining tries to upload another one after.

Best,

Nick

[TrueRank]

We saw a submission take an hour to score then yield an implausible score (roughly half the score of another submission on which it is based, with a 99% correlation in reciprocal ranks, ~90% correspondence on all top-n windows);  while there might be an ordering where this is possible, just noting this in case the retry is scoring with partial files.

[Benedikt S]

Hello Nick,

thanks for the response. It is good to know that the final submission will not be affected for it and/or it can be rerun.

We are in the final phase of the competition in which we have only 3 submission per day. Today, two of the three submission failed due to this error. 

[Sara Rabhi CA]

I got the same error  for submission d8301881-d228-4168-b460-49829ac9db5c

[Chris Deotte]

Hi Nick,

When you have time on Friday, can you please rerun erroneous submission f5299940-1c2d-415c-a949-939d3ba806ee 

Thank you

Chris

[김현우]

Hello Nick, 

can you rerun submission 8d864e81-56e1-4b2a-9855-4102a827506e ? 

Thanks 

2022년 6월 9일 목요일 오전 9시 25분 34초 UTC+9에 Chris Deotte님이 작성:

[Nick Landia]

Hi,

Thanks for reporting the half-score case, that is useful to know.

For the fastest result please try re-uploading if you have submission counts left. Tomorrow we will focus on having something that automatically retries a submission multiple times on error, and it sounds like we need to have some further verification that the files are being transferred correctly. When that is in place you will be able to re-upload and not lose a submission slot due to errors our side. I am not sure we will get to re-run individual files tomorrow but will let you know.

Cheers,

Nick

--
You received this message because you are subscribed to a topic in the Google Groups "RecSys-Challenge-2022" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/recsys-challenge-2022/O_QyAUcdBWw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to recsys-challenge...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/recsys-challenge-2022/1be0e259-b4c3-449a-a450-787c34ee6fa4n%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[TrueRank]

We're seeing random scores on exact replications (and also failed submissions still)

c7083d0d-e9b3-416a-94aa-3d84e2a394f3
blend_g9_replication
0.03993807273961077
09 Jun 23:13
4029

dd65d0ba-29d9-4a4e-9a32-c5e6b8250296
blend_g9
0.1697425939913802
08 Jun 03:29
2062

[Lorenzo Mainetti]

I got the same error. In the last days we have already wasted 4 submissions for this reason...

[Nick Landia]

@TrueRank those two files are not identical. File sizes are slightly different, and looking inside the files the content is not the same.

We're currently looking into the "not an int" error.

[TrueRank]

Those are all ~99% correlated (very minor ranking shifts); sometimes they score 0.04, sometimes 0.17.  Mathematically impossible; also the submissions labelled 'blend_g9' are exact matches, again 0.17 and 0.10 for the same file.

Unless the public leaderboard is on a sample of the first couple sessions, or things are getting cut off, these scores should be near-identical (and definitely identical for the exact re-submission).

[Nick Landia]

Ok, we have finally found the cause of the intermittent failures, it was an intermittent cutting off issue, and it can also cause random scores in some cases (so you were right @TrueRank).

This is now fixed and future evaluations should go through fine. However, to make sure everything is correct and the random scores are eliminated for everyone we are provisioning some additional compute power and are re-running all evaluations. This work is in progress and some of your results might be updating over the next hours, so bear with us for a bit, I will message again when it's done or I have more information.

[Nick Landia]

This has all gone through. All existing files have been re-evaluated properly and the system should not fail any more.

There are no changes at the top of the leaderboard but some of your evaluations might have changed in score if they were affected by the intermittent issue. I have checked a couple of the reported cases where the same file failed and then succeeded or the same file gave different results, they now all have a result and it's the same number.

Sorry for the inconvenience and have a good weekend!

Nick

You received this message because you are subscribed to the Google Groups "RecSys-Challenge-2022" group.
To unsubscribe from this group and stop receiving emails from it, send an email to recsys-challenge...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/recsys-challenge-2022/cf05f147-968e-4909-9097-fb1c67ebb91an%40googlegroups.com.