Technical questions from competition organisers

[Polina Turishcheva]

Dear CodaLab community,

I am new to codalab and trying to setup a competition here. However, our submission files are planned to be relatively big (~2Gb, after forcing them to be in parquet format). Therefore, I have 2 questions:

- What is the maximum submission size for the codalab? And is there a way to extend it? We are going to provide our own amazon machine for computing the results and s3 to store them.

- We plan to evaluate only the best submissions on the private leaderboard. Is there a way to set up the automatic deletion of the previous submissions, which are worse from the team?

I also have one more question about organisation. There is a way to limit the amount of submissions per user per 24h with the max_submissions_per_period flag. Is there a way to do the same per team?

Thanks a lot in advance!

Yours sincerely,

Polina

[Polina Turishcheva]

According to this webpage up for 10Gb should fit and I can automatically remove all submissions apart from the latest and best ones. However, I am still not sure how to move only the best one on the next stage (not latest one, like in the Iris demo challenge). Does anyone know the answer for this?

Thanks a lot in advance!

Yours sincerely,

Polina

--
You received this message because you are subscribed to a topic in the Google Groups "codalab-competitions" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/codalab-competitions/KSalHy_8M6Y/unsubscribe.
To unsubscribe from this group and all its topics, send an email to codalab-competit...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/codalab-competitions/243a7911-e2c3-4c38-86ca-8d58e5e1b719n%40googlegroups.com.

[Adrien Pavao]

Dear Polina,

The max submission size for the codalab.lisn.fr instance is 300 MB. This value can be redefined for each competition by the administrators of the platform (including me).

To organize a competition that requires big submissions, there are two approaches right now:

- Discussing with the admins to have your limit increased but a reasonable use of the storage (easy way),

- Setting up your own instance of CodaLab (harder way but with more freedom).

In the first case, we'll need to know a bit more about the anticipated size of your competition: the number of participants, the size of each submission, etc.

An interesting option that may help is the "keep only last and best submission", which will delete all submissions except the last and best of each participant, ensuring that the storage won't be too cluttered by the competition.

>>> Is there a way to do the same per team?

I don't think so. Of course you can put this in the rules of the competition and encourage the participants to follow the rules (typically by telling them that teams which don't follow the rules can't qualify for the prize).

Feel free to ask any questions you may have. On my side, I will discuss your project with the other administrators of the platform.

Best regards,

Adrien Pavao

CodaLab Team

--
You received this message because you are subscribed to the Google Groups "codalab-competitions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to codalab-competit...@googlegroups.com.

[Polina Turishcheva]

Dear Adrien,

We tried to set up our own instance of codalab with s3 bucket. However, we faced the issues with the amazon signature - https://github.com/codalab/codalab-competitions/issues/3374 . Could you please let me know why this could be a problem? 

Thanks a lot in advance!

Yours sincerely,

Polina

[Adrien Pavao]

Dear Polina,

>>> So if we would take 30 teams, 2 submissions per team, it pretty fast explodes to 300Gb

Indeed it is quite heavy but I think it is something that we can allow. I still need to have some input from other members of my team.

>>> We can provide s3 buckets to handle this storage, that's fine. Same for the manager queue, we can add an amazon EC2 worker for this. Can we do it without running our own instance?

You can add compute workers (computers running the submissions) to your competition on codalab.lisn.fr, without having to run your own instance. However, for storage, it is currently not possible to add your own volumes / VM to your competition to increase the storage space.

>>> If we want to setup our own instance of codalab, what does this mean? I guess that I probably need to configure my own docker container, with s3, etc. But what about frontend and domain?

Indeed, setting up your own instance means having your own clone of the platform CodaLab. It is perfectly fine on our side as the project is free and open-source, but it is clearly more complex to put in place: you need a storage, a domain name, to setup everything, and your competition won't appear in the listing of the public instance (as it would be published on your own version of the platform).

>>> does  'keep only last and best submission' work per user or per team as well ('Enable Competition level teams' is set to true)? 

>>> does 'force_best_submission_to_leaderboard' work per user or per team ('Enable Competition level teams' is set to true)?

I am not sure but I think these features, unfortunately, currently only work per user. Let me know if you know more after performing some tests.

>>> I want to have 2 stages in the competition [...]

This looks like a quite classical setting. And yes, the Iris example is up-to-date and functional. I am not sure about the auto-migration as I never used this feature as an organizer. One way to handle this setup is to have two different versions of the scoring program, one for each phase. Indeed, you can set a different scoring program for each phase, either in the competition.yaml file or using the editor.

>>> While I would still prefer to work without running our own instance, I have started looking into how to do it. [...] Did I get it right?

Yes, it is a good summary of the procedure. The docker-compose command sets everything up automatically, and I don't think you need to use the passwords, except for precise procedures that may not be required. Of course, as discussed earlier, setting up a whole instance requires some effort and may be too much for just running one competition (although some organizers have done it in the past).

>>> Is there a way to match our competition with the codalab.lisn.fr webpage?

If you upload your competition on your own instance of the platform, it won't be available on codalab.lisn.fr. A work-around is to upload a competition here with a link to the actual competition on your platform.

>>> for the CODALAB_SITE_DOMAIN, can I somehow give it a subdomain?

Yes, I assume this would work.

>>> We tried to set up our own instance of codalab with s3 bucket. However, we faced the issues with the amazon signature - https://github.com/codalab/codalab-competitions/issues/3374 . Could you please let me know why this could be a problem?

I've seen that you were able to solve this problem. By the way, it is also possible to set up your own storage on any VM using MinIO.

I will discuss with my team the possibility of hosting your competition on the public instance. In the meantime, feel free to give us an update about the setup of your own instance. It is always interesting for us to have feedback about this.

On a side note, I wanted to let you know that we are currently developing a modernized version of CodaLab called Codabench. It is still in beta right now, but quite functional, so you might be interested in playing around with it. Letting organizers link their own storage resources to the platform is a feature we are going to implement soon on Codabench.

Best regards,

Adrien

[Adrien Pavao]

Dear Polina,

After discussion with my team, we agreed to host your competition on codalab.lisn.fr.

Once your competition is uploaded, please share the URL (the secret URL as the competition will be private I guess) with us so we can increase your submission size limit, and check that the "keep only last or best submissions" option is enabled.

Note that everything can be edited after uploading the competition (scoring program, data, documentation, etc.) except the leaderboard columns (it is a known bug).

So be careful with the leaderboard, or you might need to re-upload the bundle from scratch.

Let me know if you have any questions.

Best regards,

Adrien