Announcing the Perceptual Challenge Dev Kit

[George Toderici]

Hello CLIC participants,

With apologies for the delay, we are happy to announce the development kit for the perceptual challenge. With this development kit you will be able to write a thin wrapper around any perceptual metric you might have and test how well it works (locally).

Step 1: Make sure you  clone our development repo https://github.com/fab-jul/clic2021-devkit.git so that you can get the latest scripts.

Step 2: Read the information here. You will need to download a large ZIP file with the PNG files (crops) and oracle (we provide the full oracle for the validation set), and validation input file.

Step 3: We provide a very simple baseline (PSNR RGB) that should make it easy for you to get started.

If you have any questions (I am sure there will be many, as this was a rather rushed release), please let me know.

George

[Abhijay Ghildyal]

Hi George,

I hope this message finds you well and safe. I wanted to know more about the scores/ratings in oracle.csv. 0 means that A was preferred by humans while 1 means that B was preferred. Based on that, the validation accuracy of LPIPS seems to be worse than that of PSNR. Upon manually checking the images in the validation set it seems like the oracle ratings are exactly opposite, i.e., 0 is 1 and vice versa. Please let me know if this is the case. It is possible that I may have misunderstood the scores. Does the oracle contain ratings provided by humans, or are they just dummy values given for code preparation?

In the final submission, will we submit models that will run on a server at your end or just a csv with scores like the psnr.csv given in the devkit?

Thank you

Regards,

Abhijay

Graduate Student

Portland State University

[Lucas Theis]

Hi Abhijay,

The ratings in oracle.csv were indeed flipped and should be updated soon. They are human ratings and not just dummy values.

Regarding submissions, we will collect CSVs via our website but you do not need to upload the model. You can now test this by uploading a CSV targeting the validation file (the corrected oracle.csv):

http://compression.cc/submit/

Unlike our other challenges, creating a submission during the validation phase is optional. We will still accept all submissions to the perceptual metric track during the test phase.

Lucas

[gtoderici]

Hi Abhijay,

We've updated the CSV file, and added it to the github directly. We also updated the zip file to contain the correct CSV as well, but it's unlikely you'll want that since it's a large download.

For now,  you can get the correct oracle csv at:

https://github.com/fab-jul/clic2021-devkit/blob/main/data/oracle.csv

Please keep in mind that while we found this bug, and we thank you for reporting it, it's not affecting anything prior to the public leaderboard being released (wheew!).

The leaderboard is for verification only in any case because the final one will be based on the actual submissions for the compression track/human raters' ratings which we haven't gotten yet, since the compression challenge hasn't ended yet.

George

[Abhijay Ghildyal]

George and Lucas,

Thank you for the update.

--
You received this message because you are subscribed to a topic in the Google Groups "Workshop and Challenge on Learned Image Compression (CLIC)" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/clic-list/0-kyGtecLhU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to clic-list+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/clic-list/b74bb04b-9ab8-4b1e-af3a-63c0bb468139n%40googlegroups.com.

[zhengxue cheng]

Hi George and Lucas,

Thanks for your update. It seems only CSVs are expected from participants, but I would like to know more about test phase.

If the model uploading is not needed, does it mean that it will be allowed for participants to conduct a local human evaluation or create a CSV file with manual check for a high accuracy when the test sets will be released during the test phase? (although this might involve a lot of labour work and very time-consuming...)  Maybe I misunderstand some parts, and I'm looking forward to your reply.

Thank you,

Zhengxue

[George Toderici]

Hi Zhengxue,

Nothing is finalized yet, but we were considering a very short amount of time for the test phase. Let's say that once we release the test data, it would be required to upload the CSV within 24-48 hours.

In order to mitigate the possibility of using human labor to perform the task, we'll make sure that the number of images (as you can see we're using crops) will be high enough that it wouldn't be cost effective for the participants to go that route. Moreover, in order to be eligible for a prize, we'll require a writeup that describes the method.

We've considered running the perceptual task just like we do the compression challenge, but we felt that since the timeline is so tight, it might be better to run in an easier mode for the participants. Next year we'll use the usual way (running a binary in a Docker environment).

George

--
You received this message because you are subscribed to a topic in the Google Groups "Workshop and Challenge on Learned Image Compression (CLIC)" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/clic-list/0-kyGtecLhU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to clic-list+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/clic-list/b7373264-56e4-4d80-8592-46bb28367493n%40googlegroups.com.

[zhengxue cheng]

Hi George,

Thanks for your explanation. I understand. 

Yes, it's reasonable and indeed makes it easier for submission. 

Best regards.

Zhengxue