Regarding some details of human perceptual evaluation

[zhengxue cheng]

Dear Organizer, 

As described in the task, we notice the image compression will be rated by human perceptual evaluation. Regarding some details, we're wondering:

1) Are the reference images available? Are there any instructions to guide the subjects which is the reference and which is compressed one?

2) When conducting pairwise comparison, will the distorted images from different codecs be placed side by side or displayed in a sequential order? Is it possible to give a simple snapshot or an example how the distorted images are displayed to subjects?

Thank you very much for your reply in advance.

Best regards,

Zhengxue

[gtoderici]

Dear Zhengxue, please find my answers inline.

On Sunday, January 31, 2021 at 10:28:44 PM UTC-8 sherrych...@gmail.com wrote:

Dear Organizer, 

As described in the task, we notice the image compression will be rated by human perceptual evaluation. Regarding some details, we're wondering:

1) Are the reference images available?

Yes. Humans will see the original, and be able to switch between the two compressed methods. Please see our dev kit which we just released.

 

Are there any instructions to guide the subjects which is the reference and which is compressed one?

The raters will know that the original is the target. The question is "choose the method that is closest to the original". We will provide further clarifications to raters, and we expect that if certain images are very far away from the original, the raters will choose the one they would prefer for their own photos.

 

2) When conducting pairwise comparison, will the distorted images from different codecs be placed side by side or displayed in a sequential order?

When raters compare images, they will focus on a single crop for all methods. The order of comparisons is both random, and also guided by an algorithm to speed up processing.

 

Is it possible to give a simple snapshot or an example how the distorted images are displayed to subjects?

We'll provide screenshots/video capture of the environment at the workshop. If you're curious, it'll be *extremely* similar to what's been described last year here: https://youtu.be/iXzgFrRWNEg?t=797

The main difference is that the raters will have the original displayed on the right hand side for reference.

[zhengxue cheng]

Dear George, 

Thanks for your clear explanation. I will check the newly released dev kit and video of last year.

Best regards,

Zhengxue

[Shangyin Gao]

Dear George,

Regarding the evaluation process, I wonder will the human rater zoom in 512x512 image patches to find some tiny difference between two methods?  

Best,

Shangyin

[George Toderici]

It's possible they might zoom. We will advise against it in the instructions. With that said the issue is that we will use a heterogenous set of raters, so we don't know for sure whether they'll follow instructions.

We also plan to use 768x768 patches. 

--
You received this message because you are subscribed to a topic in the Google Groups "Workshop and Challenge on Learned Image Compression (CLIC)" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/clic-list/II73Jft1wMQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to clic-list+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/clic-list/e687d227-677b-4b31-b407-0ae918b00eden%40googlegroups.com.