Understanding of the results of v3.0.7 when using 'side-by-side'

[Markus Loeffler]

Hello all,
I am having a hard time to understand the Q_JOD results in my test scenario.
This is my code:

ref  = im2single(imread('/Users/loeffler/Movies/0000000.tiff'));
test = im2single(imread('/Users/loeffler/Movies/0000001.tiff'));

ref_l  = pq2lin(ref);
test_l = pq2lin(test);

colorspace = 'rgb-bt.2020';
resolution = [3840 2160];
diagonal_size_in = 205;
viewing_distance_m = 3;

ppd = hdrvdp_pix_per_deg(diagonal_size_in, resolution, viewing_distance_m);
mode = 'side-by-side';
res = hdrvdp3( mode, ref_l, test_l, colorspace, ppd, { } );

We are viewing test footage in a grading theatre and trying to judge the visibility of dark noise before and after noise reduction. How can I detect if a change in the noise pattern is visible to the viewer or not?
The current test results give these values:

    P_map: [3164×4608 double]
    S_max: 0.2166
    P_det: 0.0998
    C_max: 0.2916
    C_map: [3164×4608 double]
        Q: 9.9466
    Q_JOD: 9.9878

We currently see no difference between the 2 images and I think a Q_JOD of 9.988 reflects that as well. How low has the Q_JOD value to drop to indicate a visible difference?
Can I use any of the other values as a better indicator?

Thanks in advance,
Markus

[Rafal]

Markus,

When using 'side-by-side' mode, I suggest that you check the value of P_det instead - it should be the probability of spotting the difference between the two images/frames. 

The 'side-by-side' mode was calibrated on this dataset: 

https://www.cl.cam.ac.uk/research/rainbow/projects/dpvm/

which extends on:

https://visibility-metrics.mpi-inf.mpg.de/

So the metric was calibrated (in that mode) to predict maps stored in P_map. P_det is the maximum value from that map. 

The parameters passed to the metric look fine. 

Best,

Rafal

[Markus Loeffler]

Rafal,
Thanks for your answer. I was running a quick check to see how the metric reacts to a change by increasing the saturation of the test image. To my surprise there is very little change in the resulting values although the difference between the 2 image is now obvious:

Testimage original:

    P_map: [3164×4608 double]
    S_max: 0.1668
    P_det: 0.0764
    C_max: 0.2269
    C_map: [3164×4608 double]
        Q: 9.9541
    Q_JOD: 9.9900

Testimage increased saturation:

    P_map: [3164×4608 double]
    S_max: 0.1481
    P_det: 0.0740
    C_max: 0.1985
    C_map: [3164×4608 double]
        Q: 9.9390
    Q_JOD: 9.9856

P_det is even going down. Am I doing something wrong or is that not able to be detected by the metric?

Best,
Markus

[Rafal]

Markus,

Can you post the pair of images so that I can reproduce the issue on my side? 

Because the P_det is so small, I suspect that the images are not scaled in absolute units:

https://hdrvdp.sourceforge.net/wiki/index.php/Frequently_Asked_Questions

BTW, SourceFoge pages are broken (but still legible) - I need to move the project  to GitHub. 

Best,

Rafal