changes between 3.0.6 and 3.0.7 using 'civdm'

[Stephen Wasilewski]

Hello,

I am using the civdm metric to compare various TMOs (conditioned to an output VR headset) to reference HDRs (in real world luminance). I initially ran the analysis with version 3.0.6 and just noticed a new version was posted. Upon rerunning the analysis, it appears that the loss and amplification now have minimum non-zero values  of 0.25. I had been using a 10% cutoff to judge maintenance, but am wondering about the justification of using 25% and if that is the safest threshold to use. Attached are two plots showing the distribution of detection probabilities (negative is loss) across my scenes using the two different versions of the code with the same input:

compvr = hdrvdp3('civdm', tmooutvr, refhdr, 'rgb-native', ppdvr);

where:

tmooutvr is the tone mapped image multiplied by the VR headset response (measured)

refhdr is the source hdr (in real world luminance)

Thank you very much!

Stephen

[Rafal]

Stephen,

The predictions will change between 3.0.6 and 3.0.7, but it should be only a moderate change. 

I am not sure if I fully follow your plots - Is it the distribution of values returned in `res.civdm.loss` and/or `res.civdm.ampl`? What do the negative probabilities denote? Is it for a single image or a set of images? 

Could you please send me a minimal example, perhaps using the `nancy_church.hdr` in the examples folder, showing the problem? 

Thank you,

Rafal

[Stephen Wasilewski]

Hi Rafal,

Thank you for your quick response! Sorry that I did not describe the plots very well. The plots show the inverse CDF of contrast amplitude and loss for the entire image (I set loss to negative, the sort all the data). So for example, reading from the left side of the plot 20% of the "video" image has contrast loss >= 0.6, and reading from the right side of the plot 20% of ward97 has contrast amplification of >=0.55.

This difference is reproducible in the compare_hdr_vs_tonemapped.m example. After running compare_hdr_vs_tonemapped.m to examine the mai11 result:

clf

sorted = sort(reshape(res{1}.civdm.ampl.',1,[]));

plot(sorted);

for kk=1:length(I_tmo)

fprintf("average ampl: %.03f\n", mean(res{kk}.civdm.ampl, 'all'));

end

with 3.0.6:

average ampl: 0.135

average ampl: 0.115

average ampl: 0.147

with 3.0.7:

average ampl: 0.063

average ampl: 0.053

average ampl: 0.085

I actually have no way of knowing which is right and which is wrong. I just know they are different. I assumed you programmed in some minimum detection threshold around 0.25, but could not find the difference in the code. It also seems to lower the probabilities across the range (hitting a vertical asymptote at 0.25):

with 3.0.7:

numel(res{1}.civdm.ampl(res{1}.civdm.ampl>0 & res{1}.civdm.ampl<.25))

ans =

      0

but with 3.0.6:

numel(res{1}.civdm.ampl(res{1}.civdm.ampl>0 & res{1}.civdm.ampl<.25))

ans =

      616600

I downloaded 3.0.6 from sourceforge on march 9th. I downloaded 3.0.7 on may 5th. I'm running on apple silicon so had to compile the mex libraries, but otherwise made no edits to the directories. 

best,

Stephen

PS I emailed you from my @epf.ch account, but didn't note your anti-spam instructions so I assume my message got eaten, if it did not, sorry for the spam?

[Stephen Wasilewski]

It appears the difference is introduced by changes to util/hdrvdp_lpyr.m. Using the 3.0.6 version of this file with the rest of 3.0.7 the two results between versions are quite similar (with the small differences you suggested would be there)

This seems to be the key difference:

20a21

>             ms.base_ppd = 2^ceil(log2(ppd));

24a26

>             ms.band_freqs = 2.^-(0:(height)) * ms.base_ppd / 2;

26,29c28,31

<             % Frequency peaks of each band

<             ms.band_freqs = [1 0.3228*2.^-(0:(height-1))] * ms.ppd/2;            

<                         

<             ms.P = laplacian_pyramid( I, height+1 );

---

>             % Resample to fix the frequency of the bands

>             I_res = imresize( I, ms.base_ppd/ms.ppd, 'bilinear' );

>             

>             ms.P = laplacian_pyramid( I_res, height+1 );

36a39

>             I = imresize( I, ms.img_sz ); 

For the compare_hdr_vs_tonemapped example 3.0.6 has band frequencies:

32    16     8     4     2 

3.0.7 has band frequencies:

26.3612    8.5094    4.2547    2.1274    1.0637

If I had to guess, this is causing some issue on reconstruction causing the probabilities to get clipped.

[Rafal]

I am sorry for the late answer.

>             ms.band_freqs = 2.^-(0:(height)) * ms.base_ppd / 2;

26,29c28,31

<             ms.band_freqs = [1 0.3228*2.^-(0:(height-1))] * ms.ppd/2;        

Yes, I forgot about the change in the peak of spatial frequencies (this should have been mentioned in the ChangeLog.txt). This was a long-standing bug, which affected only civdm. The frequencies used in 3.0.7 are correct.

civdm is a "qualitative" metric as it has never been calibrated with ground-truth data. It should predict something correlated with the visible reduction or amplification of contrast, but we have never had a chance to prove that it correlates well with visibility measured in a psychophysical experiment. Because of that, the values reported in both versions are fine as long as you use the same version to show the relative performance of different tone-mapping operators. Please mention the version you used when reporting the results. 

Best,

Rafal