Is there a best way to get absolute XYZ (nits) from camera YUV?

[Richard Kirk]

I am trying to use a phone as a colour measuring instrument.

This is not a colour science question. I know the any conversion will be approximate, and I will get different results for different illuminant spectra. However, I hope if I can get close enough, the user can use their eyes to fine-tunead the result.

I am getting proxy images previews from the camera. I am averaging the YUV values from the middle 1/5th square of the view. I tried turning off white balance, but the image came out very green, so I set it to daylight, as most monitors are D65. I could convert from Rec.601 YUV to RGB, and linearise the RGB assuming the sRGB tone curve. I read the exposure time and the sensitivity. I scaled the result by 250.0 / (sensitivity * exposure time in seconds) and I am getting XYZ values that look sensible.

I fixed the white balance to daylight because I did not want the white balance in the region I was measuring to be affected by what lay outside this region. Might I be better off getting the white balance settings and correcting?

The 250.0 constant was set by hand for my phone. Is there a value I ought to be using? Or will it vary with focus, lens zoom, and other parameters? The light level does seem to decrease when I put the camera very close to a display

I know every camera may be different, but I would be grateful for any general hints that might save me some time.

Here's a bit of background, which may show what I am aiming for...

I have done the other half of the job on the display. I adjust the screen gamma and offset (it does not always for the sRGB defaults). If I put up a digital Macbeth image on a display I trust, and get a visual match for the red, green, blue, and a grey on the display, I can calibrate the display white and primaries (again, not quite sRGB). Then, if I match a patch of colour on my phone display, I can read off the XYZ colours. It is not as good as my spectrometer, but it works, and maybe I can make it better. I imagine I may have to put the same sort of effort into calibrating the camera, or have it measure its own display in a mirror, or something. But a good set of starting values would help. 

[Eino-Ville Talvala]

Hi Richard,

With YUV, you can only get approximate values, since the processing from raw sensor values to YUV, in a modern phone, does many nonlinear transforms. (lens shading adjustment, local tone mapping, 3D color lookup tables, etc).

That said, it probably won't be terribly far off.

The other factor you need to account for is the device's lens aperture.  The ISO sensitivity standard is defined so that two cameras with identical settings and the same sensitivity setting, will get images of the same 'brightness', more or less. So the sensitivity accounts for differences in pixel efficiency, A/D conversion, and so on.  But you have to account for all the other standard settings that affect the amount of light collected, which include exposure time and the lens aperture.  Zoom doesn't matter unless it changes the aperture.

Also note that the YUV images are in the JFIF colorspace, which is Rec.601 with full-range values (https://www.w3.org/Graphics/JPEG/jfif3.pdf, page 3).

White balance probably doesn't matter significantly here, either, given all the other YUV processing already being done.

To do this with substantially more accuracy, you'd need to start with RAW buffers, which CameraX doesn't currently support (but are often available from the hardware via the camera2 API). Then WB is irrelevant, since no processing has been done on the pixels you receive.

Beyond that, working out the correct conversion constant from normalized sensor readout value to illuminance recorded by the camera pixels, to the luminance emitted by the screen, should be a one-time calculation, but it's not something I have handy as a derivation.

Good luck!

- Eddy

--
You received this message because you are subscribed to the Google Groups "Android CameraX Discussion Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to camerax-develop...@android.com.
To view this discussion on the web visit https://groups.google.com/a/android.com/d/msgid/camerax-developers/1203336b-7262-450a-873a-ce9ff5fb01afn%40android.com.

[Richard Kirk]

Thanks. I have some early results. It would seem that the display white is something like 80 nits on my phone (a Samsung S7) when  automatic brightness control is disabled. The white for the camera was determined by the RGB, linearised by the sRGB tone curve, has to be scaled by something like 80 nits / (s * t) where s is the sensitivity, and t is the exposure time in seconds. This is not an exact fit, but it seems a likely sort of thing to do if we want to get the display to match what the camera sees. Cameras are also naturally linear devices, and we can vary the integration time linearly. It also seems that the camera data is a fair fit to the sRGB tone curve. If anything, I am having more troubles with the display - I can calibrate for the actual primaries, and the tone curves do not seem to fit the standard gamma. If I change the gamma and offset values, I get closer, but I still have the shadows coming out about 2x too light at 20% RGB, so there is something three-dimensional happinng in there..

My provisional conclusion seems to be that the camera is measuring colours in a convincing way. I would not expect this to be accurate when a camera is measuring a display, with separate RGB primaries, but it is good enough for the job, and I am having more trouble getting predictable colour values from the display.