Optimising image processing for ZXing

[Tim]

Hi
I'm an imaging engineer responsible for optimising image processing
chipsets in mobile handsets. I'm keen to understand the image
processing carried out by ZXing prior to the actual barcoding step to
improve overall system performance.

I'm specifically looking at how to improve the barcode recognition
with EDOF cameras (these are effectively fixed focus with intelligent
sharpening algorithms to give wider depth of field) either by
customising the image processor tuning or optimising parameters in
ZXing.

Does anyone have

- a diagram or document showing the image processing used by ZXing
- recommendations of how the input system should be tuned *
- information on ZXing parameters that may be optimised for this
usecase

* E.g. sharpening, contrast, black+white vs colour, gamma, saturation,
zoom, defect correction, noise etc

Regards

Tim Kerby

[Sean Owen]

It's open source, so you can see exactly what it does. There is no other documentation available beyond what you see in the source, though you're welcome to ask questions here.

The core library does none of these things you mention; these are delegated to platform specific clients. Speaking of Android / Barcode Scanner in particular -- EDOF is not available on Android 1.5, which is what is supported now. A future build does use EDOF when available and auto-focus is not available. The camera performs its own contrast/brightness adjustments. There is no need for gamma/color correction as none of the formats use color information.

Zoom is used where available, but since it is only digital zoom it sometimes hurts scanning.

[Lachezar Dobrev]

The ZXing project uses Black-And-White raster to conduct detection
and decoding. There are a number of interfaces and abstractions, that
are expected to be used to produce one. Most of them have more than
one implementation/subclass. One can create a new implementation or
subclass to provide tuning or add a new algorithm. These are as
follows:

- com.google.zxing.common.BitMatrix
This class represents a Black-And-White raster.
Not meant for sub-classing.

- com.google.zxing.LuminanceSource
This abstract class wraps the needed functionality for a Gray-Scale raster.
Every platform (Android, Java 2 SE, Java 2 ME) has an extension of this class.

- com.google.zxing.Binarizer
This is an interface for an algorithm that converts a Luminance
Source into a black-and-white raster.
There are a few implementations, most notably the:

= com.google.zxing.common.GlobalHistogramBinarizer
Fast implementation that detects globally the boundary between
black and white.

= com.google.zxing.common.HybridBinarizer
Implementation that detects the black and white boundary in small
regions, thus providing better handling for shadows.

I believe your «best» bet would be to create a new Binarizer
implementation. There you can implement whatever techniques you
believe will be necessary.

2011/8/8 Tim <timoth...@gmail.com>:

[Tim]

Thanks to both of you for the information

If, I may, I have a few more questions that may help me before
trawling through the code.

I have two goals
1 - improve the tuning of our image processing to improve the results
with zxing
2 - look at improvements in software that could be executed prior to
or as part of zxing

To achieve (1), I currently test with a number of different pitch and
error correction levels within barcodes and vary the distance to the
camera. This gives me a number of barcodes decoded vs distance figure
that I can plot. Changing image processing within the camera varies
the result. It is however not the most accurate test. I would like
to obtain some error statistics from within zxing when it decodes (I
was hoping a verbose mode may contain this but it appears not to)

In particular I would like
- Access to the raw bit stream from after the binarizer and sampling
to compare with the known ideal bit stream of the barcode I am
decoding
- An error figure from the reed solomon stage (e.g. number of
correctable / uncorrectable errors)

Having access to the error data would allow me to fine tune the
processing at both near and far distance (limited by lens defocus and
pixel resolution respectively) to optimise for the zxing binarizer

If I can setup the tests for (1), I can then look at (2) in more
detail and I have some thoughts on how to improve the decode, probably
by missing some steps in the camera pipe and performing these later
when we have access to a cropped region in the decoder (in software).
The reason for this is that the image processor is a stream processor
which only has a few lines of data in its pipe at a time. I'm also
considering whether estimating distortion to create a distorted
sampling grid would be useful (in addition to skew correction),
perhaps aspart of the binarizer as a knowledge of pitch vs pixel
resolution could adjust processing such as local contrast enhancement
etc

What would be useful for this is a few lines of overview giving me the
flow through the zxing pipe (e.g. is perspective correction done
before binarization for instance).

If anyone has a few minutes to spare to help with the error correction
and flow it would save me a lot of time going through the code and
what calls what

Thanks

> 2011/8/8 Tim <timothy.ke...@gmail.com>:

[Sean Owen]

Are you looking at the unit test framework? It seems to do most of what you're talking about. It runs a load of test images, in differing modes and rotations, can dump intermediate output, and gives stats on success rate.

What you see is what you get here -- you'll need to add whatever other hooks you want for testing. But the current test framework may be a good place to start.A

Are you looking at QR codes in particular? there's not much room for improvement as they're generally very easy to detect and decode. In contrast, stuff like PDF 417 is very hard. So, the question really turns on what format you're looking at.

Error correction is also rarely the issue. It's either detected more or less correctly, and there are very few errors if any, or it's not detected at all to begin with.

I can't write up a guide for you -- you'd need to look at the code. But I think you want to zero in on LuminanceSource implementations (image -> luminance) and Binarizer implementations (luminance -> black-white transitions).

The hard part is the binarizer, going from luminance to some notion of where the black and white regions end. It's complicated by blur, and at some level even by resolution as the transitions occur at sub-pixel boundaries and the exact position is important for 1D formats. The approaches you will find here are mostly based on local or global histograms. It's fast and effective for mostly-clear shots.

I have a much better implementation for 1D formats based on a proper edge detector that will read blurry images too. I may not be in a position to open-source it but can tell you about what I know of what works and doesn't.