Anatomy of the Human Eye, Evolving to Digital Cameras

[James M. Atkinson]

�The Photographic Eye,� written by Allan Weitz is an interesting article
about how the human eye and the modern camera have several similar
attributes. What I found quite interesting in the article was how the
human eye constructs the entire scene through narrow (.5 degree)
snapshots of images. With some simple math, I was able to determine that
each image that we observe is actually 240-280 smaller images (120-140
degrees combined field of view), each involving 120 million panchromatic
pixels, and 7 million color pixels. This would reflect that to capture a
single image the eye collects 3.36 billion panchromatic pixels, and 196
million color pixels (and the 576 MP mentioned in the article may be a
little incorrect). The poetic message in the rods versus cones
comparison could be taken to mean that a black and white photograph
conveys 17 times more powerful a message then one in color.

The article by Weitz intrigued me further as I had recalled from my
Anatomy and Physiology courses that the cones do not respond evenly to
color in that there are more red responsive cones then green, and more
green then blue.

After reading the Weitz article an issue then came to mind (or actually
more of a hypothesis of mine) that modern digital cameras maintain an
even balance of red, green, and blue pixels in their imaging sensors,
when they actually should mimic the cone densities of the human eye and
be more biased to one color then another for optimal efficiency and
fidelity. To this end a modern digital camera should have over 70% of
the pixels dedicated to red, 23% to green, and 7% to blue, or the
inverse of these numbers depending on how you look at it.

The digital camera image sensor should also match the cone density
patterns of the human eye, which is a task that is actually far easier
then it initial seems, as the sensors can be etched in virtually any
color combination or density at the factory. My hypothesis is that by
matching the cone density of the human eye to the position or patterns
of certain pixels on the imaging sensor of a digital sensor that the
camera can be made to �see� more naturally, or more accurately the
camera sensor and the eye would see on a 1:1 basis. By collecting more
pixels in the spectrum then the eye can actually see, and not collecting
pixels the eye would �waste� the effective resolution of the digital
camera would be dramatically increased by doing nothing more then
re-ordering the pixels, but not actually increasing their number.

To look at this a different way, why should the camera sensor collect 7
million blue pixels, 7 million green pixels, and 7 million red pixels?
This symmetry of color pixels really is not needed, and an asymmetrical
approach to sensor design may allow cameras to actually see far better
then has previously been possible.

To take this a step beyond merely using asymmetrical sensor design, as
the rods and cones of the human eye are both mechanically, electrically,
and chemically different from one another and yet part of the same
matrix the camera sensor could also be modified to increase the diameter
of the monochromatic elements, and reduce the size of the color
elements. This would allow the monochromatic elements within the same
sensor to gain greater sensitivities and thus preserve image detail,
while the color elements would fill in the gaps and mimic the
performance of the human retina. Optionally, one sensor could be
dedicated to color imagery, and a second to monochromatic, and perhaps
additional sensors for alternate wavelengths similar to what is done
with reconnaissance satellites.

With silicon wafers now reaching 18-inches in diameter, and the new back
lit digital camera sensors well exceeding 50 million pixels per square
inch (i.e.: Kodak KAF-5K series) it will be possible for imaging sensors
to reach into the hundreds of millions of pixels within a few years time
using only a single chip.

While it may be costly to do so, it is currently within reason for a
single silicon wafer to be etched into a single digital sensor involving
1.12+ billion pixels; divided into whatever spectrum the designers may
wish. While an 18-inch wafer (Taiwan Semiconductor) and support circuits
may only fit onto the back of a 11x14 or larger camera the results would
be breathtaking, and may well exceed the abilities of the human eye, or
for that matter any other creature on Earth.

--
James M. Atkinson
President, Scientist and Sr. Engineer
"Leonardo da Vinci of Bug Sweeps and Spy Hunting"
Granite Island Group
jm...@tscm.com
http://www.tscm.com/
(978) 546-3803

[Thomas Shaddack]

> The Photographic Eye, written by Allan Weitz is an interesting article about

> how the human eye and the modern camera have several similar attributes. What
> I found quite interesting in the article was how the human eye constructs the
> entire scene through narrow (.5 degree) snapshots of images. With some simple
> math, I was able to determine that each image that we observe is actually
> 240-280 smaller images (120-140 degrees combined field of view), each
> involving 120 million panchromatic pixels, and 7 million color pixels. This
> would reflect that to capture a single image the eye collects 3.36 billion
> panchromatic pixels, and 196 million color pixels (and the 576 MP mentioned in
> the article may be a little incorrect). The poetic message in the rods versus
> cones comparison could be taken to mean that a black and white photograph
> conveys 17 times more powerful a message then one in color.

Not only that. The rods are also more light-sensitive; hence the trick of
observing weaker stars by naked eye; do not look directly at it, watch it
with the part of retina that is rich with rods and comparatively poor with
cones.

The distribution of the individual cell types is very uneven across the
retina; most cones are concentrated in the fovea.

Rods are also insensitive to red color. (Trick: if you want to not lose
darkness adaptation, swap the white LED in your keychain light with a red
one. Then the flashlight will not interfere with your scotopic vision.)

> The article by Weitz intrigued me further as I had recalled from my Anatomy
> and Physiology courses that the cones do not respond evenly to color in that
> there are more red responsive cones then green, and more green then blue.

And despite that, the eye is more sensitive to green. (Maybe the abundance
of reds is compensating for lower sensitivity to lower-energy photons?
Maybe the low amount of blue cones is compensated with higher sensitivity
of individual sensors and also doubling of functionality with
mostly-blue-sensitive rods?)

(The higher green-sensitivity causes the greenish tint of the
energy-saving CFLs, as then the same visible-radiated energy gives higher
brightness value when weighted with the eye spectral response.)

Then there is the phenomenon of tetrachromacy, occuring in some females.
(Who have naturally better color discrimination, which manifests e.g. as
seeing dozens of different hues of wallpapers or tiles when they all look
to you like the same off-orange.)
http://en.wikipedia.org/wiki/Tetrachromacy

This leads me to a thought if it could be possible to use topical gene
therapy to modify some of the light-sensitive cells in retina to become
sensitive in slightly different wavelengths, thus artificially engineering
such ability in vivo. (Could that work with infrared too? To how long
wavelengths could a biological structure go, given the limits of sensors
based on organic chemicals where the photons have to carry some minimal
energy to excite the molecule enough to be sensed?)

(A similar thought of mine is using the same approach for modifying smell
sensors, giving human nose the ability to sense chemicals normally not
perceived, e.g. carbon monoxide or some chemical warfare agents.)


> ...

> To look at this a different way, why should the camera sensor collect 7
> million blue pixels, 7 million green pixels, and 7 million red pixels? This
> symmetry of color pixels really is not needed, and an asymmetrical approach to
> sensor design may allow cameras to actually see far better then has previously
> been possible.

Actually, the cameras usually have twice as much greens. The Bayer filter
contains 4-element cells, two of which are green and the rest are one red
and one blue. (Can it be due to the highest-in-green eye sensitivity?)

> To take this a step beyond merely using asymmetrical sensor design, as the
> rods and cones of the human eye are both mechanically, electrically, and
> chemically different from one another and yet part of the same matrix the
> camera sensor could also be modified to increase the diameter of the
> monochromatic elements, and reduce the size of the color elements. This would
> allow the monochromatic elements within the same sensor to gain greater
> sensitivities and thus preserve image detail, while the color elements would
> fill in the gaps and mimic the performance of the human retina. Optionally,
> one sensor could be dedicated to color imagery, and a second to monochromatic,
> and perhaps additional sensors for alternate wavelengths similar to what is
> done with reconnaissance satellites.

Interesting idea!

There could be one high-resolution monochromatic sensor and a
lower-resolution color one. (Or the Bayer mask could contain gaps with
some pixels being left unmasked. Perhaps at least leaving one of the two
green pixels per mask block unmasked instead?) This would provide higher
sensitivity as well, as the color filters are wasting a lot of photons not
belonging to the given pixel's filter window.

Such cameras could also see in infrared without distorting colors, if the
infrared filter is integrated to the color filters; the pixels with mask
would then see no IR, the nonmasked pixels would have sensitivity across
the band as the silicon permits and the IR component could get calculated
from the pixel signal differences.

One of possible patterns for a modified Bayer mask could be a grid with
basic cell of
----
-R-G
----
-G-B
instead of
RG
GB
(where RGB are individual color filters and - are pixels without filters).

The unfiltered pixels would provide black-and-white high-sensitivity
image, while the sparse color ones would provide color hints. (Human eye
also has lower spatial resolution in colors than in brightness, a trick
used in analog TV by encoding chroma in a way that has lower bandwidth
requirements and piggybacks on the luma signal.)

The question is if such tricks are necessary as e.g. the Foveon sensors
stack the RGB sensors vertically, so there is no mosaic and each pixel
gets all three colors. (The 3-sensor video cameras use dichroic mirrors to
achieve the same result.)

(Of course, a brief lookup shown me that these tricks are already being used.)
http://en.wikipedia.org/wiki/Color_filter_array
http://en.wikipedia.org/wiki/Bayer_filter#Alternatives

There is even some company that provides mask removal from some camera
chips as a service, catering mostly to the astrophotography market.

> With silicon wafers now reaching 18-inches in diameter, and the new back lit
> digital camera sensors well exceeding 50 million pixels per square inch (i.e.:
> Kodak KAF-5K series) it will be possible for imaging sensors to reach into the
> hundreds of millions of pixels within a few years time using only a single
> chip.

There is another possible development. Human eye is a pretty lousy optical
device, and all the sharpness is done by postprocessing of the
aberrations-rich retinal projection. (The retina itself contains a lot of
image processing circuitry, so the brain is not getting a raw bitmap
signal but already "precompressed", preprocessed data stream.)

Such high-resolution sensors can be coupled with signal processing that
could correct the distortions caused by cheap optics, and yield
lower-resolution higher-quality image.

An early development along these lines is the Lytro light-field camera.
http://en.wikipedia.org/wiki/Light-field_camera
http://en.wikipedia.org/wiki/Lytro

> While it may be costly to do so, it is currently within reason for a single
> silicon wafer to be etched into a single digital sensor involving 1.12+
> billion pixels; divided into whatever spectrum the designers may wish. While
> an 18-inch wafer (Taiwan Semiconductor) and support circuits may only fit onto
> the back of a 11x14 or larger camera the results would be breathtaking, and
> may well exceed the abilities of the human eye, or for that matter any other
> creature on Earth.

What about going the self-assembly way? Use some patterned substrate on
which the individual sensors would attach to the right positions (e.g.
using hydrophilic/hydrophobic parts of molecules, or even short DNA chains
that would match the complementary chains on the substrate) by just
immersing the substrate in their solution? These tricks could bypass the
costly photolithography process entirely and potentially allow virtually
unlimited sensor array sizes and reel-to-reel processing with flexible
substrates...


As of hyperspectral imaging, I am harboring an odd thought. Some people
made themselves ultra-high resolution scanning cameras from document
scanners, converting the sensor holder from sliding to rotation.

A modification of this, with a 2D-sensor camera instead of a linear CCD,
could work as a hyperspectral camera. Add a narrow slit paired with a
diffraction grating, sense one image line at a time (like the linear-CCD
scanner does), with each pixel being expanded by the grating to a line of
pixels, forming a full visible (and perhaps partially UV and near-IR)
spectrum. Each image line would then be presented as a 2D-photograph,
where each pixel is expanded to its spectrum. Assuming no resolution
reduction with a color filter mask, and 400-900nm resolution, a 1000x1000
pixels sensor would provide 1000 pixels wide scanline with 0.5 nm
theoretic spectral resolution.

The acquisition process would likely take a lot of time, due to long
exposures needed for each image line, and the collected amount of data
would be fairly huge (before compression). However it could have
significant scientific value, e.g. for art restoration, or for color
matching (would allow modeling how a given object would look under
different kinds of light).


Just some random sleepy thoughts...
It's so late here it's already early.

[Eric Schmiedl]

Regarding color sensitivity, keep in mind that camera sensors (CCDs at
least, and I believe CMOS as well) are much more sensitive to longer
wavelengths than shorter. If you white balance a camera indoors,
you'll see more noise than you will if you white balance the camera
outdoors. (and if you don't white balance, you'll get a yellow
picture)

Also, digital cameras by and large use Bayer pattern filters, which
means they have twice as many green-sensitive pixels as they do red-
and blue- sensitive pixels. This is because the human eye tends to get
most of the detail information in a scene from greenish colors. That
means extra resolution in red or blue would be wasted -- try it
yourself, bathe a room in red or blue and you'll see less detail.

Different size pixels has also been done, by Fuji for their S5 Pro
camera and an accompanying (very rare) medium format digital back. But
those were for a different purpose -- the Fuji varied size (without
respect to color?), in order to create a chip that had more- and
less-sensitive pixels and which, combined together, could therefore
see a wider range from light to dark.


On Tue, May 8, 2012 at 6:34 AM, James M. Atkinson <jm...@tscm.com> wrote:
> “The Photographic Eye,” written by Allan Weitz is an interesting article

> “see” more naturally, or more accurately the camera sensor and the eye would

> see on a 1:1 basis. By collecting more pixels in the spectrum then the eye

> can actually see, and not collecting pixels the eye would “waste” the