Lossless/Near lossless, Bayer/Pixel-shift and Noise removal advice for community oriented Digital Cinema Camera projects?
Wayne
My name is Wayne, I am new to this group. I have read the compression
faqs sometime ago but still have only a rudimentary knowledge of
compression, and would like to know more for a community oriented project.
Digital Cinema camera Projects:
I have been involved with a group of people over at the dvinfo.net forums
Alternative Imaging sub-forum, developing low cost HD Digital Cinema
Camera alternatives. This was an effort to beat the huge price margins of
existing cameras that can cost hundreds of thousands. Two commercial
cameras grew out of this, first the Drake, which seems to have gone away,
used to film the feature film Drachen Feder (German, Dragon's Feather) and
the Silicon Imaging's SI-1920HDVR, just used for the filming of the
feature Spoon, using the Cineform Bayer codec. David Newman, the CTO from
Cineform, has developed the Cineform Bayer visually lossless codec from
discussions in our group. I have to admit, I am yet to get around to
reading the white paper to determine if it is just visually lossless, and
not truly lossless as is better for Bayer interpolation usually requires.
But it can achieve upto around 6:1 on bayer, which is 18 times better than
4:4:4 video. We have previously found properly debayered uncompressed
images stunning.
http://www.drachenfeder.com/aktuelles/drake_hd_en.htm
http://www.siliconimaging.com/DigitalCinema/
http://cineform.blogspot.com/2006/04/cineform-raw-this-skunkworks-project.html
Current:
I am here trying to research some long held queries on compression that
would help the latest projects and need expert guidance. Compression and
NLE editability of the format are two of the big issues to do with storage
and work flow.
We are yet to develop the originally intended sub $5K camera system
(though this can go much lower). At the moment, people are testing a
number of uncompressed USB cameras, but the camera more in question is the
Elphel 353 security oriented camera using Jpeg and a basic version of
Theora. Visually, this is less quality than the Drake or SI cameras, but
still potentially significantly above a lot of the HD cameras below their
near $20K prices. People are testing the current Linux oriented, Open
FPGA based, 333 camera that is limited, but the new camera promises to
expand that and has an internal IDE interface that is due for testing.
There is also still interest in computer based USB/camera link cameras. I
also gather there is no bayer mode in Jpeg, and neither are meant to be
lossless.
The cameras and thread:
http://www.dvinfo.net/conf/showthread.php?t=63677
http://www3.elphel.com/index.php
http://www.linuxdevices.com/articles/AT3888835064.html
This has links to most of the Digital Cinema threads, and some technical
links, information etc:
http://www.dvinfo.net/conf/showthread.php?t=28781
Information:
My interest is in lossless and visually lossless formats (preferably open
source) suitable for Bayer and grey scale compression. That is the
problem, their are plenty of lossless/visually lossless tools for standard
video formats, but not for Bayer video formats. Another particular
interest is FPGA compression designs for the camera. Of course help on
the hardware and software side would be appreciated, as the video
community tends to not have too many good programmers and hardware
people. So I have compiled a few questions that would help this project,
as well as potentially one of mine:
-----------------
What are the best lossless and visually compression systems for bayer, and
their average expected compression ratio range?
What would they be for grey scale compression instead?
What quality Noise removal exists, even frame inter based temporal with
image restoration. Preferably Open Sourced. I am interested in how this
effects the compression ratio. This is the trick I am interested in, as
much noise is removable as it is has either some regular form etc or then
image is predictable where the noise is.
How would noise removal, producing a clean image, effect the compression
ratio of the above codecs?
How helpful is interframe compression without and with noise removal
first? Even simple frame difference compression (less the movement
compensation)? One person on the forums was experimenting with this frame
difference compression and reported promising results.
What sort of average compression range can be expected above, with noise
removal and or interframe compression? Using this for a lossless variable
bit rate scheme. Is it possible to get to 7:1 to 10:1 range? As a side
line, I am interested in these ratios for the application of Bayer video,
I have made numerous suggestions on the forums for further enhancements.
As you can see, this could put a lossless/visually lossless within the
range of HDTV transmission channel in 720p25. I am further interested in
pixel shift which would require less compression. Of course, the Pixel
shift maybe lossless and smaller, but suffers from natural visual loss
because of the pixel shift format, so it should not be taken as a 4:4:4
quality image, but as quality alternative to 4:2:0. I have been thinking
of ways around this.
Back on track. Due to processing demands, I am also interested in good
compression systems for low processing load for given compression ratio.
Pixel shift technology, and compression, Open sourced preferable? This is
most of my interest in grey scale compression, but also has some bearing
on Bayer.
FPGA systems for these.
NLE compatibility of these, even Cinelerra.
-----------------
I was wondering if the ideas in the public discussion on our forums, can
be used to develop a similar, but different, Open Lossless/near lossless
bayer codec to cineform?
I know all these things are not easy and require heaps of processing
power, but I am keeping my options open, and my processing options because
of the great variability in future processing platforms.
I apologies for the excessive length, but as you can see, it is aimed at
clearing up likely future directions to investigate, and the faqs do not
go into depth about grey scale, bayer or pixel shift compression codecs
and the effects of noise removal. If anybody with precise expert
knowledge can help that would be appreciated (but if nobody is answering
feel free to jump in).
Thanks
Wayne.
--
Using M2, Opera's revolutionary e-mail client: http://www.opera.com/m2/
Wayne
I forgot to mention, we have one person looking at doing new compression
for the camera and asking for help in what can be done (the developer of
the camera is concentrating on his main focus market, and leaving it to
us). His posts are at the end of the current Elphel camera thread above.
Thanks
Wayne.
Marco Al
The problem with true lossless coding is that it can't guarantee
compression. You are going to have to choose what you want, design for
some X times compression ratio just like Cineform RAW or design for
uncompressed and just use compression to save on space.
I don't quite get the stuff about pixel shift ... it's a method only
relevant if you have separate CCDs.
No frame based compression methods will present a problem for NLE.
I don't think you will find any existing open source compression which
is near lossless and optimized for editing. If you don't take into
account the way the errors they introduce interact with multiple coding
passes things can get ugly quickly.
Marco
Wayne
> Just to state the bloody obvious ...
>
> The problem with true lossless coding is that it can't guarantee
> compression. You are going to have to choose what you want, design for
> some X times compression ratio just like Cineform RAW or design for
> uncompressed and just use compression to save on space.
Marco, I'm sorry, but I asked about the average range that it mostly
hovers around, because that is how it happens in real life it varies from
frame to frame complexity. I would not state an exact compression ratio
for lossless, as that is totally invalid in lossless compression, as the
compression ratio is totally dependent on what it can do with the image at
hand, unless you want to turn the lossless compression performance down
somehow, or pad out a file for some reason.
> I don't quite get the stuff about pixel shift ... it's a method only
> relevant if you have separate CCDs.
Easy, it used to achieve higher resolution through interpolation, but I
would like options that optimally compresses in that format, and save
space over say converting it to a high res image then compressing. Yes,
you can deliberately setup a camera to do this, many HDV 3 chip cameras
are using this, recording the pixel shift image (that might be the unusual
component format of one camera I have heard of) than the interpolated HD
image. Another guy from our forums did a HD conversion on a SD panasonic
camera using its pixel shift, actually he has a editor for it, but I don't
think it will be cheap. Here is his company website for his product:
> No frame based compression methods will present a problem for NLE.
Well, I am open to suggestions, codecs that have support or transcoding
workflow to an intermediary codec that can be used.
> I don't think you will find any existing open source compression which
> is near lossless and optimized for editing. If you don't take into
> account the way the errors they introduce interact with multiple coding
> passes things can get ugly quickly.
Thanks, looks like intermediary is it then. But commercial options are
not totally out, but just not as preferred. there are some projects that
could use it.
>
> Marco
Marco Al
> Marco, I'm sorry, but I asked about the average range that it mostly
> hovers around, because that is how it happens in real life it varies
> from frame to frame complexity.
In real life you are going to hit the worst case scenarios eventually.
Expect something like 2-3 compression.
> Easy, it used to achieve higher resolution through interpolation
I understand what it is, what I don't understand is what it has to do
with the bayer sensor based camera you guys intend to use.
Marco
Michael Schöberl
I tried to find an lossless compression algorithm suitable for high
resolution and fast processing and ended up with something similar to
JPEG-LS (but without the contexts) ...
I finally built a realtime (de)compressor for my fpga and could
demonstrate the whole system ...
the compressed file has 55-60% of the original file size for typical and
sharp images ... it goes down to 45% for less sharper input data
I also developed some ideas to introduce some losses to guarantee <60% -
this could be used for a full hdtv rgb4:4:4 image over single HD-SDI
the idea behind all this was to save space *on average* in a recording
system - the allowed cost for the compression was well below your P4
processor - I used 1000 LUTs from an FPGA ...
for bayer:
You only want to store your bayer data if you want to use a really good
offline colour reconstruction algorithms ... In this case you should not
modify the bayer data ...
Most promising was a bilinear interpolation of as a prediction step for
the next colour channel
(transmit g1,
predict g2 from g1, transmit diff,
predict r from g1 g2, transmit diff,
predict b from g1 g2, transmit diff)
noise removal is a quite delicate issue ...
It is good if you correct your sensor data (gain/offset) but algorithms
that really alter your data afterwards are considered quite bad by the
artists ...
that is the same with lossy compression and I think that we ned to spend
even more effort showing that there are compression schemes with little
impact on image quality
bye,
Michael
mihai cartoaje
Wayne
> Wayne wrote:
>
>> Marco, I'm sorry, but I asked about the average range that it mostly
>> hovers around, because that is how it happens in real life it varies
>> from frame to frame complexity.
>
> In real life you are going to hit the worst case scenarios eventually.
> Expect something like 2-3 compression.
With buffering and variability much can be smoothed out, and I know the
ratios you are talking about, but I am here to find out from other people
where ever they can be improved by noise removal preprocessing and inter
frame compression on top of that. They are a complex series of questions
to answer, and this is not helping much.
>
>> Easy, it used to achieve higher resolution through interpolation
>
> I understand what it is, what I don't understand is what it has to do
> with the bayer sensor based camera you guys intend to use.
I said both bayer (most of them) and pixel shift (myself) are being looked
at.
Wayne
<MScho...@mailtonne.de> wrote:
> I was working on similar things for my diploma thesis ...
[JPEG-LS] (but without the contexts) ...
[FPGA]
[Bayer diference]
Michael,
Thanks for all this. I think that it would be worth looking into, I am
still looking into higher performance again, I have seen one commercial
codec that claims to average out at 4:1 on 4:4:4 (though I do not know how
accurate a claim that could be). But bayer. pixel shift, noise removal
and inter frame is another level again. I think I might have seen
something like your bayer difference idea on the forums in relation to
discussions with cineform, they are currently aiming around 6:1 (sued to
be 4:1). I certainly was well into predictive related suggestions on the
list myself.
That commercial 4:4:4 Codec I mentioned.
http://www.digitalanarchy.com/micro/micro_loss.html
Actually, apart from the Elphel camera (dvinfo thread link posed before)
we had a guy that is designing his own low cost commercial HDSDI recorder,
he is interested in a bayer mode, and you might like to talk with him. I
just went to get his link, not working now, but it was:
http://www.engr.mun.ca/~wakeham/
And the project name was "Charon". He didn't implement compression
because (apart from work load) he did not have enough understanding of it.
> noise removal is a quite delicate issue ...
> It is good if you correct your sensor data (gain/offset) but algorithms
> that really alter your data afterwards are considered quite bad by the
> artists ...
> that is the same with lossy compression and I think that we ned to spend
> even more effort showing that there are compression schemes with little
> impact on image quality
I have seen good film restoration, which seemed to use a similar method to
the inter frame temporal one I have been thinking of for years. I think
that if you can restore the image pixels lost to noise you are in a better
position, artist or not, noise and grain can be added to taste if they
want. For me a clean image is best, if they want noise they can design
that into playback devices ;). I would like to find out more about noise
removal (resulting in image restoration) tools Open source and effects of
lossless compression? The jump in transmission, storage, and embeddable
parallel processing capacity, really make high level
lossless/near/visually, lossless a practical thing use instead nowadays
(given massive increase in storage capacity per cost coming on line in the
next year or so). What do you think Michael?
Thanks, it is a good start, I am still looking for additional programs
codecs and information.
>
>
> bye,
> Michael
Michael Schöberl
> still looking into higher performance again, I have seen one commercial
> codec that claims to average out at 4:1 on 4:4:4 (though I do not know
> how accurate a claim that could be).
I doubt that one could do a lossless compression with a ratio better
than 2:1 on serious natural input data. I ran some tests on CGI data and
got a higher compression but in natural images you will always have some
sort of noise.
> [...] I
> think that if you can restore the image pixels lost to noise you are in
> a better position, artist or not, noise and grain can be added to taste
> if they want. For me a clean image is best, if they want noise they can
> design that into playback devices ;).
It's more like "lossy compression? No - we don't want the data to be
modified!" ... well - you need to sell it in the end
Noise removal looks promising and I did some work on that for medical
images but for digital cinema the marketing is as important as the
function ... sometimes
bye,
Michael
mihai cartoaje
my image codec (libima). Here is the link:
http://geocities.com/repstsb/libima.html
Wayne
bug in Opera.
---------------
On Tue, 15 Aug 2006 02:33:07 +1000, Michael Schöberl
<MScho...@mailtonne.de> wrote:
>> think that if you can restore the image pixels lost to noise you are in
>> a better position, artist or not, noise and grain can be added to taste
>> if they want. For me a clean image is best, if they want noise they
>> can design that into playback devices ;).
>
> It's more like "lossy compression? No - we don't want the data to be
> modified!" ... well - you need to sell it in the end
For me, the objective is to film the scene, not the noise, the noise is
loss. So restoring the image loss and then losslessly compressing the
restored image is the closet you can get to no loss image, without filming
without noise (lighting and high S/N ratios fro 10bit plus). Otherwise
you are just compressing a lossy image with lossless compression, with
worse compression because of the noise. I think it is a preferable
solution.
>
> Noise removal looks promising and I did some work on that for medical
> images but for digital cinema the marketing is as important as the
> function ... sometimes
Can you advise me of directions, products, open source, or compression
gains you have found from this?
Don't worry about marketing, the end product talks, then you market (even
though Hollywood likes to market hard for products that don't talk so
well).
>
>
> bye,
> Michael
Thanks
Wayne.
Wayne
advanced on his own algorithm now, I will try to encourage him to have a
look at this.
Thanks
Wayne.
On Thu, 17 Aug 2006 21:25:06 +1000, mihai cartoaje <mcar...@gmail.com>
wrote:
> I have added support for Bayer color array and (lousy) demosaicing to
> my image codec (libima). Here is the link:
> http://geocities.com/repstsb/libima.html
>
--
Michael Schöberl
>> images but for digital cinema the marketing is as important as the
>> function ... sometimes
>
> Can you advise me of directions, products, open source, or compression
> gains you have found from this?
from my 1/2 year working on medical denoising algorithms I know that the
docs (expert viewers) liked the results very much. As a reference I
could only direct you to software like neatimage (for digital cameras)
> Don't worry about marketing, the end product talks, then you market (even
> though Hollywood likes to market hard for products that don't talk so
> well).
I'm eager to find out - IBC is close
bye,
Michael
Wayne
<MScho...@mailtonne.de> wrote:
>>> Noise removal looks promising and I did some work on that for medical
>>> images but for digital cinema the marketing is as important as the
>>> function ... sometimes
>> Can you advise me of directions, products, open source, or compression
>> gains you have found from this?
>
> from my 1/2 year working on medical denoising algorithms I know that the
> docs (expert viewers) liked the results very much. As a reference I
> could only direct you to software like neatimage (for digital cameras)
Thanks for this Michael, I have the page up, and they have a product,
neatvideo as well, that might be of benefit to a number of people with
ordinary cameras as well.
Hopefully with this search term in google, I can narrow down other
products, but long way to go yet.
>> Don't worry about marketing, the end product talks, then you market
>> (even
>> though Hollywood likes to market hard for products that don't talk so
>> well).
>
> I'm eager to find out - IBC is close
Keep an eye out on that thread in coming months, a prototype of the camera
is due for testing and development shortly.
vnurin
compression technology. My web site is http://www.algopix.com.
Currently I'm working on video version (video codec) of the
technology. If you're interested on it, we can go into details.
I'm also interested on effective noise reducing technique and I'd
be glad if you'd recommend me open sources for defining which one is
effective for my technique (which one could increase comp. ratio more).
vnurin
compression technology. My web site is http://www.algopix.com.
Currently I'm working on video version (video codec) of the
technology. If you're interested on it, we can go into details.
> Noise removal looks promising and I did some work on that for medical
> images but for digital cinema the marketing is as important as the
> function ... sometimes
I'm also interested on effective noise reducing technique and I'd
be glad if you'd recommend me open sources (if you could collaborate
with me) for defining which one is effective for my technique (which
one could increase comp. ratio more).
THanks, Vahagn
Wayne
Hi Vahagn
Thanks for this, I will have a closer look at that hopefully latter.
I looked up sourceforge and can see nothing really there for noise
removal, maybe somebody here knows of something, web links to information
of texts. If you look at noise it is fairly simple as long as you have
the memory bandwidth to do that multi frame comparisons.
Noise comes in a few varieties, pixel specific noise (usually solved on
sensor by fixed Pattern noise removal) but I suspect certain pixels will
be susceptible to other noise characteristics. I'll avoid thermal/dark
current noise, and amplifier noise. for a different classification
system. Re-edit I'll break this down to dot points:
-------------
We get consistent noise, where is keeps coming up in the same place frame
by frame.
then there is intermittent noise that comes and goes
then we get more random noise
then these can be variable from frame to frame
(this stuff is my own naive simplified observations, I have no formal
training, and have not done research into it)
Then localised noise is another characteristic, where a certain region of
the picture will display random/intermittent/variable/consistent noise.
Possible solutions:
Recognising that there is a noise like interference pattern out of
characteristic to the surrounding background allows you to isolate and
repair, or one that is localised, or a region prone to it (in reality, a
good camera should not be very susceptible to this, and if evident only
very minor).
Comparing between frames, identifying that unmoving regions now have
different values frame to frame.
Tracking moving region/objects position frame by frame, and identifying
that the blended pixels (because they are blurring out from movement) are
getting non consistent values (realising that turning closer/further
change this in 3D ways, that don't look consistent in 2D)
Identifying that the pixel goes through a cycle of the same values caused
by an oscillating/flashing/moving visual feature (this is where you can
actually loss real information).
Now for a trick, you can determine some of the values by local pixels, and
by the pixels from frame to frame (and who says you wont get noise in a
pixel on a few frames in a row). But a pixel carries can carries a
portion of noise. We get averaged SN ratios, which tells us something,
but the portion can be 100% or lower then the smallest bit at anytime.
But because you have the pixel in multiple frames over time (and with the
aid of the surrounding pixel values) you should be able to determine close
to the real pixel value by the portion that keeps consistently coming up
through the noise. I trust you know what I mean here, that as the noise
goes up and down like a curtain, and as noisy bits (something that maybe
not obvious, certain bits in a sample might be more prone to noise) goes
from noise to non-noise, they reveal the true value over time (which can
even then be back tracked to calculate noise in times past, from
surrounding pixels, this process being able to be done back and forth
multiple times to increase precision).
Yes this looks like a FPGA like processing option, unless have massive
parallel processing function, like GPU in DirectX10/11, or processor
array. Some though is simple and doable on less processing intensive
basis.
-------------
After discussions, our local guy has done a differential codec in ten
lines of code. he is claiming is good for 3:1 on his tests. I am still
waiting to see how it will turn out on actual RAW Bayer images though with
some compression optimisations. I suspect that 3:1 is possible, but with
improvements. If cineform can do 6:1, then I suspect event hat is
possible lossless.
I think that raw bayer compression is a big hardly tapped market. I
sometimes say we could push out visually lossless 720p25/24 TV channel in
40Mb/s using cineform (well now 32MB/s). I suspect that it is possible to
do this in 20Mb/s. I further think that 1080p might be possible with good
quality unlike current HDTV Mpeg2 (but I think the Americans need a lot
less TV channels in the States). If you tap this with the reality that
many Digital Cinema cameras use Bayer/RGB structures, and that this is a
good strategy for many cameras etc. It becomes the preferred option.
Please feel free to join in the exchange of ideas. If their is also no
open source for noise removal in compression, then their is open
opportunity for somebody like you to do it.
Wayne
Michael Schöberl
> Hi Michael. My name is Vahagn. I've developed new lossless image
> compression technology. My web site is http://www.algopix.com.
If you target for lossless gray-scale/indexed/colour then you should at
least compare your algorithm with JPEG-LS
"exploiting local intrinsic image properties" is not very detailed - why
should it be better than existing and standardized solutions like jpeg-ls?
from my side - I'm not interested in marketing but my interest is in
hardware-efficient algorithms ...
bye,
Michael
vnurin
Wayne, I need noise cleaning algorithm for having the same I, i+1
frames if motion is absent. I hope to determine value of motion after
applying such noise cleaning algorithm. In my site I introduced
Capturra video surveillance s/w, where motion detection is absent (it
must be improved). Having such noise cleaning algorithm I can get both
- big compression & motion detection.
vnurin
> least compare your algorithm with JPEG-LS
> "exploiting local intrinsic image properties" is not very detailed - why
> should it be better than existing and standardized solutions like jpeg-ls?
>
format's comparison with JPEG-LS.
Wayne
I agree, I see what you mean, I was just giving a full range of options.
So your looking at using noise removal to reveal movement (which can then
be shifted to better match) rather than comparison revealing movement,
then doing noise removal on that basis. Looks good for stream lining the
noise removal and interframe compression into one.
Michael Schöberl
I did look at your website - I only found some compression comparisons
on the R&D Page http://www.algopix.com/r-d.html
I did not find anything about JPEG-LS
bye,
Michael