FillSky February update

[Jeff Welty]

After I posted my previous update, I went and tryed the new code on a new image, which immediately failed miserably.

The problem was detection of the end of sky.  The new image was much noisier, so the old algorithm was badly fooled by a pixel with a high error due to noise.

The solution was a completely new approach -- the new algorithm builds a 2d model of a small region of the sky -- actually 3 models, one each for R,G,B.  The model also tracks errors of pixels in the sample used for the model, then examines the next pixel "down" (i.e. a larger y value), and tests to see if the next pixel, and the one following both fall outside a reasonable margin of error.   There are nuances beyond that but that's the basic idea.  The new algorithm is much more robust, and probably more precise too.

I have also replaced the Nelder-Meade optimization with Levenberg-Marquardt.

I have also vectorized all the colorspace type of calls.

It now runs a zillion times faster.

I have modified models that predict the Hue, Saturation and Value for a given pixel in the sky.  I looked at lots of plots of actual HSV, and predicted HSV.

A good solid month of work on this, many dead ends, and I hope it is functional in most cases now.

At this point, I'm going to call it alpha code, and move on to documentation.

Current, best branch is at:

https://github.com/weltyj/SkyFill/tree/2022.02.11

I would greatly appreciate reports on compilation problems.  I have NOT worked with the CMake files, I've only updated my local Makefile.

[T. Modes]

Hi Jeff,

eljef...@gmail.com schrieb am Samstag, 12. Februar 2022 um 02:07:30 UTC+1:

I would greatly appreciate reports on compilation problems.  I have NOT worked with the CMake files, I've only updated my local Makefile.

 please find attached a patch for a update of the CMake build system. It fixes also a compilation issue on MSVC (in line float g_d[width+1] ; -> constant expression expected).

A first test looks good.

A main issue I have is that it works only with 16 bit images. I know you answered that questions already. But maybe after your latest optimisation it works better also with 8 bit images?

Thomas

[T. Modes]

T. Modes schrieb am Samstag, 12. Februar 2022 um 10:28:29 UTC+1:

A first test looks good.

PS: It create now some temporary/debug files? (some .dat files and a original file from exiftool)

[Jeff Welty]

Hi Thomas,

Thanks for checking it out and the CMake patch!

I have a general idea how to handle 8 bit images -- when reading the image, upscale to 16 bit, do all the processing, then on output downscale back to 8 bit.   The extra bits would make all the interim processing better.   Dithering might need a tweak though.  Does that seem like a reasonable approach?

Yes, it makes a couple of data files -- I need to turn that off by default.  Those are the data files I have been examining to create a better sky HSV model.   I also need to tell exiftool not to back up the original.  Exiftool is how I'm getting the ICC profile copied from the input tiff to the output tiff.

Jeff

[Jeff Welty]

Added 8 bit support on the 2022.02.11 branch...  Only for TIFF images.

On Saturday, February 12, 2022 at 1:28:29 AM UTC-8 T. Modes wrote:

[T. Modes]

Hi Jeff,

eljef...@gmail.com schrieb am Sonntag, 13. Februar 2022 um 04:05:39 UTC+1:

Added 8 bit support on the 2022.02.11 branch...  Only for TIFF images.

Thanks a lot. Works fine here.

[ckoerner]

Sorry for the dense question, but is this a fork of Skyfill? 

https://gitlab.com/dr.sybren/skyfill

[Jeff Welty]

No relationship at all.  Based on a first test and a quick look at the code for dr.sybrin/skyfill, it appears to handle only extremely simple cases with entirely blue sky with very thin margins to fill above the sky.

It won't provide a modelled gradient from horizon to zenith for sky hsv.

It did not detect the sky properly in my first test -- it averaged a tree image into the surrounding sky area).

[johnfi...@gmail.com]

On Friday, February 11, 2022 at 8:07:30 PM UTC-5 eljef...@gmail.com wrote:

I would greatly appreciate reports on compilation problems.  I have NOT worked with the CMake files, I've only updated my local Makefile.

It compiled fine for me (using the CMake changes Thomas gave you) on Windows MSYS / mingw64 , which is a very different environment from the tools Thomas uses for Windows.

[johnfi...@gmail.com]

On Saturday, March 5, 2022 at 7:55:26 PM UTC-5 johnfi...@gmail.com wrote:

It compiled fine for me (using the CMake changes Thomas gave you) on Windows MSYS / mingw64 , which is a very different environment from the tools Thomas uses for Windows.

But the first time I used it I got a disappointing behavior:

 I used -fsr -d3 (as in the sample script you did for me).  I was working on a smaller example than the one of mine you looked at.  I'll post that one if you like, though it was just some half done work I chose for this test.

I cropped a tiny bit of skyline from the output (attached).  If the green line (from -d3) is supposed to represent what I think it is supposed to represent, then its tracking relative to the actual skyline is not good enough to use.

[Jeff Welty]

That'll work fine.    In -fsr mode, it will compare that brown color to the expected blue color, determine a zero probability that it is actual sky, and thus leave the color as is.

In normal mode, blending starts 50% of the way between the green line and the yellowish line at the top, so it will never reach where end of sky went to far down.

 If you want to tune it up change the end of sky detection tolerance.  Default is 3.5, so see how 3.0 works with "-tol 3.0"