Filling a clear blue sky

[Jeff Welty]

I sent a message to Bruno about this on sourceforge,  but not sure if it went.

I have some code developed to fill in the top of a clear blue sky on 16bit tiff images stitched with Hugin.

I have to run out for the day, but the code and some sample images are here:

https://www.dropbox.com/sh/uhoan6lma8yp3kk/AADl-Mf0iuBSxVHeUEOYAM-Ia?dl=0

Code has lots of global variables as a few years ago I thought it would be simple.   So, I need to work on cleaning that up!

Cheers and Happy Holidays,

Jeff Welty

[Bruno Postle]

Hi Jeff, thanks for sharing. This looks like a very useful tool.

The best way to get this out there for people to use is to put it in a public git repository (github, gitlab, sourceforge or similar).

It compiles here, but when I run it, the input tiff files are detected as having zero width:

    $ ./skyfill_tif 2021_pan19.tif 2021_pan19_out.tif
    ptofilename is 2021_pan19.pto
    setting tag number 256 in output file
    setting tag number 257 in output file
    setting tag number 258 in output file
    setting tag number 277 in output file
    setting tag number 278 in output file
    setting tag number 274 in output file
    setting tag number 284 in output file
    setting tag number 262 in output file
    setting tag number 339 in output file
    setting tag number 259 in output file
    input image is 0 x 1060, 16 bits, 4 channels, 0 = 22824
    All columns are masked!, no output generated

There is no obvious problem with the code that would cause this, `tiffinfo` reports the files as having width, and I see the same problem with a tiff file I created myself, so I'm stumped, I have libtiff 4.3.0

--

Bruno

[Jeff Welty]

Hi Bruno,

Any preference for sourceforge or github?   I don't have a preference.  I have a very, very old project on sourceforge (Gnome Wave Cleaner), so I have a user profile there as well as github.

Looks like I have libtiff 5.3.0.   Maybe there is a more rubust way to detect the image width than the example code I found.  I'll look at tiffinfo for clues.

Thanks for being the first bug reporter!

Jeff

[Bruno Postle]

On Tue, 21 Dec 2021 at 23:23, Jeff Welty wrote:
>
> Any preference for sourceforge or github? I don't have a preference. I have a very, very old project on sourceforge (Gnome Wave Cleaner), so I have a user profile there as well as github.

I have no preference for git hosting, hugin/panotools are hosted with
mercurial on sourceforge for historical reasons, I have code on
bitbucket and github - they all have advantages and disadvantages.

--
Bruno

[Jeff Welty]

https://github.com/weltyj/SkyFill

There is the github project.  I took a quick look at the width not being detected.  Looking at tiffinfo.c, I think the problem was skyfill_tif.c used 16 bit ints for tiff width and heightfields, and it needed to use 32 bit ints.  That correction is on the source file on github now.

[T. Modes]

Hi,

I tried to compile on Windows with MSVC. But there are a lot of errors:

1.) A lot of errors "libtiff type deprecated; please use corresponding C99 stdint.h type" for many places.

2.) Unknown type uint at several places

3.) " C2057: expected constant expression" in skyfill_f.c, line 1356, 1357 and 4767

4.) M_PI is unkown.

Point 1+2 can be easily fixed.

Point 4 requires an addition #define _USE_MATH_DEFINES before #include <math.h>

but point 3 require some more changes.

So  I can't test further.

But I have some more questions:

* Would it be possible to extend it to also work with 8 bit images?

* What projection is expected for the images? Only equirectangular? Or cylindrical? Does it also work with rectilinear? (It tries to read the fov from the pto file, but ignores the projection.)

* What happens when the sky contains clouds? Sorry can't test this for myself.

Thomas

[Jeff Welty]

Hi Thomas,

Thanks for giving a try on Windows.   I hadn't even considered Windows, as I've been exclusively linux ever since I retired a few years ago.

But since it doesn't have a GUI, it will be simple to address those errors you found. 

***

I looked at the source code lines for error 3) and have no idea at all why the compiler would be giving that warning -- did you modify the source file so my line numbers don't match yours?

***

To answer your questions:

* At some point it might be worth looking at modifying it for 8 bit images, I'm hesitant to do it now because I've already run into visible banding issues with 16 bit images.  The answer to banding is to dither the pixels before output, and what I did was very crude -- I just increased the dither amount until I couldn't see the banding.   But the dithering will produce bad results for pixels that are within the dithering amount of pure white or pure black.

* It doesn't use the projection from the pto file -- only the FOV.   What it needs to do the unblended sky model is the degrees in horizontal, so it can calculate the degrees per pixel.  The assumption is that the pixels are square, not rectangular.  The other assumption is that the degrees per pixel is a constant.   I'm pretty sure that's not true for many projections, but it only needs to be "true enough" to work.

* Clouds are not handled directly -- but you can mask off whole vertical sections of the image from any analysis or modification with the -m <left> <right> column.   The next step on clouds would probably be something like using Celeste to identify clouds and ignore them in processing.

Thanks!

Jeff

[T. Modes]

Hi Jeff,

eljef... schrieb am Mittwoch, 22. Dezember 2021 um 15:01:23 UTC+1:

I looked at the source code lines for error 3) and have no idea at all why the compiler would be giving that warning -- did you modify the source file so my line numbers don't match yours?

***

Yes, I modified already some lines to get further.

The lines with error " C2057: expected constant expression" are (all in skyfill_tif.c):

    float sum_h_by_row[n_max_rows] ;
    int n_h_by_row[n_max_rows] ;

        uint16 *output_buf[IMAGE_HEIGHT] ;

and here the link to your repo:

https://github.com/weltyj/SkyFill/blob/d27ce93688512d2279b823d7d8e6996339ee3dd8/skyfill_tif.c#L1353
https://github.com/weltyj/SkyFill/blob/d27ce93688512d2279b823d7d8e6996339ee3dd8/skyfill_tif.c#L1354
https://github.com/weltyj/SkyFill/blob/d27ce93688512d2279b823d7d8e6996339ee3dd8/skyfill_tif.c#L4802

To answer your questions:

* It doesn't use the projection from the pto file -- only the FOV.   What it needs to do the unblended sky model is the degrees in horizontal, so it can calculate the degrees per pixel.  The assumption is that the pixels are square, not rectangular.  The other assumption is that the degrees per pixel is a constant.   I'm pretty sure that's not true for many projections, but it only needs to be "true enough" to work.

Yes, a constant degree per pixel in horizontal and vertical direction has only the equirectangular projection. In cylindrical this holds only true horizontal, but not vertically.

Thomas

[Jeff Welty]

Thomas,

I made the changes to address your compile errors.  It compiles and runs for me.  Let me know how it goes.  New code on github now.

Jeff

[T. Modes]

Hi Jeff,

eljef... schrieb am Mittwoch, 22. Dezember 2021 um 15:50:10 UTC+1:

I made the changes to address your compile errors.  It compiles and runs for me.  Let me know how it goes.  New code on github now.

 

with some minor changes it compiles here now. I'm attaching a patch with all changes.

I'm also attaching a first draft for building with CMake. Place the CMakeLists.txt file beside the source file.

Then

>mkdir build

>cd build

>cmake /path/to/skyfill_source

>make

>make install

should build and install skyfill.

Enough for today. Will do more tests in the next days.

Thomas

[Bruno Postle]

It certainly works with this simple test, it isn't a panorama, but I was able to crop out some clouds in GIMP and replace them with blue sky:

[Jeff Welty]

I'm shocked it worked that well (though I can see obvious issues in the sky hue and saturation). I suspect the clouds may be incorrectly determined to be part of the sky -- and are affecting the estimate of sky hue and saturation.

To see where it estimates the start of sky and end of sky, run it with "-d3" -- the start of sky will be marked with red, end of sky with green, and the depth of the blending area marked with black.   Changing the tolerance of where it chooses the end of the sky may then be helpful.   If you look near the end of the huge amount of screen output, you will find a line like:

---

sky detect tolerances (hsv) are 360.000000, 0.300000, 0.030000

---

Using the tolerance flag can then allow the end of sky to stop sooner, by making the tolerances smaller.   Maybe lower the saturation tolerance a bit from the last run will help.

When i get in these situations I generally keep running with the "-d3" flag until the sky detection is acceptable, so I'd run it with: "-d3 -tol ...".  When you like it, just leave out the "-d3" and it will complete the process.

[T. Modes]

Hi Jeff,

you missed some necessary changes in skyfill_tiff.c - see attached patch.

Also your changes to the CMake system breaks building on Windows:

* The cmake policy was explicitly set to silence warning in underlying FindPackage(TIFF).

* You commented out the install target. It does not install the files automatically. Instead it adds only an install target. So by default it does nothing. But it's nice to have it available to easier get access to the generated executables.

* Never hardcode library names as with libm. You need to check it before usage.

See attached second patch.

Thomas

[Jeff Welty]

Hi Thomas,

Thanks for your patience with me.   And thanks for catching those final uints.  New skyfill_tif.c is back on github now.

I am not familiar with CMake other than as a user of it.  I had commented out those couple of lines to see if I could get it to run at all.  Then I discovered it needed the math library.

The CMakelists.txt you have provided won't work on my system (Linux Mint 19.3, CMake version 3.10.2)  As it is a stable version of Mint, it can be pretty out of date on some packages.  Here is the output when I run cmake:

---

CMake Error at CMakeLists.txt:5 (cmake_policy):
  Policy "CMP0074" is not known to this version of CMake.


-- Looking for exp in m
-- Looking for exp in m - found
CMake Error at CMakeLists.txt:46 (install):
  install TARGETS given no RUNTIME DESTINATION for executable target
  "skyfill".


-- Configuring incomplete, errors occurred!
See also "/home/welty/images/src/skyfill/build/CMakeFiles/CMakeOutput.log".

---

No rush to fix this.   I'm using the original Makefile I had which is lets me keep working with the code and trying it out.

Jeff

[T. Modes]

Hi Jeff,

you seem using a very old version of CMake. I tried to fix the remaining issue, so it should also work with the older version.

Also I found 4 more deprecated uint32 - this is also included in the patch.

Thomas

[Jeff Welty]

Hi Thomas,

That new Cmake works perfectly now.   Thanks very much.

I hope all those uint's are now found.

I made a significant improvement last night -- I directly modelled each of the sky H,S,V compoents as functions of pixel x and y from existing pixels.  I had tried that approach before but with a much simpler model.   The results are now MUCH better.  It has lost the feature to have a little "sun glow" from the edge of the frame, and I intend to add that back in.   For now, it's Christmas Eve and time to be with my family.   Happy Holidays!

Cheers,

Jeff

[Jeff Welty]

For those interested -- new and improved code on github.  The old CIE sky model as the basis for the sky has been discarded in favor of modeling H,S,V components separately as functions of X, and Y from the input image.  The CIE sun illuminance model is added, but it is still and early attempt and is at best "experimental".  The results are far superior than the previous method.  Look in the examples subdirectory to see how it is working -- along with a shell script that shows the exact command line parameters I used.

[Tobias]

Helo,

I'm not really an expert in colors, but last time I looked into HSV (HSB, HSL) I came to this Wikipedia page:

https://en.wikipedia.org/wiki/HSL_and_HSV#Disadvantages

An I think the conclusion it to use https://en.wikipedia.org/wiki/CIELUV or https://en.wikipedia.org/wiki/CIELAB_color_space#CIELAB instead.

Just my two cents. Feel free to ignore me. ;)

Regards,

Tobias

[Jeff Welty]

Hi Tobias,

The CIE I was incompletely refering to was not the CIELAB colorspace, but a CIE standards model for predicting the watts/m**2 at a given point in the sky.

Thanks for the interest though -- I don't mind input at all!

Cheers,

Jeff