Is FillSky an appropriate tool for fixing ...

[johnfi...@gmail.com]

I have many small isolated gaps in the sky of my photos from my recent vacation.  That is a bit different from the example use of FillSky.

Given how many different things I'm trying to learn to use at once now, I'd appreciate knowing whether FillSky is the right tool before I take time to figure out how to use it.  (At a quick first look, it was not obvious to me how to use it).

I got dirt on the sensor of my camera at the start of my vacation (I'm not used to this type of camera) and the air bulb I had could not blow the dirt off and I didn't have a cleaning kit.  So all my photos have dirt spots.

I can mask out the dirt spots where the matter and in non-sky areas of panoramas there is typically overlap with the photo above that can replace the masked out bit.  But for sky sections (where the dirt is most visible) there is no photo above.  So the mask leaves a gap in the final panorama.

Does FillSky automatically find and fill gaps in the sky?  Or can it easily be told to?  Or is it just for sky gaps created by a ragged top to the whole panorama?

[David W. Jones]

On March 4, 2022 8:51:50 AM HST, "johnfi...@gmail.com" <johnfi...@gmail.com> wrote:

> I got dirt on the sensor of my camera at the start of my vacation (I'm not
> used to this type of camera) and the air bulb I had could not blow the dirt
> off and I didn't have a cleaning kit. So all my photos have dirt spots.
>
> I can mask out the dirt spots where the matter and in non-sky areas of
> panoramas there is typically overlap with the photo above that can replace
> the masked out bit. But for sky sections (where the dirt is most visible)
> there is no photo above. So the mask leaves a gap in the final panorama.

When my sensor got dirty like that, I manually cleaned dirt spots from images before putting them in Hugin. Wasn't too hard for the sky, just copypasted clean patches over the spots.

Another possibility just occurred to me: make a synthetic "empty sky" frame the same dimensions etc as a real frame and add it to your panorama as many times as needed to be "clear sky backgrounds" for real frames that have dust masked out on them. I think you can adjust positions and such as need to do that in Hugin.

--
David W. Jones
gnome...@gmail.com
exploring the landscape of god
http://dancingtreefrog.com

Sent from my Android device with F/LOSS K-9 Mail.

[Jeff Welty]

Fillsky should work  for those problems --  any area *above* the detected end of sky which is black (i.e. r,g,b all 0), or alpha < 1. will be filled with the modelled esimate of the sky color.   But the trick will be to apply a test mask (-tm <left> <right> <top> <bottom>) to the area so the end of sky detection doesn't stop on those areas.   As I write this I realize there'd be some situations like yours where Skyfill should by default doesn't stop on black or transparent areas during end of sky detection.   Currently it does stop because there can be areas at the left and right edges of panaromas that are transparent after the crop, and the barrel distortion correction has left a concave area which I'd assume a use might want to repair in postprocessing after running Skyfill.

I have to get part 2 of the Tutorial done, but there is an example of the test mask usage in the first part of the Tutorial.   If your dirt spots are always in the same place it should be pretty easy to "can" the test mask flag in a script and run on multiple files.

I'd be happy to help show you on a particular image if you'd like.   I wouldn't mind having another test case too ;-)

[Jeff Welty]

... Currently it does stop because there can be areas at the left and right edges of panaromas that are transparent after the crop, and the barrel distortion correction has left a concave area which I'd assume a **user**

 might want to repair in postprocessing after running Skyfill...

[John Fine]

My existing examples are absurdly large.  I don't mind putting one on a google drive freely accessible.  But you might mind trying to download and work with it.

For several reasons, I prefer to fix the gaps after blending the panorama, rather than fix individual photos before.  One reason is that gaps might go away during blending because images overlap.  In that case I would prefer to end up with original from the image that doesn't have a gap there, rather than synthetic from the image that does.

I put a 680MB image on a google drive in case you want to look.  I'll try to make something more reasonable within the next few days.  If you look at that, there are three tiny gaps that I think are transparent (look white in a viewer) plus one made exactly the same way but looks partially white and mostly black.  I have to figure out why that one looks that way (unless your tool just fixes it).

https://drive.google.com/file/d/1eLvJsyDOrV06bc1YLAQRoG-1x4vv5gb4/view?usp=sharing

The whole panorama (taken in valley of fire NV) is still a work in progress.  There are a few blurred seams I want to figure out how to fix.  So I'm looking for how-to for this kind of thing, rather than fix this one.

[Jeff Welty]

I'll grab that, woah, yes that's large.  As an afterthought -- if you have darktable -- just use the retouch tool.  It'll be quick, easy, and like magic!

[Jeff Welty]

https://drive.google.com/drive/folders/1ds5QHI4qNYqXOLNexS0Ov6Ll14HnhkSV?usp=sharing

Hopefully you can see those files.   I retract my advice to use darktable's retouch tool -- there is a lot of sensor dust.

I ended up scaling the image by 50% because it was taking so long, but skyfill did happily work on the original full sized image.

I used the full sky replacement option (-fsr), which basically examines every pixel in the detected sky and assigns a probability that the pixel should be considered true sky.  If the probability is large enough it will be overwritten -- so magically all the sensor dust disappears.   That one spot near the horizon on the right where there is a black mask leaves some artifacts, because of some feathering at the edge of the mask where pixels were not completely black.

I will better document the -fsr option in the next tutorial for skyfill.

There is a shell script which shows the flags I used (just -fsr and a number of -tm flags).  I had to mask a few of the sensor dust spots because they were too severe and caused end of sky detection to stop at them.  The -tm mask tells skyfill to ignore that area for testing to see if the end of sky is there, and to not use that area as sample data to build the HSV sky model.

There are two runs of skyfill in the script, one with the -d3 option (debug level 3), which creates an image that shows the start of sky detected, end of sky detected, and where the masks are at.

A couple of other things:

Notice how that brown smear in the sky near the black-mask at the right is gone -- full sky replacement fixed that too.

The original full sized image caused a real problem to show up -- the large vertical shift in the sky data about at about 60% of the image width.   The problem is (and has been the biggest problem for skyfill), is pixels near the edges of the data can be just "wrong".  I'm not sure where the problem comes from, but in this example it is because the pixels on the edge near x=16278,y=551 are too dark.  Skyfill in the first few steps will fill in pixels just to the left of that vertical edge because it is not perfectly vertical, and is using a localized rgb model to extrapolate and create pixels to make the edge perfectly vertical.  Those dark pixels pollute the localized estimate of sky color...

[johnfi...@gmail.com]

Thankyou.

On Saturday, March 5, 2022 at 11:01:24 AM UTC-5 eljef...@gmail.com wrote:

https://drive.google.com/drive/folders/1ds5QHI4qNYqXOLNexS0Ov6Ll14HnhkSV?usp=sharing

Hopefully you can see those files.

Not yet.  I clicked on the "request access" button.

 

   I retract my advice to use darktable's retouch tool -- there is a lot of sensor dust.

You are right.  I thought I had used masks and the overlap between images to get rid of most of that.  But I did something wrong and lots ended up in the panorama.

 

I used the full sky replacement option (-fsr), which basically examines every pixel in the detected sky and assigns a probability that the pixel should be considered true sky.  If the probability is large enough it will be overwritten -- so magically all the sensor dust disappears.   That one spot near the horizon on the right where there is a black mask leaves some artifacts, because of some feathering at the edge of the mask where pixels were not completely black.

I still haven't found time to figure out where the black came from or where the feathering of the black came from.  None of that was in the original images.  That should have been just another gap in the final image.

I will better document the -fsr option in the next tutorial for skyfill.

Does it replace all of the sky that was already good?  Or does it detect what should be sky but isn't?

[Jeff Welty]

Hmm, give that link another try -- I haven't used google drive in that way, and I think I needed to change the access so anyone with the link can get it.

Answering your final question -- yes, full sky replacement will replace all pixels that are likely to be sky.   Internally, a very sophisticated model of the sky has been built.  It is usually a very, very good model in simple situations like this example.  But even the best model will not be good enough to prevent a noticeable edge at repair points, unless some sort of blending is employed.  Full sky replacement gets around this problem by replacing all pixels -- that are determined to be likely sky pixels.  By replacing all likely sky pixels, no spatial blending is required.  Full sky replacement uses probability of sky as a blending function at the pixel level.  Something like:

sky_prob = a value from 0 to 1.  It is how close the pixel color is to the modelled sky color at that x,y location.  It is a nonlinear function.

final_pixel_color = (1.-sky_prob)*original_pixel_color + (sky_prob)*modelled_sky_color