EXR output breaks stitching

[Matija Kogoj]

I have opened this issue before, but have since done many "things" and hope to continue here.

The main problem are black areas that appear in the EXR files while TIFF outputs seem fine.

I need the EXR format for 32b hdr, and there is no way around that.

It seems to occur randomly, becaues I have successfully stitched a small number of panoramas regardless. Those had similar problems but at one point the holes weren't present, so I just saved the images then.

System A - Desktop:

Windows 7 Pro - 64bit

intel i5

16GB RAM

Nvidia Quadro k2200 - 4GB Vram

System B - Laptop:

Windows 7 Ultimate - 64bit

intel i3

8GB RAM

Nvidia 635M - 2GB VRAM

On system A the black areas occur, while on B the stitching process just stops. I don't think it gave me an explicit error message. There were some logs left over after the process failed.

So far I have tried:

• increasing cache
• disabling GPU
• changing blender and interpolation
• adding control points
• removing control points
• photometric optimisation on HDR, LDR, and variable WB (unsure when to use var. WB, but that's for another post)
• Automatic align/stitch (simple interface mode)
  
• stitched from any arrangement,
• remap into layers and blend
• exclusion masks on stacks
• inclusion masks on stacks

Things that have been solved so far:

sun orb becomes grey at centre - usually one of the images is remapped improperly, requires checking of individual stacks in preview to notice (CP errors can still be <1px)

black patches in preview - probably intersecting masks. This can be solved almost realtime by identifying and adjusting masks with auto refresh.

merged stacks output - manually merge and output each stack in a separate project file instead of doing so from the whole panorama. The latter method drained all of the 16GB of computer's memory and crashed (while the total of all the TIF files is 3,5GB). I have yet to perform this procedure on the rest of the stacks (total 5), but am reluctant as it is time consuming and I have a feeling that once I put all the EXR stacks to be blended it will crash, hang, fail or something else I won't understand. Anyway, will try and report back.

I can only use the default engines for fusion and blending - I do not know how to use or install any external tools.

Because of the file size I can't effectively upload the project files. For now I will just link the results, and a video clip detailing the procedure in full, from my dropbox.

TIF stitch:

https://www.dropbox.com/s/r4f8dsnhh7yyp3u/Tiff_stitch.JPG?dl=1

exr stitch:

https://www.dropbox.com/s/e8h129bq9qa06hj/EXR_stitch.JPG?dl=1

exr stitcth after corrections:

https://www.dropbox.com/s/3lqy6mep9tfjvkq/EXR_stitch_aftercorrection.JPG?dl=1

Video:

https://www.dropbox.com/s/cstqxcp1pi6dt6h/Hugin_EXR_ISSUE.mp4?dl=1

Project file:

https://www.dropbox.com/s/87mrb8zfz44kqdl/KobiljaGlava_A.pto?dl=1

[T. Modes]

Am Donnerstag, 29. November 2018 20:42:58 UTC+1 schrieb Matija Kogoj:

On system A the black areas occur, while on B the stitching process just stops. I don't think it gave me an explicit error message. There were some logs left over after the process failed.

 

So far I have tried:

Sorry, but from this it is nearly impossible to give an advice. Most of the mentioned things have nothing directly to do with the output.

But the main question is unanswered. Which tool produces the black spots?

For HDR output there are several tools involved:

1. remapping with nona

2. merging stacks with hugin

3. blending the stacks with enblend or verdandi (internal blender)

So the first step is to check the intermediate images. Have the remapped images already the black spots? Or have the merged stacks these defects? Or only the blended panorama?

So this is the first questions to be answered.

Thomas

PS: If only the blended panorama has the black spots then there are 3 further possible options to test.

There are 4 blending algorithms available in default installation:

* blending with standard enblend (graph-cut algorithm)

* blending with enblends nearest-feature transformation algorithm (add --primary-seam-generator=nft to the enblend options)

* blending with internal blender (verdandi, hard seam)

* blending with internal blender and soft seam (select under internal blender options).

[Matija Kogoj]

"so far I have tried" refers to actions that had no mentionable effect on the result, meaning that the issue is probably not with them individually... which you confirmed, but well, I tried.

So the first step is to check the intermediate images. Have the remapped images already the black spots? Or have the merged stacks these defects? Or only the blended panorama?

So this is the first questions to be answered.

 

For clarity I will use the word "gap" to refer to what you call "spot." I regularly see other spots from remapped images in the preview which are not present in the final render.

When I output layers today one of the layers ended up with a gap. Adding control points to that layer moved the gap to a different layer, and adding control points to that layer moved it back to the first. At this point there is an average 0.4 error on the stitcher.

But the main question is unanswered. Which tool produces the black spots?

For HDR output there are several tools involved:

1. remapping with nona

2. merging stacks with hugin

3. blending the stacks with enblend or verdandi (internal blender)

 

Since there were no error messages or logs that would return an error, I believed the problem was with enblend in this particular panorama - it broke an image with at least 80% of layers remapped properly. It could be a remapping issue, but that would probably show gaps when inspecting images in the preview/drag/move tab?

I need to point out that here two of the image stacks overlap just barely, probably 1/5th right on an area with low contrast (hazy area on the left from the sun). While they stitch badly there are no gaps there.

...

* blending with internal blender and soft seam (select under internal blender options).

In this case both - enblend and builtin had the same problem, though I don't think I tried with soft seam (will do). Blending with nearest-feature did not have noticeable effect. On some of previous images builtin returned a panorama without gaps (spots), but all seams were strikingly visible...

Other notes for anyone experiencing similar issues:

Photometric optimization results in high values - consistently I find that when photometric optimizer calculates an EV range around 20 stops or more, it means that the exr panorama will have a spot [gap]. I suspect this is because the colour under it is much darker than anything in the actual scene.

Stitch panorama from EXR stacks - I was unable to finish through this method due to not knowing which output projections to set and how to read them in again properly. Projecting each stack to to equirectangular and raeding back from it seemed like a good idea but did not align properly (yet).

[T. Modes]

Am Freitag, 7. Dezember 2018 14:24:59 UTC+1 schrieb Matija Kogoj:

For clarity I will use the word "gap" to refer to what you call "spot." I regularly see other spots from remapped images in the preview which are not present in the final render.

When I output layers today one of the layers ended up with a gap.

This is not becoming clearer. I thought you wanted HDR output? Exposure layers are only outputted for LDR output. ???

But the main question is unanswered. Which tool produces the black spots?

For HDR output there are several tools involved:

1. remapping with nona

2. merging stacks with hugin

3. blending the stacks with enblend or verdandi (internal blender)

 

Since there were no error messages or logs that would return an error,

This question is still unanswered. To make it more clearer: on the stitcher tab select

* HDR output

* remapped images HDR

* stacks HDR.

This will keep all intermediate images and you should check which of the intermediate images have the black gap.

* the remapped images prefix_hdr_00xx.exr

* the merged stacks prefix_stack_hdr_00xx.exr

* or the final pano prefix_hdr.exr

 

[Matija Kogoj]

First of all - I have no way of knowing which tool produces the error directly.

• soft-seam algorithm on verdandi blender renders with same issue
• khan deghosting has no noticable effect
• changing Nona remapping algorithms between nearest-neighbor, sinc1024 and bicubic(poly3) resulted with same issue

As for -

Am Freitag, 7. Dezember 2018 14:24:59 UTC+1 schrieb Matija Kogoj:

For clarity I will use the word "gap" to refer to what you call "spot." I regularly see other spots from remapped images in the preview which are not present in the final render.

When I output layers today one of the layers ended up with a gap.

This is not becoming clearer. I thought you wanted HDR output? Exposure layers are only outputted for LDR output. ???

Yes, I need the EXR, but the results so far coupled with my understanding of hdr imaging suggested I could merge separate layers into a complete exr by simply stacking finished layers atop one another. Regardless, one of the layers consistently breaks as described in the previous post.

Please observe the error in the middle of the bottom image. None of the other layers have this problem.

https://www.dropbox.com/s/h5p69ocoyq2ztdc/layer_stitcherror.JPG?dl=1

At the next link you can see there is no such gap in the panorama itself.:

https://www.dropbox.com/s/t726r9vzf1264cc/nogap_toomuchoverlap.JPG?dl=1

Anyway, I output the images you suggested - HDR, remapped, and stacks. In summary:

• there were no strange gaps or other issues in individual stacks that I could see through Luminance - https://www.dropbox.com/s/beiq32lde1ui861/stacks_luminance.JPG?dl=1
• Bringing one of the stacked exr back into hugin showed some sort of rip - https://www.dropbox.com/s/dfo001bdpd4dphu/stacks_ripped.JPG?dl=1

Is there anything particular that I should be looking for?

Furthermore, a stack of 5 images covered an areea opposite from the sun that was already covered with other images (looked redundant in preview). After removing that stack:

• gap moved from layer 04 to layer 03 on merged layers of similar exposure
  
• large gap appeared 90° from the sun (more or less covered by 2 stacks of images) in final exr panorama
  
• sun orb became grey (possibly due to sun now missing from layer 03) in final panorama
  
• sample: https://www.dropbox.com/s/beuepobsvcnrx9g/remove_redundant.JPG?dl=0
  

While making another output to LDR layers I noticed an error.

Blending exposure layer 0...

enblend: info: loading next image: KobiljaGlava_A2_exposure_layers_0000.tif 1/1
enblend: info: loading next image: KobiljaGlava_A2_exposure_layers_0005.tif 1/1
enblend: warning: unable to run Dijkstra optimizer
enblend: note: seam-line end point outside of cost-image

I don't think I've seen this error so far, though the stitcher window was usually returning lines much faster and I could have missed it. This also happened only once; I likely reverted back to an earlier saved file later.

IActually, could this be caused by cropped edges in the source images? I have a circular fisheye lens which is not perfectly round, but is instead cut off at top and bottom because it falls slightly outside of the sensor area. See the link https://www.dropbox.com/s/dfo001bdpd4dphu/stacks_ripped.JPG?dl=1 again please (it is the two semicircular indentations on each side) - is it possible that Hugin would calculate the empty edges as well, somehow, and that it would cause errors?

[Matija Kogoj]

Update:

I tried a new project file - same day, similar location, but with clouds; 5 sets of 6 images with 2 stop difference and exported to TIFF with identical settings as previous. It stitched perfecty after I had to recentre the sun orb once. Finishede EXR came out perfectly in less than an hour.

This means that there should be an issue with the images I took for the problems described in this thread, but visually I haven't seen any difference in the two months since I've taken them.

At the following link is the file I made in Hugin by exporting to LDR layers and trying to merge those layers into EXR. I show this as a curiosity, as I've no idea what happened. There is another thing though - please find a black triangle near the lower-right corner of the image. That piece was empty in every layer and could have caused this - I had the images retouched by filling in the triangle and merged in photoshop through the usual HDR dialogue, and the result was good.

https://www.dropbox.com/s/dxg8qax3hr9161e/stack_and_merge.JPG?dl=1

[T. Modes]

Am Samstag, 8. Dezember 2018 16:51:15 UTC+1 schrieb Matija Kogoj:

Anyway, I output the images you suggested - HDR, remapped, and stacks. In summary:

• there were no strange gaps or other issues in individual stacks that I could see through Luminance - https://www.dropbox.com/s/beiq32lde1ui861/stacks_luminance.JPG?dl=1
• Bringing one of the stacked exr back into hugin showed some sort of rip - https://www.dropbox.com/s/dfo001bdpd4dphu/stacks_ripped.JPG?dl=1

Is there anything particular that I should be looking for?

If there are no gaps in the stacks (at the same place as in the final pano), then the culprit is the blender. And changing the blender should change the output.

The point is all stacks. The example you have shown covers only these places where no gaps are in the final pano.

[Matija Kogoj]

Remapped group (one image is masked):

https://www.dropbox.com/s/dp92f6thry8hiuq/builtin_softseam.JPG?dl=1

Separate images will be referred to as:

1, 2, 3,
4, 5, 6.

Enblend

https://www.dropbox.com/s/0pfxi7gkpidwap9/A3_exr_Enblend.JPG?dl=1

Enblend with NFT:

https://www.dropbox.com/s/behos2r38ru0vze/enblend_nft.JPG?dl=1

Builtin (Verdandi?) with hard seam setting:

https://www.dropbox.com/s/8h9saqgsem2obf8/builtin_hardseamfast.JPG?dl=1

Builtin (Verdandi?) with soft seam setting:

https://www.dropbox.com/s/dp92f6thry8hiuq/builtin_softseam.JPG?dl=1

As you can see all images seem to be projected to equirectangular and displaced properly.

I had all stacks combined into 32b hdr EXR files in Photoshop, and aligned in the process. I have noticed a small discrepancy in one of the stacks - guessing about 3-5 pixels of difference between the merged images.

It seems to me that one of the images is writen as transparency-only at the blending step before any actual blending or pixel interpolation happens.

I suspect that due to very high overlap between images 2 and 6 Hugin decides to remove one of them as a performance optimization attempt. https://www.dropbox.com/s/h5p69ocoyq2ztdc/layer_stitcherror.JPG?dl=1 was an interesting factor to consider as only one of the layers was broken, and if fixed a different layer would break.

Adding large inclusion and exclusion masks to the images in question had no effect, however. It is possible that this could be circumvented/hacked by overlapping various masks - because the opposite problem of actual black spots appearing where masks intersected.

I would like to request someone else attempt to build this panorama again. At the following wetransfer links I have uploaded the relevant files along with the project file for the TIF group. Due to their size the packages were split in two.

1. https://wetransfer.com/downloads/feca2675739da2ee38c939e88a16250a20181212200354/bb2590f09141a3c9039719e205d59e4020181212200354/b3faf5
2. https://wetransfer.com/downloads/61acae57b35b7f2bee1744f6363a7d5b20181212201526/a70a5f47e40e08d184de5f1d944c414420181212201526/e2a5ce

The EXR stacks are included. Two things are important to test: try to build a panorama from the included project file with the settings as they are, and then again with the EXR. There is no need for precision as the error should manifest in the same way it has so far - either one of the images will be obviously invisible and the panorama broken or it will be present in some capacity.

The first time I mentioned an issue with Hugin someone said that the error was not reproducible... I have noticed that my computers have some unique, endemic problems so it is possible that the reoccurring error is systemic on my end. If possible I would like someone to try to stitch this panorama and post their general method and result. My gear and method:

• Samyang 8mm f/3.5 fisheye (old model)
  
• Canon EOS 5D mark III full frame

For the method you can follow the aforementioned screencapture video from Dropbox. Otherwise:

1. load directly into expert or advanced window
2. stacks are detected, lens parameters set to: 8mm circular fisheye, crop factor (note: sometimes setting focal length to 8mm sets HFOV to 250° instead of 172 - cause unknown)
3. Set Crop on all images (a few pixels in from the edge) (for the EXR stacks it is necessary to specify the lens as well)
   
4. Control points, F3 & manually clean points with largest distance or in low contrast areas etc,
5. Everything except translation geometric optimisation
   
6. HDR fixed exposure photometric optimisation
7. stitcher to EXR hdr output, everything else is irrelevant on my system.
   

[T. Modes]

Oh man, you sent me in the wrong direction. The stacks show clearly the same gaps as the final image.

So there are 2 issues: The first one is the blender

Am Mittwoch, 12. Dezember 2018 22:35:08 UTC+1 schrieb Matija Kogoj:

Enblend

https://www.dropbox.com/s/0pfxi7gkpidwap9/A3_exr_Enblend.JPG?dl=1

Enblend with NFT:

https://www.dropbox.com/s/behos2r38ru0vze/enblend_nft.JPG?dl=1

Enblend with NFT is ok (and all with the internal blender). The gaps are already in the merged HDR stacks.

These gaps are caused by a combination of 2 factors: By default Hugin is looking at the overlap to determine with images belongs to one stack. With the high overlap between 2 stacks in your project these images from 2 shooting stacks will be merged into one (output) stack. The assigned stack numbers will be ignored by default. See https://wiki.panotools.org/Hugin_Stitcher_tab

Now a second issue adds it contribution. Under HDR merge option the option "-c consider only pixel that are defined in all images" is activated.

These 2 aspects together result in the gaps.

To solve this switch to expert mode, go to photo tab and select grouped by output stacks. Under "Minimum overlap" say -1 to use your assigned stacks. (I will extend the tooltip for this input field.)

Maybe when the issue with the gap is solved the blending issue is also solved, because there are now other seams.

[Matija Kogoj]

SOLVED: gaps

Thank you mr. Modes, you were right - the output stacks were different from what I set initially, and it was possible to change pictures to a new stack. That solved the gap problem. It should be noted that for output stacks to become available I had to stitch one panorama first.

This just leaves the grey clipping in the sun orb and related highlights. While I don't know what causes this problem yet it seems to be related to image mapping within a stack; placing control points on the sun itself fixed the problem twice or three times so far.

[T. Modes]

Am Dienstag, 18. Dezember 2018 15:38:08 UTC+1 schrieb Matija Kogoj:

This just leaves the grey clipping in the sun orb and related highlights. While I don't know what causes this problem yet it seems to be related to image mapping within a stack; placing control points on the sun itself fixed the problem twice or three times so far.

Disable GPU remapping. With CPU remapping it should be fine.

There was a bug in the GPU remapping code. This is fixed now in the repository.