2015 beta2 ram usage

[Fotografia]

Hello everyone,

 i have just installed 2015 beta2 from ppa ubuntu next on lubuntu 14.04 64 bit and found out a strange problem (and also a crash reported in the 2015 beta2 thread).

This time i choosed control points manually, it needed a lot of manual adjustments but at the end alignment was almost perfect ;) .

Trying to stich a panorama consisting of 2 horizontal images (16 megapixel each one, tiff files developed from raw with rawtherapee) i have found out that ptbachergui uses more than 4gb of ram (all i have in this pc) and it's incredibly slow (in fact i had to kill it since it was swapping a lot), but it took more than 6 minutes without a result.

Also with that tiff files the preview windows shows wrong exif values (68.882 focal length and 0.599 crop factor) while exiftool shows this:

exiftool DSCF3061.tif |grep -i focal
Focal Length                    : 25.6 mm
Focal Length                    : 25.6 mm (35 mm equivalent: 38.0 mm)

I can upload images and pto if needed.

Thank you in advance.

[Carlos Eduardo G. Carvalho (Cartola)]

Usually the RAM problem during stitch happens in the enblend step. You can try the new internal blender by changing it at the Stitcher tab. enblend is the default.

You can also try to use enblend with "-m 3000" option to limit RAM usage, but it will use disk.

The bigger the final output you configure the bigger this problem will be. You can also try smaller final outputs. I usually stitch 23megapixel tif files with 16pp from raw also treated at rawtherapee and I use 4 images to make a fullsphere panorama. I usually configure the final output as 12000 x 6000 and I can stitch it in 4GB machines running ubuntu. I've had a strange experience that I don't know if is related to hardware. On ubuntu 14.04 a specific hugin project took 52 minutes to stitch, while at the same machine, rebooted with 12.04, stitching the same project took 2.5 minutes. I couldn't discover the reason. Some people suggested that it was due to changes on xorg implementation, but I couldn't solve it. At this machine nowadays I use ubuntu 12.04.

Bests,

Carlos E G Carvalho (Cartola)
http://cartola.org/360
http://www.panoforum.com.br/

--
A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ
---
You received this message because you are subscribed to the Google Groups "hugin and other free panoramic software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hugin-ptx+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hugin-ptx/f79cac72-b1bf-45c4-827d-4b518d6687ee%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Fotografia]

Il giorno martedì 2 giugno 2015 22:50:48 UTC+2, Cartola ha scritto:

Usually the RAM problem during stitch happens in the enblend step. You can try the new internal blender by changing it at the Stitcher tab. enblend is the default.

You can also try to use enblend with "-m 3000" option to limit RAM usage, but it will use disk.

Thank you for the idea and the suggestion.

I tried again, it's not enblend but it's ptbatcher gui. From preferences i choosed stiching - processor - hugin_stich_project instead of ptbatchergui and in few seconds the panorama was made, enblend took at max 320mb or so of ram.

So i'm thinking there is some memory problem with ptbatchergui. With 2013 ubuntu packaged version i used till few days ago there weren't ram usage problems.

The bigger the final output you configure the bigger this problem will be. You can also try smaller final outputs. I usually stitch 23megapixel tif files with 16pp from raw also treated at rawtherapee and I use 4 images to make a fullsphere panorama. I usually configure the final output as 12000 x 6000 and I can stitch it in 4GB machines running ubuntu. I've had a strange experience that I don't know if is related to hardware. On ubuntu 14.04 a specific hugin project took 52 minutes to stitch, while at the same machine, rebooted with 12.04, stitching the same project took 2.5 minutes. I couldn't discover the reason. Some people suggested that it was due to changes on xorg implementation, but I couldn't solve it. At this machine nowadays I use ubuntu 12.04.

Thank you for the info. 

[T. Modes]

Am Mittwoch, 3. Juni 2015 09:28:59 UTC+2 schrieb Fotografia:

I tried again, it's not enblend but it's ptbatcher gui. From preferences i choosed stiching - processor - hugin_stich_project instead of ptbatchergui and in few seconds the panorama was made, enblend took at max 320mb or so of ram.

So i'm thinking there is some memory problem with ptbatchergui. With 2013 ubuntu packaged version i used till few days ago there weren't ram usage problems.

I can't remember big changes between 2013.0 and 2014.0 to PTBatcher. Also hugin_stitch_project and PTBatcherGUI use the same code to do the stitching. Also PTBatcherGUI does not open any images or save images. So I can't imagine how PTBatcherGUI should occupy so much memory. So I assume there is a change in an underlying library responsible for the observed behaviour.

Also with that tiff files the preview windows shows wrong exif values (68.882 focal length and 0.599 crop factor) while exiftool shows this:

exiftool DSCF3061.tif |grep -i focal
Focal Length                    : 25.6 mm
Focal Length                    : 25.6 mm (35 mm equivalent: 38.0 mm)

I can upload images and pto if needed.

Please upload the image, so I can check this.

(Hugin is using exiv2 and not exiftool when loading images.)

[Fotografia]

Il giorno mercoledì 3 giugno 2015 19:55:45 UTC+2, T. Modes ha scritto:

Am Mittwoch, 3. Juni 2015 09:28:59 UTC+2 schrieb Fotografia:

I tried again, it's not enblend but it's ptbatcher gui. From preferences i choosed stiching - processor - hugin_stich_project instead of ptbatchergui and in few seconds the panorama was made, enblend took at max 320mb or so of ram.

So i'm thinking there is some memory problem with ptbatchergui. With 2013 ubuntu packaged version i used till few days ago there weren't ram usage problems.

I can't remember big changes between 2013.0 and 2014.0 to PTBatcher. Also hugin_stitch_project and PTBatcherGUI use the same code to do the stitching. Also PTBatcherGUI does not open any images or save images. So I can't imagine how PTBatcherGUI should occupy so much memory. So I assume there is a change in an underlying library responsible for the observed behaviour.

Thank you, for now i'll solve the problem using hugin_stich_project .

Also with that tiff files the preview windows shows wrong exif values (68.882 focal length and 0.599 crop factor) while exiftool shows this:

exiftool DSCF3061.tif |grep -i focal
Focal Length                    : 25.6 mm
Focal Length                    : 25.6 mm (35 mm equivalent: 38.0 mm)

I can upload images and pto if needed.

Please upload the image, so I can check this.

(Hugin is using exiv2 and not exiftool when loading images.)

https://goo.gl/zGOyJ5

Thank you.

[T. Modes]

Am Mittwoch, 3. Juni 2015 21:15:23 UTC+2 schrieb Fotografia:

Also with that tiff files the preview windows shows wrong exif values (68.882 focal length and 0.599 crop factor) while exiftool shows this:

exiftool DSCF3061.tif |grep -i focal
Focal Length                    : 25.6 mm
Focal Length                    : 25.6 mm (35 mm equivalent: 38.0 mm)

I can upload images and pto if needed.

Please upload the image, so I can check this.

(Hugin is using exiv2 and not exiftool when loading images.)

Okay, the problem are inconsistent metadata in your file:
exiftool -Focal* -ImageWidth -ImageHeight -e DSCF3061.tif
Focal Length                    : 25.6 mm
Focal Plane X Resolution        : 820
Focal Plane Y Resolution        : 820
Focal Plane Resolution Unit     : cm
Focal Length In 35mm Format     : 38 mm
Image Width                     : 4938
Image Height                    : 3274

From the focal length a crop factor of 1.5 can be calculated.
But from the sensor size calculated from the image size and focal plane resolution a crop factor of 0.6 can be calculated
(In jpegs from the same camera a Focal Plane X|Y resolution of 2092 1/cm is stored. This gives also the correct crop factor.)

In Hugin we could change the code to prefer the crop factor from the focal length and fall back to the second variant in case the one of the focal length is missing.
But I don't know if the breaks the calculation of other camera. Opinions?

(The correct case would be to fix this in the raw converter. It should write the correct informations.)

Thomas

[Fotografia]

I have checked and with darktable the exiftool output is the same.

Focal Length                    : 25.6 mm
Focal Plane X Resolution        : 820
Focal Plane Y Resolution        : 820
Focal Plane Resolution Unit     : cm
Focal Length In 35mm Format     : 38 mm

Image Width                     : 4952
Image Height                    : 3288

I post a link to rawtherapee bug tracking system: https://code.google.com/p/rawtherapee/issues/list
and to darktable's one: http://darktable.org/redmine/projects/darktable/issues

(but maybe the problem is in dcraw...)

Thank you.

[Rogier Wolff]

On Thu, Jun 04, 2015 at 11:43:41AM -0700, T. Modes wrote:

> But I don't know if the breaks the calculation of other camera. Opinions?

IMHO, we can/might "work around" bugs in commercial software or
firmware in the cameras. Those take ages to get fixed.But if it is
another open source package that wrote this we should tell those guys
to fix their software. If we silently work around bugs in other open
source software you're going to get an escalation of "fix for bug" ->
have to maintain compatiblity -> even weirder shit.

So if what we're doing is "provably" right. That's how it should be.

(Sure we COULD use the other parameters leading to the correct result,
but we SHOULD also be allowed to use the parameters that were actually
used.)

Roger.

--
+-- Rogier Wolff -- www.harddisk-recovery.nl -- 0800 220 20 20 --
- Datarecovery Services Nederland B.V. Delft. KVK: 30160549 -
| Files foetsie, bestanden kwijt, alle data weg?!
| Blijf kalm en neem contact op met Harddisk-recovery.nl!

[T. Modes]

Am Donnerstag, 4. Juni 2015 22:32:28 UTC+2 schrieb Fotografia:

I post a link to rawtherapee bug tracking system: https://code.google.com/p/rawtherapee/issues/list
and to darktable's one: http://darktable.org/redmine/projects/darktable/issues

(but maybe the problem is in dcraw...)

I changed the order in Hugin. But I will not fill a ticket in one of the mentioned trackers.
This should do somebody who is using the program and can provide more information when needed and can do test if the fix it.

[Fotografia]

I can do this for Raw Therapee. Can i post there the text in your message about the focal plane resolution problem?

Thank you.

[Fotografia]

Sorry for another message but i think i've found another related problem.

Still using beta2 from ppa, i have tried to stitch 3 images (this time jpegs straight from my compact camera) but this time i get a negative field of view and i'm unable to proceed: https://goo.gl/zGOyJ5

Thank you again.

[T. Modes]

Am Sonntag, 7. Juni 2015 22:15:59 UTC+2 schrieb Fotografia:

Can i post there the text in your message about the focal plane resolution problem?

Yes. You can use it.
 

Sorry for another message but i think i've found another related problem.

Still using beta2 from ppa, i have tried to stitch 3 images (this time jpegs straight from my compact camera) but this time i get a negative field of view and i'm unable to proceed: https://goo.gl/zGOyJ5

This seems to be another problem. I assume an issue with the cam/lens database. Can you please post your camera and lens database file? (You will find the name in the about screen. It should be ~\.hugin\camlens.db)

Thanks

[Fotografia]

Il giorno lunedì 8 giugno 2015 17:23:08 UTC+2, T. Modes ha scritto:

Am Sonntag, 7. Juni 2015 22:15:59 UTC+2 schrieb Fotografia:

Can i post there the text in your message about the focal plane resolution problem?

Yes. You can use it.

Ok.
 

 

Sorry for another message but i think i've found another related problem.

Still using beta2 from ppa, i have tried to stitch 3 images (this time jpegs straight from my compact camera) but this time i get a negative field of view and i'm unable to proceed: https://goo.gl/zGOyJ5

This seems to be another problem. I assume an issue with the cam/lens database. Can you please post your camera and lens database file? (You will find the name in the about screen. It should be ~\.hugin\camlens.db)

Uploaded to  https://goo.gl/zGOyJ5 in the ExifError folder.

Thank you.

[T. Modes]

Am Montag, 8. Juni 2015 18:39:54 UTC+2 schrieb Fotografia:

Uploaded to  https://goo.gl/zGOyJ5 in the ExifError folder.

Thank you.

Should be fixed now in repository.

[Fotografia]

Thank you.

I was preparing the files for posting a message in the rt issues list, and i have found out a really strange thing: JPG and RAF exifs are different and that should explain why rt and darktable are writing partially incorrect exif data to the output files.

https://goo.gl/zGOyJ5 FujiRafJpg dir.

======== DSCF3061.JPG
Focal Length                    : 25.6 mm
Focal Plane X Resolution        : 2092
Focal Plane Y Resolution        : 2092

Focal Plane Resolution Unit     : cm
Focal Length In 35mm Format     : 38 mm

Image Width                     : 4896
Image Height                    : 3264
======== DSCF3061.RAF

Focal Length                    : 25.6 mm
Focal Plane X Resolution        : 820
Focal Plane Y Resolution        : 820
Focal Plane Resolution Unit     : cm
Focal Length In 35mm Format     : 38 mm

Image Width                     : 1920
Image Height                    : 1280

Thank you.

[T. Modes]

Am Montag, 8. Juni 2015 23:40:23 UTC+2 schrieb Fotografia:

I was preparing the files for posting a message in the rt issues list, and i have found out a really strange thing: JPG and RAF exifs are different and that should explain why rt and darktable are writing partially incorrect exif data to the output files.

======== DSCF3061.JPG
[EXIF:ExifIFD]  Focal Plane X Resolution        : 2092
[EXIF:ExifIFD]  Focal Plane Resolution Unit     : cm
[EXIF:ExifIFD]  Exif Image Width                : 4896

with this you get a sensor width: 23.4 mm

for the jpeg inside the raw
======== DSCF3061.RAF
[EXIF:ExifIFD]  Focal Plane X Resolution        : 820
[EXIF:ExifIFD]  Focal Plane Resolution Unit     : cm
[EXIF:ExifIFD]  Exif Image Width                : 1920

you get also a sensor width of 23.4 mm

So the Focal Plane resolution is for the jpeg embedded in the raw (when you extract the jpeg with exiftool you see that it contains these focal plane resolutions).
The raw converter reads the exif from the embedded jpeg, updates the Exif Image Width but not the Focal Plane resolution when saving as tif

======== DSCF3061.tif
[EXIF:ExifIFD]  Focal Plane X Resolution        : 820
[EXIF:ExifIFD]  Focal Plane Resolution Unit     : cm
[EXIF:IFD0]     Image Width                     : 4938

and so you get a sensor width of 60.2 mm

So the raw converter should update the focal plane resolution when it updates the image size tags.
But this needs to checked if this is valid for different raw formats. This is beyond my scope.

Thomas

[Rogier Wolff]

On Tue, Jun 09, 2015 at 08:29:46AM -0700, T. Modes wrote:
> So the raw converter should update the focal plane resolution when it
> updates the image size tags.
> But this needs to checked if this is valid for different raw formats. This
> is beyond my scope.

Again, this is the "obviously right" thing to do. Under my clause: if
it makes interfacing with closed-source-hard-to-change-hard/software
difficult, then a workaround might be implemented. But by default such
a bug should be fixed first, and where it isn't applicable due to bugs
in commercial/closed-source/embedded stuff, THEN a case-by-case
exception could be made.

The code likely does: copy the whole exif, add/overwrite the things we
know (resolution in pixels) and done! Alas, it is not that simple. :-(