nona causes server reboot

[Erik Krause]

Hello,

I use nona to extract cubefaces, thumbnails and preview images from
equirectangular source images on the WWP server (Ubuntu 14.04.5 LTS).
This used to work very well until recently, when I encountered a special
jpeg file. This file opened normally in Irfanview and photoshop, but the
windows 10 properties dialog showed "Compression: LZW", something I
didn't see before. nona crashed on that and caused a system reboot any
time I tried. After I re-compressed the image using photoshop CS6 this
didn't happen any more.

However, local nona in WSL (Ubuntu 18.04.3 LTS) didn't crash on that
strange file. This might be due to the fact that I didn't install nona
on the server using a package manager, since I don't have sudo or root
privileges there. Instead I unpacked the package and copied the files
manually one by one until nona didn't complain about missing libraries
any more. Of course I had to change LD_LIBRARY_PATH in .bashrc for this
to work.

Could it be that I missed a module but nona doesn't complain?

Does anyone know about that strange jpeg format apparently created on a
mac with photoshop?

What would be the correct and easiest way to install nona on a server
without sudo privilegs (if that is the reason for the crash)? Can I use
homebrew?

--
Erik Krause
http://www.erik-krause.de

[Bruno Postle]

Hi Erik, 'compression: lzw' sounds like this might be a tiff file with a jpg filename extension. I hope you still have the file so we can investigate.

This doesn't explain why it crashed the machine though. The only way this can happen is if the process uses too many resources, and some management system rebooted the machine to recover. Resources being memory, CPU, disk space etc..

Is the nona process being run by the web-server user as part of a web POST? This could easily bring down Apache (or whichever httpd server you are using). You might be able to reduce the resources available to each process/thread, this would at least keep the server running for other users.

An alternative design would have a separate daemon process that picked up the image files one at a time and executed nona outside of the httpd software altogether.

But your comment about .bashrc suggests that you are running this conversion process manually, are you connecting via SSH and running a script?

Without root access you are limited to copying files and libraries, pretty much as you have been doing :(
You could use PTmender instead of Nona, the scripting would be the same, but PTmender has fewer dependencies.

--
Bruno

[Erik Krause]

Am 20.10.2020 um 00:31 schrieb Bruno Postle:

> Hi Erik, 'compression: lzw' sounds like this might be a tiff file
> with a jpg filename extension.

I suspected the same, but irfanview usually complains in this case. In
the hex editor it looks like any other jpeg.

> I hope you still have the file so we
> can investigate.

Yes, get it from https://we.tl/t-lJNIVgNBly

> This doesn't explain why it crashed the machine though. The only way
> this can happen is if the process uses too many resources, and some
> management system rebooted the machine to recover. Resources being
> memory, CPU, disk space etc..

I checked that. Memory went down of course, but not as far as with other
images, at least until logging stopped. I logged in 1 second intervals.
CPU wasn't heavily loaded either. Disk space should be no problem.

> Is the nona process being run by the web-server user as part of a web
> POST?

No, it's run asynchronously. The conversion script call is added to a
screen command, so calls are queued to execute one after the other. I
tested this with up to 12 pending conversions and it ran fine.

> Without root access you are limited to copying files and libraries,
> pretty much as you have been doing:( You could use PTmender instead
> of Nona, the scripting would be the same, but PTmender has fewer
> dependencies.

I used PTMender before, but it is much slower. Will go back if I can't
solve it otherwise.

[Erik Krause]

Am 20.10.2020 um 00:31 schrieb Bruno Postle:

> 'compression: lzw' sounds like this might be a tiff file with a jpg filename extension.

Compression LZW is in the exif data. It's probably not actually LZW
compressed. Is nona affected by EXIF data?

And I forgot to mention: I use nona 2018.0.0.5abfb4de7961

[Bruno Postle]

On Tue, 20 Oct 2020 at 09:45, Erik Krause wrote:
>
> Am 20.10.2020 um 00:31 schrieb Bruno Postle:
>
> > 'compression: lzw' sounds like this might be a tiff file with a jpg filename extension.
>
> Compression LZW is in the exif data. It's probably not actually LZW
> compressed. Is nona affected by EXIF data?
>
> And I forgot to mention: I use nona 2018.0.0.5abfb4de7961

It opens fine here and nona has no problem. It is a bit odd, `file` says:

obb-nightjet_wwp.jpg: JPEG image data, Exif standard: [TIFF image
data, big-endian, direntries=16, height=10549, bps=0, compression=LZW,
PhotometricIntepretation=RGB, width=5925], progressive, precision 8,
9848x4924, frames 3

Which suggests there is a 10549x5925 LZW TIFF image in the Exif data,
but all that exiftool finds is two small thumbnails (the panorama
itself is 9848x4924). There is also no JFIF header, I don't know if
this indicates any problem.

A remaining possibility is that there is some incompatible mixture of
system libraries and libraries that you have uploaded, you can check
to see which libraries are actually being used by an executable or
library with the `ldd` command:

ldd /path/to/nona
ldd /path/to/libwhatever.so

--
Bruno

[Erik Krause]

Am 20.10.2020 um 11:57 schrieb Bruno Postle:

> A remaining possibility is that there is some incompatible mixture of
> system libraries and libraries that you have uploaded, you can check
> to see which libraries are actually being used by an executable or
> library with the `ldd` command

The complete output of ldd -v for both nona instances can be found on
https://we.tl/t-Jsw6BqEckG (4kB)

The libraries I copied to the server (all in the same directory as nona)
are binary identical to the local installation.

I'd be grateful if anyone could have a look...

[Bruno Postle]

On 20 October 2020 21:23:10 BST, Erik Krause <wrote:
>
>The complete output of ldd -v for both nona instances can be found on
>https://we.tl/t-Jsw6BqEckG (4kB)
>
>The libraries I copied to the server (all in the same directory as nona)
>are binary identical to the local installation.
>
>I'd be grateful if anyone could have a look...

Maybe somebody else can see something I can't, this looks fine to me.

You have copied over a lot of libraries, but you are using the libtiff and libjpeg that are already on the server - this really should not be a problem - as a test you could try uploading these too, then the set of libraries would be basically the same on both machines.

--
Bruno

[Gunter Königsmann]

To me "server reboot" and "unfinished file" both look like the server ran out-of-RAM, somehow which caused its kernel to kill a vital process.

[Erik Krause]

Am 21.10.20 um 14:34 schrieb Gunter Königsmann:

> To me "server reboot" and "unfinished file" both look like the server ran
> out-of-RAM, somehow which caused its kernel to kill a vital process.

I thought of that too and logged a free command run in a loop in one
second intervals. When it stopped due to reboot it was far from out of
ram. Also the same file (same size) worked after I re-saved it from
photoshop. Other files used more ram.

--
Erik Krause

[Bruno Postle]

On Tue, 20 Oct 2020 at 10:57, Bruno Postle wrote:
>
> Which suggests there is a 10549x5925 LZW TIFF image in the Exif data,
> but all that exiftool finds is two small thumbnails (the panorama
> itself is 9848x4924). There is also no JFIF header, I don't know if
> this indicates any problem.

Thinking about this, the problem is wider than just this file: nona
(and Hugin) use standard libraries and temporary files in a reasonably
safe way, but they are not designed to behave gracefully with every
kind of possible input file, in particular it is conceivably possible
someone could craft an evil file that tricks nona into doing bad
things. So my concern is really that you are accepting random files
from the internet, although the World Wide Panorama is a community
that makes this unlikely (BTW everybody should check out the World
Wide Panorama, especially if you shoot spherical panoramas and want to
share them: https://worldwidepanorama.org/ ).

So I would recommend filtering these files through a tool that has had
lots of attack exposure, like ImageMagick. If you use ImageMagick to
convert all your incoming JPEG files to PNG and then use this
temporary PNG file as input to nona, this will likely solve your
server reboot problem, while making the software more robust overall.

--
Bruno

[Erik Krause]

Am 21.10.2020 um 18:59 schrieb Bruno Postle:

> So I would recommend filtering these files through a tool that has had
> lots of attack exposure, like ImageMagick. If you use ImageMagick to
> convert all your incoming JPEG files to PNG and then use this
> temporary PNG file as input to nona, this will likely solve your
> server reboot problem, while making the software more robust overall.

That's an exceptionally good idea. We have ImageMagick on the server,
and it will of course add a bit to the processing time, but I think
that's worth it...

[Gunter Königsmann]

Only very few things should be able to cause a reboot:

 - a kernel bug 

 - an out of memory killing sysvinit/upstart/systemd

 - root killing sysvinit/upstart/systemd 

 - root calling the appropriate system command 

 - or an eventual watchdog.

Nona can only do the second of these things => I would guess it is the kernel (hopefully fixed by an update) or the watchdog.

[Erik Krause]

Am 21.10.2020 um 21:28 schrieb Gunter Königsmann:

> - an out of memory killing sysvinit/upstart/systemd

If nona eats up all the memory, shouldn't the swap be used then? Because
swap used was always 0...

[Erik Krause]

Am 21.10.2020 um 18:59 schrieb Bruno Postle:

> So I would recommend filtering these files through a tool that has had
> lots of attack exposure, like ImageMagick

Tried to convert one of the questionable images to PNG on the server.
ImageMagick caused a server reboot as well. Does this indicate it's one
of the libraries?

[Gunter Königsmann]

Or a faulty RAM: libraries can crash. But they should not be able to reboot the server even it they actively tried to.

[Erik Krause]

Am 22.10.2020 um 18:42 schrieb Gunter Königsmann:
> Or a faulty RAM

ImageMagick is official part of the VPS. So I submitted a support ticket
to dreamhost. Let's see what they say...

[Jens Scheidtmann]

Erik Krause <erik....@gmx.de> schrieb am Do. 22. Okt. 2020 um 18:28:

Am 21.10.2020 um 18:59 schrieb Bruno Postle:

> So I would recommend filtering these files through a tool that has had
> lots of attack exposure, like ImageMagick

Tried to convert one of the questionable images to PNG on the server.
ImageMagick caused a server reboot as well. Does this indicate it's one
of the libraries?

Have you tried on a different server? And does it reproduce there? 

[Erik Krause]

Am 22.10.2020 um 22:19 schrieb Jens Scheidtmann:

> Have you tried on a different server? And does it reproduce there?

Only locally under WSL ubuntu 18.04 LTS, where it works flawlessly.