OpenJPEG and Floating Point Exceptions

[Stephan Mühlstrasser]

Hi,

I wanted to ask about the position of the OpenJPEG project regarding floating points exceptions.

The background is this: We are using OpenJPEG without problems on multiple platforms. The exception is IBM iSeries, where we are getting floating point underflow traps for certain test files. These underflows occur in opj_v4dwt_decode_step1() in dwt.c.

I was now able to prepare a non-SSE build of OpenJPEG on Linux where floating point traps are enabled by calling the GNU libc function feenableexcept(FE_UNDERFLOW). After that I can reproduce with the same input files as on iSeries that the program gets a SIGFPE signal in the same function opj_v4dwt_decode_step1(), apparently caused by floating point underflow.

So my current theory is that the floating point underflow exceptions are actually occurring on all platforms, but only on IBM iSeries we notice them because there a trap for floating point underflow occurs by default.

Is the occurrence of floating point exceptions considered a bug in OpenJPEG, or is the position of the OpenJPEG project that floating point exceptions must be ignored?

Best regards

Stephan

[Even Rouault]

Looking at this draft of the C standard : https://port70.net/~nsz/c/c11/n1570.html ,
I see
"The ''underflow'' floating-point exception is raised whenever a result is tiny (essentially subnormal or zero) and suffers loss of accuracy."
So yes probably emitting an exception is expected, and that most Intel based platforms
will ignore them by default.

We could probably avoid the exception by doing the operation on double, and then
checking the range of the result against float range before casting, but that would be costly.
So it might thus be relevant in all openjpeg entry points that cause image encoding/decoding
to
- backup the exception state
- disable exceptions
- do the job
- restore exception state

Might require some care as I see the feXXXX functions are C99 so this is likely
to cause issues with older compilers (we could just do nothing regarding this for those)

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com

[Stephan Mühlstrasser]

Hello Even,

Am Freitag, 5. Juli 2019 15:38:52 UTC+2 schrieb Even Rouault:

Looking at this draft of the C standard : https://port70.net/~nsz/c/c11/n1570.html ,
I see
"The ''underflow'' floating-point exception is raised whenever a result is tiny (essentially subnormal or zero) and suffers loss of accuracy."
So yes probably emitting an exception is expected, and that most Intel based platforms
will ignore them by default.

We could probably avoid the exception by doing the operation on double, and then
checking the range of the result against float range before casting, but that would be costly.
So it might thus be relevant in all openjpeg entry points that cause image encoding/decoding
to
- backup the exception state
- disable exceptions
- do the job
- restore exception state

Might require some care as I see the feXXXX functions are C99 so this is likely
to cause issues with older compilers (we could just do nothing regarding this for those)

Even

I think there is some confusion about the term "floating-point exception". If I understand the C99 specification correctly, it is implementation-defined behavior whether a floating-point exception causes a trap (e.g. a SIGFPE signal) or not:

"H.3.1 Notification alternatives

1 LIA−1 requires at least the following two alternatives for handling of notifications: setting indicators or trap-and-terminate. LIA−1 allows a third alternative: trap-and-resume.

2 An implementation need only support a given notification alternative for the entire program. An implementation may support the ability to switch between notification alternatives during execution, but is not required to do so. An implementation can provide separate selection for each kind of notification, but this is not required.

3 C allows an implementation to provide notification. C’s SIGFPE (for traps) and FE_INVALID, FE_DIVBYZERO, FE_OVERFLOW, FE_UNDERFLOW (for indicators) can provide LIA−1 notification."

"LIA-1" is "ISO/IEC 10967, Language independent arithmetic (LIA)".

I had chosen in my initial experiment to turn on trap notification via the GNU C library's "feenableexcept(FE_UNDERFLOW)" function, with the result that a floating-point exception causes a SIGFPE signal. In a second experiment I only used the C99 "fetestexcept()" function to test whether the "FE_UNDERFLOW" floating-point exception had occurred in the critical functions, and this was also the case.

So the floating-point exceptions apparently do occur in OpenJPEG during decompression of specific input data, but normally they are not noticed on the common platforms because the default behavior of most C implementations is to only indicate the floating-point exception through the FE_... flags, which must be explicitly queried to have an effect.

So nothing needs to be done on the common platforms, and the bad luck is that the C implementation of IBM iSeries behaves differently. There it would require platform-specific code to disable the floating-point exceptions.

Are you interested in JPEG2000 files that trigger the floating-point exceptions?

Stephan 

[Even Rouault]

Stephan,

> So nothing needs to be done on the common platforms, and the bad luck is
> that the C implementation of IBM iSeries behaves differently. There it
> would require platform-specific code to disable the floating-point
> exceptions.
>
> Are you interested in JPEG2000 files that trigger the floating-point
> exceptions?

Honestly I won't tackle any development regarding this, but the project would
I believe welcome a reasonable fix in a pull request to address this.

[Stephan Mühlstrasser]

Am Montag, 8. Juli 2019 14:44:43 UTC+2 schrieb Even Rouault:

Stephan,

> So nothing needs to be done on the common platforms, and the bad luck is
> that the C implementation of IBM iSeries behaves differently. There it
> would require platform-specific code to disable the floating-point
> exceptions.
>
> Are you interested in JPEG2000 files that trigger the floating-point
> exceptions?

Honestly I won't tackle any development regarding this, but the project would
I believe welcome a reasonable fix in a pull request to address this.

A real solution for this problem would require modifications to the decompression algorithms so that floating-point exceptions are avoided. I have to admit that don't have the necessary expertise in JPEG2000 algorithms and in numerics in general for this task.

I understand that there are probably more important problems in OpenJPEG than this problem where the only currently known effect occurs on an admittedly exotic platform, but in theory it can hit any OpenJPEG user who enables floating-point traps in his application. Is it worth to file a GitHub issue so this is at least documented?

Stephan

[Even Rouault]

> A real solution for this problem would require modifications to the
> decompression algorithms so that floating-point exceptions are avoided.

That could be computationaly costly. For each multiplication, you'd had to do
something like
if( fabs((double)a * (double)b) < (double)FLT_MIN )

Disabling temporarily underflow traps in the high level entry points of the
library would likely be more efficient and less invasive

>
> I understand that there are probably more important problems in OpenJPEG
> than this problem where the only currently known effect occurs on an
> admittedly exotic platform, but in theory it can hit any OpenJPEG user who
> enables floating-point traps in his application. Is it worth to file a
> GitHub issue so this is at least documented?

Sure

[Stephan Mühlstrasser]

Am Montag, 8. Juli 2019 15:44:41 UTC+2 schrieb Even Rouault:

> A real solution for this problem would require modifications to the
> decompression algorithms so that floating-point exceptions are avoided.

That could be computationaly costly. For each multiplication, you'd had to do
something like
if( fabs((double)a * (double)b) < (double)FLT_MIN )

Disabling temporarily underflow traps in the high level entry points of the
library would likely be more efficient and less invasive

We have researched this approach for the IBM iSeries platform, and there the <fenv.h> header and corresponding functions are not available. Floating-point traps can be disabled on the iSeries with "Machine Interface Instructions". It is currently unclear whether this is possible on the thread level or only on a per-process level. If it is is only possible at the process level then we can't do it because we produce a library that has OpenJPEG embedded, and we cannot control the per-process level setting as this could interfere with other threads.

 

>
> I understand that there are probably more important problems in OpenJPEG
> than this problem where the only currently known effect occurs on an
> admittedly exotic platform, but in theory it can hit any OpenJPEG user who
> enables floating-point traps in his application. Is it worth to file a
> GitHub issue so this is at least documented?

Sure

 

Ok, will do.

Stephan

[Eitan Mizrahi]

Hi.

I had done several things with no library for jpeg2 protocol, without using the openjp2 library.

i.e merging and spliting big file images.

It took few days for both - I don't think it's quite a big issue.

Try search: http://hosting.astro.cornell.edu/~carcich/LRO/jp2/ISO_JPEG200_Standard/INCITS+ISO+IEC+15444-1-2000.pdf

‫בתאריך יום ב׳, 8 ביולי 2019 ב-17:08 מאת ‪Stephan Mühlstrasser‬‏ <‪s...@pdflib.com‬‏>:‬

--
You are subscribed to the mailing-list of the OpenJPEG project (www.openjpeg.org)
To post: email to open...@googlegroups.com
To unsubscribe: email to openjpeg+u...@googlegroups.com
For more options: visit http://groups.google.com/group/openjpeg
OpenJPEG is mainly supported by :
* UCL Image and Signal Processing Group (http://sites.uclouvain.be/ispgroup)
* IntoPIX (www.intopix.com)
---
You received this message because you are subscribed to the Google Groups "OpenJPEG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openjpeg+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/openjpeg/cc88137f-febb-4a66-a775-a6adbc260838%40googlegroups.com.