Ulps and non-floating point values

[Charlie Poole]

Hi All,

As implemented, EqualConstraint(expected).Within(x).Ulps works just
like a normal linear tolerance when expected is int, long, uint,
decimal, etc.

Is that what it should do? I'm not pushing one way or the other,
just asking. :-)

Charlie

[Thierry Lach]

Well, to me, ulps only makes sense in the context of float or double.

One suggestion would be to throw an ArgumentException if there's not a double or float.  Looking at the code in Numerics.cs method AreEqual(), after the first two ifs for double and float and before the one for decimal, you could add an if to throw the exception if the tolerance is ulps.

--
---
Do not follow where the path may lead. Go instead where there is no path and leave a trail.
~Ralph Waldo Emerson

[Markus Ewald]

As per definition one ulp (= unit in the last place / units of last
precision, depending on who you ask) is the smallest change a numeric
type can make. For chars, ints and longs that smallest change is, of
course, always 1, which incidentally yields the exact same behavior as a
linear comparison.

Using Ulps on integral types is probably not a typical use case at all,
but that's the behavior I think is appropriate.

-

The Ulps-based methods in the Java Class Library do indeed only provide
overloads for float and double: Math.Ulp(float) and Math.Ulp(double).
That could either be because it's accepted practice that Ulps are only
used with floats and doubles or because creating an overload that does,
well, nothing and just returns its argument would be nonsensical.

Opinions?

> Charlie
>
>
>
-Markus-

[Charlie Poole]

Yup, that seems like a reasonable alternative.

 

So we have (I think) three choices...

 

1) Ignore it (basically we do that now)

2) Convert the arguments to double if possible

3) Throw

 

Charlie

---

From: nunit-...@googlegroups.com [mailto:nunit-...@googlegroups.com] On Behalf Of Thierry Lach
Sent: Wednesday, January 21, 2009 8:03 PM
To: nunit-...@googlegroups.com
Subject: [nunit-discuss] Re: Ulps and non-floating point values

[Markus Ewald]

Charlie Poole wrote:

Yup, that seems like a reasonable alternative.

 

So we have (I think) three choices...

 

1) Ignore it (basically we do that now)

2) Convert the arguments to double if possible

3) Throw

I don't think 2) would be a good idea. For example, the smallest change an integer with value 1 can make is 1, but when seen as a double, there are 4,503,599,627,370,496 ulps between 1 and 2, for example, which is quite unexpected behvaior and outside of any reasonable tolerance you might specify.

So it's either 1) or 3), where my personal preference is option 1). It's not ignored, it just produces the exact same behavior as the linear comparison.

Charlie

-Markus-

[Charlie Poole]

Hi Markus,

> I don't think 2) would be a good idea. For example, the
> smallest change an integer with value 1 can make is 1, but
> when seen as a double, there are 4,503,599,627,370,496 ulps
> between 1 and 2, for example, which is quite unexpected
> behvaior and outside of any reasonable tolerance you might specify.

OK, good point.

> So it's either 1) or 3), where my personal preference is
> option 1). It's not ignored, it just produces the exact same
> behavior as the linear comparison.

True, but one can view it either way. It's as if we had
a modifier "Units" with no impact at all.

Here's a rather pathological case. What should happen when
the third item is reached? What about the fourth?

object[] expected = new object[] { 1.0, 2.0, 3, TimeSpan.FromSeconds(20) };
object[] actual = new object[] { 1.00001, 2, 3, TimeSpan.FromSeconds(19) };
Assert.That(actual, Is.EqualTo(expected).Within(1).Ulps);

Charlie

[Markus Ewald]

On 22.01.2009 08:37, Charlie Poole wrote:

So it's either 1) or 3), where my personal preference is 
option 1). It's not ignored, it just produces the exact same 
behavior as the linear comparison.
    

True, but one can view it either way. It's as if we had
a modifier "Units" with no impact at all.

Here's a rather pathological case. What should happen when
the third item is reached? What about the fourth?

object[] expected = new object[] { 1.0, 2.0, 3, TimeSpan.FromSeconds(20) };
object[] actual = new object[] { 1.00001, 2, 3, TimeSpan.FromSeconds(19) };
Assert.That(actual, Is.EqualTo(expected).Within(1).Ulps);

  

The third should clearly pass. Even if it was 2 and 3 being compared, I'd say it should pass based on my earlier reasoning.

About the fourth item, an ulp in a TimeSpan would be a 'tick', as that is its smallest possible change (in steps of 100 nanoseconds I think). That wouldn't be unexpected at all from my point of view, although someone not aware of the implementation details of a TimeSpan might possibly expect the test to pass.

I'd say, .Ulps for a time span might even make more sense than using the .Percent modifier here.

Charlie

  

-Markus-

[Thierry Lach]

Well, if you want to go with option 3, here's a patch.  I guess it's apparent which approach I'm in favor of.

[Charlie Poole]

Hi Markus,

> Here's a rather pathological case. What should happen when
> the third item is reached? What about the fourth?
>
> object[] expected = new object[] { 1.0, 2.0, 3, TimeSpan.FromSeconds(20)
};
> object[] actual = new object[] { 1.00001, 2, 3, TimeSpan.FromSeconds(19)
};
> Assert.That(actual, Is.EqualTo(expected).Within(1).Ulps);
>
>
>
> The third should clearly pass. Even if it was 2 and 3 being
> compared, I'd say it should pass based on my earlier reasoning.

I understand the reasoning, but it could be surprising for
someone who used it casually. There's some logic to bias the
interpretation toward failure, since that way you at least
see an error message.

> About the fourth item, an ulp in a TimeSpan would be a
> 'tick', as that is its smallest possible change (in steps of
> 100 nanoseconds I think). That wouldn't be unexpected at all
> from my point of view, although someone not aware of the
> implementation details of a TimeSpan might possibly expect
> the test to pass.

That's particularly problematic since the value of a tick
is hardware-dependent.

What about decimal? The same logic that makes (int) 2 == 3
within 1 Ulp would appear to make 2m != 3m. That sounds really
confusing.

I am agreeing with you on one point: taking Ulps to work like
Linear is not the same as ignoring. So that gives me a a total
of four options when Ulps is used with non-floats:

1) Ignore the tolerance completely

2) Interpret it as Linear

3) Convert the arguments to double if possible

4) Throw

#4 can be implemented in the constuctor if a scalar or a
collection with known element type is passed as the expected
value. Otherwise, we would wait until the value is actually tested.

If we believe that using Ulps with an int is most likely a
mistake on the part of the programmer, then I think throwing
would give the best information. OTOH, if we believe that
such usage serves a purpose that can't be duplicated in
other ways, then it's harder to decide.

We've only heard from three of us on the issue - anybody
else have an opinion?

> I'd say, .Ulps for a time span might even make more sense
> than using the .Percent modifier here.

Agreed. But we now have .Ticks for that usage, which makes
even more sense when you're talking about time.

Charlie

>
>
> Charlie
>
>
>
> -Markus-
>
>
> >
>
>

[Charlie Poole]

I did a review of all the literature I can find on the subject. While Markus' use of

Ulps (units in the last place) with integers makes sense linguistically, all the

literature about it refers to floating point calculations.

 

I hesitate to do something that would appear wrong to experts, even if

it might arguably make "common sense."

Charlie

---

From: nunit-...@googlegroups.com [mailto:nunit-...@googlegroups.com] On Behalf Of Thierry Lach

Sent: Thursday, January 22, 2009 9:27 AM

To: nunit-...@googlegroups.com
Subject: [nunit-discuss] Re: Ulps and non-floating point values

[Markus Ewald]

On 22.01.2009 21:27, Charlie Poole wrote:
> 1) Ignore the tolerance completely
>
> 2) Interpret it as Linear
>
> 3) Convert the arguments to double if possible
>
> 4) Throw
>
> #4 can be implemented in the constuctor if a scalar or a
> collection with known element type is passed as the expected
> value. Otherwise, we would wait until the value is actually tested.
>
> If we believe that using Ulps with an int is most likely a
> mistake on the part of the programmer, then I think throwing
> would give the best information. OTOH, if we believe that
> such usage serves a purpose that can't be duplicated in
> other ways, then it's harder to decide.
>
> We've only heard from three of us on the issue - anybody
> else have an opinion?
>
>

Well, you're slowly starting to convince me, at least ;-)

An Ulps comparison is probably really not something the user would want
to do.

-Markus-

[Markus Ewald]

On 22.01.2009 21:51, Charlie Poole wrote:

I did a review of all the literature I can find on the subject. While Markus' use of

Ulps (units in the last place) with integers makes sense linguistically, all the

literature about it refers to floating point calculations.

 

I hesitate to do something that would appear wrong to experts, even if

it might arguably make "common sense."

Wow, thanks!
Okay, that's one more good argument for throwing.

On the basis the the user probably wouldn't intentionally want to do an Ulps comparison on integral types and that there is no established use of the technique on anything else than floats, I'd agree if we took option three, throwing.

Charlie

-Markus-

[Charlie Poole]

Hi Markus and Everyone,

 

Looks like a concensus has been reached!

 

So, I checked in changes to Markus' work as follows:

1) Specifying Ulps for non-floating comparisons is an error. Note that it's still OK

    if either of the args is float or double, since the other arg will be converted.

2) Specifying Ulps, Percent, Hours, Days, Minutes, etc. for a non-numeric

    tolerance is an error.

3) Specifying any of the above /before/ the Within is an error.

4) There is now a Tolerance class, which encapsulates an amount and mode.

    The class is immutable and the constructor taking a mode is private,

    so I eliminated some tests for error on invalid modes.

 

The Tolerance class doesn't actually do the equality tests, it just carries around

the information and allows creating of new tolerances (e.g. Linear => Percent).

To do that, I'd need to do some further work that didn't seem needed at this time.

 

If you'd like to review the code, you can get the changes from the commit

list or look at Tolerance.cs and Numerics.cs in CVS.

 

Charlie

---

From: nunit-...@googlegroups.com [mailto:nunit-...@googlegroups.com] On Behalf Of Markus Ewald
Sent: Thursday, January 22, 2009 10:28 PM

To: nunit-...@googlegroups.com
Subject: [nunit-discuss] Re: Ulps and non-floating point values