Floating point errors in quantity extension

[Martin Jones]

I was surprised to find that quantity(random(2..10 #0.1), "m/s") produces the occasional value like 7.800000000000001.

I had to force it to 1 decimal place by using precround(quantity(random(2..10 #0.1), "m/s"),1) but perhaps there is a small bug in there which could be fixed?

Martin

[Christian Lawson-Perfect]

The problem is that the quantity data type uses the `decimal` data type for the scalar part, while `random` produces `number` values. When converting from `number` to `decimal`, there's no information about the precision of the number, so the extra 0.00000000000001 from floating point imprecision is kept.

I suppose that values produced by the `random` function could look at the precision of the step value, but I'd have to think more deeply about how robust that is. For now, I think you'll have to do precround.

--
You received this message because you are subscribed to the Google Groups "Numbas Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to numbas-users...@googlegroups.com.
To view this discussion, visit https://groups.google.com/d/msgid/numbas-users/993f632b-bc21-47f6-afdb-939b3f78c612n%40googlegroups.com.