Extracting bit fields from an IEEE-784 float

[Steven D'Aprano]

I wish to extract the bit fields from a Python float, call it x. First I
cast the float to 8-bytes:

s = struct.pack('=d', x)
i = struct.unpack('=q', s)[0]

Then I extract the bit fields from the int, e.g. to grab the sign bit:

(i & 0x8000000000000000) >> 63


Questions:

1) Are there any known implementations or platforms where Python floats
are not C doubles? If so, what are they?

2) If the platform byte-order is reversed, do I need to take any special
action? I don't think I do, because even though the float is reversed, so
will be the bit mask. Is this correct?

3) Any other problems with the way I am doing this?



Thanks in advance,



Steven

[Chris Gonnerman]

I've been making some minor updates to the PollyReports module I
announced a while back, and I've noticed that when I upload it to PyPI,
my changelog (CHANGES.txt) doesn't appear to be integrated into the site
at all. Do I have to put the changes into the README, or have I missed
something here? It seems that there should be some automatic method
whereby PyPI users could easily see what I've changed without
downloading it first.

[Dan Sommers]

On 2012-07-30 at 00:44:04 +0000,

Steven D'Aprano <steve+comp....@pearwood.info> wrote:

> I wish to extract the bit fields from a Python float, call it x. First I
> cast the float to 8-bytes:
>
> s = struct.pack('=d', x)
> i = struct.unpack('=q', s)[0]
>
> Then I extract the bit fields from the int, e.g. to grab the sign bit:
>
> (i & 0x8000000000000000) >> 63

> 3) Any other problems with the way I am doing this?

No, but perhaps this would be clearer:

import math
sign = math.copysign(1.0, x)

There are solutions that use math.frexp, too, but IMO they're more
obtuse.

HTH,
Dan

[Ben Finney]

Chris Gonnerman <ch...@gonnerman.org> writes:

> I've been making some minor updates to the PollyReports module

Your post is showing up as a reply to a thread about IEEE-784 floats,
because you created your message as a reply. Consequently, it's rather
confusing why you suddenly start talking about PollyReports.

If you want to attract attention to an unrelated topic, it's best if you
don't reply to an existing thread; instead, start a new thread by
composing a new message to the forum.

--
\ “I'd take the awe of understanding over the awe of ignorance |
`\ any day.” —Douglas Adams |
_o__) |
Ben Finney

[Terry Reedy]

On 7/29/2012 8:44 PM, Steven D'Aprano wrote:
> I wish to extract the bit fields from a Python float, call it x. First I
> cast the float to 8-bytes:
>
> s = struct.pack('=d', x)
> i = struct.unpack('=q', s)[0]
>
> Then I extract the bit fields from the int, e.g. to grab the sign bit:
>
> (i & 0x8000000000000000) >> 63
>
>
> Questions:
>
> 1) Are there any known implementations or platforms where Python floats
> are not C doubles? If so, what are they?

CPython floats are C doubles, which should be IEEE doubles. Other
implementations have a different to probably the same thing.

> 2) If the platform byte-order is reversed, do I need to take any special
> action? I don't think I do, because even though the float is reversed, so
> will be the bit mask. Is this correct?

The math modules functions to disassemble floats will not care.

--
Terry Jan Reedy

[Ulrich Eckhardt]

Am 30.07.2012 02:44, schrieb Steven D'Aprano:
> I wish to extract the bit fields from a Python float, call it x. First I
> cast the float to 8-bytes:
>
> s = struct.pack('=d', x)
> i = struct.unpack('=q', s)[0]
>
> Then I extract the bit fields from the int, e.g. to grab the sign bit:
>
> (i & 0x8000000000000000) >> 63
>
>
> Questions:
>
> 1) Are there any known implementations or platforms where Python floats
> are not C doubles? If so, what are they?

The struct docs refer to C's double type, so it depends on that type
probably. However, regardless of C's double type, the same docs refer to
the IEEE form when packed into a byte array. Is it just the
representation you are after or some specific behaviour?

> 2) If the platform byte-order is reversed, do I need to take any special
> action? I don't think I do, because even though the float is reversed, so
> will be the bit mask. Is this correct?

Yes, the code is fine. If you have doubts, I have a big-endian system at
home (Linux/PowerPC) where I could run tests.

> 3) Any other problems with the way I am doing this?

Python docs refer to IEEE-754, not 784? Typo?


Uli

[Mark Dickinson]

On Monday, July 30, 2012 1:44:04 AM UTC+1, Steven D'Aprano wrote:
> 1) Are there any known implementations or platforms where Python floats
> are not C doubles? If so, what are they?

Well, IronPython is presumably using .NET Doubles, while Jython will be using Java Doubles---in either case, that's specified to be the IEEE 754 binary64 type.

For CPython, and I guess PyPy too, we're using C doubles, which in theory are in whatever format the platform provides, but in practice are always IEEE 754 binary64 again.

So you're pretty safe assuming IEEE 754 binary64 format. If you ever meet a current Python running on a system that *doesn't* use IEEE 754 for its C doubles, please let me know---there are a lot of interesting questions that would come up in that case.

> 2) If the platform byte-order is reversed, do I need to take any special
>
> action? I don't think I do, because even though the float is reversed, so
>
> will be the bit mask. Is this correct?

True; on almost all current platforms, the endianness of int types matches the endianness of float types. But to be safe, why not use '<d' and '<q' in your formats instead of '=d' and '=q'? That way you don't have to worry.

> 3) Any other problems with the way I am doing this?

You might consider whether you want to use '<q' or '<Q' --- i.e. whether you want a signed integer or an unsigned integer to be returned. For grabbing bits, '<Q' seems a bit cleaner, while '<q' has the nice property that you can tell the sign of the original double by looking at the sign of the integer.

--
Mark

[Dieter Maurer]

"CHANGES.txt" is not automatically presented.
If necessary, you must integrate it into the "long description".

However, personally, I am not interested in all the details (typically
found in "CHANGES.txt") but some (often implicit) information is
sufficient for me: something like "major API change", "minor bug
fixes". Thus, think carefully what you put on the overview page.

I find it very stupid to see several window scrolls of changes for
a package but to learn how to install the package, I have to download its
source...

[Chris Gonnerman]

On 07/30/2012 04:20 AM, Dieter Maurer wrote:
> "CHANGES.txt" is not automatically presented.
> If necessary, you must integrate it into the "long description".
>
> However, personally, I am not interested in all the details (typically
> found in "CHANGES.txt") but some (often implicit) information is
> sufficient for me: something like "major API change", "minor bug
> fixes". Thus, think carefully what you put on the overview page.

I see your point. I'm just lazy, I guess. I already put a description
of what I've changed into git, so why, I muse, must I also edit the
overview page separately? I was hoping there was an automatic way that
"setup.py sdist upload" could handle it for me.

> I find it very stupid to see several window scrolls of changes for
> a package but to learn how to install the package, I have to download its
> source...

Not sure I get this. The installation procedure for PollyReports is the
same as for, what, 99% of Python source packages?

sudo python setup.py install

What else are you saying I should do?

-- Chris.

[Chris Gonnerman]

On 07/29/2012 11:00 PM, Ben Finney wrote:
> Your post is showing up as a reply to a thread about IEEE-784 floats,
> because you created your message as a reply. Consequently, it's rather
> confusing why you suddenly start talking about PollyReports. If you
> want to attract attention to an unrelated topic, it's best if you
> don't reply to an existing thread; instead, start a new thread by
> composing a new message to the forum.

My apologies. I did not consider that headers I can't see might be
being sent along.

[Grant Edwards]

On 2012-07-30, Steven D'Aprano <steve+comp....@pearwood.info> wrote:

> 1) Are there any known implementations or platforms where Python floats
> are not C doubles? If so, what are they?

And the question you didn't ask: are there any platforms where a C
double isn't IEEE-754?

The last ones I worked on that where the FP format wasn't IEEE were
the DEC VAX and TI's line if 32-bit floating-point DSPs. I don't
think Python runs on the latter, but it might on the former.

--
Grant Edwards grant.b.edwards Yow! I was born in a
at Hostess Cupcake factory
gmail.com before the sexual
revolution!

[Roy Smith]

In article <jv64v5$g2n$2...@reader1.panix.com>,

Grant Edwards <inv...@invalid.invalid> wrote:

> The last ones I worked on that where the FP format wasn't IEEE were
> the DEC VAX

According to http://en.wikipedia.org/wiki/Vax#History, the last VAX was
produced 7 years ago. I'm sure there's still more than a few chugging
away in corporate data centers and manufacturing floors, but as an
architecture, it's pretty much a dead parrot.

IEEE floating point is as near to a universal standard as it gets in the
computer world. About the only thing that has it beat for market
penetration and longevity are 2's complement integers and 8-bit bytes.

[Mark Lawrence]

On 30/07/2012 15:16, Grant Edwards wrote:
> On 2012-07-30, Steven D'Aprano <steve+comp....@pearwood.info> wrote:
>
>> 1) Are there any known implementations or platforms where Python floats
>> are not C doubles? If so, what are they?
>
> And the question you didn't ask: are there any platforms where a C
> double isn't IEEE-754?
>
> The last ones I worked on that where the FP format wasn't IEEE were
> the DEC VAX and TI's line if 32-bit floating-point DSPs. I don't
> think Python runs on the latter, but it might on the former.
>

Support for Python on VMS has been dropped for v3.3 see
http://bugs.python.org/issue11918

--
Cheers.

Mark Lawrence.

[Grant Edwards]

I imagine that VAXes running Unix went extinct in the wild long before
VAXes running VMS.

--
Grant Edwards grant.b.edwards Yow! Did YOU find a
at DIGITAL WATCH in YOUR box
gmail.com of VELVEETA?

[Mark Dickinson]

On Monday, July 30, 2012 3:16:05 PM UTC+1, Grant Edwards wrote:
> The last ones I worked on that where the FP format wasn't IEEE were
>
> the DEC VAX and TI's line if 32-bit floating-point DSPs. I don't
>
> think Python runs on the latter, but it might on the former.

For current hardware, there's also IBM big iron: the IBM System z10 apparently has hardware support for IBM's hexadecimal floating-point format in addition to IEEE 754 binary *and* decimal floating-point. But IIUC, a typical Linux installation on one of these machines uses the IEEE 754 stuff, not the hexadecimal bits. So unlikely to be an issue for Python.

--
Mark

[Prasad, Ramit]

> > However, personally, I am not interested in all the details (typically
> > found in "CHANGES.txt") but some (often implicit) information is
> > sufficient for me: something like "major API change", "minor bug
> > fixes". Thus, think carefully what you put on the overview page.

> I see your point. I'm just lazy, I guess. I already put a description
> of what I've changed into git, so why, I muse, must I also edit the
> overview page separately? I was hoping there was an automatic way that
> "setup.py sdist upload" could handle it for me.

Even if you could include commit messages I would not recommend it.
Commit messages are for developers of _your_ package not users. It is
like leaving a memo (cough or message) for future developers.

As a user, I want to know that you added a feature AAA. I do *not*
want to know that in order to add AAA you also had to modify BBB
and create CCC. Maybe you had to revert BBB because it conflicted
with DDD and then you refactored some code and added/updated
comments. Finally you managed to fix BBB (maybe also modifying DDD?)
and so the feature AAA is now available.


Ramit

This email is confidential and subject to important disclaimers and
conditions including on offers for the purchase or sale of
securities, accuracy and completeness of information, viruses,
confidentiality, legal privilege, and legal entity disclaimers,
available at http://www.jpmorgan.com/pages/disclosures/email.

[Roy Smith]

In article <jv6ab7$jne$1...@reader1.panix.com>,

Grant Edwards <inv...@invalid.invalid> wrote:

> I imagine that VAXes running Unix went extinct in the wild long before
> VAXes running VMS.

Of course they did. VMS is all about vendor lock-in. People who
continue to run VAXen don't do so because they're wedded to the
hardware. They do so because they're wedded to some specific VMS
application (and the business processes which depend on it).

[Dieter Maurer]

Chris Gonnerman <ch...@gonnerman.org> writes:

> On 07/30/2012 04:20 AM, Dieter Maurer wrote:

> ...

>> I find it very stupid to see several window scrolls of changes for
>> a package but to learn how to install the package, I have to download its
>> source...
> Not sure I get this. The installation procedure for PollyReports is
> the same as for, what, 99% of Python source packages?
>
> sudo python setup.py install
>
> What else are you saying I should do?

This remark was not targeted at "PollyReports" but (in general) at packages
with non-trivial installation procedures which nevertheless
state on the overview page "for installation
read the separate installation instructions (in the source distribution)".

As a side note: playing well with python package managers
("easy_install", "pip", "zc.buildout", ...) could make it even
simpler than "sudo python setup.py install".