write binary with struct.pack_into

[palmeira]

Dear pythonists,

I'm having a problem with read/write binary in python.
I have a binary file that I need to read information, extract a array,
modify this array and put these values into file again in same binary
format.
I need to use unpack_from and pack_into because sometimes gonna need
read/write in the middle of file.

Script:

import struct
bloco='>%df' %(252) #Binary format

# READ
fa=open('testIN.bin')
my_array=struct.unpack_from(bloco,fa.read()[0*4:251*4]) # my_aray = 252
elements array
## This read is OK!

#WRITE
fb=open('testOUT.bin')
test=struct.pack_into(bloco,fb.write()[0*4:251*4]) # ERROR in this WRITE




Regards,

Ronaldo Palmeira.




--
View this message in context: http://python.6.n6.nabble.com/write-binary-with-struct-pack-into-tp4991234.html
Sent from the Python - python-list mailing list archive at Nabble.com.

[88888 Dihedral]

palmeira於 2012年10月6日星期六UTC+8上午11時27分47秒寫道：

Are you writing and reading files produce by different
languages?

The pickle part is better for OOP and glue logics in python.

The heavy computing part should be done in CYTHON.

[88888 Dihedral]

palmeira於 2012年10月6日星期六UTC+8上午11時27分47秒寫道：

[Chris Angelico]

On Sat, Oct 6, 2012 at 1:27 PM, palmeira <palm...@gmail.com> wrote:
> import struct
> bloco='>%df' %(252) #Binary format
>
> # READ
> fa=open('testIN.bin')
> my_array=struct.unpack_from(bloco,fa.read()[0*4:251*4]) # my_aray = 252
> elements array
> ## This read is OK!
>
> #WRITE
> fb=open('testOUT.bin')
> test=struct.pack_into(bloco,fb.write()[0*4:251*4]) # ERROR in this WRITE

You have a beautiful parallel here, but I think that's where your
problem is. In the READ section, you have fa.read() which will read
the whole file, and then you slice the resulting string. That's pretty
inefficient for large files, but it'll work.

But when you write, that completely does not work. (Even assuming
you've opened read/write, per Dennis's comment.)
>>> fb.write()
Traceback (most recent call last):
File "<pyshell#37>", line 1, in <module>
fb.write()
TypeError: write() takes exactly 1 argument (0 given)

It needs an argument of what to write out, it doesn't return a
writable buffer suitable for pack_into.

I recommend you completely rewrite your file handling to use actual
seeks and file writes. Also, you'll want (I think) to use binary mode
on your files; character encodings don't mean much when you're working
with arrays of numbers.

Finally, when you post issues, "ERROR in this WRITE" isn't very
helpful. Please please post the full traceback (like in my trivial
example above), as problems will be much more visible.

Hope that's of some value!

ChrisA

[Peter Otten]

palmeira wrote:

> Dear pythonists,
>
> I'm having a problem with read/write binary in python.
> I have a binary file that I need to read information, extract a array,
> modify this array and put these values into file again in same binary
> format.
> I need to use unpack_from and pack_into because sometimes gonna need
> read/write in the middle of file.

Use pack/unpack and file.seek() instead.

> Script:

>
> import struct
> bloco='>%df' %(252) #Binary format
>
> # READ
> fa=open('testIN.bin')
> my_array=struct.unpack_from(bloco,fa.read()[0*4:251*4]) # my_aray = 252
> elements array
> ## This read is OK!
>
> #WRITE
> fb=open('testOUT.bin')
> test=struct.pack_into(bloco,fb.write()[0*4:251*4]) # ERROR in this WRITE

However, I think you have picked the wrong API. So:

# untested

import sys
import array

offset = 0
N = 252

a = array.array("f")
with open("testIN.bin", "rb") as f:
f.seek(offset)
a.read(f, N)
if sys.byteorder == "little":
a.byteswap()

# process a

if sys.byteorder == "little":
a.byteswap()
with open("testOUT.bin", "wb") as f:
f.seek(offset)
a.write(f)

[Grant Edwards]

On 2012-10-06, Dennis Lee Bieber <wlf...@ix.netcom.com> wrote:
> On Fri, 5 Oct 2012 20:27:36 -0700 (PDT), palmeira <palm...@gmail.com>
> declaimed the following in gmane.comp.python.general:
>
>>
>> #WRITE
>> fb=open('testOUT.bin')
>
> Unless you specify otherwise, open() defaults to read-only

It also defaults to 'text' mode which does cr/lf translaction. That
will break both reads and writes on any binary file containing 0x0a
and 0x0d bytes.

--
Grant

[Chris Angelico]

And, in Python 3, it'll be returning Unicode characters too, unless
the decode fails. You definitely want binary mode.

ChrisA

[Alexander Blinne]

First, you should consider reading the documentation of
struct.unpack_from and struct.pack_into at
http://docs.python.org/library/struct.html quite carefully. It says,
that these commands take a parameter called offset, which names the
location of the data in a buffer (e.g. an opened file).

example:

bloco='>%df' % (252) # Format string (252 floats)
fa = open('testIN.bin', 'rb') # open for reading in binary mode
off = 0 # suppose i want to read block at beginning of file
my_array=struct.unpack_from(bloco, fa, off) #read data


now write them to another file:

fb = open('testOUT.bin', 'r+b') # open for updating in binary mode
off = 0 # suppose i want to write block at beginning of file
struct.pack_into(bloco, fb, off, *my_array) #write data to testOUT