How to write list of integers to file with struct.pack_into?

[Jen Kris]

Iwant to write a list of 64-bit integers to a binary file. Everyexample I have seen in my research convertsit to .txt, but I want it in binary. I wrote this code,based on some earlier work I have done:




buf= bytes((len(qs_array)) * 8)

foroffset in range(len(qs_array)):


item_to_write= bytes(qs_array[offset])


struct.pack_into(buf,"<Q", offset, item_to_write)





ButI get the error "struct.error: embedded null character."





Maybethere's a better way to do this?





Anyhelp will be very appreciated.


Thanks. 

[Barry]

On 2 Oct 2023, at 16:02, Jen Kris via Python-list

<pytho...@python.org> wrote:

Iwant to write a list of 64-bit integers to a binary file.  Everyexample
I have seen in my research convertsit to .txt, but I want it in binary.
 I wrote this code,based on some earlier work I have done:
buf= bytes((len(qs_array)) * 8)

buf is not writable so cannot be used by pack_into. I think you need to
use bytesarray not bytes.

foroffset in range(len(qs_array)):
item_to_write= bytes(qs_array[offset])
struct.pack_into(buf,"<Q", offset, item_to_write)

You have the parameters in the wrong order.

struct.pack_into(format, buffer, offset, v1, v2, ...)[1]¶

Pack the values v1, v2, … according to the format string format and
write the packed bytes into the writable buffer buffer starting at
position offset. Note that offset is a required argument.

ButI get the error "struct.error: embedded null character."

The nul is because you past buf of 0’s as the format.

Maybethere's a better way to do this?

Anyhelp will be very appreciated.
Thanks. 

Barry

--
https://mail.python.org/mailman/listinfo/python-list

References

Visible links
1. Permalink to this definition
https://docs.python.org/3/library/struct.html#struct.pack_into

[MRAB]

On 2023-10-01 23:04, Jen Kris via Python-list wrote:
>
> Iwant to write a list of 64-bit integers to a binary file.
Everyexample I have seen in my research convertsit to .txt, but I want
it in binary. I wrote this code,based on some earlier work I have done:
>
> buf= bytes((len(qs_array)) * 8)
>

> foroffset in range(len(qs_array)):
>
> item_to_write= bytes(qs_array[offset])
>
> struct.pack_into(buf,"<Q", offset, item_to_write)
>

> ButI get the error "struct.error: embedded null character."
>

> Maybethere's a better way to do this?
>

You can't pack into a 'bytes' object because it's immutable.

The simplest solution I can think of is:

buf = struct.pack("<%sQ" % len(qs_array), *qs_array)

[Jen Kris]

Thanks very much, MRAB.  I just tried that and it works.  What frustrated me is that every research example I found writes integers as strings.  That works -- sort of -- but it requires re-casting each string to integer when reading the file.  If I'm doing binary work I don't want the extra overhead, and it's more difficult yet if I'm using the Python integer output in a C program.  Your solution solves those problems. 



Oct 2, 2023, 17:11 by pytho...@python.org:

> --
> https://mail.python.org/mailman/listinfo/python-list
>

[Dieter Maurer]

Jen Kris wrote at 2023-10-2 00:04 +0200:
>Iwant to write a list of 64-bit integers to a binary file. Everyexample I have seen in my research convertsit to .txt, but I want it in binary. I wrote this code,based on some earlier work I have done:
>
>buf= bytes((len(qs_array)) * 8)
>

>for offset in range(len(qs_array)):

> item_to_write= bytes(qs_array[offset])
> struct.pack_into(buf,"<Q", offset, item_to_write)
>

>But I get the error "struct.error: embedded null character."

You made a lot of errors:

* the signature of `struct.pack_into` is
`(format, buffer, offset, v1, v2, ...)`.
Especially: `format` is the first, `buffer` the second argument

* In your code, `offset` is `0`, `1`, `2`, ...
but it should be `0 *8`, `1 * 8`, `2 * 8`, ...

* The `vi` should be something which fits with the format:
integers in your case. But you pass bytes.

Try `struct.pack_into("<Q", buf, 0, *qs_array)`
instead of your loop.


Next time: carefully read the documentation and think carefully
about the types involved.

[Jen Kris]

Dieter, thanks for your comment that:

* In your code, `offset` is `0`, `1`, `2`, ...
but it should be `0 *8`, `1 * 8`, `2 * 8`, ...

But you concluded with essentially the same solution proposed by MRAB, so that would obviate the need to write item by item because it writes the whole buffer at once. 

Thanks for your help. 


Oct 2, 2023, 17:47 by die...@handshake.de:

[Jen Kris]

My previous message just went up -- sorry for the mangled formatting.  Here it is properly formatted:

I want to write a list of 64-bit integers to a binary file.  Every example I have seen in my research converts it to .txt, but I want it in binary.  I wrote this code, based on some earlier work I have done:

    buf = bytes((len(qs_array)) * 8)
    for offset in range(len(qs_array)):
        item_to_write = bytes(qs_array[offset])

        struct.pack_into(buf, "<Q", offset, item_to_write)

But I get the error "struct.error: embedded null character."  

Maybe there's a better way to do this?  

Any help will be very appreciated.  

[Roel Schroeven]

Jen Kris via Python-list schreef op 2/10/2023 om 17:06:

> My previous message just went up -- sorry for the mangled formatting.  Here it is properly formatted:
>
> I want to write a list of 64-bit integers to a binary file.  Every example I have seen in my research converts it to .txt, but I want it in binary.  I wrote this code, based on some earlier work I have done:
>
>     buf = bytes((len(qs_array)) * 8)
>     for offset in range(len(qs_array)):
>         item_to_write = bytes(qs_array[offset])
>         struct.pack_into(buf, "<Q", offset, item_to_write)

Shouldn't the two first parameters be swapped, like this?

        struct.pack_into("<Q", buf, offset, item_to_write)


Also it wouldn't surprise me if you had to use offset*8 instead of just
offset in that pack_into() call -- I would think the offset is specified
in bytes.

--
"In the old days, writers used to sit in front of a typewriter and stare out of
the window. Nowadays, because of the marvels of convergent technology, the thing
you type on and the window you stare out of are now the same thing.”
-- Douglas Adams