[Numpy-discussion] Saving and loading a structured array from a TEXT file

[Emmanuel Mayssat]

Is there a way to save a structured array in a text file?
My problem is not so much in the saving procedure, but rather in the
'reloading' procedure.
See below

In [3]: import numpy as np

In [4]: r = np.ones(3,dtype=[('name', '|S5'), ('foo', '<i8'), ('bar', '<f8')])

In [5]: r.tofile('toto.txt',sep='\n')

bash-4.2$ cat toto.txt
('1', 1, 1.0)
('1', 1, 1.0)
('1', 1, 1.0)

In [7]: r2 = np.fromfile('toto.txt',sep='\n',dtype=r.dtype)
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
/home/cls1fs/clseng/10/<ipython-input-7-b07ba265ede7> in <module>()
----> 1 r2 = np.fromfile('toto.txt',sep='\n',dtype=r.dtype)

ValueError: Unable to read character files of that array type

--
Emmanuel
_______________________________________________
NumPy-Discussion mailing list
NumPy-Di...@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Derek Homeier]

On 23 Jan 2012, at 21:15, Emmanuel Mayssat wrote:

> Is there a way to save a structured array in a text file?
> My problem is not so much in the saving procedure, but rather in the
> 'reloading' procedure.
> See below
>
>
> In [3]: import numpy as np
>
> In [4]: r = np.ones(3,dtype=[('name', '|S5'), ('foo', '<i8'), ('bar', '<f8')])
>
> In [5]: r.tofile('toto.txt',sep='\n')
>
> bash-4.2$ cat toto.txt
> ('1', 1, 1.0)
> ('1', 1, 1.0)
> ('1', 1, 1.0)
>
> In [7]: r2 = np.fromfile('toto.txt',sep='\n',dtype=r.dtype)
> ---------------------------------------------------------------------------
> ValueError Traceback (most recent call last)
> /home/cls1fs/clseng/10/<ipython-input-7-b07ba265ede7> in <module>()
> ----> 1 r2 = np.fromfile('toto.txt',sep='\n',dtype=r.dtype)
>
> ValueError: Unable to read character files of that array type

I think most of the np.fromfile functionality works for binary input; for reading text
input np.loadtxt and np.genfromtxt are the (currently) recommended functions.
It is bit tricky to read the format generated by tofile() in the above example, but
the following should work:

cnv = {0: lambda s: s.lstrip('('), -1: lambda s: s.rstrip(')')}
r2 = np.loadtxt('toto.txt', delimiter=',', converters=cnv, dtype=r.dtype)

Generally loadtxt works more smoothly together with savetxt, but the latter unfortunately
does not offer an easy way to save structured arrays (note to self and others currently
working on npyio: definitely room for improvement!).

HTH,
Derek

[Derek Homeier]

On 23 Jan 2012, at 22:07, Derek Homeier wrote:

>> In [4]: r = np.ones(3,dtype=[('name', '|S5'), ('foo', '<i8'), ('bar', '<f8')])
>>
>> In [5]: r.tofile('toto.txt',sep='\n')
>>
>> bash-4.2$ cat toto.txt
>> ('1', 1, 1.0)
>> ('1', 1, 1.0)
>> ('1', 1, 1.0)
>>
>

> cnv = {0: lambda s: s.lstrip('('), -1: lambda s: s.rstrip(')')}
> r2 = np.loadtxt('toto.txt', delimiter=',', converters=cnv, dtype=r.dtype)
>
> Generally loadtxt works more smoothly together with savetxt, but the latter unfortunately
> does not offer an easy way to save structured arrays (note to self and others currently
> working on npyio: definitely room for improvement!).

For the record, in that example

np.savetxt('toto.txt', r, fmt='%s,%d,%f')

would work as well, saving you the custom converter for loadtxt - it could just become tedious
to work out the format for more complex structures, so an option to construct this automatically
from r.dtype could certainly be a nice enhancement.
Just wondering, is there something like the inverse operator to np.format_parser, i.e.
mapping each dtype to a default print format specifier?

Cheers,