[SciPy-User] scipy.io.loadmat error when trying to read a .mat file

1,471 views
Skip to first unread message

Dharhas Pothina

unread,
Oct 4, 2010, 9:30:14 AM10/4/10
to <SciPy Users List
Hi,

I'm using scipy 0.8.0 and having problems using the loadmat function. I've looked through the forum and found a few posts that seem related separately to this warning and error but not the solution. Any help is appreciated.

I'm receiving the following warning when importing scipy.io

In [4]: import scipy.io
C:\Python26\lib\site-packages\scipy\io\matlab\mio5.py:90: RuntimeWarning: __builtin__.file size changed, may indicate binary incompatibility from mio5_utils import VarReader5

And then the following error when trying to read a .mat file

In [5]: a = scipy.io.loadmat('20100819102442.mat')
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)

C:\Documents and Settings\dpothina\My Documents\Projects\InstreamFlows\M9\<ipython console> in <modu
le>()

C:\Python26\lib\site-packages\scipy\io\matlab\mio.pyc in loadmat(file_name, mdict, appendmat, **kwar
gs)
138 '''
139 MR = mat_reader_factory(file_name, appendmat, **kwargs)
--> 140 matfile_dict = MR.get_variables()
141 if mdict is not None:
142 mdict.update(matfile_dict)

C:\Python26\lib\site-packages\scipy\io\matlab\mio5.pyc in get_variables(self, variable_names)
416 continue
417 try:
--> 418 res = self.read_var_array(hdr, process)
419 except MatReadError, err:
420 warnings.warn(

C:\Python26\lib\site-packages\scipy\io\matlab\mio5.pyc in read_var_array(self, header, process)
385 `process`.
386 '''
--> 387 return self._matrix_reader.array_from_header(header, process)
388
389 def get_variables(self, variable_names=None):

C:\Python26\lib\site-packages\scipy\io\matlab\mio5_utils.pyd in scipy.io.matlab.mio5_utils.VarReader
5.array_from_header (scipy\io\matlab\mio5_utils.c:4821)()

C:\Python26\lib\site-packages\scipy\io\matlab\mio5_utils.pyd in scipy.io.matlab.mio5_utils.VarReader
5.array_from_header (scipy\io\matlab\mio5_utils.c:4491)()

C:\Python26\lib\site-packages\scipy\io\matlab\mio5_utils.pyd in scipy.io.matlab.mio5_utils.VarReader
5.read_struct (scipy\io\matlab\mio5_utils.c:6713)()

C:\Python26\lib\site-packages\scipy\io\matlab\mio5_utils.pyd in scipy.io.matlab.mio5_utils.VarReader
5.read_mi_matrix (scipy\io\matlab\mio5_utils.c:4079)()

C:\Python26\lib\site-packages\scipy\io\matlab\mio5_utils.pyd in scipy.io.matlab.mio5_utils.VarReader
5.array_from_header (scipy\io\matlab\mio5_utils.c:4491)()

C:\Python26\lib\site-packages\scipy\io\matlab\mio5_utils.pyd in scipy.io.matlab.mio5_utils.VarReader
5.read_struct (scipy\io\matlab\mio5_utils.c:6713)()

C:\Python26\lib\site-packages\scipy\io\matlab\mio5_utils.pyd in scipy.io.matlab.mio5_utils.VarReader
5.read_mi_matrix (scipy\io\matlab\mio5_utils.c:4079)()

C:\Python26\lib\site-packages\scipy\io\matlab\mio5_utils.pyd in scipy.io.matlab.mio5_utils.VarReader
5.array_from_header (scipy\io\matlab\mio5_utils.c:4406)()

C:\Python26\lib\site-packages\scipy\io\matlab\mio5_utils.pyd in scipy.io.matlab.mio5_utils.VarReader
5.read_char (scipy\io\matlab\mio5_utils.c:5555)()

TypeError: buffer is too small for requested array


thanks

- dharhas

_______________________________________________
SciPy-User mailing list
SciPy...@scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-user

Pauli Virtanen

unread,
Oct 4, 2010, 10:16:04 AM10/4/10
to scipy...@scipy.org
Mon, 04 Oct 2010 08:30:14 -0500, Dharhas Pothina wrote:
> I'm using scipy 0.8.0 and having problems using the loadmat function.
> I've looked through the forum and found a few posts that seem related
> separately to this warning and error but not the solution. Any help is
> appreciated.

(i) Does it happen with a specific .mat file only, or with all files? If
a specific file, it would be useful if you can put it available somewhere.

(ii) How did you install Scipy 0.8.0 (which binaries?)

(iii) Try running the test suite,

>>> import scipy.io
>>> scipy.io.test("full", verbose=2, extra_argv=["--exe"])

Does it pass?

Dharhas Pothina

unread,
Oct 4, 2010, 11:24:33 AM10/4/10
to scipy...@scipy.org
Pauli,

It looks like it fails with files generated by one of our instruments. I was able to read in another type of matlab file qithout a problem. I think I was able to read this type of file in a previous version of scipy (can't tell you which one now, it was a long time ago), when I had tested our ability to read this type of file. I also tried scipy v0.7.0 installed on one of our linux boxes just know and it gave me the same error (but not the warning). Running the test suite on the windows box gave me three errors listed below, on linux (w/ ver scipy 7.0) all tests passed. I've posted two sample files at the links below. On windows I'm installed with the scipy-0.8.0-win32-superpack-python2.6.exe binary.

sample files:

http://midgewater.twdb.state.tx.us/bays_estuaries/Publications\20100819125947.mat (~330KB)
http://midgewater.twdb.state.tx.us/bays_estuaries/Publications\20100819102442.mat (~18mb)

scipy.io test errors on windows 32 bit:

======================================================================
ERROR: test_complex (test_array_import.TestReadArray)
----------------------------------------------------------------------
Traceback (most recent call last):
File "C:\Python26\lib\site-packages\scipy\io\tests\test_array_import.py", line 38, in test_complex

io.write_array(fname,a)
AttributeError: 'module' object has no attribute 'write_array'

======================================================================
ERROR: test_float (test_array_import.TestReadArray)
----------------------------------------------------------------------
Traceback (most recent call last):
File "C:\Python26\lib\site-packages\scipy\io\tests\test_array_import.py", line 46, in test_float
io.write_array(fname,a)
AttributeError: 'module' object has no attribute 'write_array'

======================================================================
ERROR: test_integer (test_array_import.TestReadArray)
----------------------------------------------------------------------
Traceback (most recent call last):
File "C:\Python26\lib\site-packages\scipy\io\tests\test_array_import.py", line 55, in test_integer

io.write_array(fname,a)
AttributeError: 'module' object has no attribute 'write_array'

----------------------------------------------------------------------
Ran 386 tests in 8.061s

FAILED (errors=3)
Out[7]: <nose.result.TextTestResult run=386 errors=3 failures=0>

- dharhas

Pauli Virtanen

unread,
Oct 4, 2010, 12:04:35 PM10/4/10
to scipy...@scipy.org
Mon, 04 Oct 2010 10:24:33 -0500, Dharhas Pothina wrote:
> It looks like it fails with files generated by one of our instruments. I
> was able to read in another type of matlab file without a problem. I

> think I was able to read this type of file in a previous version of
> scipy (can't tell you which one now, it was a long time ago), when I had
> tested our ability to read this type of file. I also tried scipy v0.7.0
> installed on one of our linux boxes just know and it gave me the same
> error (but not the warning). Running the test suite on the windows box
> gave me three errors listed below, on linux (w/ ver scipy 7.0) all tests
> passed. I've posted two sample files at the links below. On windows I'm
> installed with the scipy-0.8.0-win32-superpack-python2.6.exe binary.

Ok, access to the failing file is probably needed to sort this out.

I'm not able to access these -- maybe your institute has a firewall in
between? You can also e.g. send the smaller file to me via mail.

> scipy.io test errors on windows 32 bit:

[clip]


> "C:\Python26\lib\site-packages\scipy\io\tests\test_array_import.py",

There's no file named "test_array_import.py" in Scipy 0.8.0. You probably
have some left-over files from Scipy 0.7.x in the directory. Anyway, the
test suite seems to be in order.

--
Pauli Virtanen

Dharhas Pothina

unread,
Oct 4, 2010, 12:26:04 PM10/4/10
to scipy...@scipy.org

hmm. That site should be outside our firewall. I'll send you the smaller file via email to your personal email, since I'm not sure what the list policy on attachment sizes.

- dharhas

>>> Pauli Virtanen <p...@iki.fi> 10/4/2010 11:04 AM >>>

josef...@gmail.com

unread,
Oct 4, 2010, 12:29:20 PM10/4/10
to SciPy Users List
On Mon, Oct 4, 2010 at 12:26 PM, Dharhas Pothina
<Dharhas...@twdb.state.tx.us> wrote:
>
> hmm. That site should be outside our firewall. I'll send you the smaller file via email to your personal email, since I'm not sure what the list policy on attachment sizes.
>
> - dharhas
>
>>>> Pauli Virtanen <p...@iki.fi> 10/4/2010 11:04 AM >>>
> Mon, 04 Oct 2010 10:24:33 -0500, Dharhas Pothina wrote:
>> It looks like it fails with files generated by one of our instruments. I
>> was able to read in another type of matlab file without a problem. I
>> think I was able to read this type of file in a previous version of
>> scipy (can't tell you which one now, it was a long time ago), when I had
>> tested our ability to read this type of file. I also tried scipy v0.7.0
>> installed on one of our linux boxes just know and it gave me the same
>> error (but not the warning). Running the test suite on the windows box
>> gave me three errors listed below, on linux (w/ ver scipy 7.0) all tests
>> passed. I've posted two sample files at the links below. On windows I'm
>> installed with the scipy-0.8.0-win32-superpack-python2.6.exe binary.
>
> Ok, access to the failing file is probably needed to sort this out.
>
>> sample files:
>>
>> http://midgewater.twdb.state.tx.us/bays_estuaries/Publications
> \20100819125947.mat
>> (~330KB)
>> http://midgewater.twdb.state.tx.us/bays_estuaries/Publications
> \20100819102442.mat
>> (~18mb)

a forward slash does it for me:
http://midgewater.twdb.state.tx.us/bays_estuaries/Publications/20100819102442.mat

Josef

Dharhas Pothina

unread,
Oct 4, 2010, 12:43:17 PM10/4/10
to SciPy Users List

Thanks, I didn't notice that.

- dharhas

Matthew Brett

unread,
Oct 4, 2010, 1:31:43 PM10/4/10
to SciPy Users List
Hi,

On Mon, Oct 4, 2010 at 9:04 AM, Pauli Virtanen <p...@iki.fi> wrote:
> Mon, 04 Oct 2010 10:24:33 -0500, Dharhas Pothina wrote:
>> It looks like it fails with files generated by one of our instruments. I
>> was able to read in another type of matlab file without a problem. I
>> think I was able to read this type of file in a previous version of
>> scipy (can't tell you which one now, it was a long time ago), when I had
>> tested our ability to read this type of file. I also tried scipy v0.7.0
>> installed on one of our linux boxes just know and it gave me the same
>> error (but not the warning). Running the test suite on the windows box
>> gave me three errors listed below, on linux (w/ ver scipy 7.0) all tests
>> passed. I've posted two sample files at the links below. On windows I'm
>> installed with the scipy-0.8.0-win32-superpack-python2.6.exe binary.
>
> Ok, access to the failing file is probably needed to sort this out.

It's very possible that the behavior of loadmat has changed because I
refactored it radically between 0.7 and 0.8.

I'll have a look your file this morning in case it will save anyone some time,

See you,

Matthew

Pauli Virtanen

unread,
Oct 4, 2010, 2:11:36 PM10/4/10
to scipy...@scipy.org
Mon, 04 Oct 2010 11:26:04 -0500, Dharhas Pothina wrote:

> hmm. That site should be outside our firewall. I'll send you the smaller
> file via email to your personal email, since I'm not sure what the list
> policy on attachment sizes.

Ok, I seem to be able to download these with Google Chrome, but not with
Wget. Apparently, the site does some filtering based on the user agent
string...

Matthew Brett

unread,
Oct 4, 2010, 6:56:27 PM10/4/10
to SciPy Users List
Hi,

> It's very possible that the behavior of loadmat has changed because I
> refactored it radically between 0.7 and 0.8.
>
> I'll have a look your file this morning in case it will save anyone some time,

I see that, if I load the 20100819102442.mat file in matlab and save
it again, loadmat can read it. This is in matlab r2008a.

I think the problem is arising in the 'System' variable in that file.

How did this mat file arise? What version of matlab is it being saved
from? With what flags?

Best,

Matthew Brett

unread,
Oct 5, 2010, 12:41:10 AM10/5/10
to SciPy Users List
Hi,

> I see that, if I load the 20100819102442.mat file in matlab and save
> it again, loadmat can read it.   This is in matlab r2008a.

Just a little more on the file 20100819102442.mat

I suspect it must be being written from outside matlab.

The error you are getting is because there is a 1-length string which
has a 0 byte length. Matlab copes OK with this, and returns a string
' ' - here for the System.Units struct 'Sample' field:

Pitch: 'deg'
Roll: 'deg'
Heading: 'deg'
Temperature: 'degC'
Time: ' '
SNR: 'dB'
Sample: ' '
Voltage: 'V'
Cell_Start: 'm'
Cell_Size: 'm'
Step: ' '
Pings: ' '

scipy and octave barf. I don't know what scipy used to do in that
situation, but I would be surprised if it hasn't always barfed. It's
easy to fix that one, but then, once fixed, you get this:

/Users/mb312/usr/local/lib/python2.6/site-packages/scipy/io/matlab/mio5_utils.so
in scipy.io.matlab.mio5_utils.VarReader5.read_struct
(./mio5_utils.c:6836)()

ValueError: two fields with the same name

That turns out to be because the 'Summary' struct has 4 fields all
named 'Station_Q':

Top_Q: [34x1 double]
Middle_Q: [34x1 double]
Bottom_Q: [34x1 double]
Left_Q: [34x1 double]
Right_Q: [34x1 double]
Total_Q: [34x1 double]
Depth: [34x1 double]
Cells: [34x1 double]
Track: [34x2 double]
Mean_Vel: [34x2 double]
Boat_Vel: [34x4 double]
Station_Q: [34x1 double]
Station_Q: [34x1 double]
Station_Q: [34x1 double]
Station_Q: [34x1 double]

Matlab can handle that, although I don't know how you can make a
structure that has two fields with the same name - as far as I know
there's no way to do that with matlab syntax, e.g:

>> a = struct('field1', 1, 'field1', 2)
??? Error using ==> struct
Duplicate field name "field1"

I don't know how to return those guys to you either - structs come
back either as record arrays or objects, and neither allow duplicate
names. Of course we could rename the duplicates, but that would start
to get a bit tangled.

Is there any way you can load these into matlab and fix them up before
loading into scipy?

I'll commit the changes to allow the 1 length 0 byte strings tomorrow.

Pauli Virtanen

unread,
Oct 5, 2010, 5:09:32 AM10/5/10
to scipy...@scipy.org
Mon, 04 Oct 2010 21:41:10 -0700, Matthew Brett wrote:
[clip]

> That turns out to be because the 'Summary' struct has 4 fields all named
> 'Station_Q':
[clip]

> Matlab can handle that, although I don't know how you can make a
> structure that has two fields with the same name - as far as I know
> there's no way to do that with matlab syntax, e.g:
>
>>> a = struct('field1', 1, 'field1', 2)
> ??? Error using ==> struct
> Duplicate field name "field1"
>
> I don't know how to return those guys to you either - structs come back
> either as record arrays or objects, and neither allow duplicate names.
> Of course we could rename the duplicates, but that would start to get a
> bit tangled.
[clip]

Renaming to "originalname_counter" sounds like an OK solution. Skipping
duplicates + emitting a warning might also be OK...

--
Pauli Virtanen

Dharhas Pothina

unread,
Oct 5, 2010, 9:55:53 AM10/5/10
to scipy...@scipy.org

@ Mathew

> I suspect it must be being written from outside matlab.

You are right, this file is being written by a instrument software (Sontek River Surveyor Live). Unfortunately, the new version of this instrument and software no longer allows export of data in any format but .mat.

> Is there any way you can load these into matlab and fix them up before
> loading into scipy?

I'm trying to get rid of the matlab dependency since we only have one copy available and I'm trying to write some automated processing scripts.

> Matlab can handle that, although I don't know how you can make a
> structure that has two fields with the same name - as far as I know
> there's no way to do that with matlab syntax, e.g:

I opened the files in matlab (v R14) and I can see the 4 duplicate StationQ arrays but I don't seem to be able access them separately. The Array Editor view and the tab complete shows the 4 Station_Q but I can only access a Summary.StationQ and I don't know which one of the 4 it is accessing. The documentation for software that describes the matlab variables/structures generated by the matlab export command does not list Summary.StationQ at all. It isn't one of the primary variables we need and I'm not sure what it contains. I am assuming the four copies is a bug in their software, I'm going to call them and submit a bug report, not sure how much traction it will get though. The other option is that it represents data from the 4 sonar beams but in all the other structures that have data from the 4 beams the data is represented by a 2D array.

@Pauli & Mathew


> Renaming to "originalname_counter" sounds like an OK solution. Skipping
> duplicates + emitting a warning might also be OK...

Since it looks like matlab cannot handle duplicate field names (at least w/ R14 I haven't tried more recent versions) in terms of accessing the variable once loaded in, and also since it is impossible to make a structure with duplicate field names within matlab, I think this is an error in the way the data file is being written. My vote would be for skipping the duplicates and issuing a warning. That would allow for a more robust way loading the file into scipy directly and also avoid matlab completely.

Thanks for all your help on this.

- dharhas

Benjamin Root

unread,
Oct 5, 2010, 10:13:19 AM10/5/10
to SciPy Users List

I did a little bit of testing on this issue since I was curious about how this is even possible.

It appears that the Station_Q that you are accessing is the last one.  If I modify the contents of x.Summary.Station_Q, and then display what is in x.Summary, the last Station_Q is changed.  It is *possible* to get access to the other 3 Station_Q variables if you use the "struct2cell" function, which gets you back everything in a cell array

I wonder if Matlab is loading the data into a cell array first and then -- somehow --- converting that into a structure.

BTW, this was checked using Matlab R2008b.

Ben Root

Matthew Brett

unread,
Oct 5, 2010, 2:01:48 PM10/5/10
to SciPy Users List
Hi,

> I'm trying to get rid of the matlab dependency since we only have one copy available and I'm trying to write some automated processing scripts.

That's easy to imagine...

> @Pauli & Mathew
>> Renaming to "originalname_counter" sounds like an OK solution. Skipping
>> duplicates + emitting a warning might also be OK...
>
> Since it looks like matlab cannot handle duplicate field names (at least w/ R14 I haven't tried more recent versions) in terms of accessing the variable once loaded in, and also since it is impossible to make a structure with duplicate field names within matlab, I think this is an error in the way the data file is being written.

Yes, I'm sure it's an error, but I suppose we should set ourselves the
goal of being able to read what matlab can read.

I don't like the 'originalname_counter' option because that might
conceivably still clash with the intended fieldnames (imagine the
original fieldnames of ['field1_1', 'field1', 'field1'] for example).
I see that matlab intends names prepended with an underscore to be
invalid:

>> a = [];
>> a._field1 = 1
??? a._field1 = 1
|
Error: The input character is not valid in MATLAB statements or expressions.

So one option would be ['field1_1', 'field1', 'field1'] ->
['field1_1', 'field1', '_0_field1'] . It would be a problem when
saving back to matlab, but that seems like a sensible problem to have.

Best,

Matthew

Matthew Brett

unread,
Oct 5, 2010, 2:25:20 PM10/5/10
to SciPy Users List
Hi,

> I did a little bit of testing on this issue since I was curious about how
> this is even possible.
>
> It appears that the Station_Q that you are accessing is the last one.  If I
> modify the contents of x.Summary.Station_Q, and then display what is in
> x.Summary, the last Station_Q is changed.  It is *possible* to get access to
> the other 3 Station_Q variables if you use the "struct2cell" function, which
> gets you back everything in a cell array

Thanks - nice forensics ;) . It makes a difference, because it means
you can get the data out, and that we should probably allow that too,

See you,

Matthew

Matthew Brett

unread,
Oct 5, 2010, 9:13:13 PM10/5/10
to SciPy Users List
Hi,

> Yes, I'm sure it's an error, but I suppose we should set ourselves the
> goal of being able to read what matlab can read.

You will be pleased to know that, on further investigation, there are
also two 'System' variables in the file. You only see the second when
you read it into Matlab. The first has fields:

('SerialNumber',
'FirmwareVersion',
'FirmwareRevision',
'Frequency',
'InstrumentMatrix_0',
'InstrumentMatrix_1',
'InstrumentMatrix_2')

It's a fun file ;)

Dharhas Pothina

unread,
Oct 6, 2010, 9:36:40 AM10/6/10
to SciPy Users List

fun fun fun. I can't see the second 'System' variable in Matlab R14. Also, from what I could tell the 4 duplicate Summary.StationQ arrays contain different data. The Summary.Units array that contains the units for each variable in summary only has a single Summary.Units.Station_Q value = m3/s. So I still think the duplicates are a mistake. I let you know what I find out from the manufacturer when I'm able to get a hold of them.

In the meanwhile is there a work around to load the rest of the data file into scipy. The duplicate variables that are causing the problems are not ones I need for my analysis. Is it possible from a technical point of view to mimic the matlab load command that lets you specify which variables to load:

Examples for pattern matching:
load fname a* % Load variables starting with "a"
load fname -regexp ^b\d{3}$ % Load variables starting with "b" and
% followed by 3 digits
load fname -regexp \d % Load variables containing any digits

i.e. something along the lines of

scipy.io.loadmat(fname, return_vars='BottomTrack, GPS, Station')

Of course this would require knowing what the field names where. So to be completely independent from matlab we would need a helper function that just returned the names of variables contained in the .mat file.

i.e. something like

scipy.io.loadmat(fname, return_headers=True)

where the result would be a list of variable names.

- dharhas

>>> Matthew Brett <matthe...@gmail.com> 10/5/2010 8:13 PM >>>

Matthew Brett

unread,
Oct 12, 2010, 1:40:46 AM10/12/10
to SciPy Users List
Hi,

On Wed, Oct 6, 2010 at 6:36 AM, Dharhas Pothina
<Dharhas...@twdb.state.tx.us> wrote:
>
> fun fun fun.

Oh yes.

> I can't see the second 'System' variable in Matlab R14.

No, I guess matlab does the same as we do, and just overwrites with
the last occurrence of the variable.

> In the meanwhile is there a work around to load the rest of the data file into scipy.

Yes - I can now read both your files into scipy. I just did
'fieldname', '_1_fieldname', '_2_fieldname' ... for the duplicates, so
it's a bit of a hidden feature, but it does seem to work. You need to
clone my github copy:

git clone git://github.com/matthew-brett/scipy-work.git
git checkout -b mio-read-fix origin/mio-read-fix

and install scipy from there. Does that work for you?

Incidentally, if you see the scipy/io/matlab/mio5.py file you'll see a
function 'varmats_from_mat' that takes a given open mat file and
returns a list of individual variables as (name, mat_stream) tuples,
which you can then read out with loadmat. On the lines of:

from scipy.io.matlab.mio5 import varmats_from_mat
varmats = varmats_from_mat(open('strange_file.mat', 'rb'))

and then

first_vars = loadmat(varmats[0][1])

That last will give you the contents of the first variable.

> Is it possible from a technical point of view to mimic the matlab load command that lets you specify which variables to load:

Yes, almost anything is possible, but, oh, a little bit annoying to
implement unless it's really a killer feature for someone. I'm hoping
that now we can read your files, maybe it's a bit less killer for you?

You can already specify a variable name (not a regexp), so for example:

from scipy.io.matlab.mio5 import MatFile5Reader
rdr = MatFile5Reader(open('20100819125947.mat', 'rb'))
var = rdr.get_variables(['System'])

will already return you the first 'System' variable, that would
otherwise be overwritten when you read the full set of variables. You
are right, that should be in the loadmat API somewhere.

See you,

Matthew Brett

unread,
Oct 12, 2010, 2:51:16 AM10/12/10
to SciPy Users List
Hi again,

> You can already specify a variable name (not a regexp), so for example:
>
> from scipy.io.matlab.mio5 import MatFile5Reader
> rdr = MatFile5Reader(open('20100819125947.mat', 'rb'))
> var = rdr.get_variables(['System'])
>
> will already return you the first 'System' variable, that would
> otherwise be overwritten when you read the full set of variables.  You
> are right, that should be in the loadmat API somewhere.

Actually, I just did add that - in the same branch ('mio-read-fix')

In [5]: import scipy.io as sio
In [6]: vars = sio.loadmat('20100819125947.mat', variable_names=['System'])

will return just the first occurrence of a variable named 'System' in the file.

Best,

Dharhas Pothina

unread,
Oct 12, 2010, 9:12:23 AM10/12/10
to SciPy Users List
Mathew,

Many thanks. I'll test this out as soon as I can. I think for now I will go with the MatFile5Reader approach and just load the variables I need, since from what I understand this approach will work with the current version of scipy. I need to deploy this script across several users and I prefer to install release versions of Scipy on their machines. I'll transition to loadmat once that makes it into the stable release.

Will these fixes make it into the next release of scipy?

- dharhas

>>> Matthew Brett <matthe...@gmail.com> 10/12/2010 1:51 AM >>>

Matthew Brett

unread,
Oct 12, 2010, 3:34:52 PM10/12/10
to SciPy Users List
Hi,

> Many thanks. I'll test this out as soon as I can. I think for now I will go with the MatFile5Reader approach and just load the variables I need, since from what I understand this approach will work with the current version of scipy. I need to deploy this script across several users and I prefer to install release versions of Scipy on their machines. I'll transition to loadmat once that makes it into the stable release.

Ah - no - so there are three separate problems in your example file:

a) Duplicate variable names
b) Strings that have declared length > 0, and allocated storage of 0.
c) Structs with duplicate fieldnames.

For a) you can do something about this in current scipy if you want
the first (rather than the last) version of a particular variable, by
doing the MatFileReader trick. But that's your smallest problem
because b) and c) crash the scipy reader. So, you can probably pull
out any variables not suffering from b) or c), but that's all - with
current scipy.

> Will these fixes make it into the next release of scipy?

Yes, they should be. I will push them into the current svn version
for others to try and we'll see what damage I have done ;)

Cheers,

Dharhas Pothina

unread,
Oct 12, 2010, 4:00:22 PM10/12/10
to SciPy Users List

> Ah - no - so there are three separate problems in your example file:
>
> a) Duplicate variable names
> b) Strings that have declared length > 0, and allocated storage of 0.
> c) Structs with duplicate fieldnames.
>
> For a) you can do something about this in current scipy if you want
> the first (rather than the last) version of a particular variable, by
> doing the MatFileReader trick. But that's your smallest problem
> because b) and c) crash the scipy reader. So, you can probably pull
> out any variables not suffering from b) or c), but that's all - with
> current scipy.

>From my initial examination of the data files all of these issues occur with variables I am not planning to use in my script, so I *should* be ok.

Again thanks for the sleuthing and the fixes. I think the responses I get from this community are an order of magnitude more helpful than those I get for products we have commercial support contracts for.

- dharhas

Matthew Brett

unread,
Oct 12, 2010, 4:11:35 PM10/12/10
to SciPy Users List
Hi,

> Again thanks for the sleuthing and the fixes. I think the responses I get from this community are an order of magnitude more helpful than those I get for products we have commercial support contracts for.

Well - I hope we can live up to that recommendation ;).

Best,

Matthew

Reply all
Reply to author
Forward
0 new messages