scipy.io.loadmat error for large MAT file on network drive

[Tim Williams]

Hi,

I'm getting an error trying to read a large (100MB) MAT file using scipy.io.loadmat (version 0.10.0b2, Python 2.7). Here is the message I get:

>>> os.getcwd()
'X:\\Exp003\\DATA\\Porecounts\\Face'
>>> matdata=scipy.io.loadmat(os.path.join(os.getcwd(),'Exp003_025_A009_IR1_01_0_porecount'))
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "C:\Python27\Lib\site-packages\scipy\io\matlab\mio.py", line 175, in loadmat
    matfile_dict = MR.get_variables(variable_names)
  File "C:\Python27\Lib\site-packages\scipy\io\matlab\mio5.py", line 292, in get_variables
    res = self.read_var_array(hdr, process)
  File "C:\Python27\Lib\site-packages\scipy\io\matlab\mio5.py", line 255, in read_var_array
    return self._matrix_reader.array_from_header(header, process)
  File "mio5_utils.pyx", line 624, in scipy.io.matlab.mio5_utils.VarReader5.array_from_header (scipy\io\matlab\mio5_utils.c:5280)
  File "mio5_utils.pyx", line 671, in scipy.io.matlab.mio5_utils.VarReader5.array_from_header (scipy\io\matlab\mio5_utils.c:4940)
  File "mio5_utils.pyx", line 900, in scipy.io.matlab.mio5_utils.VarReader5.read_struct (scipy\io\matlab\mio5_utils.c:7455)
  File "mio5_utils.pyx", line 622, in scipy.io.matlab.mio5_utils.VarReader5.read_mi_matrix (scipy\io\matlab\mio5_utils.c:4533)
  File "mio5_utils.pyx", line 653, in scipy.io.matlab.mio5_utils.VarReader5.array_from_header (scipy\io\matlab\mio5_utils.c:4720)
  File "mio5_utils.pyx", line 706, in scipy.io.matlab.mio5_utils.VarReader5.read_real_complex (scipy\io\matlab\mio5_utils.c:5469)
  File "mio5_utils.pyx", line 424, in scipy.io.matlab.mio5_utils.VarReader5.read_numeric (scipy\io\matlab\mio5_utils.c:3303)
  File "mio5_utils.pyx", line 360, in scipy.io.matlab.mio5_utils.VarReader5.read_element (scipy\io\matlab\mio5_utils.c:3032)
  File "streams.pyx", line 119, in scipy.io.matlab.streams.cStringStream.read_string (scipy\io\matlab\streams.c:1827)
IOError: could not read bytes

I thought it was a problem with just the file being on a mounted windows share drive, because when I copy over to a local drive, I can read the file fine, I then found a thread on this list about problems reading large MAT files, so I went inside of MatLab and deleted the one of the structure members that had the majority of the file's size. I could then read the smaller file with loadmat, so it seems to be a combination of the file being large, and being on a network.

For now, I can copy all the files I need to process to a local drive, but I'd like to be able to access these files from where they are.

Thanks for any help.