MetaArray in Python 3

[Paul Manis]

As we move towards using Python 3, I have encountered a problem related to the MetaArrays. Although not related to the graphics portion of pyqtgraph, MetaArrays are part of pyqtgraph. I suspect they are not widely used, but they are used extensively in acq4 (a separate project that depends on pyqtgraph).

MetaArrays written in Python 2 or Python 3 are readable in Python 2. However, neither seems to be readable in Python 3.  This seems to be related to the difference between strings and bytes in the two when interpreting part of the data. 

The specific error (test script of minimal working version is attached; see the code for usage)  is:

user: sandbox $ python test_ma.py read 2

Running under Python 3:  3.6.5

Reading metaarray_py2.ma

Traceback (most recent call last):

  File "test_ma.py", line 57, in <module>

    mdata = ma.MetaArray(file=fn)

  File "/Users/pbmanis/Desktop/Python/pyqtgraph/pyqtgraph/metaarray/MetaArray.py", line 135, in __init__

    self.readFile(file, **kwargs)

  File "/Users/pbmanis/Desktop/Python/pyqtgraph/pyqtgraph/metaarray/MetaArray.py", line 752, in readFile

    meta = MetaArray._readMeta(fd)

  File "/Users/pbmanis/Desktop/Python/pyqtgraph/pyqtgraph/metaarray/MetaArray.py", line 777, in _readMeta

    meta += line

TypeError: must be str, not bytes

user: sandbox $ python test_ma.py read 3

Running under Python 3:  3.6.5

Reading metaarray_py3.ma

Traceback (most recent call last):

  File "test_ma.py", line 57, in <module>

    mdata = ma.MetaArray(file=fn)

  File "/Users/pbmanis/Desktop/Python/pyqtgraph/pyqtgraph/metaarray/MetaArray.py", line 135, in __init__

    self.readFile(file, **kwargs)

  File "/Users/pbmanis/Desktop/Python/pyqtgraph/pyqtgraph/metaarray/MetaArray.py", line 752, in readFile

    meta = MetaArray._readMeta(fd)

  File "/Users/pbmanis/Desktop/Python/pyqtgraph/pyqtgraph/metaarray/MetaArray.py", line 777, in _readMeta

    meta += line

TypeError: must be str, not bytes

Whereas in Python 2 we can read or write python 2 or 3 files just fine.

user: sandbox $ python test_ma.py read 3

Running under Python 2:  2.7.14

user metaarray_py3.ma (displays data)

The problem lies in this routine around line 777 in MetaArray.py:

    @staticmethod

    def _readMeta(fd):

        """Read meta array from the top of a file. Read lines until a blank line is reached.

        This function should ideally work for ALL versions of MetaArray.

        """

        meta = ''

        ## Read meta information until the first blank line

        while True:

            line = fd.readline().strip()

            if line == '':

                break

            meta += line

        ret = eval(meta)

        #print ret

        return ret

I have tried to clean this up to be agnostic with respect to python versions, but without success. I suspect there is a simple way to do this, but I haven't found it yet.

Any ideas?

--Paul

[Paul Manis]

Partly solved: h5py (used by MetaArray in pyqtgraph) returns bytestrings in python 3 (but just "str" in python2), and some comparisons have to be made using proper representations (it was helpful to view this : https://nedbatchelder.com/text/unipain.html). Representing a couple of constants as unicode or bytestrings, and a couple of try-except statements using decode() were sufficient to remove the error. The metaarry "info" entry still has to be massaged to convert some data elements in the dictionaries, but that is outside the scope of MetaArray at this point because the structure of those dicts can be arbitrarily defined by the user.

I will create a PR against the current pyqtgraph for Luke to examine. 

--pbm

PS - for those who care, the error below arose as an immediate effect of not recognizing the magic string for an hdf5 file in python 3. Representing the string as b'\x89HDF\r\n\x1a\n' (around line 746 of MetaArray.py) works for both py2 and 3, and causes the file to be read as an hdf5 file, rather than be parsed by _readMeta. 

On Saturday, July 21, 2018 at 6:35:43 PM UTC-4, Paul Manis wrote:

As we move towards using Python 3, I have encountered a problem related to the MetaArrays. Although not related to the graphics portion of pyqtgraph, MetaArrays are part of pyqtgraph. I suspect they are not widely used, but they are used extensively in acq4 (a separate project that depends on pyqtgraph).

MetaArrays written in Python 2 or Python 3 are readable in Python 2. However, neither seems to be readable in Python 3.  This seems to be related to the difference between strings and bytes in the two when interpreting parts of the data. 

[Luke Campagnola]

Hi Paul!

I worked on this a bit yesterday, but it's still not 100% working when I test storage / retrieval of different metadata types. I am turning your script into a unit test, and also thinking about breaking metaarray out into its own library; I agree it doesn't belong in pyqtgraph.

--
You received this message because you are subscribed to the Google Groups "pyqtgraph" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pyqtgraph+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/pyqtgraph/1c02255f-d2d4-4d7f-b55a-e2cabab1ca61%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.