Possible Bug & Fix for caffe.io.array_to_datum

[Steven]

I think there may be a problem with caffe.io.array_to_datum. It only seems to convert arrays of type uint8 or float64. I would have expected it to handle arrays of type int32 and float32.This is what I saw:

>>> import caffe
>>> from caffe.proto import caffe_pb2
>>> import numpy as np
>>> ar = np.array([[[1., 2., 3., 4.,],[5., 6., 7., 8.,],[11., 22., 33., 44.]]])
>>> ar.shape
(1, 3, 4)

>>> arU8 = ar.astype(np.uint8)
>>> arI32 = ar.astype(np.int32)
>>> arF32 = ar.astype(np.float32)
>>> arF64 = ar.astype(np.float64)

>>> a =caffe.io.array_to_datum(arU8)

>>> b = caffe.io.array_to_datum(arI32)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/smgutstein/Caffe/caffe/python/caffe/io.py", line 89, in array_to_datum
    datum.float_data.extend(arr.flat)  #SG sub
  File "/usr/local/lib/python2.7/dist-packages/google/protobuf/internal/containers.py", line 128, in extend
    new_values.append(self._type_checker.CheckValue(elem))
  File "/usr/local/lib/python2.7/dist-packages/google/protobuf/internal/type_checkers.py", line 103, in CheckValue
    raise TypeError(message)
TypeError: 1 has type <type 'numpy.int32'>, but expected one of: (<type 'float'>, <type 'int'>, <type 'long'>)

>>> c = caffe.io.array_to_datum(arF32)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/smgutstein/Caffe/caffe/python/caffe/io.py", line 89, in array_to_datum
    datum.float_data.extend(arr.flat)  #SG sub
  File "/usr/local/lib/python2.7/dist-packages/google/protobuf/internal/containers.py", line 128, in extend
    new_values.append(self._type_checker.CheckValue(elem))
  File "/usr/local/lib/python2.7/dist-packages/google/protobuf/internal/type_checkers.py", line 103, in CheckValue
    raise TypeError(message)
TypeError: 1.0 has type <type 'numpy.float32'>, but expected one of: (<type 'float'>, <type 'int'>, <type 'long'>)

>>> d = caffe.io.array_to_datum(arF64)
>>>

The problem, I believe, lies in the use of the extend function to store any non-uint8 data in the float_data field of a Datum object. Although this is very robust for lists, the float_data field for a Datum object is a google.protobuf.internal.containers.RepeatedScalarFieldContainer, which is more type sensitive when using the extend function, as can be seen in the error trace:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/smgutstein/Caffe/caffe/python/caffe/io.py", line 89, in array_to_datum
    datum.float_data.extend(arr.flat)  
  File "/usr/local/lib/python2.7/dist-packages/google/protobuf/internal/containers.py", line 128, in extend
    new_values.append(self._type_checker.CheckValue(elem))
  File "/usr/local/lib/python2.7/dist-packages/google/protobuf/internal/type_checkers.py", line 103, in CheckValue
    raise TypeError(message)
TypeError: 1.0 has type <type 'numpy.float32'>, but expected one of: (<type 'float'>, <type 'int'>, <type 'long'>)

The two ways of I see to fix this are:

1. Change protobuf's type checking, so that it allows for numpy.types. I believe this would involve modifying the following section of code (lines 196-208 in type_checkers.py):
 

# Type-checkers for all scalar CPPTYPEs.
_VALUE_CHECKERS = {
    _FieldDescriptor.CPPTYPE_INT32: Int32ValueChecker(),
    _FieldDescriptor.CPPTYPE_INT64: Int64ValueChecker(),
    _FieldDescriptor.CPPTYPE_UINT32: Uint32ValueChecker(),
    _FieldDescriptor.CPPTYPE_UINT64: Uint64ValueChecker(),
    _FieldDescriptor.CPPTYPE_DOUBLE: TypeChecker(
        float, int, long),
    _FieldDescriptor.CPPTYPE_FLOAT: TypeChecker(
        float, int, long),
    _FieldDescriptor.CPPTYPE_BOOL: TypeChecker(bool, int),
    _FieldDescriptor.CPPTYPE_STRING: TypeChecker(bytes),
    }

However, that is probably best left to someone who understands protobuf better than I do.

2. Modify array_to_datum, so that when it flattens an array, the returned iterator does the type casting for protobuf:

def array_to_datum(arr, label=0):
    """Converts a 3-dimensional array to datum. If the array has dtype uint8,
    the output data will be encoded as a string. Otherwise, the output data
    will be stored in float format.
    """

    class npIterCast():                         # +                  
      '''Class added so that iterator           # +
         over numpy array will return           # +
         values of float or int type,           # +
         not np.float or np.int'''              # +
      def __init__(self, myIter, myCast):       # +
        self.myIter = myIter                    # +
        self.myCast = myCast                    # +
      def __iter__(self):                       # +
        return self                             # +
      def next(self):                           # +
        return myCast(self.myIter.next())       # +
   
    if not isinstance(arr,np.ndarray):             # +
        raise TypeError('Expecting a numpy array') # +
    if arr.ndim != 3:
        raise ValueError('Incorrect array shape.')
    datum = caffe_pb2.Datum()
    datum.channels, datum.height, datum.width = arr.shape
    if arr.dtype == np.uint8:
        datum.data = arr.tostring()
    else:
        if np.issubdtype(arr.dtype, np.int) or \          # +
           np.issubdtype(arr.dtype, np.unsignedinteger):  # +
            myCast = int                                  # +
        elif np.issubdtype(arr.dtype, np.float):          # +
            myCast = float                                # +
        else:                                             # +
            raise TypeError('Expecting a numpy array of either a float or int type') # +
       
        castIter = npIterCast(arr.flat, myCast) # +
        datum.float_data.extend(castIter)       # +
        datum.float_data.extend(arr.flat)       # -
    datum.label = label
    return datum

This gave me the following results, so it appears to be working fine:

>>> import caffe
>>> from caffe.proto import caffe_pb2
>>> import numpy as np
>>> ar = np.array([[[1., 2., 3., 4.,],[5., 6., 7., 8.,],[11., 22., 33., 44.]]])
>>>
>>> arU8 = ar.astype(np.uint8)
>>> arI32 = ar.astype(np.int32)
>>> arF32 = ar.astype(np.float32)
>>> arF64 = ar.astype(np.float64)
>>>
>>> a =caffe.io.array_to_datum(arU8)
>>> b = caffe.io.array_to_datum(arI32)
>>> c = caffe.io.array_to_datum(arF32)
>>> d = caffe.io.array_to_datum(arF64)
>>>
>>> a.float_data
[]
>>> a.data
'\x01\x02\x03\x04\x05\x06\x07\x08\x0b\x16!,'
>>>
>>> b.float_data
[1, 2, 3, 4, 5, 6, 7, 8, 11, 22, 33, 44]
>>> c.float_data
[1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 11.0, 22.0, 33.0, 44.0]
>>> d.float_data
[1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 11.0, 22.0, 33.0, 44.0]
>>>

There may still be an issue that other than uint8, all ints will be the system's default int type (i.e. either 32 or 64 bit), all unsigned ints will also be the system's default int type and all floats will be the default float type. But, for now, unless there is a better solution, I think it's OK to use this. Am I correct in believing this? Is there a better approach?

Thanks,

Steven