Working with xdressed types and np.arrays

[Matthew Gidden]

Hi all,

I wanted to confirm that this is issue-worth on the list before making one. Here's some output from IPython, where I've xdressed a class called ArcFlow and have a stlcontainer addition for `('vector', 'ArcFlow')`.

In [1]: import numpy as np
In [2]: from cyclopts.execute import ArcFlow
In [3]: from cyclopts.dtypes import xd_arcflow
In [4]: a = np.array([], dtype=xd_arcflow)
In [5]: np.append(a, ArcFlow(1, 5))
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-5-b62c3bae453f> in <module>()
----> 1 np.append(a, ArcFlow(1, 5))
/home/gidden/.local/lib/python2.7/site-packages/numpy/lib/function_base.pyc in append(arr, values, axis)
   3541         values = ravel(values)
   3542         axis = arr.ndim-1
-> 3543     return concatenate((arr, values), axis=axis)
ValueError: No cast function available.
In [6]: f = ArcFlow(1, 5)
In [7]: f.id
Out[7]: 1
In [8]: np.append(a, f)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-8-b0ed85d070b0> in <module>()
----> 1 np.append(a, f)
/home/gidden/.local/lib/python2.7/site-packages/numpy/lib/function_base.pyc in append(arr, values, axis)
   3541         values = ravel(values)
   3542         axis = arr.ndim-1
-> 3543     return concatenate((arr, values), axis=axis)
ValueError: No cast function available.

--

Matthew Gidden
Ph.D. Candidate, Nuclear Engineering

The University of Wisconsin -- Madison
Ph. 225.892.3192

[Matthew Gidden]

And there's also issues with array comparisons

In [9]: a
Out[9]: array([], dtype='xd_arcflow')
In [10]: b = np.array([], dtype=xd_arcflow)
In [11]: b
Out[11]: array([], dtype='xd_arcflow')
In [12]: np.array_equal(a, b)
Out[12]: False

[Anthony Scopatz]

Hi Matt, 

This is actually a limitation of numpy.  They do not provide an API in C for casting to-from Python types and an associated dtype.  Eventually, I would be happy to champion this change but I don't have the time currently. Thus, for the moment you should understand that a dtype is not exchangeable for the python type.  Numpy fools people into believing this is the case for primitive types.  But in truth even in simple cases we can see this behavior. For example:

np.array([1, 2, 3])[0]

Here the value will be 1, but the type will be np.int32 or np.int64.  It will not be the Python type "int".  You are experience the same issue here. ArcFlow is not xd_arcflow and they are not castable.  They must be explicitly converted in a new array or scalar.  The following is more what you are looking for.

Be Well

Anthony

import numpy as np

import cyclopts

print(cyclopts.__file__)

from cyclopts.execute import ArcFlow

from cyclopts.dtypes import xd_arcflow

# convert to xd_dtype

# x is a scalar array here

a = np.array([], dtype=xd_arcflow)

x = np.array(ArcFlow(1, 5), dtype=xd_arcflow)

b = np.append(a, x)

print(b)

c = np.empty(10, dtype=xd_arcflow) 

c[0] = ArcFlow(1, 5)

print(c)

# convert to ArcFlow

y = ArcFlow(x)

print(type(x))

print(type(y))

--
You received this message because you are subscribed to the Google Groups "xdress" group.
To view this discussion on the web visit https://groups.google.com/d/msgid/xdress/CAHRdPFyHDOQx1oYkRSeGrA30Uj69YA6FYqiAR%3Dg3HyAF-H_HSA%40mail.gmail.com.

[Anthony Scopatz]

I admit that this is unexpected.

--

You received this message because you are subscribed to the Google Groups "xdress" group.

To view this discussion on the web visit https://groups.google.com/d/msgid/xdress/CAHRdPFw228zRxNMKDxJQu3uMe_%3DaN1mjkBhHVp7B-P9xv0eqiQ%40mail.gmail.com.

[Matthew Gidden]

Thanks again for your help, scopz. 

On Fri, Apr 4, 2014 at 6:46 PM, Anthony Scopatz <sco...@gmail.com> wrote:

Hi Matt, 

This is actually a limitation of numpy.  They do not provide an API in C for casting to-from Python types and an associated dtype.  Eventually, I would be happy to champion this change but I don't have the time currently.

Soon^TM! 

Thus, for the moment you should understand that a dtype is not exchangeable for the python type.  Numpy fools people into believing this is the case for primitive types.  But in truth even in simple cases we can see this behavior. For example:

np.array([1, 2, 3])[0]

Here the value will be 1, but the type will be np.int32 or np.int64.  It will not be the Python type "int".  You are experience the same issue here. ArcFlow is not xd_arcflow and they are not castable.  They must be explicitly converted in a new array or scalar.  The following is more what you are looking for.

Be Well

Anthony

import numpy as np

import cyclopts

print(cyclopts.__file__)

from cyclopts.execute import ArcFlow

from cyclopts.dtypes import xd_arcflow

# convert to xd_dtype

# x is a scalar array here

a = np.array([], dtype=xd_arcflow)

x = np.array(ArcFlow(1, 5), dtype=xd_arcflow)

b = np.append(a, x)

print(b)

c = np.empty(10, dtype=xd_arcflow) 

c[0] = ArcFlow(1, 5)

print(c)

# convert to ArcFlow

y = ArcFlow(x)

print(type(x))

print(type(y))

Thank you very much for the example. I should have added that these examples were part of an investigation into a larger issue -- returning stl containers of xdressed types. I've updated `cyclopts.execute.test()`  to show such an example (it returns `std::vector<ArcFlow>`, printing out the values it `push_back`s each time. 

In short, the line `y = ArcFlow(x)` works great for me if `len(x) == 1`. However, if `len(x) > 1`, then (of course), the cast "fails", i.e., it returns the cast of `x[0]`. Furthermore, casting the individual elements results in a seg fault:

In [1]: from cyclopts.execute import ArcFlow, test
In [2]: from cyclopts.dtypes import xd_arcflow
In [3]: obs = test()
testing cyclopts!
Coin0506I Presolve 1 (-2) rows, 2 (0) columns and 2 (-2) elements
Clp0006I 0  Obj 0 Primal inf 0.9999999 (1)
Clp0006I 1  Obj 5
Clp0000I Optimal - objective value 5
Coin0511I After Postsolve, objective 5, infeasibilities - dual 0 (0), primal 0 (0)
Clp0032I Optimal objective 5 - 1 iterations time 0.002, Presolve 0.00
 Warning: Use of OsiCbc is deprecated.
 To enjoy the full performance of Cbc, use the CbcSolver interface.
Clp0000I Optimal - objective value 5
Cbc0045I Heuristic rounding took 0 seconds (no good)
Clp0000I Optimal - objective value 5
Cbc0004I Integer solution of 5 found after 0 iterations and 0 nodes (0.00 seconds)
Cbc0001I Search completed - best objective 5, took 0 iterations and 0 nodes (0.00 seconds)
Cbc0035I Maximum depth 0, 0 variables fixed on reduced cost
Clp0000I Optimal - objective value 5
Clp0000I Optimal - objective value 5
Adding arc
        i: 0
     flow: 5
In [4]: print len(obs)
2
In [5]: obs
Out[5]: 
array([<cyclopts.execute.ArcFlow object at 0x7ffd31136fd0>,
       <cyclopts.execute.ArcFlow object at 0x7ffd31136fd0>], dtype='xd_arcflow')
In [6]: obs[0]
Out[6]: <cyclopts.execute.ArcFlow object at 0x35970f0>
In [7]: obs[1]
Out[7]: <cyclopts.execute.ArcFlow object at 0x3597130>
In [8]: obs[0].id
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-8-9be77bb1847a> in <module>()
----> 1 obs[0].id
AttributeError: 'xd_arcflow' object has no attribute 'id'
In [9]: cast0 = ArcFlow(obs[0])
Segmentation fault (core dumped)

Note that as long as I use slices, I can access the underlying elements, e.g., 

In [1]: from cyclopts.execute import ArcFlow, test
In [2]: from cyclopts.dtypes import xd_arcflow
In [3]: obs = test()
testing cyclopts!
Coin0506I Presolve 1 (-2) rows, 2 (0) columns and 2 (-2) elements
Clp0006I 0  Obj 0 Primal inf 0.9999999 (1)
Clp0006I 1  Obj 5
Clp0000I Optimal - objective value 5
Coin0511I After Postsolve, objective 5, infeasibilities - dual 0 (0), primal 0 (0)
Clp0032I Optimal objective 5 - 1 iterations time 0.002, Presolve 0.00
 Warning: Use of OsiCbc is deprecated.
 To enjoy the full performance of Cbc, use the CbcSolver interface.
Clp0000I Optimal - objective value 5
Cbc0045I Heuristic rounding took 0 seconds (no good)
Clp0000I Optimal - objective value 5
Cbc0004I Integer solution of 5 found after 0 iterations and 0 nodes (0.00 seconds)
Cbc0001I Search completed - best objective 5, took 0 iterations and 0 nodes (0.00 seconds)
Cbc0035I Maximum depth 0, 0 variables fixed on reduced cost
Clp0000I Optimal - objective value 5
Clp0000I Optimal - objective value 5
Adding arc
        i: 0
     flow: 5
In [4]: cast = ArcFlow(obs)
In [5]: print cast.id
0
In [6]: print cast.flow
5.0
In [7]: cast = ArcFlow(obs[1:])
In [8]: cast.id
Out[8]: 99
In [9]: cast.flow
Out[9]: 4.5

Is this the expected behavior? Thanks again for your effort explaining these issues to me.

[Anthony Scopatz]

That is weird behaviour.  Segfaults are always bad.  The slicing vs indexing thing I don't understand.  This bears further investigation.

[Matthew Gidden]

I made an issue, FYI.