Hi Alexy,
Sounds like a very interesting project!
I think you have two decent options if you want mutable arrays of arbitrary objects:
1) core.matrix should support Java Object[] arrays directly. Probably the best option for 1D arrays, you can also nest them to get higher dimensional arrays.
2) There is also the core.matrix NDArray which supports arbitrary n-dimensional object arrays. Also mutable and pretty efficient, will be better than Object[] arrays if you do a lot of stuff like slicing, transposes etc. that benefit from the NDArray representation.
Note however that I haven't yet spent that much time testing and optimising these use cases.... I suggest you try it out, and file a GitHub issue (or PR) if you run into any problems.
Mike.