How to feed multiple input arrays with different dimensions into a deep neural network model

[msabda...@gmail.com]

Good day All,

I'm building a deep recurent network-like model for signal processing using chainer. The training data as well as the testset data are prepared using "TupleDataset" method as: train_data = TupleDataset(x, indices, y, x_zf, HtH, Hty), where the dimensions of the inputs are respectively: (100000, 10), (100000, 10), (100000, 4), (100000, 10), (100000, 10, 10) and (100000, 10). Here, the number of examples is 100000 from which the minibatches are obtained. Waht I'm not clear about is that why must other dimensions be the same? The following is the error it returned:

Exception in main training loop: all the input arrays must have same number of dimensions
Traceback (most recent call last):
  File "/Users/mac/miniconda3/lib/python3.6/site-packages/chainer/training/trainer.py", line 316, in run
    update()
  File "/Users/mac/miniconda3/lib/python3.6/site-packages/chainer/training/updaters/standard_updater.py", line 175, in update
    self.update_core()
  File "/Users/mac/miniconda3/lib/python3.6/site-packages/chainer/training/updaters/standard_updater.py", line 181, in update_core
    in_arrays = convert._call_converter(self.converter, batch, self.device)
  File "/Users/mac/miniconda3/lib/python3.6/site-packages/chainer/dataset/convert.py", line 73, in _call_converter
    return converter(batch, device)
  File "/Users/mac/miniconda3/lib/python3.6/site-packages/chainer/dataset/convert.py", line 58, in wrap_call
    return func(*args, **kwargs)
  File "/Users/mac/miniconda3/lib/python3.6/site-packages/chainer/dataset/convert.py", line 223, in concat_examples
    [example[i] for example in batch], padding[i])))
  File "/Users/mac/miniconda3/lib/python3.6/site-packages/chainer/dataset/convert.py", line 254, in _concat_arrays
    [array[None] for array in arrays])
Will finalize trainer extensions and updater before reraising the exception.
[JTraceback (most recent call last):
  File "/Users/mac/Documents/idp_detnet/examples/run_mlp.py", line 14, in <module>
    mlp.run(args)
  File "/Users/mac/Documents/idp_detnet/examples/mlp.py", line 38, in run
    util.load_or_train_model(model, train, test, args)
  File "/Users/mac/Documents/idp_detnet/examples/util.py", line 209, in load_or_train_model
    train_model(model, train, test, args)
  File "/Users/mac/Documents/idp_detnet/examples/util.py", line 183, in train_model
    trainer.run()
  File "/Users/mac/miniconda3/lib/python3.6/site-packages/chainer/training/trainer.py", line 349, in run
    six.reraise(*exc_info)
  File "/Users/mac/miniconda3/lib/python3.6/site-packages/six.py", line 693, in reraise
    raise value
  File "/Users/mac/miniconda3/lib/python3.6/site-packages/chainer/training/trainer.py", line 316, in run
    update()
  File "/Users/mac/miniconda3/lib/python3.6/site-packages/chainer/training/updaters/standard_updater.py", line 175, in update
    self.update_core()
  File "/Users/mac/miniconda3/lib/python3.6/site-packages/chainer/training/updaters/standard_updater.py", line 181, in update_core
    in_arrays = convert._call_converter(self.converter, batch, self.device)
  File "/Users/mac/miniconda3/lib/python3.6/site-packages/chainer/dataset/convert.py", line 73, in _call_converter
    return converter(batch, device)
  File "/Users/mac/miniconda3/lib/python3.6/site-packages/chainer/dataset/convert.py", line 58, in wrap_call
    return func(*args, **kwargs)
  File "/Users/mac/miniconda3/lib/python3.6/site-packages/chainer/dataset/convert.py", line 223, in concat_examples
    [example[i] for example in batch], padding[i])))
  File "/Users/mac/miniconda3/lib/python3.6/site-packages/chainer/dataset/convert.py", line 254, in _concat_arrays
    [array[None] for array in arrays])
ValueError: all the input arrays must have same number of dimen

Please I need your help, because I have been on this issue for over three months and searched everywhere for possible solutions, but without sucess.

Thank you,

Abdullahi

[Kenichi Maehashi]

The default `converter` of StandardUpdater is `concat_example`, which concatenates inputs into NumPy/CuPy array, which requires the same dimensions for all inputs.

https://docs.chainer.org/en/stable/reference/generated/chainer.training.updaters.StandardUpdater.html

Please refer to the examples using custom converter, e.g.:

https://github.com/chainer/chainer/blob/v6.4.0/examples/seq2seq/seq2seq.py#L350-L351