Inception model with custom input_tensor

[Annika Hänsch]

Hi,

I want to finetune the InceptionV3 model on my own data. However I get errors when trying to use the input_tensor option.

This is my minimal example code:

from keras.applications.inception_v3 import InceptionV3
from keras.layers import Input
import keras.backend as K
import numpy as np

K.set_image_data_format('channels_first')

input_tensor = Input(shape=(3,224, 224))  # this assumes K.image_data_format() == 'channels_first'
model = InceptionV3(input_tensor=input_tensor, weights='imagenet', include_top=True)


def get_dummy_data(batch_size,input_shape,n_classes):
   
    imgs = np.zeros((batch_size,input_shape[0],input_shape[1],input_shape[2]))
    labels = np.zeros((batch_size,n_classes))
   
    return imgs, labels
   


train_imgs, train_labels = get_dummy_data(10,(3,224,224),1000)


# compile the model (should be done *after* setting layers to non-trainable)
model.compile(optimizer='rmsprop', loss='categorical_crossentropy')

# train the model on the new data for a few epochs
model.fit(x=train_imgs,y=train_labels)

This is the error message:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-2-6aaaccc85c4b> in <module>()
     26 
     27 # train the model on the new data for a few epochs
---> 28 model.fit(x=train_imgs,y=train_labels)

/usr/local/lib/python2.7/dist-packages/Keras-2.0.1-py2.7.egg/keras/engine/training.pyc in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, **kwargs)
   1403             class_weight=class_weight,
   1404             check_batch_axis=False,
-> 1405             batch_size=batch_size)
   1406         # prepare validation data
   1407         if validation_data:

/usr/local/lib/python2.7/dist-packages/Keras-2.0.1-py2.7.egg/keras/engine/training.pyc in _standardize_user_data(self, x, y, sample_weight, class_weight, check_batch_axis, batch_size)
   1293                                     self._feed_input_shapes,
   1294                                     check_batch_axis=False,
-> 1295                                     exception_prefix='model input')
   1296         y = _standardize_input_data(y, self._feed_output_names,
   1297                                     output_shapes,

/usr/local/lib/python2.7/dist-packages/Keras-2.0.1-py2.7.egg/keras/engine/training.pyc in _standardize_input_data(data, names, shapes, check_batch_axis, exception_prefix)
     98             raise ValueError('The model expects ' + str(len(names)) +
     99                              ' input arrays, but only received one array. '
--> 100                              'Found: array with shape ' + str(data.shape))
    101         arrays = [data]
    102 

ValueError: The model expects 0 input arrays, but only received one array. Found: array with shape (10, 3, 224, 224)

Does anyone have an idea what could be the problem?

Thanks!

[Annika H.]

Forgot to mention that I'm using keras 2.0.2 with theano backend...

[Daπid]

Thank you for providing a nice, short example.

On 28 March 2017 at 09:33, Annika H. <annika....@gmail.com> wrote:

I'm using keras 2.0.2 with theano backend

Your paths say it is 2.0.1, but I am getting the same error with 2.0.2. Here is what I have been able to dig:

* Keras saves the input nodes under model._feed_inputs. They are take from model.input_layers, but only if the attribute is_placeholder is True. [1]
* An Input layer is set as is_placeholder iff no tensor input is given [2].

* Inception_v3 wraps the input_tensor in an Input layer, hence it is not a placeholder. [3]

* It then delegates on get_source_inputs [4] to find them, but then the ouput is a TensorVariable, not an Input layer. Is this relevant?

[1] https://github.com/fchollet/keras/blob/master/keras/engine/topology.py#L1566
[2] https://github.com/fchollet/keras/blob/master/keras/engine/topology.py#L1294-L1302
[3] https://github.com/fchollet/keras/blob/master/keras/applications/inception_v3.py#L160
[4] https://github.com/fchollet/keras/blob/master/keras/engine/topology.py#L2569

I also noticed that the constructor for inception doesn't accept giving just the input_shape while including the top layers, but since the top layers are behind a global pooling, I don't see why it shouldn't. Maybe your best bet is to take away that part [5].

If this is a simplified example and you are planning to do some operations on the network before, you can always call the hacked version of inception_v3 just as if it were a layer on your master model.

[5] https://github.com/fchollet/keras/blob/master/keras/applications/inception_v3.py#L150-L155

Hope this helps, at least to some extent.

/David.

[Annika H.]

Hi David,

thank you for your answer!
I'm not sure whether I fully understand this:

you can always call the hacked version of inception_v3 just as if it were a layer on your master model.

If say 'model' is the Inception model in the code above and I also have a tensor x that is the output tensor of some base_model, how do I call the Inception model on this tensor?

I also digged through the code and found that if I give the input tensor a name, e.g.

input_tensor = Input(shape=input_shape,name='test_input')

and then call the fit method with a dictionary using that name

model.fit(x={'test_input': train_imgs},y=train_labels)

I don't get the error above anymore. However now I get the following:

---------------------------------------------------------------------------
MissingInputError                         Traceback (most recent call last)
<ipython-input-2-ffe58377976c> in <module>()
     49 # train the model on the new data for a few epochs
     50 if custom_input:
---> 51     model.fit(x={'test_input': train_imgs},y=train_labels)
     52     #model.fit(x=train_imgs,y=train_labels)
     53 else:

/usr/local/lib/python2.7/dist-packages/Keras-2.0.1-py2.7.egg/keras/engine/training.pyc in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, **kwargs)

   1455         else:
   1456             ins = x + y + sample_weights
-> 1457         self._make_train_function()
   1458         f = self.train_function
   1459

/usr/local/lib/python2.7/dist-packages/Keras-2.0.1-py2.7.egg/keras/engine/training.pyc in _make_train_function(self)
   1005                                              [self.total_loss] + self.metrics_tensors,
   1006                                              updates=updates,
-> 1007                                              **self._function_kwargs)
   1008
   1009     def _make_test_function(self):

/usr/local/lib/python2.7/dist-packages/Keras-2.0.1-py2.7.egg/keras/backend/theano_backend.pyc in function(inputs, outputs, updates, **kwargs)
   1102                 msg = 'Invalid argument "%s" passed to K.function' % key
   1103                 raise ValueError(msg)
-> 1104     return Function(inputs, outputs, updates=updates, **kwargs)
   1105
   1106

/usr/local/lib/python2.7/dist-packages/Keras-2.0.1-py2.7.egg/keras/backend/theano_backend.pyc in __init__(self, inputs, outputs, updates, **kwargs)
   1088                                         allow_input_downcast=True,
   1089                                         on_unused_input='ignore',
-> 1090                                         **kwargs)
   1091
   1092     def __call__(self, inputs):

/home/annika/.local/lib/python2.7/site-packages/theano/compile/function.pyc in function(inputs, outputs, mode, updates, givens, no_default_updates, accept_inplace, name, rebuild_strict, allow_input_downcast, profile, on_unused_input)
    324                    on_unused_input=on_unused_input,
    325                    profile=profile,
--> 326                    output_keys=output_keys)
    327     # We need to add the flag check_aliased inputs if we have any mutable or
    328     # borrowed used defined inputs

/home/annika/.local/lib/python2.7/site-packages/theano/compile/pfunc.pyc in pfunc(params, outputs, mode, updates, givens, no_default_updates, accept_inplace, name, rebuild_strict, allow_input_downcast, profile, on_unused_input, output_keys)
    484                          accept_inplace=accept_inplace, name=name,
    485                          profile=profile, on_unused_input=on_unused_input,
--> 486                          output_keys=output_keys)
    487
    488

/home/annika/.local/lib/python2.7/site-packages/theano/compile/function_module.pyc in orig_function(inputs, outputs, mode, accept_inplace, name, profile, on_unused_input, output_keys)
   1792                    profile=profile,
   1793                    on_unused_input=on_unused_input,
-> 1794                    output_keys=output_keys).create(
   1795             defaults)
   1796

/home/annika/.local/lib/python2.7/site-packages/theano/compile/function_module.pyc in __init__(self, inputs, outputs, mode, accept_inplace, function_builder, profile, on_unused_input, fgraph, output_keys)
   1444             # OUTPUT VARIABLES)
   1445             fgraph, additional_outputs = std_fgraph(inputs, outputs,
-> 1446                                                     accept_inplace)
   1447             fgraph.profile = profile
   1448         else:

/home/annika/.local/lib/python2.7/site-packages/theano/compile/function_module.pyc in std_fgraph(input_specs, output_specs, accept_inplace)
    175
    176     fgraph = gof.fg.FunctionGraph(orig_inputs, orig_outputs,
--> 177                                   update_mapping=update_mapping)
    178
    179     for node in fgraph.apply_nodes:

/home/annika/.local/lib/python2.7/site-packages/theano/gof/fg.pyc in __init__(self, inputs, outputs, features, clone, update_mapping)
    178
    179         for output in outputs:
--> 180             self.__import_r__(output, reason="init")
    181         for i, output in enumerate(outputs):
    182             output.clients.append(('output', i))

/home/annika/.local/lib/python2.7/site-packages/theano/gof/fg.pyc in __import_r__(self, variable, reason)
    349         # Imports the owners of the variables
    350         if variable.owner and variable.owner not in self.apply_nodes:
--> 351                 self.__import__(variable.owner, reason=reason)
    352         elif (variable.owner is None and
    353                 not isinstance(variable, graph.Constant) and

/home/annika/.local/lib/python2.7/site-packages/theano/gof/fg.pyc in __import__(self, apply_node, check, reason)
    395                                      % (node.inputs.index(r), str(node)))
    396                         error_msg += get_variable_trace_string(r)
--> 397                         raise MissingInputError(error_msg, variable=r)
    398
    399         for node in new_nodes:

MissingInputError: Input 0 of the graph (indices start from 0), used to compute AbstractConv2d{convdim=2, border_mode='valid', subsample=(2, 2), filter_flip=True, imshp=(None, 3, 299, 299), kshp=(32, 3, 3, 3), filter_dilation=(1, 1)}(/test_input, InplaceDimShuffle{3,2,0,1}.0), was not provided and not given a value. Use the Theano flag exception_verbosity='high', for more information on this error.
Backtrace when that variable is created:

  File "/usr/local/lib/python2.7/dist-packages/ipykernel/zmqshell.py", line 501, in run_cell
    return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/IPython/core/interactiveshell.py", line 2717, in run_cell
    interactivity=interactivity, compiler=compiler, result=result)
  File "/usr/local/lib/python2.7/dist-packages/IPython/core/interactiveshell.py", line 2821, in run_ast_nodes
    if self.run_code(code, result):
  File "/usr/local/lib/python2.7/dist-packages/IPython/core/interactiveshell.py", line 2881, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-ffe58377976c>", line 22, in <module>
    input_tensor = Input(shape=input_shape,name='test_input')
  File "/usr/local/lib/python2.7/dist-packages/Keras-2.0.1-py2.7.egg/keras/engine/topology.py", line 1388, in Input
    input_tensor=tensor)
  File "/usr/local/lib/python2.7/dist-packages/Keras-2.0.1-py2.7.egg/keras/engine/topology.py", line 1299, in __init__
    name=self.name)
  File "/usr/local/lib/python2.7/dist-packages/Keras-2.0.1-py2.7.egg/keras/backend/theano_backend.py", line 184, in placeholder
    x = T.TensorType(dtype, broadcast)(name)


Backtrace when the variable is created:
  File "/usr/local/lib/python2.7/dist-packages/ipykernel/zmqshell.py", line 501, in run_cell
    return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/IPython/core/interactiveshell.py", line 2717, in run_cell
    interactivity=interactivity, compiler=compiler, result=result)
  File "/usr/local/lib/python2.7/dist-packages/IPython/core/interactiveshell.py", line 2821, in run_ast_nodes
    if self.run_code(code, result):
  File "/usr/local/lib/python2.7/dist-packages/IPython/core/interactiveshell.py", line 2881, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-ffe58377976c>", line 22, in <module>
    input_tensor = Input(shape=input_shape,name='test_input')
  File "/usr/local/lib/python2.7/dist-packages/Keras-2.0.1-py2.7.egg/keras/engine/topology.py", line 1388, in Input
    input_tensor=tensor)
  File "/usr/local/lib/python2.7/dist-packages/Keras-2.0.1-py2.7.egg/keras/engine/topology.py", line 1299, in __init__
    name=self.name)
  File "/usr/local/lib/python2.7/dist-packages/Keras-2.0.1-py2.7.egg/keras/backend/theano_backend.py", line 184, in placeholder
    x = T.TensorType(dtype, broadcast)(name)

[Daπid]

On 28 March 2017 at 16:52, Annika H. <annika....@gmail.com> wrote:

I'm not sure whether I fully understand this:

you can always call the hacked version of inception_v3 just as if it were a layer on your master model.

If say 'model' is the Inception model in the code above and I also have a tensor x that is the output tensor of some base_model, how do I call the Inception model on this tensor?

This is what I meant:

https://keras.io/getting-started/functional-api-guide/#all-models-are-callable-just-like-layers

So, this would be (untested):

z = Input(shape=(1000,))

image = base_model(z)  # This model generates an image

class_ = inception(image)

final_model = Model(inputs=z, outputs=class_)

/David.

[Annika H.]

Hi David,

thanks! I didn't know that this is possible.

[Annika H.]

I did some more investigations on this. When the model is called without a custom input_tensor, then the Input is a placeholder and everything works fine. As you pointed out, a custom input_tensor is wrapped in an Input layer and is no longer a placeholder, here the problems start...

From what I investigated, part [5] that you pointed out didn't do any harm, it just checks if the input_shapes are correct for the model.

Even if it's ugly I think for now I'll use a hacked version of the Architecture and hack my custom input preprocessing directly into the code.

[gagarw...@gmail.com]

Hi Annika

I am running into the same issue. how did you resolve this?

here is my code:

input_img = Input(shape=(150, 150, 3)) 

pre_trained_model = InceptionV3(input_tensor=input_img, include_top=False, weights='imagenet')

for layer in pre_trained_model.layers:

  layer.trainable = False

last_layer = pre_trained_model.get_layer('mixed7')

print (last_layer.output_shape)

last_output = last_layer.output

x = Flatten()(last_output)

x = Dense(1024, activation='relu')(x)

x = Dropout(0.2)(x)

x = Dense(1, activation='sigmoid')(x)

model = Model(input=input_img, output=x)

When i try to train I get this error:

ValueError: The model expects 0 input arrays, but only received one array. Found: array with shape (32, 150, 150, 3)

[Annika H.]

Hi,

I replaced this line

img_input = Input(tensor=input_tensor, shape=input_shape)Code hier eingeben...

in https://github.com/fchollet/keras/blob/master/keras/applications/inception_v3.py#L160 by

img_input = input_tensor

In your case this should work. It doesn't work if the input_tensor is the result of more than 1 layer, because then the number of layers in the model and the number of weights that we want to load into the model mismatch. In that case I think you have to do a combination of this hack and what David suggested above, using a base_model for preprocessing.

[Daπid]

By the way, I encountered some similar errors (missing inputs) applying TimeDistributed over a model embedded in a bigger one, that were fixed by upgrading to Keras master. Not a solution to Annika's problem, though.

--
You received this message because you are subscribed to the Google Groups "Keras-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to keras-users+unsubscribe@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/keras-users/fbdb785f-698a-4975-9627-a6fffec1826a%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Garima Agarwal]

I was able to get past it by downgrading to Keras  1.0.2. :)