VGG-16 memory requirements

[Pawel Kozela]

Hi guys,

I'm struggling to understand the memory requirements for a standard VGG-16 implementation, using tensorflow backend. For instance, based on this:

http://cs231n.github.io/convolutional-networks/#case

I would expect the parameters to take around 0.5GB only. However, when I try to use a pre-trained model on my GPU (GTX 960, 4GB) I'm getting a memory allocation error for the first FC layer (I've put the batch_size to 2 so my understanding is it's almost weights only). Could somebody give a few pointers on this ?

Here is the code I use:

model = Sequential()
model.add(ZeroPadding2D((1, 1), input_shape=(color_type,
                                             img_rows, img_cols)))
model.add(Convolution2D(64, 3, 3, activation='relu'))
model.add(ZeroPadding2D((1, 1)))
model.add(Convolution2D(64, 3, 3, activation='relu'))
model.add(MaxPooling2D((2, 2), strides=(2, 2)))

model.add(ZeroPadding2D((1, 1)))
model.add(Convolution2D(128, 3, 3, activation='relu'))
model.add(ZeroPadding2D((1, 1)))
model.add(Convolution2D(128, 3, 3, activation='relu'))
model.add(MaxPooling2D((2, 2), strides=(2, 2)))

model.add(ZeroPadding2D((1, 1)))
model.add(Convolution2D(256, 3, 3, activation='relu'))
model.add(ZeroPadding2D((1, 1)))
model.add(Convolution2D(256, 3, 3, activation='relu'))
model.add(ZeroPadding2D((1, 1)))
model.add(Convolution2D(256, 3, 3, activation='relu'))
model.add(MaxPooling2D((2, 2), strides=(2, 2)))

model.add(ZeroPadding2D((1, 1)))
model.add(Convolution2D(512, 3, 3, activation='relu'))
model.add(ZeroPadding2D((1, 1)))
model.add(Convolution2D(512, 3, 3, activation='relu'))
model.add(ZeroPadding2D((1, 1)))
model.add(Convolution2D(512, 3, 3, activation='relu'))
model.add(MaxPooling2D((2, 2), strides=(2, 2)))

model.add(ZeroPadding2D((1, 1)))
model.add(Convolution2D(512, 3, 3, activation='relu'))
model.add(ZeroPadding2D((1, 1)))
model.add(Convolution2D(512, 3, 3, activation='relu'))
model.add(ZeroPadding2D((1, 1)))
model.add(Convolution2D(512, 3, 3, activation='relu'))
model.add(MaxPooling2D((2, 2), strides=(2, 2)))

model.add(Flatten())
model.add(Dense(4096, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(4096, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(1000, activation='softmax'))

model.load_weights(os.path.join(MODELS_DIR, 'vgg16_weights.h5'))

# Code above loads pre-trained data and
model.layers.pop()

model.add(Dense(10, activation='softmax'))

sgd = SGD(lr=1e-3, decay=1e-6, momentum=0.9, nesterov=True)
model.compile(optimizer=sgd, loss='categorical_crossentropy')
return model

[marku...@gmail.com]

Hi Pawel & others,

It might be a while a go but I have similar issues trying to get a VGG-like model to run. Even a reduced 5-layer version is already pushing it using Tensorflow as backend (Theano doesnt play too nice with Windows which I'm stuck at).

Did you ever find a solution to this?

Mark

Op dinsdag 31 mei 2016 17:37:11 UTC+12 schreef Pawel Kozela:

[Pawel Kozela]

Hi Mark,

unfortunately not. I ended up loading only weights for conv layers and this worked OK for my problem. 

My understanding is that the cs321n only gives a lower bound - what memory is needed just to store the weights and images.

Take a look here: 

http://timdettmers.com/2014/08/14/which-gpu-for-deep-learning/

He goes into some detail about the implementations and why it takes more memory than the lower bound.

I would love to fully understand it too. 

Regards,
Pawel

--
You received this message because you are subscribed to a topic in the Google Groups "Keras-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/keras-users/2LafaBB7bIU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to keras-users+unsubscribe@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/keras-users/0c412ec8-215a-4abb-b3c0-65d2b262abf5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.