Regression model barely use the GPU when fitting
52 views
Skip to first unread message
Gabriele Proietti Mattia
Apr 12, 2020, 5:56:23 AM4/12/20
to Keras-users
I am training a DNN with keras for a regression model, I have a n inputs and n outputs and the model is the following
# init the model
self._model = Sequential()
self._model.add(Dense(self._n, input_dim=self._n, kernel_initializer='normal', activation='relu'))
self._model.add(Dense(self._n + 10, activation='relu'))
self._model.add(Dense(self._n + 10, activation='relu'))
self._model.add(Dense(self._n, activation='linear'))
# self._model.summary()
self._model.compile(loss='mse', optimizer='adam', metrics=['mse', 'mae'])I read the data from a simple .txt file that has a fixed lines size, so I can do seek for quickly retrieving the line. Therefore, with a custom data generator I create batches of data by shuffling the line indexes at every epoch. The problem is that when I start training the network, whatever is the batch size from 16 to 2048, I get intensive tasks on CPU and for 1 epoch of about 500'000 lines it estimates about 3 hours (the txt file size is about 60mb), is that normal? I mean, I have 16gb of RAM, could it be convenient to load the entire file in memory? And why I get that the most of the operation are CPU bound? Is there something wrong in the way I create the dataset?
The placement logging is 98% with the following lines
2020-04-12 11:41:16.829552: I tensorflow/core/common_runtime/eager/execute.cc:573] Executing op MapDataset in device /job:localhost/replica:0/task:0/device:CPU:0
2020-04-12 11:41:16.829917: I tensorflow/core/common_runtime/eager/execute.cc:573] Executing op PrefetchDataset in device /job:localhost/replica:0/task:0/device:CPU:0
2020-04-12 11:41:16.837311: I tensorflow/core/common_runtime/eager/execute.cc:573] Executing op FlatMapDataset in device /job:localhost/replica:0/task:0/device:CPU:0
2020-04-12 11:41:16.837884: I tensorflow/core/common_runtime/eager/execute.cc:573] Executing op TensorDataset in device /job:localhost/replica:0/task:0/device:CPU:0
2020-04-12 11:41:16.838019: I tensorflow/core/common_runtime/eager/execute.cc:573] Executing op RepeatDataset in device /job:localhost/replica:0/task:0/device:CPU:0
2020-04-12 11:41:16.838178: I tensorflow/core/common_runtime/eager/execute.cc:573] Executing op ZipDataset in device /job:localhost/replica:0/task:0/device:CPU:0
2020-04-12 11:41:16.841638: I tensorflow/core/common_runtime/eager/execute.cc:573] Executing op ParallelMapDataset in device /job:localhost/replica:0/task:0/device:CPU:0
2020-04-12 11:41:16.842229: I tensorflow/core/common_runtime/eager/execute.cc:573] Executing op ModelDataset in device /job:localhost/replica:0/task:0/device:CPU:0
2020-04-12 11:41:16.849076: I tensorflow/core/common_runtime/eager/execute.cc:573] Executing op RangeDataset in device /job:localhost/replica:0/task:0/device:CPU:0
2020-04-12 11:41:16.849240: I tensorflow/core/common_runtime/eager/execute.cc:573] Executing op RepeatDataset in device /job:localhost/replica:0/task:0/device:CPU:0The remaining 2% of these ones
input_iterator: (_Arg): /job:localhost/replica:0/task:0/device:CPU:0
input_iterator_1: (_Arg): /job:localhost/replica:0/task:0/device:CPU:0
sequential_1_dense_4_matmul_readvariableop_resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0
sequential_1_dense_4_biasadd_readvariableop_resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0
sequential_1_dense_5_matmul_readvariableop_resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0
sequential_1_dense_5_biasadd_readvariableop_resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0
sequential_1_dense_6_matmul_readvariableop_resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0
sequential_1_dense_6_biasadd_readvariableop_resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0
sequential_1_dense_7_matmul_readvariableop_resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0
sequential_1_dense_7_biasadd_readvariableop_resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0
IteratorGetNext: (IteratorGetNext): /job:localhost/replica:0/task:0/device:CPU:0
Cast: (Cast): /job:localhost/replica:0/task:0/device:GPU:0
sequential_1/dense_4/MatMul/ReadVariableOp: (ReadVariableOp): /job:localhost/replica:0/task:0/device:GPU:0
sequential_1/dense_4/MatMul: (MatMul): /job:localhost/replica:0/task:0/device:GPU:0
sequential_1/dense_4/BiasAdd/ReadVariableOp: (ReadVariableOp): /job:localhost/replica:0/task:0/device:GPU:0
sequential_1/dense_4/BiasAdd: (BiasAdd): /job:localhost/replica:0/task:0/device:GPU:0
sequential_1/dense_4/Relu: (Relu): /job:localhost/replica:0/task:0/device:GPU:0
sequential_1/dense_5/MatMul/ReadVariableOp: (ReadVariableOp): /job:localhost/replica:0/task:0/device:GPU:0
sequential_1/dense_5/MatMul: (MatMul): /job:localhost/replica:0/task:0/device:GPU:0
sequential_1/dense_5/BiasAdd/ReadVariableOp: (ReadVariableOp): /job:localhost/replica:0/task:0/device:GPU:0
sequential_1/dense_5/BiasAdd: (BiasAdd): /job:localhost/replica:0/task:0/device:GPU:0
sequential_1/dense_5/Relu: (Relu): /job:localhost/replica:0/task:0/device:GPU:0
sequential_1/dense_6/MatMul/ReadVariableOp: (ReadVariableOp): /job:localhost/replica:0/task:0/device:GPU:0
sequential_1/dense_6/MatMul: (MatMul): /job:localhost/replica:0/task:0/device:GPU:0
sequential_1/dense_6/BiasAdd/ReadVariableOp: (ReadVariableOp): /job:localhost/replica:0/task:0/device:GPU:0
sequential_1/dense_6/BiasAdd: (BiasAdd): /job:localhost/replica:0/task:0/device:GPU:0
sequential_1/dense_6/Relu: (Relu): /job:localhost/replica:0/task:0/device:GPU:0
sequential_1/dense_7/MatMul/ReadVariableOp: (ReadVariableOp): /job:localhost/replica:0/task:0/device:GPU:0
sequential_1/dense_7/MatMul: (MatMul): /job:localhost/replica:0/task:0/device:GPU:0
sequential_1/dense_7/BiasAdd/ReadVariableOp: (ReadVariableOp): /job:localhost/replica:0/task:0/device:GPU:0
sequential_1/dense_7/BiasAdd: (BiasAdd): /job:localhost/replica:0/task:0/device:GPU:0
Identity: (Identity): /job:localhost/replica:0/task:0/device:GPU:0
identity_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0
Const: (Const): /job:localhost/replica:0/task:0/device:GPU:0These are the most important parts of my custom DataGenerator, it is essentially inspired from this example https://towardsdatascience.com/keras-data-generators-and-how-to-use-them-b69129ed779c
def __getitem__(self, index):
"""Generate one batch of data
:param index: index of the batch
:return: X and y when fitting. X only when predicting
"""
# Generate indexes of the batch
last_item_for_batch = (index + 1) * self._batch_size if (index + 1) * self._batch_size < len(
self._indexes) else len(self._indexes) - 1
indexes = self._indexes[index * self._batch_size: last_item_for_batch]
# Find list of IDs
data_ids_temp = indexes # [self._indexes[k] for k in indexes]
# Generate data
x_arr = self._generate_x_arr(data_ids_temp)
self._total_requested_batches += 1
if self._to_fit:
y_arr = self._generate_y_arr(data_ids_temp)
return x_arr, y_arr, [None]
else:
return x_arr
def on_epoch_end(self):
"""Updates indexes after each epoch"""
self._total_requested_batches = 0
if self._shuffle:
np.random.shuffle(self._indexes)
def _generate_x_arr(self, data_ids_temp):
"""Generates data containing batch_size images (shape [batch_size, n])
:param list_IDs_temp: list of label ids to load
:return: batch of images
"""
# Initialization
x_arr = np.empty((self._batch_size, self._n_nodes))
for i, line_number in enumerate(data_ids_temp):
line = self._get_data_file_line(line_number)
components = line.split(" ")
input_data = [] # the state of all nodes
# parse input
for j in range(self._n_nodes):
input_data.append(float(components[j]))
x_arr[i,] = np.asarray(input_data)
return x_arr
def _generate_y_arr(self, data_ids_temp):
"""Generates data containing batch_size masks
:param list_IDs_temp: list of label ids to load
:return: batch if masks
"""
# Initialization
y_arr = np.empty((self._batch_size, self._n_nodes))
for i, line_number in enumerate(data_ids_temp):
line = self._get_data_file_line(line_number)
components = line.split(" ")
input_data = [] # the state of all nodes
target_data = [] # the expected values for all actions
# parse input_data
for j in range(self._n_nodes):
input_data.append(float(components[j]))
action = int(components[self._n_nodes])
reward = float(components[self._n_nodes + 1])
prediction = [[0.0 for i in range(self._n_nodes)]] # these comes from another network
# generate expected value
for j in range(self._n_nodes):
if j == action:
target_data.append(reward - prediction[0][j])
else:
target_data.append(prediction[0][j])
y_arr[i,] = np.asarray(target_data)
return y_arr
def _get_data_file_line(self, line_number) -> str: """Get a line in the data file by seeking""" self._train_file_fp.seek(line_number * self._train_file_line_size) line = self._train_file_fp.read(self._train_file_line_size).strip() return lineThanks to anyone helps me to figure it out what is going on!
Reply all
Reply to author
Forward
0 new messages