Multiple sessions and graphs in Tensorflow

[khanna...@gmail.com]

I'm training a model where the input vector is the output of another model. This involves restoring the first model from a checkpoint file while initializing the second model from scratch (using tf.initialize_varaibles()).

There is a substantial amount of code and abstraction, so I'm just pasting the relevant sections here.

The following is the restoring code:

self.variables = [var for var in all_vars if var.name.startswith(self.name)]
saver = tf.train.Saver(self.variables, max_to_keep=3)
self.save_path = tf.train.latest_checkpoint(os.path.dirname(self.checkpoint_path))

if should_restore:
    self.saver.restore(self.sess, save_path)
else:
    self.sess.run(tf.initialize_variables(self.variables))

Each model is scoped within its own graph and session, like this:

 self.graph = tf.Graph()
 self.sess = tf.Session(graph=self.graph)

 with self.sess.graph.as_default():
    # Create variables and ops.

All the variables within each model are created within the variable_scope context manager.

The feeding works as follows:

• A background thread calls sess.run(inference_op) on input scipy.misc.imread(X) and puts the result in a blocking thread-safe queue.
• The main training loop reads from the queue and calls sess.run(train_op) on the second model.

PROBLEM:
I am observing that the loss values, even in the very first iteration, of the training (second model) keep changing drastically. I confirmed that the output of the first model is exactly the same everytime. Commenting out the sess.run of the first model and replacing it with identical input from a a picked file does not show this behaviour.

This is the train_op:

    loss_op = tf.nn.sparse_softmax_cross_entropy(network.feedforward())
    # Apply gradients.
    with tf.control_dependencies([loss_op]):
        opt = tf.train.GradientDescentOptimizer(lr)
        grads = opt.compute_gradients(loss_op)
        apply_gradient_op = opt.apply_gradients(grads)

    return apply_gradient_op

I know this is vague, but I'm happy to provide more details. Any help is appreciated!

[Vikesh Khanna]

The original question (with multiple edits) is posted on StackOverflow here: http://stackoverflow.com/questions/38819576/multiple-sessions-and-graphs-in-tensorflow-in-the-same-process 

Thanks,

Vikesh

[Bojan Ploj]

Dear Vikesh,

I have similar problem. I am novice in TF environment and I would like to check my new algorithm (Bipropagation). My ANN is made from two layers which I would like  teach separately. Firstly the First layer, then the second one and finaly both together (fine tuning). Should I make all this in one or in three separated  sessions and how can I do this? Thaks in advance!

 

Dne ponedeljek, 08. avgust 2016 01.42.18 UTC+2 je oseba Vikesh Khanna napisala:

[Vikesh Khanna]

Hi Bojan,

I was able to resolve my issue by using a common session object for all models (one TF graph). My understanding is that `session` assumes exclusive access to resources, possibly causing a conflict. TF Documentation says sessions support concurrent execution, so it's OK to have multiple runs on the same session object. 

More details here: http://stackoverflow.com/questions/38819576/multiple-sessions-and-graphs-in-tensorflow-in-the-same-process/38889562#38889562

- Vikesh