How to continue training model after saving and loading in a new browser session.

[Arman Oganesian]

Hello,

I modified the Webcam-Transfer_learning sample to save the trained model and load it back from the file system in a new browser session. Works great.

Now, after reloading the json and the weights file i want to continue training the same model by providing more examples.

if i call model.fit() with a new set of examples  from camera will it add to the existing training or overwrite?

Thank you

[yass...@google.com]

It will continue training the network, this will involve overwriting/updating the weights in the network but conceptually that is like adding to the existing training since it gets to use information from the weights you loaded. 

[Arman Oganesian]

Unfortunately that's not a behavior i observe.

There are 4 classes to predict in a sample.

I train 2 classes, save the model to local drive.

Restart the browser, load the saved model and confirm that the loaded model works for 2 previously trained classes.

Then i add samples for 2 other classes, retrain the model using the code below and completely lose training for the first 2 classes.

I tried with compiling and without, same result, can't figure out what am i doing wrong.

async function retrain() {

if (controllerDataset.xs == null) {

throw new Error('Add some examples before retraining!');

}

// //Creates the optimizers which drives training of the model.

// const optimizer = tf.train.adam(ui.getLearningRate());

// // We use categoricalCrossentropy which is the loss function we use for

// // categorical classification which measures the error between our predicted

// // probability distribution over classes (probability that an input is of each

// // class), versus the label (100% probability in the true class)>

// model.compile({

// optimizer: optimizer,

// loss: 'categoricalCrossentropy'

// });

// We parameterize batch size as a fraction of the entire dataset because the

// number of examples that are collected depends on how many examples the user

// collects. This allows us to have a flexible batch size.

const batchSize =

Math.floor(controllerDataset.xs.shape[0] * ui.getBatchSizeFraction());

if (!(batchSize > 0)) {

throw new Error(

`Batch size is 0 or NaN. Please choose a non-zero fraction.`);

}

// Train the model! Model.fit() will shuffle xs & ys so we don't have to.

model.fit(controllerDataset.xs, controllerDataset.ys, {

batchSize,

epochs: ui.getEpochs(),

//initialEpoch: ui.getInitialEpochs(),

callbacks: {

onBatchEnd: async(batch, logs) => {

ui.trainStatus('Loss: ' + logs.loss.toFixed(5));

}

}

}

);

//const saveResult = await model.save('downloads://my-model-1');

}

[yass...@google.com]

Ah I see, you are trying to train new non-overlapping classes. I'm not sure that is possible, if you show the network completely new examples and nothing from the 'existing' classes the model has no incentive to remember the old classes. All of the parameters of the model are used to solve the new problem that you are giving it. Essentially you are asking it to learn a new weight distribution. 

I don't know of a way to train the same model to still perform the original task without using at least some of the original training data alongside the new data. It would probably be best to train using a fraction of the data from all classes at each retraining.

[Arman Oganesian]

Thanks for clarifying!

So if i add examples to all 4 classes every time i "retrain" including the first time training, the previous weights will be included in the updated weight distribution?

Basically, i am trying to build a in-browser training application to train and retrain the model until it generalizes well enough to accurately predict  using examples(faces) it never seen before.

The way i see it i have 2 choices 

1. As you suggested to train using a fraction of the data on all 4 classes each time i add more examples/retrain including first time training.

2. Write code to persist the ControllerDataset to disk/db every time i save the model and load it back when i am loading the model later for retraining.

Could you please advise which of the 2 choices will work better to achieve the goal.

Thank you,

Arman

On Wednesday, September 5, 2018 at 4:33:05 PM UTC-4, Arman Oganesian wrote:

[Jose Lepistö]

If you find a solution for this I am very interested too!