Bug on model-personalization demo app?

[Nikolas Stavrou]

I've been experimenting with the following model-personalization demo app for on-device training.

examples/lite/examples/model_personalization at master · tensorflow/examples (github.com)

It seems that in every cycle of sample gathering and then training, the trainingSamples of all the previous cycles remain in the list that we use to train our model. This means that every time we perform on-device training, we use all of the samples taken so far within our app.

However, I've found that the inference in a class incremental scenario is totally wrong for the newer class something which I do not understand why it occurs because we train every time with all of our samples.

If anyone wants to replicate this, gather some samples for class 1 and some for class 2, train and perform inference. Then gather samples for class 3 and train again. Performing inference will show that class 3 is classified wrongly as class 1 or 2 and the loss is always high above 1 (around 4,5 if you gather 10 samples for each class).

I do not understand why this wrong inference occurs, to my understanding so far it seems like it shouldn't wrongly classify the new class.

Any help is greatly appreciated, thanks!

[Haoliang Zhang]

+Jared Lim 

--
You received this message because you are subscribed to the Google Groups "TensorFlow Lite" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tflite+un...@tensorflow.org.
To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/tflite/cccb049a-5099-49da-9852-b45b78b1db98n%40tensorflow.org.

--

Best,

Haoliang

[Nikolas Stavrou]

Did anyone manage to get a look at this? I've been trying to find out what is wrong with the training in the past days but I couldn't figure it out.

On Thursday, April 6, 2023 at 2:12:30 AM UTC+3 haol...@google.com wrote:

+Jared Lim 

On Wed, Apr 5, 2023 at 4:01 PM Nikolas Stavrou <nikolas....@gmail.com> wrote:

I've been experimenting with the following model-personalization demo app for on-device training.

examples/lite/examples/model_personalization at master · tensorflow/examples (github.com)

It seems that in every cycle of sample gathering and then training, the trainingSamples of all the previous cycles remain in the list that we use to train our model. This means that every time we perform on-device training, we use all of the samples taken so far within our app.

However, I've found that the inference in a class incremental scenario is totally wrong for the newer class something which I do not understand why it occurs because we train every time with all of our samples.

If anyone wants to replicate this, gather some samples for class 1 and some for class 2, train and perform inference. Then gather samples for class 3 and train again. Performing inference will show that class 3 is classified wrongly as class 1 or 2 and the loss is always high above 1 (around 4,5 if you gather 10 samples for each class).

I do not understand why this wrong inference occurs, to my understanding so far it seems like it shouldn't wrongly classify the new class.

Any help is greatly appreciated, thanks!

--
You received this message because you are subscribed to the Google Groups "TensorFlow Lite" group.

To unsubscribe from this group and stop receiving emails from it, send an email to tflite+unsubscribe@tensorflow.org.

To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/tflite/cccb049a-5099-49da-9852-b45b78b1db98n%40tensorflow.org.

--

Best,

Haoliang

[Jared Lim]

Hi Nikolas,

Thanks for reaching out. This has been a known issue of the on-device training demo, and I have also observed this behavior in the past.

Although we haven't fully analyzed the real cause of the issue (because the current demo app already meets our intention and requirement; performing on-device training and inference on the fixed size of outputs), I have some hypothesis on this issue:

When we first gather some samples for class 1 and 2 and train, we already start overfitting our model to this small amount of samples for those two specific classes only. Even if we introduce similar amount of sample data for class 3 later, there is already imbalance among the number of samples that went through the training steps for each class, and the model is stuck at the state biased towards the "majority" classes (class 1 and 2), misclassifying class 3 as 1 or 2 in most cases and having a bad loss value. Usually these types of outcomes happen when we perform ML on imbalanced data.

Hope this helps.

Best regards,

Jared Lim

On Sun, Apr 9, 2023 at 11:20 AM Nikolas Stavrou <nikolas....@gmail.com> wrote:

Did anyone manage to get a look at this? I've been trying to find out what is wrong with the training in the past days but I couldn't figure it out.

On Thursday, April 6, 2023 at 2:12:30 AM UTC+3 haol...@google.com wrote:

+Jared Lim 

On Wed, Apr 5, 2023 at 4:01 PM Nikolas Stavrou <nikolas....@gmail.com> wrote:

I've been experimenting with the following model-personalization demo app for on-device training.

examples/lite/examples/model_personalization at master · tensorflow/examples (github.com)

It seems that in every cycle of sample gathering and then training, the trainingSamples of all the previous cycles remain in the list that we use to train our model. This means that every time we perform on-device training, we use all of the samples taken so far within our app.

However, I've found that the inference in a class incremental scenario is totally wrong for the newer class something which I do not understand why it occurs because we train every time with all of our samples.

If anyone wants to replicate this, gather some samples for class 1 and some for class 2, train and perform inference. Then gather samples for class 3 and train again. Performing inference will show that class 3 is classified wrongly as class 1 or 2 and the loss is always high above 1 (around 4,5 if you gather 10 samples for each class).

I do not understand why this wrong inference occurs, to my understanding so far it seems like it shouldn't wrongly classify the new class.

Any help is greatly appreciated, thanks!

--
You received this message because you are subscribed to the Google Groups "TensorFlow Lite" group.

To unsubscribe from this group and stop receiving emails from it, send an email to tflite+un...@tensorflow.org.

To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/tflite/cccb049a-5099-49da-9852-b45b78b1db98n%40tensorflow.org.

--

Best,

Haoliang

[Nikolas Stavrou]

Thanks for your fast response.

Having this observation on imbalanced data is indeed expected. However, I experiment with extreme scenarios such as gathering only  1 sample for 1 class and performing training and then inference. Then, training again by gathering a lot more samples of a new class or multiple new classes such as 50 samples and performing inference results again in totally wrong inference of the new classes (loss above 10) even tho there was only 1 sample used for training in the first sample gathering.

This makes me believe that the issue is not because of imbalanced data. 

[Jared Lim]

Hi Nikolas,

If there was only one sample used during the first training, then the training parameter might have already been overfitted to that specific training sample (because training happens continuously while the `training` is pressed). But as you said, we shouldn't overlook the possibility that there are other issues causing this behavior.

As the demo app is working well with the fixed output size, we will deprioritize this issue as of now (i.e. we cannot guarantee that it will be addressed in the near future), but once we have more bandwidth, we will figure out the root cause and address it.

Please let us know if you have any more questions.

Best regards,

Jared Lim

[Nikolas Stavrou]

Valid argument. I hope this gets addressed sometimes in the future to allow to expand on it by adding incremental learning capabilities

Cheers,

Nick