Issues in saving model

[Abhishek Sinha]

Hi 

I am using BLS2017 model for my experiment. I actually want to save the entire training state of the model and entropy layer to restart the training from same point . The model.save() function only saves the weight of the network and simply drops the entropy model's parameter. Further the warning says that the weights from entropy models are being dropped and may not loaded while loading the weights next time. Is there any way to save the entire model?? 

Regards,

Abhishek

[Johannes Ballé]

Hi Abhishek,

can you post the content of the warning?

Thanks!

Johannes

--
You received this message because you are subscribed to the Google Groups "tensorflow-compression" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tensorflow-compre...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tensorflow-compression/ba3b1ba7-46e8-49fd-a322-0ee52075b77dn%40googlegroups.com.

[Abhishek Sinha]

On using the command:  model.save('all')

Following warning appears:

WARNING:absl:Found untraced functions such as lambda_layer_call_fn, layer_0_layer_call_fn, layer_1_layer_call_fn, layer_2_layer_call_fn, layer_0_layer_call_fn while saving (showing 5 of 34). These functions will not be directly callable after loading.

INFO:tensorflow:Assets written to: m/assets

INFO:tensorflow:Assets written to: m/assets

On using command: tf.keras.models.load_model('all')

Warning appears as:

WARNING:absl:Importing a function (__inference_layer_0_layer_call_and_return_conditional_losses_122239) with ops with custom gradients. Will likely fail if a gradient is requested.

 WARNING:absl:Importing a function (__inference_layer_1_layer_call_and_return_conditional_losses_111423) with ops with custom gradients. Will likely fail if a gradient is requested.

 WARNING:absl:Importing a function (__inference_synthesis_layer_call_and_return_conditional_losses_121036) with ops with custom gradients. Will likely fail if a gradient is requested. 

WARNING:absl:Importing a function (__inference_analysis_layer_call_and_return_conditional_losses_112764) with ops with custom gradients. Will likely fail if a gradient is requested. 

WARNING:absl:Importing a function (__inference_layer_0_layer_call_and_return_conditional_losses_121911) with ops with custom gradients. Will likely fail if a gradient is requested.

 WARNING:absl:Importing a function (__inference_layer_1_layer_call_and_return_conditional_losses_122408) with ops with custom gradients. Will likely fail if a gradient is requested. 

WARNING:absl:Importing a function (__inference_layer_0_layer_call_and_return_conditional_losses_111259) with ops with custom gradients. Will likely fail if a gradient is requested. 

WARNING:absl:Importing a function (__inference_synthesis_layer_call_and_return_conditional_losses_120679) with ops with custom gradients. Will likely fail if a gradient is requested.

 WARNING:absl:Importing a function (__inference_bl_s2017model_layer_call_and_return_conditional_losses_116330) with ops with custom gradients. Will likely fail if a gradient is requested. 

WARNING:absl:Importing a function (__inference_analysis_layer_call_and_return_conditional_losses_119752) with ops with custom gradients. Will likely fail if a gradient is requested. 

WARNING:absl:Importing a function (__inference_bl_s2017model_layer_call_and_return_conditional_losses_117200) with ops with custom gradients. Will likely fail if a gradient is requested.

 WARNING:absl:Importing a function (__inference_bl_s2017model_layer_call_and_return_conditional_losses_118070) with ops with custom gradients. Will likely fail if a gradient is requested. 

WARNING:absl:Importing a function (__inference_analysis_layer_call_and_return_conditional_losses_120322) with ops with custom gradients. Will likely fail if a gradient is requested. 

WARNING:absl:Importing a function (__inference_synthesis_layer_call_and_return_conditional_losses_121750) with ops with custom gradients. Will likely fail if a gradient is requested.

 WARNING:absl:Importing a function (__inference_synthesis_layer_call_and_return_conditional_losses_114350) with ops with custom gradients. Will likely fail if a gradient is requested.

 WARNING:absl:Importing a function (__inference_synthesis_layer_call_and_return_conditional_losses_113374) with ops with custom gradients. Will likely fail if a gradient is requested.

 WARNING:absl:Importing a function (__inference_layer_0_layer_call_and_return_conditional_losses_111959) with ops with custom gradients. Will likely fail if a gradient is requested.

 WARNING:absl:Importing a function (__inference_layer_1_layer_call_and_return_conditional_losses_112148) with ops with custom gradients. Will likely fail if a gradient is requested.

 WARNING:absl:Importing a function (__inference_analysis_layer_call_and_return_conditional_losses_113740) with ops with custom gradients. Will likely fail if a gradient is requested.

 WARNING:absl:Importing a function (__inference_bl_s2017model_layer_call_and_return_conditional_losses_115212) with ops with custom gradients. Will likely fail if a gradient is requested. 

WARNING:absl:Importing a function (__inference_synthesis_layer_call_and_return_conditional_losses_121393) with ops with custom gradients. Will likely fail if a gradient is requested. 

WARNING:absl:Importing a function (__inference_analysis_layer_call_and_return_conditional_losses_119467) with ops with custom gradients. Will likely fail if a gradient is requested.

 WARNING:absl:Importing a function (__inference_layer_1_layer_call_and_return_conditional_losses_122055) with ops with custom gradients. Will likely fail if a gradient is requested. 

WARNING:absl:Importing a function (__inference__wrapped_model_111107) with ops with custom gradients. Will likely fail if a gradient is requested. WARNING:absl:Importing a function (__inference_analysis_layer_call_and_return_conditional_losses_120037) with ops with custom gradients. Will likely fail if a gradient is requested.

 WARNING:absl:Importing a function (__inference_bl_s2017model_layer_call_and_return_conditional_losses_118940) with ops with custom gradients. Will likely fail if a gradient is requested.

<tensorflow.python.keras.saving.saved_model.load.BLS2017Model at 0x7f95830ed590>

[Johannes Ballé]

Hi,

these are warnings that some of the functions could not be traced. They should be safe to ignore.

model.save() will not only try to save the weights/variables, but also trace functions of the model and save their graph representations. This is what is failing for a few of them, and the warnings tell you about that. However, it shouldn't matter in this case.

If you want to continue training a model, you probably want to recreate the model from Python code anyway, and load/save just the weights. The BackupAndRestore callback does this automatically at the end of each epoch. So if you interrupt the training script, next time you start it, it should automatically pick up from the beginning of the interrupted epoch.

Hope this helps!

Johannes.

To view this discussion on the web visit https://groups.google.com/d/msgid/tensorflow-compression/e18d5468-b48c-4ef8-82a6-3a1a7e2aa821n%40googlegroups.com.