Partial runs of models in TFLM

[Daniel Situnayake]

Hi TFLM team!

I'm interested in running a multi-output model in stages, so that I can exit early if a layer's output indicates I'm able to do so. Is this supported in TFLM right now?

I figured it wasn't, but looking through the code to see where I might add it I came across this comment:

https://github.com/tensorflow/tflite-micro/blob/main/tensorflow/lite/micro/micro_interpreter.h#L67

Not sure what "strided models" refers to in this case but it sounds intriguing! Is there some kind of undocumented support for what I'd like to do?

Warmly,

Dan

--

Daniel Situnayake
Founding TinyML Engineer, Edge Impulse

d...@edgeimpulse.com
Twitter | LinkedIn

[Advait Jain]

Hi Dan,

Strided models in the comment that you link to refers to models that do not produce an output with every call to invoke (e.g. streaming models). The way this was implemented in TFLM is via the circular buffer custom op, which can exit early with a special TfLiteStatus. The emphasis here is on "was". Such support for partial graph executions is unique to TFLM (i.e. not a standard TF or TfLite feature and is subject to change / removal). For this particular class of models, we recommend looking at the kws_streaming approach.

TFLM recently added support for the if operator (https://www.tensorflow.org/api_docs/python/tf/cond). This may be what you are looking for.

We might be able to offer better guidance if you are able to share an example of what you are trying to do.

Regards,

Advait

--
You received this message because you are subscribed to the Google Groups "SIG Micro" group.
To unsubscribe from this group and stop receiving emails from it, send an email to micro+un...@tensorflow.org.
To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/micro/CAOu%3DFxaUs2sBkBOt%3DPdFdxNbL1GDTb7yT%3D1E1WOpz2Gn3E6P%2Bw%40mail.gmail.com.

[Daniel Situnayake]

Thanks Advait, that's very helpful. I'm interested in creating something inspired by the architecture described in this paper:

https://arxiv.org/pdf/2104.13400.pdf

It's designed for classifying video and is able to early exit when it has enough confidence in its classification, even if it has not yet been run over all frames.

Warmly,

Dan

d...@edgeimpulse.com
+1 415-940-2607
Twitter | LinkedIn

[Pete Warden]

One idea I've been considering is a developer-supplied callback function that's invoked between ops, to allow pre-loading of memory areas, or calls to cooperative multitasking yields. It's still only a vague idea right now, but some sort of extensibility like this keeps coming up as a requirement, and allowing the callback to abort the calculations would be one way of enabling your use case.

I think Advait's suggestion of using an "if" op is a lot cleaner for your needs though, since then all the logic stays in the model.

To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/micro/CAOu%3DFxYL7hBP%3DcvxOiyHN1DbatYy5DhnNPdtkTCQU7bLXKR4EQ%40mail.gmail.com.

[Daniel Situnayake]

I love that idea of the developer-supplied callback; it would be super helpful in all sorts of ways, especially if it gave you the ability to poke around and change values in the tensors.

Does TF Lite converter produce the 'if' op for models using the 'cond' TF op?

https://www.tensorflow.org/api_docs/python/tf/cond

Warmly,

Dan