Partial runs of models in TFLM

32 views
Skip to first unread message

Daniel Situnayake

unread,
Aug 26, 2021, 12:19:30 PMAug 26
to SIG Micro
Hi TFLM team!

I'm interested in running a multi-output model in stages, so that I can exit early if a layer's output indicates I'm able to do so. Is this supported in TFLM right now?

I figured it wasn't, but looking through the code to see where I might add it I came across this comment:


Not sure what "strided models" refers to in this case but it sounds intriguing! Is there some kind of undocumented support for what I'd like to do?

Warmly,
Dan

--
Daniel Situnayake
Founding TinyML Engineer, Edge Impulse

Advait Jain

unread,
Aug 26, 2021, 12:55:37 PMAug 26
to Daniel Situnayake, SIG Micro
Hi Dan,

Strided models in the comment that you link to refers to models that do not produce an output with every call to invoke (e.g. streaming models). The way this was implemented in TFLM is via the circular buffer custom op, which can exit early with a special TfLiteStatus. The emphasis here is on "was". Such support for partial graph executions is unique to TFLM (i.e. not a standard TF or TfLite feature and is subject to change / removal). For this particular class of models, we recommend looking at the kws_streaming approach.

TFLM recently added support for the if operator (https://www.tensorflow.org/api_docs/python/tf/cond). This may be what you are looking for.

We might be able to offer better guidance if you are able to share an example of what you are trying to do.

Regards,
Advait


--
You received this message because you are subscribed to the Google Groups "SIG Micro" group.
To unsubscribe from this group and stop receiving emails from it, send an email to micro+un...@tensorflow.org.
To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/micro/CAOu%3DFxaUs2sBkBOt%3DPdFdxNbL1GDTb7yT%3D1E1WOpz2Gn3E6P%2Bw%40mail.gmail.com.

Daniel Situnayake

unread,
Aug 27, 2021, 8:00:21 PMAug 27
to Advait Jain, SIG Micro
Thanks Advait, that's very helpful. I'm interested in creating something inspired by the architecture described in this paper:

It's designed for classifying video and is able to early exit when it has enough confidence in its classification, even if it has not yet been run over all frames.

Warmly,
Dan

Pete Warden

unread,
Aug 27, 2021, 9:50:14 PMAug 27
to Daniel Situnayake, Advait Jain, SIG Micro
One idea I've been considering is a developer-supplied callback function that's invoked between ops, to allow pre-loading of memory areas, or calls to cooperative multitasking yields. It's still only a vague idea right now, but some sort of extensibility like this keeps coming up as a requirement, and allowing the callback to abort the calculations would be one way of enabling your use case.

I think Advait's suggestion of using an "if" op is a lot cleaner for your needs though, since then all the logic stays in the model.

Daniel Situnayake

unread,
Aug 30, 2021, 4:25:20 PMAug 30
to Pete Warden, Advait Jain, SIG Micro
I love that idea of the developer-supplied callback; it would be super helpful in all sorts of ways, especially if it gave you the ability to poke around and change values in the tensors.

Does TF Lite converter produce the 'if' op for models using the 'cond' TF op?


Warmly,
Dan


Reply all
Reply to author
Forward
0 new messages