File format import / export and protobufs

[Brad Larson]

During the TensorFlow Dev Summit talk about Swift for TensorFlow, one of the bullet points mentioned SavedModel export as a near-term project. I was curious about plans for that and other file formats, and would be glad to help with this implementation. I see loading and saving of weights / models to standard formats as a key next step in making this more viable for everyday use. Also, being able to write out things like TensorFlow events would enable integration with TensorBoard for monitoring and visualization.

Most of the standard file formats you have out there for deep learning model interchange are based on protobufs. These include:

- Caffe

- TensorFlow frozen graph

- TensorFlow SavedModel

- ONNX

- Core ML

as well as TensorBoard events. The Swift Protobuf project, hosted by Apple, provides an easy and cross-platform way of loading and saving binary and text protobuf files in native Swift code:

https://github.com/apple/swift-protobuf

However, it's an external dependency that currently isn't part of the mainline toolchain. Are there plans to more tightly integrate this with the toolchain and / or deep learning APIs? I have Swift code that I've already written to parse Caffe models, and have been building out support for others. I'd love to be able to submit that as part of the deep learning API, but it depends on the Swift Protobuf framework and won't build without that dependency. The framework is pretty mature and fairly official (with support from developers at Apple and Google), so I don't think it would be all that controversial to bundle with the toolchain.

I know that the graph is serialized into a protobuf on the C++ side at compile time, but that seems to be a separate mechanism and wouldn't be something we'd interact with here (aside from potentially inspecting the graph for model export).

Having the ability to pull in pretrained weights makes training more practical (I wrote my own importer to enable my AlexNet transfer learning example), and having a one-step model.exportCoreMLModel(to:[URL]) method (as an example) would immediately make Swift developers targeting iOS pay attention to this. TensorFlow Lite uses FlatBuffer, so it'll be a little different as an import / export format, but having native protobuf support would open up a bunch of these file formats. SavedModel -> TFLite would at least be a pathway for Android, Edge TPU, and other TFLite developers as a first step.

I figured that the mention of TensorFlow SavedModels during the talk might indicate that there was a plan for protobuf integration, so I was curious as to the roadmap there. Again, I'd be glad to help with this, because I've done a bunch of work with Swift protobufs and model files.

[Brennan Saeta]

Hey Brad!

Thanks for proposing this! I think that although we'll have to carefully scrutinize all our dependencies when mobile + IoT applications come into scope, I think that using swift-protobuf is a great idea (and for now we won't worry too much about the extra dependency). We'll probably have to be careful to ensure we don't accidentally link in common Protobuf symbols multiple times (i.e. once via TF, and once for Swift-Protobuf). It probably makes sense to use the Swift Codable infrastructure as well (although I suspect we'll need some flexibility to ensure we can define arbitrary mappings between names to ensure we can load up checkpoints).

From my perspective, I think the first priority should probably be TensorFlow SavedModel, but I would be very interested in what the community thinks makes sense for subsequent priorities. (Sounds like there's at least one vote for CoreML.)

All the best,

-Brennan

P.S. We load up MiniGo from a set of weights trained in Python. See the demo code (search for "Loading checkpoint into GoModel.") and the implementation in GitHub.

--
You received this message because you are subscribed to the Google Groups "Swift for TensorFlow" group.
To unsubscribe from this group and stop receiving emails from it, send an email to swift+un...@tensorflow.org.

[Chris Lattner]

Right.  Super +1 for leveraging Codable if we can use it on Model's.

-Chris

[Volodymyr Pavliukevych]

Hi Brad, 

 You can explore my solution I have done for Swift/C API year ago.

https://github.com/Octadero/TensorFlow/blob/master/Sources/TensorFlowKit/SavedModel.swift