Run inference on multiple models/graphs in parallel (tensorflow-gpu)

[Kahlil Abreu]

Tensorflow version: tensorflow-gpu (1.15)

I have built a service that runs in an infinite loop and pops files off a queue in order to process them (run them through some tensorflow models).
Performance (speed) is critical and hence I chose to use Rust to build it.

In order to speed things up (as loading model graphs into memory takes a fair bit of time as they seem to be fairly large models (around 200-300MB each) and I have a total of about 5 models to be run for each iteration), I load the graphs into memory first and then pass them into a looping function that checks a queue and pops files off the queue as and when they are available.

Things are running okay for now but the speed is not as good as I would like it to be.

I wanted a way to run inference in parallel.

Currently I pass a `graphSeshVec:&Vec<(Graph,Session)>)` to my "inference" function (that is `async`) that then does the following:

```
for (graphVal, seshVal) in graphSeshVec {
// my model inference code here

// (this function that is called is also async but this does not seem to affect anything as the model inference seems to run serially)
}

```

Is there a way by which I could try to make these operations run in parallel?

Any help on this would be appreciated, thanks!

[Adam Crume]

I don't think async alone is going to help you here, since the Rust API is fully synchronous.  To do what you want, you need to run separate threads.  Depending on exactly how TensorFlow schedules your computation, adding more threads may or may not improve performance, so you'll want to benchmark it.

--
You received this message because you are subscribed to the Google Groups "Rust for TensorFlow" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rust+uns...@tensorflow.org.
To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/rust/48e6352a-6232-4cd1-9370-cc2b1b8cdfb5%40tensorflow.org.