Running model inference in a thread

[Kriti Tulsian]

I'm trying to spawn a `thread` which runs model inference inside a `loop`. 

I get a `move` error while running model inference inside the thread - `move occurs because `graphSeshVecFC` has type `std::vec::Vec<(food_scoring_inference::tensorflow::Graph, food_scoring_inference::tensorflow::Session)>`, which does not implement the `Copy` trait`

A snippet of my code below - 

async fn main() {

let graphSeshVecFS = match init_multi_tf_graph(tfModelsFSPaths) {

Ok(graphSeshVal) => graphSeshVal,

Err(initErrMsg) => panic!(

"Error while loading/initializing Food Scoring graphs: {}",

initErrMsg

),

};

let graphSeshVecFC = match init_multi_tf_graph(tfModelsFCPath) {

Ok(graphSeshVal) => graphSeshVal,

Err(initErrMsg) => panic!(

"Error while loading/initializing Food Classification graphs: {}",

initErrMsg

),

};

info!("Models loaded into memory...");

loop {

//---------------------------------------- polling and downloading from s3----------------

let poll_result = tokio::spawn(async move {polling_downloading(foodImgsFolder.clone(), awsRegion.clone(), sqsQueueName, s3BucketB2)});

//-----------------------------------------------------------------------------------------

let currentImgCount = getFileCountInDirectory(&configInfo.localFiles_FoodImagesDirectory.clone()).unwrap();

info!(

"Current Image count in FoodImages directory: {}",

currentImgCount

);

// if currentImgCount > 0 {

let inference_result = task::spawn_blocking(move || {

runModelInferenceAndStoreResult(foodImgsFolder.clone(),redisConnStr.clone(),&graphSeshVecFC,&graphSeshVecFS,currentImgCount);});

tokio::join!(poll_result,inference_result );

info!(

"Image rating flow completed execution in: {:?}",

imgRatingFlowTimeStart.elapsed()

);

}

}

[Kyle Kosic]

I may be wrong, but I don't think you can safely send a reference to a model to another thread. Either the model needs to be loaded and owned by that thread, or you can pass a thread-safe reference using the `Arc<Mutex<_>>` pattern. Here is an example of a model inference being offloaded to another thread using a mutex: https://github.com/kykosic/actix-tensorflow-example/blob/master/server/src/main.rs#L29-L43

[Daniel McKenna]

Also if the model isn't being changes (weights or structure) you don't need the mutex it's safe to run inference from multiple threads as only the SessionRunArgs will be mutated and that will be unique for each inference call

[Kyle Kosic]

Good point, the Mutex in my example is unnecessary. I will update it. Thanks!