Running model inference in a thread

104 views
Skip to first unread message

Kriti Tulsian

unread,
Jul 24, 2020, 4:05:35 AM7/24/20
to Rust for TensorFlow

I'm trying to spawn a `thread` which runs model inference inside a `loop`. 

I get a `move` error while running model inference inside the thread - `move occurs because `graphSeshVecFC` has type `std::vec::Vec<(food_scoring_inference::tensorflow::Graph, food_scoring_inference::tensorflow::Session)>`, which does not implement the `Copy` trait`

A snippet of my code below - 
async fn main() {

let graphSeshVecFS = match init_multi_tf_graph(tfModelsFSPaths) {
Ok(graphSeshVal) => graphSeshVal,
Err(initErrMsg) => panic!(
"Error while loading/initializing Food Scoring graphs: {}",
initErrMsg
),
};

let graphSeshVecFC = match init_multi_tf_graph(tfModelsFCPath) {
Ok(graphSeshVal) => graphSeshVal,
Err(initErrMsg) => panic!(
"Error while loading/initializing Food Classification graphs: {}",
initErrMsg
),
};
info!("Models loaded into memory...");

loop {
//---------------------------------------- polling and downloading from s3----------------

let poll_result = tokio::spawn(async move {polling_downloading(foodImgsFolder.clone(), awsRegion.clone(), sqsQueueName, s3BucketB2)});

//-----------------------------------------------------------------------------------------
let currentImgCount = getFileCountInDirectory(&configInfo.localFiles_FoodImagesDirectory.clone()).unwrap();

info!(
"Current Image count in FoodImages directory: {}",
currentImgCount
);

// if currentImgCount > 0 {
let inference_result = task::spawn_blocking(move || {
runModelInferenceAndStoreResult(foodImgsFolder.clone(),redisConnStr.clone(),&graphSeshVecFC,&graphSeshVecFS,currentImgCount);});
tokio::join!(poll_result,inference_result );
info!(
"Image rating flow completed execution in: {:?}",
imgRatingFlowTimeStart.elapsed()
);
}
}

Kyle Kosic

unread,
Jul 24, 2020, 8:47:14 AM7/24/20
to Rust for TensorFlow
I may be wrong, but I don't think you can safely send a reference to a model to another thread. Either the model needs to be loaded and owned by that thread, or you can pass a thread-safe reference using the `Arc<Mutex<_>>` pattern. Here is an example of a model inference being offloaded to another thread using a mutex: https://github.com/kykosic/actix-tensorflow-example/blob/master/server/src/main.rs#L29-L43

Daniel McKenna

unread,
Jul 24, 2020, 8:56:12 AM7/24/20
to Rust for TensorFlow, kyle...@gmail.com
Also if the model isn't being changes (weights or structure) you don't need the mutex it's safe to run inference from multiple threads as only the SessionRunArgs will be mutated and that will be unique for each inference call

Kyle Kosic

unread,
Jul 24, 2020, 9:01:42 AM7/24/20
to Rust for TensorFlow, kyle...@gmail.com
Good point, the Mutex in my example is unnecessary. I will update it. Thanks!
Reply all
Reply to author
Forward
0 new messages