Hi :) I am currently building a small Model to predict some text if it is spam or not and want to build a high level api in rust with it.
In python to invoke/predict things I called via this piece of code: https://github.com/MTRNord/matrix-spam-ml/blob/main/model_v2.py#L241-L247
however in Rust I can't seem to find how I am supposed to tokenize my strings to get the data I can pass into my pretrained model.
What am I missing? :) Or is there a better way of solving this?