How to accomplish tf.reduce_mean()?

[Robert Blomqvist]

Hi again, working through my MNIST example is going quite well so far. I have reached a point now where the graph is fully constructed and I'm beginning to translate the training phase from Python. The cost function used in my book is tf.reduce_mean, I searched through the documentation and understand this specific one is not available in the C bindings. Is there any way to accomplish the same result as tf.reduce_mean using the C-bindings only?

[Adam Crume]

You can call tensorflow::ops::mean if you're using the new Scope-based graph building.  Note that this currently requires the experimental_training feature.  If you're using the lower-level graph building (i.e. using tensorflow::Operation), you can use the "Mean" op_type.

On Mon, May 25, 2020 at 10:29 AM 'Robert Blomqvist' via Rust for TensorFlow <ru...@tensorflow.org> wrote:

Hi again, working through my MNIST example is going quite well so far. I have reached a point now where the graph is fully constructed and I'm beginning to translate the training phase from Python. The cost function used in my book is tf.reduce_mean, I searched through the documentation and understand this specific one is not available in the C bindings. Is there any way to accomplish the same result as tf.reduce_mean using the C-bindings only?

--
You received this message because you are subscribed to the Google Groups "Rust for TensorFlow" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rust+uns...@tensorflow.org.
To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/rust/d7bd7431-d153-4826-a09f-dd9cc47b7754%40tensorflow.org.

[Robert Blomqvist]

Thanks for the response!

I'm struggling a bit with translating the cost function from my book to the Rust bindings. Below shows the (hopefully) relevant parts:

target_output = tf.placeholder(tf.float32, [None, 10])

...

ntwk_output_1 = tf.matmul(l3_output, out_layer) + out_layer_bias

...

cf = td.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=ntwk_output_1, labels=target_output))

So network_output_1 should, according to the guide in the book, be the raw logits which is later fed to a Softmax.

First, am I translating the combined matmul and addition from Python code above correctly? There are a lot of these in the book of different variations. Or is there a better way?

Second, I don't feel confident in what I should send in to the ops::mean reduction_indices. tf.reduce_mean seems to handle this without needing to specify the reduction_indices, that same result is what I'm after.

I'm trying to output Rust code for easiest possible readability and learning, not necessarily optimized Rust code.

    // raw logits
    let network_output_mul = ops::mat_mul(
        layer_3_output.into(),
        out_layer.output().clone(),
        &mut layer_scope.with_op_name("network_output_mul"),
    )?;

    let network_output_1 = ops::add(
        network_output_mul.clone().into(),
        out_layer_bias.output().clone(),
        &mut layer_scope.with_op_name("network_output_1"),
    )?;

    let softmax_cewl = ops::softmax_cross_entropy_with_logits(
        network_output_1.into(),
        target_output.into(),
        &mut training_scope.with_op_name("softmax_cewl"),
    )?;

    let cf = ops::mean(
        softmax_cewl.into(),
        // ?
        &mut training_scope.with_op_name("cost_function"),
    )?;

Den tisdag 26 maj 2020 kl. 21:30:30 UTC+2 skrev Adam Crume:

You can call tensorflow::ops::mean if you're using the new Scope-based graph building.  Note that this currently requires the experimental_training feature.  If you're using the lower-level graph building (i.e. using tensorflow::Operation), you can use the "Mean" op_type.

On Mon, May 25, 2020 at 10:29 AM 'Robert Blomqvist' via Rust for TensorFlow <ru...@tensorflow.org> wrote:

Hi again, working through my MNIST example is going quite well so far. I have reached a point now where the graph is fully constructed and I'm beginning to translate the training phase from Python. The cost function used in my book is tf.reduce_mean, I searched through the documentation and understand this specific one is not available in the C bindings. Is there any way to accomplish the same result as tf.reduce_mean using the C-bindings only?

--
You received this message because you are subscribed to the Google Groups "Rust for TensorFlow" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ru...@tensorflow.org.

[Adam Crume]

The mat_mul and add look fine.  You don't need to specify with_op_name if you don't want to; that simply gives the operations more understandable names which may help with debugging.  For reduction_indices, you need to pass in the list of dimensions that you want to average over.  In your case, it sounds like you should pass in all dimensions.  We don't currently have a shorthand for reducing over all dimensions.  Here's an example to demonstrate how it works:

let mut scope = Scope::new_root_scope();
let scope = &mut scope;

let placeholder = ops::Placeholder::new().dtype(DataType::Float).shape(Shape::from(&[2, 2][..])).build(scope)?;
let mean0 = ops::mean(placeholder.clone().into(), ops::constant(&[0][..], scope)?.into(), scope)?;
let mean1 = ops::mean(placeholder.clone().into(), ops::constant(&[1][..], scope)?.into(), scope)?;
let mean_all = ops::mean(placeholder.clone().into(), ops::constant(&[0, 1][..], scope)?.into(), scope)?;

let session = Session::new(&SessionOptions::new(), &scope.graph_mut())?;
let sample_input = Tensor::new(&[2, 2]).with_values(&[1.0f32, 10.0, 100.0, 1000.0])?;
let mut run_args = SessionRunArgs::new();
run_args.add_feed(&placeholder, 0, &sample_input);
let token0 = run_args.request_fetch(&mean0, 0);
let token1 = run_args.request_fetch(&mean1, 0);
let token_all = run_args.request_fetch(&mean_all, 0);
session.run(&mut run_args)?;
println!("mean0 = {}", run_args.fetch::<f32>(token0)?);
println!("mean1 = {}", run_args.fetch::<f32>(token1)?);
println!("mean_all = {}", run_args.fetch::<f32>(token_all)?);

which prints out

mean0 = [50.5, 505]
mean1 = [5.5, 550]
mean_all = 277.75

To unsubscribe from this group and stop receiving emails from it, send an email to rust+uns...@tensorflow.org.
To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/rust/36e42f00-ad04-488f-96be-6fedf70d8e45%40tensorflow.org.

[Robert Blomqvist]

Thanks a lot for the clear example. I figured it out a while after the question as well, looking at the error message and reading the tf.reduce_mean docs.

I am aware the with_op_name is optional, but it's kinda nice to know where to look when reading TensorFlow's error messages. Many operations look the same.

I finally got the entire Python MNIST example from my Neural Network book converted to Rust (not optimized/refactored at all though, ), it builds and runs fine. Unfortunately the cost function is not decreasing so I have to look into some more details before I publish it as a blog post, I think it's data set related.

These bindings are actually quite logical to work with, it just becomes VERY verbose. That's ok for learning though, but I wish I had some spare time to put into some kind of wrapper crate or module around this one to simplify some things.

Into:ing numerals (1, 2, 3) into ops::constants would be nice, not sure how clean this can become since a scope needs to be supplied most of the time. Also some shorthand for slices into Tensors would be cool. Maybe the TensorFlow scope variable could be sent into a Rust scope somehow and just become the standard for all operations in that Rust scope. Perhaps with a closure or something. Not even sure if that's possible.

like:

|scope| {

     // all ops in here use the same scope, no need to write it out

}

Or maybe using macros, probably lots of ways to solve this.

Most operations also require the actual structure passed in, which means a lot of cloning stuff that is used a lot (like network inputs etc), it kinda hurts my Rust-feelings but I don't claim to understand the inner details of the API and realize this is probably required. Using references would be nice though, if possible. Again, maybe this is something that could be simplified with a layer on top of all this.

I really appreciate the work you put in to this Adam, and thanks for helping out with answering questions. 👍 

To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/rust/36e42f00-ad04-488f-96be-6fedf70d8e45%40tensorflow.org.

[Robert Blomqvist]

Thanks a lot for the clear example. I figured it out a while after the question as well, looking at the error message and reading the tf.reduce_mean docs.

I am aware the with_op_name is optional, but it's kinda nice to know where to look when reading TensorFlow's error messages. Many operations look the same.

I finally got the entire Python MNIST example from my Neural Network book converted to Rust (not optimized/refactored at all though, ), it builds and runs fine. Unfortunately the cost function is not decreasing so I have to look into some more details before I publish it as a blog post, I think it's data set related.

These bindings are actually quite logical to work with, it just becomes VERY verbose. That's ok for learning though, but I wish I had some spare time to put into some kind of wrapper crate or module around this one to simplify some things.

Into:ing numerals (1, 2, 3) into ops::constants would be nice, not sure how clean this can become since a scope needs to be supplied most of the time. Also some shorthand for slices into Tensors would be cool. Maybe the TensorFlow scope variable could be sent into a Rust scope somehow and just become the standard for all operations in that Rust scope. Perhaps with a closure or something. Not even sure if that's possible.

like:

|scope| {

     // all ops in here use the same scope, no need to write it out

}

Or maybe using macros, probably lots of ways to solve this.

Most operations also require the actual structure passed in, which means a lot of cloning stuff that is used a lot (like network inputs etc), it kinda hurts my Rust-feelings but I don't claim to understand the inner details of the API and realize this is probably required. Using references would be nice though, if possible. Again, maybe this is something that could be simplified with a layer on top of all this.

I really appreciate the work you put in to this Adam, and thanks for helping out with answering questions. 👍 

Den fredag 29 maj 2020 kl. 17:36:18 UTC+2 skrev Adam Crume:

To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/rust/36e42f00-ad04-488f-96be-6fedf70d8e45%40tensorflow.org.

[mason....@gmail.com]

Would you consider checking in the MNIST example to the rust repo? Maybe under examples/?

I think it’d be helpful for a lot of people. (Adam: would you take that patch?)

 

Sent from Mail for Windows 10

To unsubscribe from this group and stop receiving emails from it, send an email to rust+uns...@tensorflow.org.
To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/rust/b0acbbbb-231d-442b-b0a6-3142c62928f5%40tensorflow.org.

 

[Robert Blomqvist]

Yeah, I can create a pull request as soon as I get it working and cleaned up a bit.

To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/rust/b0acbbbb-231d-442b-b0a6-3142c62928f5%40tensorflow.org.

 

[Adam Crume]

I would definitely love to have that added as an example.  Besides the direct benefit to users, the more real-world examples I have of this API, the more I can tweak it to make it more ergonomic.

As for the other concerns, there's a reason why this bit of the API is still marked as experimental.  Automatically tying everything to the scope is not simple.  We can't use macros, because Rust macro hygiene prevents us from reading variables with an arbitrary name.  I'd also like to avoid macros unless they're a clear win.  The only other solution I can think of is thread locals, but that's a very big and dangerous hammer to use for a problem like this.  For all the necessary clones, I might play around with AsRef, which might increase the total number of clone calls (although Operation and Output are fairly cheap to clone), but it should reduce the number of clone calls the user has to make manually.

To unsubscribe from this group and stop receiving emails from it, send an email to rust+uns...@tensorflow.org.
To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/rust/71aa0ffa-c17c-46d1-8972-8d0a56d90707%40tensorflow.org.

[Robert Blomqvist]

How about building operations from the scope? So instead of

   

ops::constant(&[INPUT_SIZE, HIDDEN_LAYER_SIZE][..], layer_scope);

something like

layer_scope.constant(&[INPUT_SIZE, HIDDEN_LAYER_SIZE][..]);

Not sure if it's applicable throughout the entire API, but at least in my opinion it improves readability.

To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/rust/71aa0ffa-c17c-46d1-8972-8d0a56d90707%40tensorflow.org.

[Adam Crume]

It's possible, but it makes things a lot more awkward to have a single type with that many methods.  For one, they would all have to go in the same source file, which makes incremental compilation harder.  We'd have to mix regular and generated code, which makes writing the regular code harder, and it also means that any op names which happen to collide with other Scope method names would have to be renamed, which adds complexity and confusion.  It also means that built-in and user-provided ops would look different, since user-provided ops wouldn't be able to be Scope methods.

Some of these might be addressable by introducing another type or trait, but that would add boilerplate of its own.

To unsubscribe from this group and stop receiving emails from it, send an email to rust+uns...@tensorflow.org.
To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/rust/934e110a-6142-4720-b79b-fb0127119916%40tensorflow.org.

[Robert Blomqvist]

Regarding the source file, you don't need to put all impl's in one file, it's fine to split it up in multiple source files (I do this from time to time for better file structure). I do see the other points you mention though. Probably not worth it in the end, it was just a quick idea from the top of my head.

To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/rust/934e110a-6142-4720-b79b-fb0127119916%40tensorflow.org.

[Adam Crume]

Good point; I always write monolithic impl blocks when possible, and forgot that they can be split up.

To unsubscribe from this group and stop receiving emails from it, send an email to rust+uns...@tensorflow.org.
To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/rust/8e9e62fd-691e-43d1-8924-3858c3e2383c%40tensorflow.org.

[Robert Blomqvist]

I have not finished the blog post yet, but put my complete MNIST example here: https://github.com/roqvist/rust-tensorflow-examples

To clarify, I am just a machine learning hobbyist so in now way an expert but the example builds and the loss is shrinking... so I guess it works. Feel free to take a look, I might have missed or misunderstood something. The main goal was to learn by writing and keeping the code as readable as possible.