Setting a time limit on operation/operations

[agnusm...@gmail.com]

Is it possible to set a time limit on a set of operation / operations so that after a certain time period that operation will end and return say a zero tensor? I'm trying to add a time out on the compute_gradients method so that after a specific time limit it just returns 0.

What I've tried: 

I've tried actually sending a SIGINT signal to interrupt the process, but for some reason there's a delay between the signal being sent and when it's received, and this delay basically spans the whole computation period (oddly).

I've also tried inserting conditionals between gradient operations, but for some reason this slows things down like crazy. Looking at the timeline, it looks as if the conditionals hurt prefetching of data from the parameter server (maybe it's interfering with remote prefetches of some sort?).

Any help or suggestions would be awesome!

[Yaroslav Bulatov]

You can do something like

config = tf.ConfigProto()

config.operation_timeout_in_ms=60000

sess = tf.InteractiveSession(config=config)

that'll interrupt run call after 60 seconds

this setting can go into RunOptions as well

--
You received this message because you are subscribed to the Google Groups "Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss+unsubscribe@tensorflow.org.
To post to this group, send email to dis...@tensorflow.org.
To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/discuss/ca98fbb6-8d29-4335-93fe-6c7ae14fc209%40tensorflow.org.

[agnusm...@gmail.com]

Awesome!! I can't believe I missed that! Thanks for pointing this out.

On Wednesday, December 21, 2016 at 8:55:48 AM UTC-8, Yaroslav Bulatov wrote:

You can do something like

config = tf.ConfigProto()

config.operation_timeout_in_ms=60000

sess = tf.InteractiveSession(config=config)

that'll interrupt run call after 60 seconds

this setting can go into RunOptions as well

On Wed, Dec 21, 2016 at 8:33 AM, <agnusm...@gmail.com> wrote:

Is it possible to set a time limit on a set of operation / operations so that after a certain time period that operation will end and return say a zero tensor? I'm trying to add a time out on the compute_gradients method so that after a specific time limit it just returns 0.

What I've tried: 

I've tried actually sending a SIGINT signal to interrupt the process, but for some reason there's a delay between the signal being sent and when it's received, and this delay basically spans the whole computation period (oddly).

I've also tried inserting conditionals between gradient operations, but for some reason this slows things down like crazy. Looking at the timeline, it looks as if the conditionals hurt prefetching of data from the parameter server (maybe it's interfering with remote prefetches of some sort?).

Any help or suggestions would be awesome!

--
You received this message because you are subscribed to the Google Groups "Discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to discuss+u...@tensorflow.org.

[agnusm...@gmail.com]

Just wondering, I've tested this timeout and it's very precise. Does this functionality rely on signals or something? Is there any way I could send an external signal to a running tensorflow session to replicate this effect to interrupt a sess.run procedure? (as mentioned SIGINT results in a delay between signal delivery and handling which makes it not as good of a choice)

[Yaroslav Bulatov]

You can see how it was implemented in original commit, support for timeout was added in github.com/tensorflow/tensorflow/commit/fa07bfda

To unsubscribe from this group and stop receiving emails from it, send an email to discuss+unsubscribe@tensorflow.org.

To post to this group, send email to dis...@tensorflow.org.

To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/discuss/529a060f-ff46-45e6-9488-99191b422d0c%40tensorflow.org.

[agnusm...@gmail.com]

Thanks!!!!

To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/discuss/529a060f-ff46-45e6-9488-99191b422d0c%40tensorflow.org.