stan float

[Andrew Gelman]

Hi--in debugging I made the mistake of removing all the lines form the model (the ultimate uniform prior!). This caused stan to float (within Rstudio). Is there a way to interrupt Stan when it floats, or is that just an unsolvable R problem?
A

[Jiqiang Guo]

I can reproduce in rstan as 

 b <- stan(model.code = "model{}")

This is not a problem with R or rstan.  Stan will hang in this case.  I guess we should check if there is any parameter declared and if so,  an empty model block is legitimate. 

--

Jiqiang  

[Bob Carpenter]

I don't even know what you mean by "float".
Does it just go away and not come back? The
CS term is "hang", because we use "float" for
floating point numbers.

- Bob

[Matt Hoffman]

This particular problem is fixed, but do we know the answer to
Andrew's question about interrupting R functions when they hang in
general? In Matlab the answer is (infuriatingly) in many cases that it
has to be implemented by whoever wrote the c/c++ function that's
interfacing with the environment. It's annoying, but at some point we
should probably look into how to make Stan interruptible in R.

Matt

[Ben Goodrich]

http://cran.r-project.org/doc/manuals/R-exts.html#Allowing-interrupts

[Bob Carpenter]

Interesting. But complex from an implementation point
of view because we want to interrupt a loop within Stan,
but don't want to have to depend on R to do it.

So we'll have to provide a plug-in callback interrupter.
Shouldn't be too hard.

We could add a template parameter IF for the type of interrupter
to all the sampler classes, then down in the guts of
loops, have something like:

template <class IF, ...>
class XXX_sampler {

IF& interrupter_;

if (interrupter_()) {
...handle interrupt...
... stop ...
}
}

Then the question is how often we check.

One obvious choice would be to do it each time the log
probability/gradient function is called. Or once per
sample, or once per N samples.

I don't know how slow it is, and after you call their interrupt,
we apparently can't even assume we get control back,
so we might not be able to do any cleanup like flushing
buffers and closing files (I don't know if C++ does any
of that itself -- Java does).

- Bob

On 8/10/12 4:52 PM, Ben Goodrich wrote:
> http://cran.r-project.org/doc/manuals/R-exts.html#Allowing-interrupts
>
>
> On Friday, August 10, 2012 4:49:48 PM UTC-4, Matt Hoffman wrote:
>
> This particular problem is fixed, but do we know the answer to
> Andrew's question about interrupting R functions when they hang in
> general? In Matlab the answer is (infuriatingly) in many cases that it
> has to be implemented by whoever wrote the c/c++ function that's
> interfacing with the environment. It's annoying, but at some point we
> should probably look into how to make Stan interruptible in R.
>
> Matt
>

[Andrew Gelman]

I have no strong feelings about this interrupt-thing. I mean, sure, it would be nice, but no need to kill ourselves to impelent it. When I run stan, I always start with 10 iterations and then go up from there. As long as the program doesn't "hang," I think we'll be fine.

[Ben Goodrich]

On Friday, August 10, 2012 5:37:14 PM UTC-4, Bob Carpenter wrote:

Interesting.  But complex from an implementation point
of view because we want to interrupt a loop within Stan,
but don't want to have to depend on R to do it.

I was just thinking that in RStan stan.parse() would add

#include <R_ext/Utils.h>

to the top of the C++ file and add

void R_CheckUserInterrupt(void);

somewhere useful in the C++ file.

But we could always do something more complicated, along the lines of what you suggest.

Ben

[Bob Carpenter]

If we want to make it interruptible, then we
need to put it right into the deeper loops. Those
aren't in the model file, but in the samplers.

Although we could put the interrupt inside a model,
it seems more natural to put it on the samplers.
So it'd have to go in as a callback from the command,
with different values passed in for command-line, R,
etc.

- Bob

[Ben Goodrich]

If we want to be able to interrupt during leapfrogging for example, then yes the check for an interrupt would need to go deeper. But I don't think interrupts are intended to make it possible to escape from an infinite loop; an infinite loop shouldn't be possible to fall into in the first place. I see interrupts as a solution for problem of "my model is taking too long and I don't care enough anymore", in which case it is probably okay to allow interrupting every sampled iteration.

Ben

[Bob Carpenter]

On 8/10/12 6:24 PM, Ben Goodrich wrote:
> If we want to be able to interrupt during leapfrogging for example, then yes the check for an interrupt would need to go
> deeper. But I don't think interrupts are intended to make it possible to escape from an infinite loop; an infinite loop
> shouldn't be possible to fall into in the first place. I see interrupts as a solution for problem of "my model is taking
> too long and I don't care enough anymore", in which case it is probably okay to allow interrupting every sampled iteration.

That we can do from the outside in the commands
either from R or from C++. So it might make sense
to at least do that to start in the RStan code.

I don't even know how to do it in C++ directly --
presumably there's an interrupt flag somewhere and
some way to test for it.

An infinite loop is always possible WITHIN
an iteration. In Stan:

for (n in 1:N) {
a <- 2 * 2;
n <- n - 1;
}

Checking at that level of granularity would require
threading interrupts through the model itself. Yuck.

Even if we forbid loop variables from being assigned,
as soon as we allow more general control structures
we'll be in trouble.

- Bob

[Andrew Gelman]

That's ok, if there's an infinite loop people can just colse the process. This shoudn't happen very often!

[Ben Goodrich]

On Friday, August 10, 2012 6:51:37 PM UTC-4, Bob Carpenter wrote:

An infinite loop is always possible WITHIN
an iteration.  In Stan:

    for (n in 1:N) {
        a <- 2 * 2;
        n <- n - 1;
    }

Checking at that level of granularity would require
threading interrupts through the model itself.  Yuck.

Even if we forbid loop variables from being assigned,
as soon as we allow more general control structures
we'll be in trouble.

 
Yeah, that would be a pain to handle in general. Maybe the parser could spit a warning or something. But my main point is that interrupts are not the solution to faulty code. They are the solution to impatience with the presumed correct code.

Ben

[Bob Carpenter]

It'd be nice if we could have both so that
a faulty model doesn't cause us to have to shut
down the R process (and lose all the work in it).

But I just don't see how to do that with any efficiency
when we compile user code.

The within-leapfrog vs. within-sample is just a granularity
issue, though one with tradeoffs among ease of coding,
efficiency, and immediate interruptibility.

- Bob

[Ben Goodrich]

On Friday, August 10, 2012 6:24:30 PM UTC-4, Ben Goodrich wrote:

If we want to be able to interrupt during leapfrogging for example, then yes the check for an interrupt would need to go deeper. But I don't think interrupts are intended to make it possible to escape from an infinite loop; an infinite loop shouldn't be possible to fall into in the first place. I see interrupts as a solution for problem of "my model is taking too long and I don't care enough anymore", in which case it is probably okay to allow interrupting every sampled iteration.

To this end, it would be more useful if stan() would still return some output when interrupted, which is a bit tricky. Inside a potentially long-running function, you can do something like

interrupted <- TRUE
on.exit( if(interrupted) do_something() )
output <- run_model()
interrupted <- FALSE
return(output)

That way if the user hits the interrupt signal while run_model() is running, whatever the expression after if(interrupted) gets executed. For example, this could assign whatever posterior samples we have to the global environment with an explanatory message, e.g.

do_something <- function() {
  half_posterior <<- # whatever
  warning("You interrupted Stan before he was done. The results have been placed in the 'half_posterior' object in the global environment. Interpret with caution, if at all")
  return(invisible(NULL))
}

And if the user does not hit the interrupt signal while run_model() is running, then interrupted gets set to FALSE, in which case do_something() is not called when stan() exits.

Ben

[Andrew Gelman]

I don't thiink this should be such a common problem!  People are not supposed to start with 100,000 iterations.  They should work up to that.

If youall can implement all this interrupt stuff easily, then great, but I don't think this should be a high or even moderate priority.

A

[Jiqiang Guo]

Thanks to Ben.  For the time being, I would just do this so that it could be interrupted and let us keep it stupidly simple and focus on something more important. 

--

Jiqiang 

[Bob Carpenter]

On 8/10/12 9:42 PM, Jiqiang Guo wrote:
> Thanks to Ben. For the time being, I would just do this so that it could be interrupted and let us keep it stupidly
> simple and focus on something more important.

That's not stupidly simple, it's the
right thing to do!

I actually think well-behaved interrupts are important
if you want the software to be usable. This is
especially true for us now that we're running in
the same process as R -- if you had to kill the
WinBUGS process, at least you didn't kill all of R
with it!

- Bob

[Matt Hoffman]

I agree. Not being able to interrupt a process without killing the
environment it's running in can be infuriating.

<Possibly boring examples of Matlab being infuriating>

For example, in Matlab until recently it was impossible to interrupt
audio playback, so if you accidentally forgot to specify the sampling
rate and typed
sound(signal)
instead of
sound(signal, 44100)
then you'd have to listen to your signal at the default sampling rate
of 8000 Hz, e.g., your 10-second sound file would be pitch-shifted
down by about 2.5 octaves and play for about a minute.

I still often have to quit Matlab when I accidentally type
x = zeros(100000);
(which tries to allocate a 100000-by-100000 matrix) instead of
x = zeros(100000, 1);
(which produces a 100000-by-1 vector).

</Possibly boring examples of Matlab being infuriating>

Anyway, the point is people will feel more positively about Stan if it
doesn't force them to restart R and throw away any valuable state that
they may have in their workspace.

Matt