removing rejecting warning

[Bob Carpenter]

I propose we just remove the rejection warning message
altogether in 2.0.

There are two cases where it comes up:

1. Early in adaptation and occassionally during sampling
when step sizes are too large.

2. During inititalization or during sampling when the
parameters don't have the correct constraints specified.

The warning message seems to both confuse users and undermine
their trust in their own models and Stan as a sampler. So I
think it's better if we just get rid of it.

In case (2), the fact that samples continue to be rejected
will provide some clue.

Another issue is that only some of our functions throw these
execptions --- others just return NaN silently.

-Bob

> On Aug 8, 2013, at 11:02 PM, Markian Pahuta <mark....@gmail.com> wrote:

>> The code runs, and I obtain reasonable estimates, however I get the following error

>> Informational Message: The parameter state is about to be Metropolis rejected due to the following underlying, non-fatal (really) issue (and please ignore that what comes next might say 'error'): Error in function stan::prob::lkj_cov_log(N4stan5agrad3varE): Covariance matrix is 1.#INF:0, but must be finite!
>> If the problem persists across multiple draws, you might have a problem with an initial state or a gradient somewhere.
>> If the problem does not persist, the resulting samples will still be drawn from the posterior.

>> My prior on the standard deviation seems reasonable, and eta is 1... so I am not sure why I am getting this error. Is my parameterization incorrect?

...

[Marcus Brubaker]

I'm not sure removing it entirely is a good idea.  It really shouldn't happen very often in a properly specified model except perhaps during warmup or adaptation.  Most of the time people post to the list about it, it is the result of a misspecified model making it a useful canary in the proverbial coal mine.  That said, the message does tend to be confusing to new users so maybe it's worth thinking about what we can do.

Cheers,

Marcus

--
You received this message because you are subscribed to the Google Groups "stan development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Ben Goodrich]

I still think messages like that she be off during warmup and on during post-warmup.

To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+u...@googlegroups.com.

[Marcus Brubaker]

Turning it off during warmup/adaptation is probably fine, although I've found it useful myself to detect stupid bugs in my models.

Having not looked at the code at all, how hard would it be to, by default, not display the message unless more than a certain percentage of steps get rejected with the message?  E.G., if more than 5% of steps are getting rejected, it's probably something the user should be aware of.

I am still a little concerned about what kind of pathologies or misspecified models may be masked by the combination of adaptation and something like this but I would hope nothing quite so bad would be really seen in practice.

Cheers,

Marcus

[Michael Betancourt]

I'm with Marcus -- I think the warning during warmup catches enough problems to warrant the mild frustration of seeing the messages. We just need to clean up the message itself. Here's what I put into the arg_refactor branch while cleaning up command.hpp:

*error_stream << std::endl
<< "Informational Message: The current Metropolis proposal is about to be "
<< "rejected becuase of the following issue:"
<< std::endl
<< e.what() << std::endl
<< "If this warning occurs sporadically then the sampler is fine,"
<< std::endl
<< "but if this warning occurs often then your model is either severly "
<< "ill-conditioned or even misspecified."
<< std::endl;

[Bob Carpenter]

It's not the frustration of responding to the messages
that worries me, it's that they indicate that the users
don't trust the inferences when they see this message.

What Marcus wrote looks better than what I tried to write.
(We need to switch "severly" to "severely".)

Should we hint that the problem is due to lack of constraints,
illegal initial values, or arithmetical issues?

- Bob

[Michael Betancourt]

I make no claims of decent spelling! I'll fix it and push.

I think the exception should have information specific to the exact reason for rejection. No?

[Bob Carpenter]

On 8/9/13 1:27 PM, Michael Betancourt wrote:
> I make no claims of decent spelling! I'll fix it and push.

It's hard when we're shoveling out text by the pound
and the text isn't really the focus.

> I think the exception should have information specific to the exact reason for rejection. No?

The e.what() prints the text of the exception that caused
the message. Usually a domain error on a call to something
constrained.

- Bob

[Marcus Brubaker]

The text of the exception is important information, as it (very vaguely) indicates where to start looking.  I would really love it if we could include the line number of the compiled model which produced the exception, as that would make debugging such things that much easier.  As it is right now, if you have a model with lots of calls to normal_log, it's not particularly useful to know that normal_log threw the exception.

I once tried to look into this but got lost in the parser code.  It should be possible to propagate this information when we generate the log_prob() function to make the exceptions more expressive.

Cheers,

Marcus

To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.




--
You received this message because you are subscribed to the Google Groups "stan development mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "stan development mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "stan development mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+unsubscribe@googlegroups.com.

[Michael Betancourt]

Well, I can easily add more detail about the possible reasons for failure if people think it's a good idea.