Lagrange Multipliers for contraints?

[andre.p...@googlemail.com]

Hi, 

I read various post about contraints, and come up to the idea to 

use Lagrange Multipliers in STAN. See example 1 in:

https://en.wikipedia.org/wiki/Lagrange_multiplier

I came up with the following basic model:

data {

}

parameters {

  real x;

  real y;

  real la;

}

model {

 // x  ~ normal(0,100);

//  y  ~ normal(0,100);

//  la ~ normal(0,100);

 

  increment_log_prob(-square(x+y+la*(square(x)+square(y)-1)));

  //increment_log_prob(-0.5*square(1+2*la*x));

  //increment_log_prob(-0.5*square(1+2*la*y));

//  increment_log_prob(-0.5*square(square(x)+square(y)-1));

}

I go several warnings:

Warning messages:
1: There were 27 divergent transitions after warmup. Increasing adapt_delta may help. 
2: Examine the pairs() plot to diagnose sampling problems
 
> x<-mean(extract(fit,"x")$x)
> y<-mean(extract(fit,"y")$y)
> la<-mean(extract(fit,"la")$la)
> la
[1] 0.609033
> x
[1] -0.0002591018
> y
[1] 0.06798691

The solution is not correct. Well, I didn't expected that it would work.

STAN makes use of autodiff, thus I would hope someone has an idea. 

André

[Tamas Papp]

I don't understand

1. what the inference problem is (the wikipedia example is an
optimization problem),

2. how you mean to apply Lagrange multipliers for your problem (once you
describe it to us),

3. what you are trying to do in Stan.

What you have below is a function that is quadratic in x and y and
linear in la. This is not a proper posterior, as it is not even
bounded, so Stan of course cannot sample from it --- no algorithm could,
as it is not a density of a distribution.

Best,

Tamas

[Bob Carpenter]

You can use Stan for optimization, but I don't see how
you want to use Lagrange multipliers. As Tamas said,
what you have doesn't define a density, so Stan can't
sample from it.

- Bob

> --
> You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.
> To post to this group, send email to stan-...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[andre.p...@googlemail.com]

Tam, Bob, everybody,

I was into coding a bayesian support vector machine (bSVM) in STAN. In their dual form

in common to use lagrange multiplier to cope with the inequality condition. 

I looked in the group here and thought maybe somebody else know how to handle

contraints. I found:

https://groups.google.com/d/msg/stan-users/CFbIiXAzHl0/mSRKER0WNcoJ

So I tried to make it very simple and put the lagrange example in the STAN optimizer. 

You all right. I doesn't makes sense. 

Just for informational background:

 

I have had the idea to use STAN as an Bayesian optimizer. 

Currently we do optimizing like an optimizing tunnel. 

We pick on point, optimize and get one result.  

We do that repeatedly. At the end of the other side

we get a 'picture'.

But what, if we could transmit whole distributions 

through the "tunnel"? 

Our solver have constraints. So how could I had constraints

to STAN. Thus I come up with the Lagrange multiplier idea. 

To make it more practical, in the company I worked before,

we had a simulation of the factory. We did a simulation,

picked one state out of the simulation process, optimized

that. Repeated that n-times and then at the end we 

reviewed the optimized points.

Andre

[Bob Carpenter]

I don't know much about either Lagrange multipliers or Bayesian
optimization. Stan is only able to do MCMC, variational inference,
and optimization.

Constraints on variables need to be implemented in Stan through transfroms
from R^N to a suitable natural space for modeling (usually a constrained
portion of R^M, for M >= N). An example is mapping any point in R^N to a
simplex in R^(N+1) using softmax (we use a stick-breaking construction, but
the idea's the same).

- Bob

[Daniel Lakeland]

On Friday, October 2, 2015 at 1:41:03 PM UTC-7, andre.p...@googlemail.com wrote:

Hi, 

I read various post about contraints, and come up to the idea to 

use Lagrange Multipliers in STAN. See example 1 in:

https://en.wikipedia.org/wiki/Lagrange_multiplier

Lagrange multipliers are more or less an analytic technique (ie. if you are trying to solve analytically for locations where derivatives vanish), they aren't necessarily appropriate directly for numerical usage.  A related technique is "penalty" methods. If you are willing to put up with "soft" constraints g(x) \approx 0, you can take f(x) + K*(g(x))^2. The contribution from g(x) is obviously is minimized when g(x) = 0. When K goes to infinity the solution to this penalized minimization procedure is equal to the constrained minimum of f(x) subject to g(x)=0. But this is not always a practically effective method.