cov_exp_quad: Which kernel does this Stan function represent? SE or SE-ARD?

[Daniel Emaasit]

I would like to get some insight into which kernel this Stan function below represents

cov_exp_quad(x, signal_variance, length_scale)

Is it the standard Squared Exponential (SE) kernel shown below?

Or is it the Squared Exponential Automatic Relevance Determination (SE-ARD), below?

One of the reasons for asking is that I want to get a different length_scale parameter for each explanatory variable (x) or each dimension. So that I can test the relevance of each dimension. The current implementation of cov_exp_quad returns a single length_scale even when input x is a vector i.e. vector[D] x[N];

see code below.

data {
  int<lower=1> N;
  int<lower=1> D;
  int<lower=1> N_pred;
  vector[N] y;
  vector[D] x[N];
  vector[D] x_pred[N_pred];
}
parameters {
  real<lower=1e-12> length_scale;
  real<lower=0> alpha;
  real<lower=1e-12> sigma;
  vector[N] eta;
}
transformed parameters {
  vector[N] f;
  {
     matrix[N, N] L_cov;
     matrix[N, N] cov;
     cov = cov_exp_quad(x, alpha, length_scale);
     for (n in 1:N)
       cov[n, n] = cov[n, n] + 1e-12;
     L_cov = cholesky_decompose(cov);
     f = L_cov * eta;
  }
}
model {
  length_scale ~ student_t(4,0,1); # (df, mean, sd)
  alpha ~ normal(0, 1);
  sigma ~ normal(0, 1);
  eta ~ normal(0, 1);
  y ~ normal(f, sigma);
}

Thanks,

-- Daniel

[Aki Vehtari]

Hi Daniel,

On Sunday, March 19, 2017 at 5:26:25 AM UTC+2, Daniel Emaasit wrote:

I would like to get some insight into which kernel this Stan function below represents

cov_exp_quad(x, signal_variance, length_scale)

Is it the standard Squared Exponential (SE) kernel shown below?

Stan reference manual https://github.com/stan-dev/stan/releases/download/v2.14.0/stan-reference-2.14.0.pdf

page 458 says it's this one, that is, one length scale for all dimensions. The version where lengthscale can be a vector/array is coming.

 

One of the reasons for asking is that I want to get a different length_scale parameter for each explanatory variable (x) or each dimension. So that I can test the relevance of each dimension.

You can't use lengthscale reliably to test the relevance. The lengthscale is only weakly related to the relevance and as it's more directly connected to the non-linearity. See Figures 1 and 2 in the paper

Juho Piironen and Aki Vehtari (2016). Projection predictive input variable selection for Gaussian process models. In 2016 IEEE 26th International Workshop on Machine Learning for Signal Processing (MLSP), doi:10.1109/MLSP.2016.7738829.  arXiv preprint http://arxiv.org/abs/1510.04813.

Aki

[Álvaro M]

Hi Aki,

Sorry to go a bit off-topic in this thread but, there is any project to introduce periodic or quasi-periodic kernels in the math library?

Thanks.

[Daniel Lee]

Hi Álvaro,

We're always looking for contributors. A few things:

1. Please start a new thread

2. Functions that make it into the Stan language need the function evaluation and gradients; hopefully you know how to compute the gradients of the kernels you're interested in (that's the gradient of each element of the kernel matrix with respect to each argument of the function)

3. Some knowledge of C++ as it pertains to the math library; see https://arxiv.org/abs/1509.07164

That said, we're always happy to help. Before diving too deep, it would help to scope out what you're thinking and determine if what you're thinking is feasible. Computing gradients in a numerically stable way is a skill that you'll need if you don't have it already.

Daniel

--
You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+unsubscribe@googlegroups.com.
To post to this group, send email to stan-...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mike Lawrence]

Replying off list.  Have you tried doing it by hand I Stan first? 

--
You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.

To post to this group, send email to stan-...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

--
Mike Lawrence
Graduate Student
Department of Psychology & Neuroscience
Dalhousie University

~ Certainty is (possibly) folly ~

[Mike Lawrence]

Oops, replied in-list by mistake. Sorry  folks! 

[Álvaro M]

Hi Mike,

yes I implement a quasi-periodic kernel in Stan, and it seems to work (I don't tested it throughly), but it's kind of slow.

Sorry Daniel but it was just a question, I don't have enough knowledge of C++ or automatic differentiation  up to the task.

[Daniel Emaasit]

Thanks Aki for clarifying this for me.

[Daniel Emaasit]

On Sunday, March 19, 2017 at 4:19:10 AM UTC-7, Aki Vehtari wrote:

Hi Daniel,

On Sunday, March 19, 2017 at 5:26:25 AM UTC+2, Daniel Emaasit wrote:

I would like to get some insight into which kernel this Stan function below represents

cov_exp_quad(x, signal_variance, length_scale)

Is it the standard Squared Exponential (SE) kernel shown below?

Stan reference manual https://github.com/stan-dev/stan/releases/download/v2.14.0/stan-reference-2.14.0.pdf

page 458 says it's this one, that is, one length scale for all dimensions. The version where lengthscale can be a vector/array is coming.

Aki, could you point me to the GitHub branch for this version so I can experiment with it. 

[Aki Vehtari]

On Sunday, March 19, 2017 at 10:51:34 PM UTC+1, Daniel Emaasit wrote:

Aki, could you point me to the GitHub branch for this version so I can experiment with it. 

Oh, I don't know if it's yet there or when it's there, but I think there is high enough probability that it will be there in a finite time that I wrote "is coming".  :)

Sorry for the weak information, I guess I should in the future just write "no" and not give false hopes for you or give pressure to someone who might actually code it...

Aki