weights to models ?

[Shantanu K. Karve]

I want to do a weighted model in Stan - in the very old days, you just duplicated, triplicated etc, the samples that you regarded as more important( like recent samples versus older samples, or samples where the target was known to be less noisy that others ). Now R's lm  lets you provide a weights vector. 

For Stan after reading the discussion in the dev-group  - https://groups.google.com/forum/#!topic/stan-dev/5pJdH72hoM8

I tried this - 

      increment_log_prob(wtsnormedLogged[i]*normal_log(sess[i],(  p[i]  + weekly_tv*WEEKLY_TV_SPEND[i] + weekly_rd*WEEKLY_RD_SPEND[i] + tv_coeff*tv_spend_prime[i]

                                   + rd_coeff*rd_spend_prime[i]

                                   + br_coeff*SEM_BRAND_SPEND[i]

                                   + nb_coeff*SEM_NB_SPEND[i] ),sigma_sess));

And for the weights  I provided this via R ( code below is from R ) -  Since stan works in the logged space I thought I shd log the weights vector, in addition to normalising it to 1. I got Rhats of 10**5, 10**7 and so on !

Shdn't I be logging the wts vector then ? 

============= R code that creates the weighting vector

wts<-vector(length=(nrow(dms)))

wts<-c(rep(1,N_dates))

wts[as.Date(dms$Date)>as.Date('2014-12-08') & as.Date(dms$Date)<=as.Date('2015-01-05')]<-.0001

wts[as.Date(dms$Date)==as.Date('2015-12-25')]<-.00001

wts[dms$Year>=2016]<-30

total_wt<-sum(wts)

wtsnormed<-wts/total_wt

wtsnormedLogged<-log(wtsnormed)

[Bob Carpenter]

No, definitely not. Think about what happens when you
have two observations. You multiply their likelihoods
on the linear scale, so you add them on the log scale,
so that means you multiply the log likelihood by the weight
(which would be 2 in this case).

The new notation is

target += u;

instead of

increment_log_prob(u);

and normal_lpdf(y | mu, sigma) instead of normal_log(y, mu, sigma).

You should be using Stan 2.11 because there was a bug in 2.10
that got fixed (earlier releases are OK, but they haven't
been upgraded to the new notations that we're using going forward).

Having said all this, Andrew doesn't like the weighting because
it's not a generative model, but what you do with Stan is up to you!

- Bob

P.S. By "samples" you mean what we'd call observations
or modeled data (what is modeled by the likelihood or
sampling distribution---this terminology is all super confusing).

> --
> You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.
> To post to this group, send email to stan-...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Shantanu K. Karve]

Yes, I'm using v2.11.0

Thanks. I need to mull on that to totally grok it.  Yes I'll switch to the target and normal_lpdf notations - oddly when I first had a shot at this, I kept getting parser expects "(" errors at compilation for many permutations of target, normal_log, increment_log_prob, normal_lpdf, and using | ( conditional) rather than a , inside the distribution function, putting spaces, taking out spaces after the key-words etc in my construct.  Then somehow the variant I'm using worked( albeit I get deprecated warnings ) and I stuck with it.

Yup I saw that Andrew doesn't like weighting. I'll have to look up the term "generative model" to see what he's getting at ! 

[Shantanu K. Karve]

Ahh yes by samples I meant the original observations dataset.. - with the target, the y, being the set of web-sessions per day and the predictors, the features being the advertising  and marketing spends spends that day ( and previous days )

On Tuesday, August 30, 2016 at 9:54:21 AM UTC-7, Bob Carpenter wrote:

[Jonah Gabry]

On Tuesday, August 30, 2016 at 1:15:05 PM UTC-4, Shantanu K. Karve wrote:

Yup I saw that Andrew doesn't like weighting. I'll have to look up the term "generative model" to see what he's getting at  

 

The idea is that the weights don't actually factor in to the data generating process (how the data came to be, so to speak). If you have a sample that you know isn't representative of the population then you can still use a generative model by conditioning on the relevant variables (i.e. the variables that would go into the weights). 

[Shantanu K. Karve]

Hmm, yes, instead of just mechanically translating the existing lm way of doing things, going back to first principles and doing it the Bayesian/stan way is better. In this case I do know what happened to the data that I underweight dramatically -  two Google analytics segments got mixed up. For overweighting more recent observations, I can condition on calendar time of course but not quite sure yet how to incorporate the "more representative  of the actual population" idea into that.  I'll think on it.  

[Bob Carpenter]

The other alternative is to fit a time series model
if you have some quantity evolving over time.

- Bob

[Shantanu K. Karve]

returning to this - the thing about a Auto regressive model of some type i.e.

y(i) = f(  y(i-1), y(i-2)...  ) 

 where f is some linear function, is that there's nothing about regressors in there - it just says that my target/predicted variable depends on past values of itself.. 

I have regressors/ predictors so my equation is :

y(i)  = f(   g(x(i), x (i-1), x(i-2 )...   ) etc.. there's multiple predictors - advertising in print, radio, tv, digital etc.

 In this domain its called Adstock and saturation. The g function includes both a decay ( x'(i) = x(i) + lamba * x'(-1) )  and a saturation function which can be a power of less than 1 or a neg-exponential or logistic... Not sure if using such "moving average" of the predictors breaks any assumptions in stan..  

For some g's - I get good fit, good n_eff, Rhat, plots are fine. 

For others - which in business senses seem MORE sensible, I get really bad n_eff, Rhats and I would love to understand why. I can't get anything out of the Pairs plots yet. 

[Bob Carpenter]

> On Sep 15, 2016, at 2:27 PM, Shantanu K. Karve <shantan...@gmail.com> wrote:
>
> returning to this - the thing about a Auto regressive model of some type i.e.
>
> y(i) = f( y(i-1), y(i-2)... )
> where f is some linear function, is that there's nothing about regressors in there - it just says that my target/predicted variable depends on past values of itself..
>
> I have regressors/ predictors so my equation is :
> y(i) = f( g(x(i), x (i-1), x(i-2 )... ) etc.. there's multiple predictors - advertising in print, radio, tv, digital etc.
>
> In this domain its called Adstock and saturation. The g function includes both a decay ( x'(i) = x(i) + lamba * x'(-1) ) and a saturation function which can be a power of less than 1 or a neg-exponential or logistic... Not sure if using such "moving average" of the predictors breaks any assumptions in stan..

The only assumptions we make are

1. that the program defines a proper density up to
an additive constant (i.e., the exp() of the log density
has a finite integral), and

2. the density is continuously differentiable.

Stan can run into sampling problems with too-fat tails, too-thin tails, or
extreme position-dependent curvature in the posterior.

Time series can have strong dependencies among the parameters,
which can sometimes be tackled by reparameterizing.

E.g., rather than

theta[t+1] ~ normal(theta[t], sigma);

you can use a non-centered parameterization with a parameter
theta_raw and transformed parameter theta, with


theta[t + 1] = theta[t] + sigma * theta_raw[t+1];

theta_raw[t + 1] ~ normal(0, 1);

The sampling will happen over the theta_raw, but you'll still
recover your theta.

- Bob