Dealing with partially observed ranked observations

[Julian King]

This is more of a statistics question than a Stan question per se, although ultimately my model will be done in JAGS or Stan and so this actually goes to the algorithm/code to be written.

Problem setup

Suppose I am interviewing N candidates for job offers.  I have Nstaff staff members who interview these candidates.  Each candidate is not seen by each staff member.  That is, Nobs[1:N] is the number of times a candidate is interviewed by each of the [1:Nstaff] staff members, and Nobs[i] < Nstaff.

In fact, each staff member typically only interviews a relatively small fraction of the candidates (maybe less than half).  You ask the staff members to rank the candidates from 1 (best) to X, where X is the number of people they saw.

The question is: what is the best ordering of candidates, from 1 to N?

Approach + the issue

If each staff member interviewed all candidates then this problem is solved by the Thurstonian model - see http://en.wikipedia.org/wiki/Thurstonian_model.  However the issue I have here is that each staff member does not see all candidates. 

For example, suppose there are 20 candidates, and one staff member sees 7 candidates.  You can assume that the quality of the candidates is not correlated with the particular staff member.  The 7 candidates the staff member may be bunched towards the top, so the "1,2,3,4,5,6,7" rankings may be nearer the top end.  But they could also be evenly spread, or they could be near the bottom.

How can I take the ranks of each staff member ([1:Nobs[i]]) and map them to the [1:N] scale.  Does this require some sort of cumulative link ordered response model?

Many thanks

[Bob Carpenter]

There's a huge literature on voting that I don't know, and
it depends what you mean by "best" ordering. Not every way
of sorting is going to give you the same answer.

Partial observations don't prevent you from using the
Thurstonian model. Each rater has an underlying position
for each object they rate, based on "measurement error" from
the true value, as usual, and that gives you the observed ranking,
as usual. The only difference is that not every rater ranks
every item.

Stan and JAGS would code these differently. In Stan, each rater
would get an ordered vector based on their number of observations.
In JAGS, you need to do the Chib/Albert-style truncation trick.

I would, perhaps not surprisingly, recommend building a hierarchical
model for the noise. And you have to be careful with identifiability
of the underlying true scalars, because there's both an additive
and multiplicative non-identifiability involved.

- Bob

> --
> You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Bob Carpenter]

P.S. If you need more help formulating the Thurstonian model,
I'm glad to help more. It's going to be a bit tricky in Stan.
One thing you can do is give each rater an underlying rating
of the full size, with no information provided about the items
they don't rank, but that'll be prohibitive if their ratings
are very sparse.

- Bob

On Aug 7, 2014, at 7:26 PM, Julian King <jking...@gmail.com> wrote:

>

[Julian King]

Bob - thanks for the rapid reply.

There is a good article at http://www.ncbi.nlm.nih.gov/pubmed/23539504 which has an attachment (http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4015936/bin/pone.0096431.s001.pdf) showing JAGS code for a Thurstonian model.

I understand how this model works for the fully observed case.  But how do you treat the case where some people are not observed.

In line with that article, assume that the latent allocation for each item (in this context, a person) is mu[i].  The model assumes that each jth person generates a mental sample as x[i,j] ~ normal(mu[i], sigma[j]).   Here, i is in [1,N] and j is in [1:Nstaff].  This is then compared to actual rankings for each person, y[1:N, 1:Nstaff].  How do you deal with the fact that some staff members are not observed?  For the jth staff member, the y vector has a N rows.  In my example, each person only provides X[j] < N rows.   You can certainly put the [1:X[j]] rankings in y[1:N,j] in the correct positions, and (in JAGS) put NA values in the other positions.  But these rankings will max out at X[j] < N.  Doesn't this screw the algorithm up?

Thanks

[Bob Carpenter]

On Aug 7, 2014, at 9:26 PM, Julian King <jking...@gmail.com> wrote:

> Bob - thanks for the rapid reply.
>
> There is a good article at http://www.ncbi.nlm.nih.gov/pubmed/23539504 which has an attachment (http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4015936/bin/pone.0096431.s001.pdf) showing JAGS code for a Thurstonian model.
>
> I understand how this model works for the fully observed case. But how do you treat the case where some people are not observed.
>
> In line with that article, assume that the latent allocation for each item (in this context, a person) is mu[i]. The model assumes that each jth person generates a mental sample as x[i,j] ~ normal(mu[i], sigma[j]).

That makes sense other than that I wouldn't use the variable "x" for
a parameter --- it violate's Andrew's conventions that "x" is for unmodeled
data, "y" for data, and greek letters for parameters.

Then you need to fix the location and scale of mu for identifiability --- the
model you sent uses normal(0,0.001) in JAGS, which is equivalent to normal(0,sqrt(1000))
in Stan. I'd just use normal(0,1) to make sure the scale of sigma lined up properly.
the numbers easier to scan --- the scale doesn't matter (except in its interaction
with the scale of sigma).

I'd also be inclined to build a hierarchical model for the sigma[j], but that's
another issue.

> Here, i is in [1,N] and j is in [1:Nstaff]. This is then compared to actual rankings for each person, y[1:N, 1:Nstaff]. How do you deal with the fact that some staff members are not observed? For the jth staff member, the y vector has a N rows. In my example, each person only provides X[j] < N rows. You can certainly put the [1:X[j]] rankings in y[1:N,j] in the correct positions, and (in JAGS) put NA values in the other positions. But these rankings will max out at X[j] < N. Doesn't this screw the algorithm up?

The model you sent me is wrong in the text that the rank needs to be something
like

y[j,] <- rank(x[j,])

You can't call rank(x[j,i]) because x[j,i] is a scalar!

I also don't think their JAGS code is going to be efficient because
they're using the dinterval trick for rejection, when they could just
use truncation a la Albert-Chib.

All you need to do for your model where not every individual i
is observed is to trim the implicit rankings down to the ones you observe:

y[j,] <- trim(y[j,],rank(x[j,]));

where the "trim" function just chops the result down to the individuals
for which you have observations, which are just the ones in y[j,].
(I know "trim" isn't a built-in function in anyone's system.)

The hard part is going to be coding it in either Stan or JAGS.
If all the y[i,] are the same size, it's easy. If everything was observed
in Stan, this model's trivial to code:

data {
int<lower=0> I; // number of items being ranked
int<lower=0> J; // number of raters
int<lower=1,upper=I> y[J,I]; // ranks for rater j
}
parameters {
real mu[I]; // true latent positions
real<lower=0,upper=20> sigma[J]; // accuracy of rater j
ordered[I] x[J]; // positions assigned by rater j
}
model {
mu ~ normal(0,1000);
for (j in 1:J)
for (i in 1:I)
x[j,i] ~ normal(mu[i],sigma[j]);
}

The ordered[I] data type is for I ordered values. This is
the appropriate constraint on the underlying values (though it
makes lower better --- you'd have to reverse to get it the other
way around because ordered sorts in ascending order).

I think this is much simpler than you see in the JAGS encoding in the
referenced PDF (though as I said, I don't think this is the best JAGS
encoding --- the alternative would involve truncated normal sampling
statements in JAGS).

If each rater rates the same number of items, say K, then all you
need to do to code in Stan is to add a variable for K, replace y[J,I] with
y[J,K] and ordered[I] with ordered[K] in the declarations and then loop
from 1:K rather than 1:I in the model. The truncations you'd need in
JAGS (or rejections the way the example was coded in the PDF) automatically
get handled by the ordered vector data type.

If not every rater rates the same number of items, then the model's
going to be difficult to code because neither Stan nor JAGS supports
ragged arrays. And Stan doesn't support ragged constraints. So you'd
need to either (a) break the data down ahead of time into groups with
the same number of observations, then proceed as above [although it's
hard to make general], or (b) bite the bullet and code up ragged data
structures (using the trick described in the manual for LDA or naive
Bayes) and define the transforms and Jacobians yourself (they're simple
for the ordered vectors --- see the last chapter of the manual on
variable transforms).

- Bob