Multinomial model for large samples

[Luiz Max Carvalho]

Hi there,

Imagine I have N (discrete) observations from n individuals. Each observation X_i is a vector {X_1i, X_2i, ..., X_k}, where sum(X_i) = n. The problem I have is: N is in the millions, and since we can only have n*(n +1)/2 different values for X, this means we'll have many "repeated" observations, i.e, each X_j (j  =1, 2, n*(n +1)/2) will appear f_j times, sum(f_j) = N. So far, I've been downsampling the data proportional to f_j, but I'd like to use all of the data.

How can I include the frequencies in the multinomial likelihood in stan? If I were doing this outside of stan I'd just sum the log of each frequency f_j to the (multinomial) log-likelihood of X_j. Is there a way of incrementing l_p to achieve this?

Thanks in advance,

Luiz

[Ben Goodrich]

In this case, you can do

for (j in 1:J) target += f_j * multinomial_lpmf(...);

Since the duplicative observations are actually observed, this is okay, as opposed to the situation where the f_j are estimates of how many people in the population are in stratum j.

Ben

 

[Luiz Max Carvalho]

Thanks so much, Ben!

Cheers,

Luiz