MNL model using panel data and weights

[Lotte Notelaers]

Dear prof.,

I am trying to estimate a MNL model (later I want to extend to nested logit) using panel data with weights. In one of the previous 'discussions' I read that the database needs to be flattened when working with panel data with weights. I think I was able to do this correctly. Because not all respondents answered the same number of experiments, the flattened dataset includes columns indicating the availibility of the modes per choice experiment (e.g. 1_avail_BIKE, 1_avail_CAR, 2_avail_BIKE, 2_avail_CAR).

However, when calculating the null log likelihood, I get an error that the availabilities are specified wrongly. 

Moreover, estimating the model results in all beta values being 0 (or staying at the initial value).

I am still using version 3.2.10.

My code is attached. 

Can you point me to an example that shows how to specify the availabilities and the loglikelihood? Or can you explain how to do this?

Thanks a lot!

Lotte

[Michel Bierlaire]

> On 24 Feb 2025, at 16:43, Lotte Notelaers <lo...@wearenutz.com> wrote:
>
> Dear prof.,
>
> I am trying to estimate a MNL model (later I want to extend to nested logit) using panel data with weights.

> In one of the previous 'discussions' I read that the database needs to be flattened when working with panel data with weights.

Not if you do not involve any random parameter.

> I think I was able to do this correctly. Because not all respondents answered the same number of experiments, the flattened dataset includes columns indicating the availibility of the modes per choice experiment (e.g. 1_avail_BIKE, 1_avail_CAR, 2_avail_BIKE, 2_avail_CAR).
>
> However, when calculating the null log likelihood, I get an error that the availabilities are specified wrongly.

Because it expects a dict mapping the id of the alternatives with the availability conditions.
And you provide a dict of dicts instead.

>
> Moreover, estimating the model results in all beta values being 0 (or staying at the initial value).
>

When you define 'logprob', you forget to take the log.

condprobIndiv = exp(bioMultSum(obsprob))
logprob = log(condprobIndiv)

But, then, it makes more sense to write directly
logprob = bioMultSum(obsprob)

Actually, if you do not include an agent effect, and if you do not implement a Markov model, it is pointless to consider the panel nature of the data. Therefore, I assume that you plan to include the agent effect later on. In this case, it is indeed a good idea to start specifying a simple model on the flat database.

I suggest you to write first a model that works on the non-flat database.
Once it works, adapt it to the flat database, and verify that oyu have the samme results, up to numerical approximations.

Another hint: do not use "globals". You can replace your "globals" by Variable:

V_temp += B[nb] * Variable(f'{attribute}_{mode}')

It seems that biogeme cannot calculate the gradient at your starting point. I suggest that you use it in simulation model to investigate the value of the likelihood function at the starting point, and which data is causing the problem.

I hope this helps.

Michel

> I am still using version 3.2.10.
>
> My code is attached.
>
> Can you point me to an example that shows how to specify the availabilities and the loglikelihood? Or can you explain how to do this?
>
> Thanks a lot!
>
> Lotte
>
>

> --
> You received this message because you are subscribed to the Google Groups "Biogeme" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to biogeme+u...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/biogeme/428bdb9d-3c1f-4ac0-a06a-a7ab7b4e2c35n%40googlegroups.com.
> <LogLikelihood&Estimate.JPG><FlatteningDatabase.JPG><DefiningUtilities.JPG>

Michel Bierlaire
Transport and Mobility Laboratory
School of Architecture, Civil and Environmental Engineering
EPFL - Ecole Polytechnique Fédérale de Lausanne
http://transp-or.epfl.ch
http://people.epfl.ch/michel.bierlaire