Assisted specification: segmentation with NAs, difficulties to converge. Fix to 0?

[Antonin Danalet]

Dear Michel, dear all,

I'm using the assisted specification module. I define a segmentation, let's say by mobility tool: 

segmentation_pt_subscription = database.generate_segmentation(
        variable="has_pt_subscr",
        mapping={2: "no_pt_subscr", 1: "pt_subscr"},
        reference="no_pt_subscr",
    )

Then I use it as in ASC_catalog = segmentation_catalogs(... segmentation_pt_subscription ...), V = ASC_catalog + ...

Let's say my variable has_pt_subscr contains 5 NA/-99. So it generates an ASC_-99. It is fine to estimate it, it's even a good practice, but with a very low number of NAs, the algorithm fails to converge and the maximum number of iterations is very quickly reached for a complex, real-life model.

Is there a way / a trick to fix ASC_-99 to zero?

(The trick being obviously to just remove these observations, it also works quite well, I just want to do things in a clean way.)

It might be a nice extension of the generate_segmentation. Or do you think it's not worth it and the changes would not be important enough?

Antonin

[Michel Bierlaire]

Dear Antonin,

This is a very good suggestion. The next Biogeme release, coming soon, will provide a generic way to replace automatically generated parameters by any valid Biogeme expression.

I also recommend using a category name such as minus_99, rather than including the - character in generated parameter names:

segmentation_pt_subscription = database.generate_segmentation(
variable="has_pt_subscr",
mapping={
2: "no_pt_subscr",
1: "pt_subscr",

-99: "minus_99",

},
reference="no_pt_subscr",
)

After constructing the complete likelihood, but before creating the BIOGEME object, the following code will fix the corresponding parameters to zero:

from biogeme.expressions import (
Numeric,
ParameterOverrides,
apply_parameter_overrides,
list_of_all_betas_in_expression,
)

# Construct the utilities and likelihood as usual

V = ASC_catalog + ...

log_probability = ...

missing_category_parameters = {
beta.name
for beta in list_of_all_betas_in_expression(log_probability)
if beta.name.endswith("_has_pt_subscr_minus_99")
}

overrides = ParameterOverrides()

for parameter_name in missing_category_parameters:
overrides.set(parameter_name, Numeric(0))

# In the next release, this will return the likelihood
# with the selected parameters replaced by Numeric(0).
log_probability = apply_parameter_overrides(log_probability, overrides)

biogeme = BIOGEME(database, log_probability)

For example, the generated parameter names may be:

asc_train_diff_has_pt_subscr_minus_99
asc_car_diff_has_pt_subscr_minus_99

Does it correspond to what you had in mind?

Best regards,

Michel

> --
> You received this message because you are subscribed to the Google Groups "Biogeme" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to biogeme+u...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/biogeme/cd7bb9f5-1a87-4489-8f83-56b6fc30d4can%40googlegroups.com.

Michel Bierlaire
Transport and Mobility Laboratory
School of Architecture, Civil and Environmental Engineering
EPFL - Ecole Polytechnique Fédérale de Lausanne
http://transp-or.epfl.ch
http://people.epfl.ch/michel.bierlaire