Why the running time of the biogeme code bocame so so so slow, especially for using Monte-Carlo

[chao chen]

Two years ago, i have used the Biogeme for estimating mixed logit model, and the running time is fine. Recently i have reused the Biogeme, there are many updates on the codes, and i have to rewrite a new code for estimate mixed logit model. 

But, i do not know why the running time is quite long even if only one parameter is used Monte-Carlo simulation. 

I am wondering how can i speed up the algorithm. Am i right to use the following sentences to change estimate algorithm？

the_biogeme = bio.BIOGEME(database, logprob)
the_biogeme.algorithm_name = 'simple_bounds_BFGS'
the_biogeme.number_of_draws=R
the_biogeme.modelName = 'Normal_MixtureWithSocio_AllPopulation_OnlyRandom'

i am quite upset with the function of biogeme.toml, can i directly change the code in this file?

Thank you!!

[antare...@gmail.com]

Hi Chao Chen,

What I did to specify those parameters was to create a function whose arguments are the parameters values that I needed, and its output is the text for the .toml file:

def createBioTomlFile(draws=200, threds=8):
    return f"""
[Specification]
missing_data = 99999
[TrustRegion]
dogleg = "True"
[Estimation]
bootstrap_samples = 100
max_number_parameters_to_report = 15
save_iterations = "False"
maximum_number_catalog_expressions = 100
optimization_algorithm = "simple_bounds"
[MonteCarlo]
number_of_draws = {draws}
[SimpleBounds]
second_derivatives = 1.0
tolerance = 6.06273418136464e-06
max_iterations = 100
infeasible_cg = "False"
initial_radius = 1
steptol = 1e-05
enlarging_factor = 10
[AssistedSpecification]
maximum_number_parameters = 50
number_of_neighbors = 20
largest_neighborhood = 20
maximum_attempts = 100
[MultiThreading]
number_of_threads = {threds}
[Output]
identification_threshold = 1e-05
only_robust_stats = "False"
generate_html = "True"
generate_pickle = "True"
"""


Then I saved the file as "myfile.toml" before creating the BIOGEME object and passed to this object the path of the .toml file:

# TOML FILE
filename = os.path.join(os.path.dirname(__file__), "myfile.toml")
with open(filename, "w") as myfile:
myfile.write(createBioTomlFile(draws, threds))

# ESTIMATE
myBIO = bio.BIOGEME(bioDB, logprob, parameter_file=filename)


This way you can create you own .toml file with parameters values tailored to your needs. Hope this is useful to you.

Regards,
Jorge

[Michel Bierlaire]

> On 8 Apr 2024, at 10:23, chao chen <chenc...@gmail.com> wrote:
>
> Two years ago, i have used the Biogeme for estimating mixed logit model, and the running time is fine. Recently i have reused the Biogeme, there are many updates on the codes, and i have to rewrite a new code for estimate mixed logit model.
> But, i do not know why the running time is quite long even if only one parameter is used Monte-Carlo simulation.

It is because the default values have changed. The default number of draws is 20000.
Also, the default algorithm calculates the second derivatives, which can be slow for mixtures.

> I am wondering how can i speed up the algorithm. Am i right to use the following sentences to change estimate algorithm？
>
> the_biogeme = bio.BIOGEME(database, logprob)
> the_biogeme.algorithm_name = 'simple_bounds_BFGS'
> the_biogeme.number_of_draws=R
> the_biogeme.modelName = 'Normal_MixtureWithSocio_AllPopulation_OnlyRandom'
>
> i am quite upset with the function of biogeme.toml, can i directly change the code in this file?

For many users, the use of the TOML file was a big simplification.
Some parameters can be changed once the biogeme object has been created, like the algorithm name:

biogeme.algorithm_name = 'simple_bounds_BFGS'

However, for the number of draws, it is needed by the constructor and cannot be changed afterwards.

If you want to control it from the code, you can use the class Parameters in the module biogeme.parameters
https://biogeme.epfl.ch/sphinx/parameters.html#parameters
It allows you to set the values of the parameters, and to generate the TOML file.




>
> Thank you!!
>
> --
> You received this message because you are subscribed to the Google Groups "Biogeme" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to biogeme+u...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/biogeme/cbced54f-856b-43a6-8b87-85dcd14a250an%40googlegroups.com.

Michel Bierlaire
Transport and Mobility Laboratory
School of Architecture, Civil and Environmental Engineering
EPFL - Ecole Polytechnique Fédérale de Lausanne
http://transp-or.epfl.ch
http://people.epfl.ch/michel.bierlaire

[chao chen]

Dear Professor Michel Bierlaire and Jorge,

Thanks for your help, it helps me a lot indeed. But I still have one question on how to cancel the calculation of the second derivatives because even I used the method you provided, the calculation time is still quite long. Thank you!

Best regards,

Chao

[Michel Bierlaire]

In the biogeme.toml file, set the following parameter to zero
second_derivatives = 0.0

Note that, if you set it to 0.1, say, it will calculate the second derivatives only every 10 iterations. For some models, it may considerably speeds up the convergence of the algorithm at a relatively low computational cost. To be experimented...

> On 17 Apr 2024, at 01:25, chao chen <chenc...@gmail.com> wrote:
>
> Dear Professor Michel Bierlaire and Jorge,
>
> Thanks for your help, it helps me a lot indeed. But I still have one question on how to cancel the calculation of the second derivatives because even I used the method you provided, the calculation time is still quite long. Thank you!
>
> Best regards,
> Chao

[chao chen]

Dear Jorge,

Thanks for your help, it helps me a lot indeed. I still have a question on how to cancel the calculation of the second derivatives because even i used the method you provided, the calculation time is still quite long. Thank you!

Best regards,

Chao