Multithreading with CBC

[Syed Taha Munir]

When i run the solver with this statement.

 

prob.solve(PULP_CBC_CMD(fracGap=0.01,threads=2,msg=1)) it always gives me that output on screen.

 

Coin0008I MODEL read with 0 errors
No match for threads - ? for list of commands
No match for 2 - ? for list of commands
ratioGap was changed from 0 to 0.01

 

Should i use the updated version of CBC ?

 

I want to use more threads for solving my problem.

[Stuart Mitchell]

CBC has to be compiled specially to support multi-threading.

If you want to use this option I suggest you compile cbc yourself and use the COIN_CMD solver.

Stu

--
You received this message because you are subscribed to the Google Groups "pulp-or-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pulp-or-discu...@googlegroups.com.
To post to this group, send email to pulp-or...@googlegroups.com.
Visit this group at http://groups.google.com/group/pulp-or-discuss.
For more options, visit https://groups.google.com/d/optout.

--

Stuart Mitchell

PhD Engineering Science

Extraordinary Freelance Programmer and Optimisation Guru

www.stuartmitchell.com

[Stuart Mitchell]

Also if you just want a speed up, try downloading the development version from github. I should be releasing this this week after I test on windows.

Stu

[Syed Taha Munir]

Hi stu

Thanks for reply

Have you used new version of CBC in the development version?

How can i compile COIN_CMD solver my self and use it with pulp, I am sure you must have answered this to someone already in the group :) , if you could just point out the discussion , it will be a favor

 

 kind regards

Taha

[Stuart Mitchell]

Are you on Windows or OSX/ Linux?

Linux is fairly straight forward, with Windows it is complicated.

Look at the CBC webpage on coin-or.org for a discussion on how to compile. Once you have done that add it to the path and COIN_CMD will just work.

Development on github uses uptodate (single threaded) cbc.

Stu

[Syed Taha Munir]

Hi Stu

 

Unluckily I am using Windows :( .

[Syed Taha Munir]

Hi Stu

i just wanted to know when are you releasing new  version of Pulp?

 

kind regards

Taha

[Stuart Mitchell]

Done it is on PyPI now :-)

Please test.

[Neil Wang]

Hi Stuart,

I am currently using CBC-2.9.9 and I would like to use multi-threading feature of in on Linux. I have recompiled with --enable-parallel argument passed in. In cbc command line, I can specify 4 threads, and it shows "change threads from 0 to 4", but when I ran the model, the system only showed 1 core working.

Could you show me a way to solve this problem, please? Thank you so much

[Stuart Mitchell]

I would email this to the cbc list if I were you.

Stu

Stuart Mitchell

PhD Engineering Science

Extraordinary Freelance Programmer and Optimisation Guru

www.stuartmitchell.com

To unsubscribe from this group and stop receiving emails from it, send an email to pulp-or-discuss+unsubscribe@googlegroups.com.
To post to this group, send email to pulp-or-discuss@googlegroups.com.

Visit this group at https://groups.google.com/group/pulp-or-discuss.

[J. Friedman]

Like this

m.solve(solvers.COIN_CMD(maxSeconds=400,threads=4,msg=1,fracGap = 0.0))

[Matthew Pearlson]

I'm glad there is some more discussion on multi-threading. I am working on a project that I think would benefit from multiprocessing.

[Russell Romney]

Thanks for your answers. I am trying to do a similar thing: I have compiled from source, used the path to the solver, and specified that threads should be used, but when I run it like above the threads are created but not used. 

This looks like:

threads was changed from 0 to 8

Cbc0012I Integer solution of -25507 found by DiveCoefficient after 0 iterations and 0 nodes (18.04 seconds)
Cbc0030I Thread 0 used 0 times,  waiting to start 0.291008, 0 cpu time, 0 locks, 0 locked, 0 waiting for locks
Cbc0030I Thread 1 used 0 times,  waiting to start 0.24997687, 0 cpu time, 0 locks, 0 locked, 0 waiting for locks
Cbc0030I Thread 2 used 0 times,  waiting to start 0.21034408, 0 cpu time, 0 locks, 0 locked, 0 waiting for locks
Cbc0030I Thread 3 used 0 times,  waiting to start 0.17122722, 0 cpu time, 0 locks, 0 locked, 0 waiting for locks
Cbc0030I Thread 4 used 0 times,  waiting to start 0.13530493, 0 cpu time, 0 locks, 0 locked, 0 waiting for locks
Cbc0030I Thread 5 used 0 times,  waiting to start 0.098966837, 0 cpu time, 0 locks, 0 locked, 0 waiting for locks
Cbc0030I Thread 6 used 0 times,  waiting to start 0.062871933, 0 cpu time, 0 locks, 0 locked, 0 waiting for locks
Cbc0030I Thread 7 used 0 times,  waiting to start 0.028151035, 0 cpu time, 0 locks, 0 locked, 0 waiting for locks
Cbc0030I Main thread 0 waiting for threads,  1 locks, 0.00077700615 locked, 9.5367432e-07 waiting for locks

Also, the build/Cbc/config.log includes a line about multithreading being enabled, and I'm using a 4-core MBP so it's not a CPU issue. Any ideas about how to solve this? @J or @stu

[Stuart Mitchell]

Try a larger problem, this example looks like it was solved by the presolve so no other threads were needed.

Stu

Stuart Mitchell

PhD Engineering Science

Extraordinary Freelance Programmer and Optimisation Guru

www.stuartmitchell.com

--
New posters to this group are moderated which can take up to 48 hours, so please be patient if your first post takes a while to turn up.
---

[David]

Hi Stuart,

Building CBC threaded as discussed above appears to thread CBC so an instance of an optimisation can be parallelised across multiple threads.

Can I parallelise at the pulp level? By this I mean instantiate multiple instances of an optimisation within a process with perhaps different coefficient values for one or more decision variables. I'll confess that while I've done a lot of work with parallelising C++ programs across multiple cores/CPUs I haven't parallelised a python application so that part is new to me. Also new to me would be parallelising pulp if that's supported.

Thanks

David

[Stuart Mitchell]

If you have a long build time there are probably ways to speed that up.

Unfortunately it is difficult to multi thread model build.

Stu

Stuart Mitchell

PhD Engineering Science

Extraordinary Freelance Programmer and Optimisation Guru

www.stuartmitchell.com

[David]

Hi Stuart,

Apologies for not being more clear. I wasn't referring to parallelising the model build specifically but rather having multiple instances of pulp optimisations running simultaneously. I think this could be done a job parallelisation by instantiating separate Python environments for each pulp optimisation but this is expensive in terms of repeated code and memory. What would be great is parallelising Pulp/CBC as per what you get using OpenMP with C++ on Linux boxes.

If CBC can be built thread-safe then this can be shared amongst multiple optimisations. Python as I understand it has a serial bottleneck at the interpreter. I noticed a multiprocessing Python capability that adds some possibly relevant  capability to Python but I need to spend some more time on that. What would be very handy is to parallelise at the pulp model level with threadsafe data and code. I've used the Intel HPC profiling and analysis toolkit quite a bit on C++ code. I think Intel are supporting Python to some extent now but I'm not sure if their thread analysis tools work on Python code.

Bye for now

David

[Stuart Mitchell]

Multiprocessing is what you want then.

Just remember that if you use multiprocessing from your script make sure to add an

#place function defs here

if __name__ == "__main__":

      #place script code here

structure to your code

Stuart Mitchell

PhD Engineering Science

Extraordinary Freelance Programmer and Optimisation Guru

www.stuartmitchell.com

[David]

Thanks, Stuart.

Has anyone had a crack at running multiple pulp sessions with the Python multiprocessing facility? Be very interested in any lessons learnt

David

[Franco Peschiera]

Ok, so even though I'm 4 years late to this thread...

In case windows users don't want to struggle with the horrible task of compiling things, the ampl site has 64-bit binaries for CBC. I've tried to use the windows one with the pulp option of threads=4 and it worked.

Also, Linux binaries are available.

the site: https://ampl.com/products/solvers/open-source/

The CBC version is 2.9.7 so not the newest.

regards,

Franco

El jueves, 26 de marzo de 2015, 14:10:11 (UTC-7), Stuart Mitchell escribió:

Are you on Windows or OSX/ Linux?

Linux is fairly straight forward, with Windows it is complicated.

Look at the CBC webpage on coin-or.org for a discussion on how to compile. Once you have done that add it to the path and COIN_CMD will just work.

Development on github uses uptodate (single threaded) cbc.

Stu

To unsubscribe from this group and stop receiving emails from it, send an email to pulp-or...@googlegroups.com.

To post to this group, send email to pulp-or...@googlegroups.com.
Visit this group at http://groups.google.com/group/pulp-or-discuss.
For more options, visit https://groups.google.com/d/optout.

[Matthew Pearlson]

Thanks Franco. I downloaded the binaries, but no joy with multithreading.

We are calling the CBC solver from a module that imported to our main python script.

Can anyone explain how to handle the if __name__ = "__main__"  in this case?

I saw a forum posts on "Singleton Command", but it wasn't clear what the syntax was.

Thanks! Matthew

[Franco Peschiera]

Hey Matthew,

I’m sorry. I thought I had it working at some point but upon other tests I was not able to replicate. I haven’t retried.

As long as you pass pulp the explicit, complete path to the good CBC executable when initializing the solver, it doesn’t care how you call pulp or from where. If you use COIN_CMD you require to give it the explicit path to CBC. In PULP_CBC_CMD, pulp uses the CBC that comes inside pulp.

F.

--

New posters to this group are moderated which can take up to 48 hours, so please be patient if your first post takes a while to turn up.
---

You received this message because you are subscribed to a topic in the Google Groups "pulp-or-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/pulp-or-discuss/IKE--kT6c5U/unsubscribe.
To unsubscribe from this group and all its topics, send an email to pulp-or-discu...@googlegroups.com.

To post to this group, send email to pulp-or...@googlegroups.com.

Visit this group at https://groups.google.com/group/pulp-or-discuss.
To view this discussion on the web visit https://groups.google.com/d/msgid/pulp-or-discuss/1babf194-4afe-47c0-b6a1-cd49a1c00a15%40googlegroups.com.

[daG ee]

Wow, thank you for that link! It somehow worked (with the command line in the same folder, where the exe is). The output says, that "threads was changed from 0 to 5"..

Now I've got a problem, that IF I change the threads within a script, then sometimes the run stops/does not finish.. but thats another topic I guess.

But that link was great, Franco!