Slow processing of many small LMI constraints

[ilay]

Hi,

I have a similar problem to this:

https://groups.google.com/forum/#!topic/mosek/0ejPwPE-pr0

I have tens of thousands of small LMI constraints (3x3 and 6x6 matrices).

I use Mosek as the solver. It seems that the actual Mosek solver time is a small fraction of the overall run time, and most of the time is spent on creating the constraints and on yalmiptime.

Is there a way to improve the performance?

I attached a simple working example, and its profile report on my machine. It has a similar performance profile to the actual data. (ignore the time spent on creating the initial sdpvars, it is not a bottleneck in my project).

Tested on Matlab R2015a (newer versions of Matlab have much worse performance on this problem, for some reason)

Thanks.

[Johan Löfberg]

This is a typical case which is hard to vectorize in YALMIP

If you have this large simple model and you want to improve performance, you would have to code it directly in moseks numerical format

[Johan Löfberg]

having said that, you many (expensive) calls to sdpvar can be eliminated

As = sdpvar(repmat(3,1,N),repmat(3,1,N),'full');

Ks = sdpvar(1,N);

ks = sdpvar(1,N);

of course, also Ks and ks kan be cells, I just created vectors here. No gain in placing a bunch of scalars in cells

[ilay]

Thanks for the quick reply!
Could you give me some pointers for how to use the Mosek format?
I have a bunch of different kinds of constraints, and the objective is complicated, so ideally I would like to code in Mosek just for the bottleneck constraints, and use Yalmip for the rest.

[Johan Löfberg]

All I can say is https://docs.mosek.com/8.1/toolbox/index.html

But I'll look at the code to see if I can find any simple internal bottle-necks. Once the sdpvar  was changed though, it ran in twice as fast, but it is still annoyingly slow

[ilay]

Thanks for the information!

[frankshi]

Hi, do you have matlab crash problem when dealing with large scale problems?

[Johan Löfberg]

If you run out of memory, you will crash of course

[frankshi]

Hi Professor,

I am not sure whether it is out of memory. There are really many variables and constraints in my MILP problem, and I use the gurobi solver. Most of the time matlab (2018a) will crash, it can succeed occasionally with a unreasonable solution. For a similar small scale problem, I can get the solution. Is there a stable version matlab or how could I solve the crash problem? Thank you very much.

The crash information is as follows:

Crash Decoding           : Disabled - No sandbox or build area path

  Crash Mode               : continue (default)

  Default Encoding         : GBK

  Deployed                 : false

[Johan Löfberg]

never seen that message before. You will have to consult a gurobi forum (with full display as I guess you are not showing everything displayed). crash for a large problem which works when it is small, is most often due to memory issues

[frankshi]

Hi Professor,

Yes, I only show part of the crash information. The full information is given below, and I thought that it is a common problem. We can also find the similar problem online: https://www.mathworks.com/matlabcentral/answers/409368-matlab-r2018a-crash-disabled-no-sandbox-or-build-area-path  

I will check the memory issues. Thank you very much for your suggestion.

 

Configuration:

  Crash Decoding           : Disabled - No sandbox or build area path

  Crash Mode               : continue (default)

  Default Encoding         : GBK

  Deployed                 : false

  Graphics Driver          : Unknown hardware 

  Graphics card 1          : NVIDIA ( 0x10de ) NVIDIA Quadro K620 Version 10.18.13.6213 (2016-3-18)

  Java Version             : Java 1.8.0_144-b01 with Oracle Corporation Java HotSpot(TM) 64-Bit Server VM mixed mode

  MATLAB Architecture      : win64

  MATLAB Entitlement ID    : 1981661

  MATLAB Root              : C:\Program Files\MATLAB\R2018a

  MATLAB Version           : 9.4.0.813654 (R2018a)

  OpenGL                   : hardware

  Operating System         : Microsoft Windows 7 Professional 

  Process ID               : 482352

  Processor ID             : x86 Family 6 Model 94 Stepping 3, GenuineIntel

  Session Key              : 6cc8e785-50a6-41c6-a088-fdf437da6ee0

  Window System            : Version 6.1 (Build 7601: Service Pack 1)

Fault Count: 1

Abnormal termination

Register State (from fault):

  RAX = 00000000ffa8c720  RBX = 00000000ffa8c800

  RCX = 00000000ffa8c720  RDX = 0000000000000000

  RSP = 0000000004029800  RBP = 0000000004029ea0

  RSI = 000000000402ac98  RDI = 00000000339f5f80

 

   R8 = 00000000088cbc80   R9 = 00000000ffa8c9c0

  R10 = 00000000ffa8c000  R11 = 000007fef7abc350

  R12 = 0000000004029ea8  R13 = 0000000000000000

  R14 = 0000000000000000  R15 = 0000000004029ea0

 

  RIP = 000007fef7a42d5e  EFL = 00010206

 

   CS = 0033   FS = 0053   GS = 002b

Stack Trace (from fault):

[  0] 0x000007fef7a42d5e C:\Program Files\IBM\ILOG\CPLEX_Studio1271\cplex\matlab\x64_win64\cplexlink1271.mexw64+00011614 mexFunction+00000910

[  1] 0x00000000fc626594                               bin\win64\libmex.dll+00288148 MexRetrieveVersion+00002916

[  2] 0x00000000fc62678c                               bin\win64\libmex.dll+00288652 MexRetrieveVersion+00003420

[  3] 0x00000000fc6268de                               bin\win64\libmex.dll+00288990 MexRetrieveVersion+00003758

[  4] 0x00000000fc612ec3                               bin\win64\libmex.dll+00208579 mexUnlock_800+00024147

[  5] 0x0000000010bfc51a                     bin\win64\pgo\m_dispatcher.dll+00050458 Mfh_file::dispatch_fh_impl+00001114

[  6] 0x0000000010bfc0b2                     bin\win64\pgo\m_dispatcher.dll+00049330 Mfh_file::dispatch_fh+00000062

[  7] 0x0000000010bf4674                     bin\win64\pgo\m_dispatcher.dll+00018036 Mfunction_handle::dispatch+00000420

[  8] 0x00000000123e0189                            bin\win64\pgo\m_lxe.dll+00393609

[  9] 0x00000000123ef53b                            bin\win64\pgo\m_lxe.dll+00455995

[ 10] 0x00000000123ee0a6                            bin\win64\pgo\m_lxe.dll+00450726

[ 11] 0x00000000123eb335                            bin\win64\pgo\m_lxe.dll+00439093

[ 12] 0x00000000123ea82e                            bin\win64\pgo\m_lxe.dll+00436270

[ 13] 0x00000000123e8b35                            bin\win64\pgo\m_lxe.dll+00428853

[ 14] 0x00000000123e8e9e                            bin\win64\pgo\m_lxe.dll+00429726

[ 15] 0x00000000123e8e29                            bin\win64\pgo\m_lxe.dll+00429609

[ 16] 0x0000000010bfc51a                     bin\win64\pgo\m_dispatcher.dll+00050458 Mfh_file::dispatch_fh_impl+00001114

[ 17] 0x0000000010bfc0b2                     bin\win64\pgo\m_dispatcher.dll+00049330 Mfh_file::dispatch_fh+00000062

[ 18] 0x0000000010bf4674                     bin\win64\pgo\m_dispatcher.dll+00018036 Mfunction_handle::dispatch+00000420

[ 19] 0x00000000123e2aa1                            bin\win64\pgo\m_lxe.dll+00404129

[ 20] 0x0000000012468e25                            bin\win64\pgo\m_lxe.dll+00953893 mwboost::serialization::singleton<mwboost::archive::detail::pointer_iserializer<mwboost::archive::binaryTerm_iarchive,ir::IrTree> >::get_instance+00115205

[ 21] 0x000000001246a808                            bin\win64\pgo\m_lxe.dll+00960520 mwboost::serialization::singleton<mwboost::archive::detail::pointer_iserializer<mwboost::archive::binaryTerm_iarchive,ir::IrTree> >::get_instance+00121832

[ 22] 0x000000001246b403                            bin\win64\pgo\m_lxe.dll+00963587 mwboost::serialization::singleton<mwboost::archive::detail::pointer_iserializer<mwboost::archive::binaryTerm_iarchive,ir::IrTree> >::get_instance+00124899

[ 23] 0x000000001246c836                            bin\win64\pgo\m_lxe.dll+00968758 mwboost::serialization::singleton<mwboost::archive::detail::pointer_iserializer<mwboost::archive::binaryTerm_iarchive,ir::IrTree> >::get_instance+00130070

[ 24] 0x000000001246d3f0                            bin\win64\pgo\m_lxe.dll+00971760 mwboost::serialization::singleton<mwboost::archive::detail::pointer_iserializer<mwboost::archive::binaryTerm_iarchive,ir::IrTree> >::get_instance+00133072

[ 25] 0x000000001246c97f                            bin\win64\pgo\m_lxe.dll+00969087 mwboost::serialization::singleton<mwboost::archive::detail::pointer_iserializer<mwboost::archive::binaryTerm_iarchive,ir::IrTree> >::get_instance+00130399

[ 26] 0x000000001246ca7e                            bin\win64\pgo\m_lxe.dll+00969342 mwboost::serialization::singleton<mwboost::archive::detail::pointer_iserializer<mwboost::archive::binaryTerm_iarchive,ir::IrTree> >::get_instance+00130654

[ 27] 0x00000000123ee8a5                            bin\win64\pgo\m_lxe.dll+00452773

[ 28] 0x00000000123eb335                            bin\win64\pgo\m_lxe.dll+00439093

[ 29] 0x00000000123ea82e                            bin\win64\pgo\m_lxe.dll+00436270

[ 30] 0x00000000123f1b9a                            bin\win64\pgo\m_lxe.dll+00465818

[ 31] 0x00000000123f133d                            bin\win64\pgo\m_lxe.dll+00463677

[ 32] 0x00000000123f144e                            bin\win64\pgo\m_lxe.dll+00463950

[ 33] 0x0000000012591d9c                            bin\win64\pgo\m_lxe.dll+02170268 mwboost::serialization::singleton<mwboost::archive::detail::pointer_oserializer<mwboost::archive::binary_oarchive,MathWorks::lxe::PreLineExecutionEvent> >::get_instance+00475644

[ 34] 0x0000000012591d4a                            bin\win64\pgo\m_lxe.dll+02170186 mwboost::serialization::singleton<mwboost::archive::detail::pointer_oserializer<mwboost::archive::binary_oarchive,MathWorks::lxe::PreLineExecutionEvent> >::get_instance+00475562

[ 35] 0x00000000123c051d                            bin\win64\pgo\m_lxe.dll+00263453

[ 36] 0x0000000011028acf                    bin\win64\pgo\m_interpreter.dll+00297679 inEvalCmdWithLocalReturn+00000063

[ 37] 0x00000000fb60bedb                          bin\win64\libmwbridge.dll+00114395 mnParser+00001595

[ 38] 0x0000000011e46564                                  bin\win64\mcr.dll+00222564 mcr::runtime::setInterpreterThreadSingletonToCurrent+00091332

[ 39] 0x0000000011e497ae                                  bin\win64\mcr.dll+00235438 mcr::runtime::setInterpreterThreadSingletonToCurrent+00104206

[ 40] 0x0000000011e66c8a                                  bin\win64\mcr.dll+00355466 mcrRegisterExternalFunction+00007930

[ 41] 0x0000000011e689bc                                  bin\win64\mcr.dll+00362940 mcrRegisterExternalFunction+00015404

[ 42] 0x0000000011e65bff                                  bin\win64\mcr.dll+00351231 mcrRegisterExternalFunction+00003695

[ 43] 0x0000000011e66c46                                  bin\win64\mcr.dll+00355398 mcrRegisterExternalFunction+00007862

[ 44] 0x0000000011e65ce3                                  bin\win64\mcr.dll+00351459 mcrRegisterExternalFunction+00003923

[ 45] 0x0000000011e68ff6                                  bin\win64\mcr.dll+00364534 mcrRegisterExternalFunction+00016998

[ 46] 0x00000000fd014f67                                  bin\win64\iqm.dll+00544615 iqm::IqmPlugin<cmddistributor::PackagedTaskIIP>::getIIP+00016215

[ 47] 0x00000000fd01567f                                  bin\win64\iqm.dll+00546431 iqm::PackagedTaskPlugin::execute+00000911

[ 48] 0x00000000fd014fed                                  bin\win64\iqm.dll+00544749 iqm::IqmPlugin<cmddistributor::PackagedTaskIIP>::getIIP+00016349

[ 49] 0x00000000fd0154c3                                  bin\win64\iqm.dll+00545987 iqm::PackagedTaskPlugin::execute+00000467

[ 50] 0x00000000fcfed9ea                                  bin\win64\iqm.dll+00383466 iqm::Iqm::setupIqmFcnPtrs+00071994

[ 51] 0x00000000fcfed84c                                  bin\win64\iqm.dll+00383052 iqm::Iqm::setupIqmFcnPtrs+00071580

[ 52] 0x00000000fcfce722                                  bin\win64\iqm.dll+00255778 iqm::Iqm::deliver+00003730

[ 53] 0x00000000fcfcf749                                  bin\win64\iqm.dll+00259913 iqm::Iqm::deliver+00007865

[ 54] 0x000000000536a211                        bin\win64\libmwservices.dll+01810961 services::system_events::PpeDispatchHook::dispatchOne+00030865

[ 55] 0x0000000005377ff3                        bin\win64\libmwservices.dll+01867763 sysq::addProcessPendingEventsUnitTestHook+00004659

[ 56] 0x00000000053781e0                        bin\win64\libmwservices.dll+01868256 sysq::addProcessPendingEventsUnitTestHook+00005152

[ 57] 0x0000000005379576                        bin\win64\libmwservices.dll+01873270 sysq::getCondition+00003446

[ 58] 0x000000000537b0e6                        bin\win64\libmwservices.dll+01880294 svWS_ProcessPendingEvents+00000230

[ 59] 0x0000000011e50737                                  bin\win64\mcr.dll+00263991 mcr::runtime::setInterpreterThreadSingletonToCurrent+00132759

[ 60] 0x0000000011e52360                                  bin\win64\mcr.dll+00271200 mcr::runtime::setInterpreterThreadSingletonToCurrent+00139968

[ 61] 0x0000000011e2e9b7                                  bin\win64\mcr.dll+00125367 mcr_process_events+00001959

[ 62] 0x0000000010b4fef5                             bin\win64\MVMLocal.dll+00327413 mvm_server::inproc::LocalFactory::terminate+00139589

[ 63] 0x00000000fa9605b9                                  bin\win64\mvm.dll+01246649 mvm::detail::initLocalMvmHack+00000793

[ 64] 0x00000000fa960dcb                                  bin\win64\mvm.dll+01248715 mvm::detail::SessionImpl::privateSession+00000555

[ 65] 0x00000000fa960ff1                                  bin\win64\mvm.dll+01249265 mvm::detail::SessionImpl::privateSession+00001105

[ 66] 0x000000014000780d                               bin\win64\MATLAB.exe+00030733

[ 67] 0x000000014000862f                               bin\win64\MATLAB.exe+00034351

[ 68] 0x00000000775759cd                   C:\windows\system32\kernel32.dll+00088525 BaseThreadInitThunk+00000013

[ 69] 0x00000000776d385d                      C:\windows\SYSTEM32\ntdll.dll+00342109 RtlUserThreadStart+00000029

This error was detected while a MEX-file was running. If the MEX-file

is not an official MathWorks function, please examine its source code

for errors. Please consult the External Interfaces Guide for information

on debugging MEX-files.

[Johan Löfberg]

you talked about gurobi, but this is cplex 12.7 crashing.

cplex 12.7 and earlier are known to be completely unstable in matlab and cannot be used

[frankshi]

Hi Professor,

Yes, I know cplex is unstable. But I do use the command: ops=sdpsettings('solver','gurobi','verbose',1,'debug',1) in my script, and gurobi solver is installed. Why is the bug information about cplex? Is anything wrong with the "sdpsettings" command? Thanks. 

[Johan Löfberg]

either you haven't used the ops structure when calling optimize, or cplex has crashed when it has created a default options structure in the call to sdpsettings.

remove cplex from your path

[frankshi]

Hi Professor,

Got it. I will remove cplex from the path. Thank you very much.