Fusion is slow setting parameters

[Damien Couture]

Hi all,

I'm using mosek to solve a bog-standard portfolio optimization problem.

Now, I'm facing an issue where mosek takes 4s to solve my problem. Upon looking at the logs presolve takes about 0.01s, optimizer takes 0.03s, and a full 4s seem to be spent in the `__flush_1parameters_` function of fusion.

Further down, it's in `_task_1putafefijlist__3I_3I_3D`, which seems responsible for setting affine expressions. That code in turns has a horrible native python loop:

      # TODO: use putafefenrtylist
      for i,j,c in zip(idxs,subj,cof):
        self.__task.putafefentry(i,j,c)

In my case, the number of coefficients set this way is about 2e6, which is the cause of slow speed.

Two questions:

- Is anybody aware of a fix for that (I presume Mosek has a plan, given the TODO comment).

- Until that's fixed, is it possible for me to rewrite my problem differently to avoid these terms, and use others? Fusion does not give much insight about how expressions are converted, but I presume most of my affine expressions fall within this block?

Best,

Damien

[Michal Adamaszek]

Hi,

That is easy enough to change. We'll investigate and let you know.

But if you have 2e+6 parameterized entries then you may need to rethink the model anyway. Perhaps you are parameterizing too much. It is not only the time to actually input the values, but even more so to recompute the values, which happens before that loop. See the "Parameterize relevant parts of the model" paragraph in https://docs.mosek.com/latest/pythonfusion/guidelines-fusion.html#efficiency  It would help to know the model to say something more.

If you move all parametrization from conic to linear constraints then a different function is called.

Michal

[Damien Couture]

Thanks for your reply.

Regarding the parameters, they mostly come from exposures between stocks and factors (again, standard portfolio stuff). Having 200 factors, 10k stocks will have this number of values.

Now, they don't change between every single optimization, but the goal is a tight loop where they change every 10 iterations or so, that's why it seems wasteful to re-build a full model every 10 loops.

I'll have a look at moving to linear constraints instead, and see if it improves speed, but in any case if the change could happen, that would be great (and push us to switch from mosek 10 to mosek 11 :) )

Cheers

D

[Michal Adamaszek]

If you can, replace the loop with

self.__task.putafefentrylist(idxs,subj,cof)

and soon we'll do that in 11.0 Thanks for spotting

I should point out that even if you don't change parameter values, all parameterized entries are recomputed before each solve(). There is possible room for improvement there but that is a more involved TODO. So whether this will be less or more efficient than rebuilding the model every 10 iterations is something you may have to experiment with.

Michal

[Damien Couture]

Thanks, will experiment with this change, possibly with a local patch.

I've just refactored the code, to introduce a new variable (position_factors) with a linear constraint (position_factors = exposures @ position_stocks), then reduced the conic constraint (which instead of taking the old [n_stocks x n_factors] matrix only takes [n_factors x n_factors]), and it seems to speed up things dramatically.

Will test the old version with your change as well, thanks a lot.

D

[Michal Adamaszek]

Fixed in versions 10.2.16 and 11.0.11

Michal