Minimum variance optimization with CVXPY...

[Joël Hubert]

I'm trying to get the code below to work in the context of minimum variance optimization. Unfortunately I'm inexperienced with CVXPY and things are not going smooth.

It's for my thesis but I really need help... Can I beg anyone to take a look? The error I'm getting with this setup is ValueError: Invalid sign for Parameter value.

R are the expected returns, C the covariances, rf the risk-free rate, w the weights and r selected means on the Efficient Frontier for which variance is minimized.

def solve_frontier(R, C, rf, context):
    frontier_mean, frontier_var, frontier_weights, var, penalty = [], [], [], 0, 0
    n = len(R)
    w = cvx.Variable(n)
    r = cvx.Parameter(sign='positive')
   
    prob = cvx.Problem(cvx.Minimize(var + penalty),
                   [sum(w)-context.allowableMargin == 0])
   
    r_vals = linspace(max(min(R), rf), max(R), num=20)
    for i in range(20):
        r.value = r_vals[i]
        var = dot(dot(w, C), w)
        penalty = (1/100)*abs(mean_1-r)
        prob.solve()
        frontier_mean.append(r.value)
        frontier_var.append(compute_var(prob.value, C))
        frontier_weights.append(prob.value)
        print "status:", prob.status
    return array(frontier_mean), array(frontier_var), frontier_weights

[Steven Diamond]

There are a few issues with this code. To get rid of the "ValueError: Invalid sign for Parameter value." declare r as "r = cvx.Parameter()". This means the parameter can have any sign. If you do "Parameter(sign="positive")", you're asserting/requiring that the parameter value is always positive. You do this when it's needed to satisfy the DCP rules. But I don't see any reason that it has to be positive for your problem.

Also, the way you're updating "var" won't work. You can't use "dot" in cvxpy. It looks like you mean "w.T*C*w". This violates DCP, but you can use "quad_form(w, C)", which is equivalent.

Hope this helps,

Steven

[Joël Hubert]

Awesome Steven, thanks for this. I actually used quad_form(w, C) before, but got "CvxPyDomainError: P is not symmetric".

The P in question refers to C, and if I do C.shape I get (10, 10). The issue is probably due to C consisting of NaN (only) for the first data fetch. I'll work on that. Thanks again. I'll get back to you if there are any further problems, clearly they're not overly complicated :)

[Steven Diamond]

Yeah, it would definitely break if there were NaNs. If C has proper entries and is not symmetric, you can always replace it with "(C + C.T)/2" in quad_form. cvxpy should probably do that automatically.

[Joël Hubert]

Alright. The penalty variable is giving me trouble (bad operand type for abs(): 'AddExpression') because mean_1 is considered 'AddExpression' and r 'Parameter'. I guess subtracting one from the other is a problematic  task.

The point of the penalty variable is to serve as an extra constraint; not only is variance minimized, but it is minimized for that particular mean, r, to form a specific portfolio for that return.

Any ideas to get around this?

[Joël Hubert]

In the meantime I realized r should be r.value. Still trying to figure out mean_1.

[Joël Hubert]

Fixed that too. At this point the code seems to be running, but I get a rather cryptic error "Exception: Cannot evaluate the truth value of a constraint" while the constraints look pretty harmless to me.

In fact they're identical to the portfolio minimization example on the CVXPY website. That means we don't even know what exactly is causing the problem now (I removed Margin from the constraint for the sake of clarity).

Current code:

def solve_frontier(R, C, rf, context):
    frontier_mean, frontier_var, frontier_weights, var, penalty = [], [], [], 0, 0
    n = len(R)
    w = cvx.Variable(n)

    r = cvx.Parameter()
    mean_1 = sum(R*w)
    #allowableMargin = 1

   
    prob = cvx.Problem(cvx.Minimize(var + penalty),

                   [cvx.sum_entries(w) == 1,
                   w >= 0])

   
    r_vals = linspace(max(min(R), rf), max(R), num=20)
    for i in range(20):
        r.value = r_vals[i]

        var = cvx.quad_form(w, C)
        penalty = (1/100)*max(mean_1-r.value, -mean_1-r.value) #find better alternative to abs  

[Steven Diamond]

You're using the built in Python max in "max(mean_1-r.value, -mean_1-r.value)" rather than the cvxpy max_elemwise function. You absolutely cannot use non cvxpy functions on cvxpy objects.

[Joël Hubert]

Whoops, wow, now it runs... Almost there, but all prob.values are 0 I'm afraid.

def solve_frontier(R, C, rf, context):
    frontier_mean, frontier_var, frontier_weights, var, penalty = [], [], [], 0, 0
    n = len(R)
    w = cvx.Variable(n)
    r = cvx.Parameter()
    mean_1 = sum(R*w)
    #allowableMargin = 1
   
    prob = cvx.Problem(cvx.Minimize(var + penalty),
                   [cvx.sum_entries(w) == 1,
                   w >= 0])
   
    r_vals = linspace(max(min(R), rf), max(R), num=20)
    for i in range(20):
        r.value = r_vals[i]
        var = cvx.quad_form(w, C)

        penalty = (1/100)*cvx.max_elemwise(mean_1-r.value, -mean_1-r.value) #find better alternative to abs  
        prob.solve()
        print(logging.debug("r: " + str(r)))
        print(logging.debug("prob.value: " + str(prob.value)))

        frontier_mean.append(r.value)
        frontier_var.append(compute_var(prob.value, C))
        frontier_weights.append(prob.value)
        print "status:", prob.status
    return array(frontier_mean), array(frontier_var), frontier_weights

Debug log:

DEBUG:root:r: param81
DEBUG:root:prob.value: 0.0
DEBUG:root:r: param81
DEBUG:root:prob.value: 0.0
DEBUG:root:r: param81
DEBUG:root:prob.value: 0.0
...

Even though the status is

optimal

For reference, this is what my R and C look like for the first non-nan run:

R:
[ -5.68298038e-02   2.43611735e-01   3.53116429e-01  -3.36094631e-01
  -9.39206819e-02  -1.31529637e-01  -5.26763590e-02  -1.14016331e-04
  -2.38461275e-01  -6.85108757e-02]

C:
[[ 0.01883194 -0.01755008 -0.01829476 -0.02154588 -0.01421845 -0.01293689
  -0.0069792  -0.00916988 -0.00980294 -0.01649447]
 [-0.01755008  0.10091211  0.08605381  0.06845271  0.0572482   0.04779117
   0.01994855  0.02979787  0.02669141  0.04569414]
 [-0.01829476  0.08605381  0.11210214  0.0491932   0.05090629  0.04346153
   0.01746663  0.03068978  0.02128769  0.03266539]
 [-0.02154588  0.06845271  0.0491932   0.16031082  0.06840978  0.0686063
   0.03502345  0.04160541  0.0410681   0.09184603]
 [-0.01421845  0.0572482   0.05090629  0.06840978  0.05459739  0.0452198
   0.01965265  0.02430314  0.02312082  0.04735021]
 [-0.01293689  0.04779117  0.04346153  0.0686063   0.0452198   0.06039362
   0.02306852  0.0233749   0.02322384  0.05399279]
 [-0.0069792   0.01994855  0.01746663  0.03502345  0.01965265  0.02306852
   0.02132792  0.01655368  0.0149683   0.02641823]
 [-0.00916988  0.02979787  0.03068978  0.04160541  0.02430314  0.0233749
   0.01655368  0.04100387  0.0211659   0.02743472]
 [-0.00980294  0.02669141  0.02128769  0.0410681   0.02312082  0.02322384
   0.0149683   0.0211659   0.02890283  0.02732543]
 [-0.01649447  0.04569414  0.03266539  0.09184603  0.04735021  0.05399279
   0.02641823  0.02743472  0.02732543  0.07594181]]

.

[Steven Diamond]

In these lines:

        var = cvx.quad_form(w, C)

        penalty = (1/100)*cvx.max_elemwise(mean_1-r.value, -mean_1-r.value) #find better alternative to abs 

You're not actually changing the problem. The problem is fixed when you define it, except for parameter values. So the problem you're solving has an objective of 0.

[Joël Hubert]

Alright, I guess I was a little optimistic. How would you minimize var dependent on a particular mean r?

[Steven Diamond]

You could make mean_l a parameter or the simplest thing would be to recreate the problem inside the for loop.

Also, this line "mean_1 = sum(R*w)" looks very wrong. You can't use the built-in sum on a cvxpy object, just like you can't use max or min.

[Joël Hubert]

Like this? It runs, but I only get one weight for one asset (there are 10) and it's the same for all iterations.

def solve_frontier(R, C, rf, context):
    frontier_mean, frontier_var, frontier_weights, var, penalty = [], [], [], 0, 0
    n = len(R)
    w = cvx.Variable(n)
    r = cvx.Parameter()

    #allowableMargin = 1

   
    r_vals = linspace(max(min(R), rf), max(R), num=20)
    for i in range(20):
        r.value = r_vals[i]
        var = cvx.quad_form(w, C)

        mean_1 = cvx.sum_entries(R*w)

        penalty = (1/100)*cvx.max_elemwise(mean_1-r.value, -mean_1-r.value) #find better alternative to abs  

        prob = cvx.Problem(cvx.Minimize(var + penalty),
                   [cvx.sum_entries(w) == 1,
                   w >= 0])

        prob.solve()
        print(logging.debug("r: " + str(r)))
        print(logging.debug("prob.value: " + str(prob.value)))
        frontier_mean.append(r.value)
        frontier_var.append(compute_var(prob.value, C))
        frontier_weights.append(prob.value)
        print "status:", prob.status
    return array(frontier_mean), array(frontier_var), frontier_weights

Result:

DEBUG:root:prob.value: 0.00517471016738 
DEBUG:root:r: param213 
DEBUG:root:prob.value: 0.00517471016738 
DEBUG:root:r: param213 
DEBUG:root:prob.value: 0.00517471016738 
DEBUG:root:r: param213

Normally the minimization should come up with an array of different weights for each iteration. 

[Steven Diamond]

What is ``R`` exactly? Is mean_l supposed to be the same every iteration? cvxpy has an ``abs`` function, BTW.

[Joël Hubert]

 R are the expected returns for the 10 assets:

[ -5.68298038e-02   2.43611735e-01   3.53116429e-01  -3.36094631e-01
  -9.39206819e-02  -1.31529637e-01  -5.26763590e-02  -1.14016331e-04
  -2.38461275e-01  -6.85108757e-02]

 

and C the covariances.

[[ 0.01883194 -0.01755008 -0.01829476 -0.02154588 -0.01421845 -0.01293689
  -0.0069792  -0.00916988 -0.00980294 -0.01649447]
 [-0.01755008  0.10091211  0.08605381  0.06845271  0.0572482   0.04779117
   0.01994855  0.02979787  0.02669141  0.04569414]
 [-0.01829476  0.08605381  0.11210214  0.0491932   0.05090629  0.04346153
   0.01746663  0.03068978  0.02128769  0.03266539]
 [-0.02154588  0.06845271  0.0491932   0.16031082  0.06840978  0.0686063
   0.03502345  0.04160541  0.0410681   0.09184603]
 [-0.01421845  0.0572482   0.05090629  0.06840978  0.05459739  0.0452198
   0.01965265  0.02430314  0.02312082  0.04735021]
 [-0.01293689  0.04779117  0.04346153  0.0686063   0.0452198   0.06039362
   0.02306852  0.0233749   0.02322384  0.05399279]
 [-0.0069792   0.01994855  0.01746663  0.03502345  0.01965265  0.02306852
   0.02132792  0.01655368  0.0149683   0.02641823]
 [-0.00916988  0.02979787  0.03068978  0.04160541  0.02430314  0.0233749
   0.01655368  0.04100387  0.0211659   0.02743472]
 [-0.00980294  0.02669141  0.02128769  0.0410681   0.02312082  0.02322384
   0.0149683   0.0211659   0.02890283  0.02732543]
 [-0.01649447  0.04569414  0.03266539  0.09184603  0.04735021  0.05399279
   0.02641823  0.02743472  0.02732543  0.07594181]]

The code should be replicable with this. I forgot to make abs "cvx.abs".

[Joël Hubert]

Without the penalty function i.e. mean_1 the minimization would yield the overall portfolio with the least variance.

However, we want different portfolios for different means (r), hence the loop, so we add a penalty function dependent on the weights calculated that minimizes variances for a point close to the iteration of r in question.

E.g. for r == 0.015 we don't just want the variance var minimized, but also the gap between 0.015 and the mean_1 corresponding to the calculated weights/variance. So variance is minimized for a certain set of weights, we look at the mean that corresponds to that set, and we try to minimize the gap between that mean and the initial r, which involves changing w again.

In essence what I want is simply to know, for twenty different r's, what the associated weights are that minimize the variance. I realize the setup is confusing.

[Joël Hubert]

So, somehow Steven's reply is not listing but he pointed out that the 1/100 in the penalty function acts as 0 in Python 2, and so a

 from __future__ import division

should be added at the beginning of the code. Cheers.