Optimise the diagonals of a set of matrices

[a.m....@gmail.com]

I have a set of matrices that represent networks, and a function that determines network prominence per row ( = network node ). The optimal value for each separate matrix depends on the values on the diagonal, and I can find the optimal diagonal for the whole set of matrices by maximising (or minimising) the total for the set (by adding the matrix-wise values together).

When I pass the diagonal to the group function, the values on the diagonal, passed as lmfit Parameters, don't change between iterations, I cannot figure out why.

I have tried to make a minimal representative example, which aims to find the best diagonal D for a matrix M so that M - (D*M) is minimal, by minimising this over a group of matrices. This is a very roundabout way of finding the identity matrix, but it represents how I pass on my parameters. I wanted to test whether the parameters were going to be passed on between different iterations here:

However, I get an error for this example saying:

  File "/home/amwink/cloud/surfdrive/work/projects/epad/analyses/multipad/optimaldiagonal.py", line 36, in <module>
   best_fit = minimize ( fcn        = diagonal_all,
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/amwink/software/python/conda/envs/network-analyses/lib/python3.11/site-packages/lmfit/minimizer.py", line 2601, in minimize
   return fitter.minimize(method=method)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/amwink/software/python/conda/envs/network-analyses/lib/python3.11/site-packages/lmfit/minimizer.py", line 2345, in minimize
   return function(**kwargs)
          ^^^^^^^^^^^^^^^^^^
 File "/home/amwink/software/python/conda/envs/network-analyses/lib/python3.11/site-packages/lmfit/minimizer.py", line 1541, in least_squares
   ret = least_squares(self.__residual, start_vals,
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/amwink/software/python/conda/envs/network-analyses/lib/python3.11/site-packages/scipy/optimize/_lsq/least_squares.py", line 796, in least_squares
   raise ValueError("`x0` must have at most 1 dimension.")
ValueError: `x0` must have at most 1 dimension.

I do not get this in my other program, even though the "group_optimal" function there also returns a scalar "group difference".

Does anyone know:

- if it is possible to pass on the diagonal in a matrix optimisation problem as Parameters?
- if it is a problem to pass a function to minimize() that returns a scalar?

[Matt Newville]

On Wed, Nov 22, 2023 at 6:32 AM a.m....@gmail.com <a.m....@gmail.com> wrote:

I have a set of matrices that represent networks, and a function that determines network prominence per row ( = network node ). The optimal value for each separate matrix depends on the values on the diagonal, and I can find the optimal diagonal for the whole set of matrices by maximising (or minimising) the total for the set (by adding the matrix-wise values together).

When I pass the diagonal to the group function, the values on the diagonal, passed as lmfit Parameters, don't change between iterations, I cannot figure out why.

I do not really follow what you are trying to do. Matrices representing networks?  group function? 

I have tried to make a minimal representative example, which aims to find the best diagonal D for a matrix M so that M - (D*M) is minimal, by minimising this over a group of matrices. This is a very roundabout way of finding the identity matrix, but it represents how I pass on my parameters. I wanted to test whether the parameters were going to be passed on between different iterations here:

It would be way better if you posted actual code.  

 

However, I get an error for this example saying:

  File "/home/amwink/cloud/surfdrive/work/projects/epad/analyses/multipad/optimaldiagonal.py", line 36, in <module>
   best_fit = minimize ( fcn        = diagonal_all,
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/amwink/software/python/conda/envs/network-analyses/lib/python3.11/site-packages/lmfit/minimizer.py", line 2601, in minimize
   return fitter.minimize(method=method)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/amwink/software/python/conda/envs/network-analyses/lib/python3.11/site-packages/lmfit/minimizer.py", line 2345, in minimize
   return function(**kwargs)
          ^^^^^^^^^^^^^^^^^^
 File "/home/amwink/software/python/conda/envs/network-analyses/lib/python3.11/site-packages/lmfit/minimizer.py", line 1541, in least_squares
   ret = least_squares(self.__residual, start_vals,
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/amwink/software/python/conda/envs/network-analyses/lib/python3.11/site-packages/scipy/optimize/_lsq/least_squares.py", line 796, in least_squares
   raise ValueError("`x0` must have at most 1 dimension.")
ValueError: `x0` must have at most 1 dimension.

Without code, we can only guess what you are doing and what could be going wrong.  This looks like maybe a Parameter has some non-scalar value.  

For sure, Parameter values must be scalar, double-precision floating point numbers, not any other data type. All of the documentation, examples, and any working code you will ever find will have Parameters that are floats.

I do not get this in my other program, even though the "group_optimal" function there also returns a scalar "group difference".

Does anyone know:

- if it is possible to pass on the diagonal in a matrix optimisation problem as Parameters?

Parameter values must be double precision floats.

- if it is a problem to pass a function to minimize() that returns a scalar?

It is not a problem. Some minimization methods support that, but some do not.  It is not a problem, but it can cause some problems. 

[a.m....@gmail.com]

Sure I am happy to send code, thanks Matt for looking into this.

My previous example was very minimal, but it did not represent very well what I want to do. This attachment is a bit longer, still simplified but closer to the original. Now I generate the matrices rather than reading them from file, which makes for more manageable plots.

The idea is that 3 the matrices each represent undirected network connections (i.e. they are symmetric) between a set of nodes, and the networks are connected at the nodes as a multiplex network. My goal is to change the centrality of the multiplex network by varying the interlayer connections:

• In the main loop
  • lines 46-57 create a matrix similar to the ones I load in my code: a block diagonal of 3 symmetric matrices
    
  • lines 59-68 define the values to go in the 3 diagonals in the off-diagonal blocks (symmetric copies) as fit parameters
  • lines 70-79 call the minimize function, so in this case it should find diagonals that return a minimal cost function
    
• The function multicentrality()
  • creates a multiplex network from the block diagonal original (line 17)
  • puts the interlayer links from the Parameters object to the off-diagonal block on either side (lines 19-30)
  • computes the centrality at every node and returns the mean over all the nodes as the result (lines 32-33)
  • prints some diagnostics for my benefit (lines 35-40)

When I debug this in pycharm / spyder, the plots of the matrix inside the multicentrality() function don't change, and the interlayer weight weighted_multiplex[10,0], the first interlayer link added to the matrix, is stuck at 0.1 (the initial value).

That makes me think the matrices / Parameters don't change during the optimisation process?

Thanks for your help!

Best wishes,

Alle Meije

[Matt Newville]

Your code runs for me and updates the values of the Parameters.  The `weights.json` file has values that are not the starting values.

Printing out a fit report is always recommended.   ;)

Plotting intermediate results with a blocking i/o call from your objective function is anti-recommended. ;)

--
You received this message because you are subscribed to the Google Groups "lmfit-py" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lmfit-py+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lmfit-py/15724292-9e76-495d-95d5-56cc457f63ean%40googlegroups.com.

--

--Matt Newville <newville at cars.uchicago.edu> 630-327-7411

[a.m....@gmail.com]

Thanks! That is a relief, I guess.

I'm not sure I understand your last comment. Was it expected that although the parameters are updated by minimizer(), the intermediate plots do not sow these updated values?

[Matt Newville]

On Sun, Nov 26, 2023 at 9:41 AM a.m....@gmail.com <a.m....@gmail.com> wrote:

Thanks! That is a relief, I guess.

Hm, I cannot tell whether that means it does not work for you, or if you did not run the code that you posted.

I'm not sure I understand your last comment. Was it expected that although the parameters are updated by minimizer(), the intermediate plots do not sow these updated values?

The intermediate plots will dramatically slow down the code. That `plt.show()` will block further code execution.  

This makes me think that you did not run your own code.   

This is all really weird....

[a.m....@gmail.com]

I definitely ran the code and saw the plots :), in pycharm they appear in a separate browser-type window on the right without stopping the run.

Too many iterations (and as a result, too many plots) do end in an error though.

Thanks for looking into it.