Full Covariance Matrix

[Jacqualine Henington]

Intuitivelythe covariance matrix generalizes the notion of variance to multiple dimensions. As an example, the variation in a collection of random points in two-dimensional space cannot be characterized fully by a single number, nor would the variances in the x \displaystyle x and y \displaystyle y directions contain all of the necessary information; a 2 2 \displaystyle 2\times 2 matrix would be necessary to fully characterize the two-dimensional variation.

A covariance matrix with all non-zero elements tells us that all the individual random variables are interrelated. This means that the variables are not only directly correlated, but also correlated via other variables indirectly. Often such indirect, common-mode correlations are trivial and uninteresting. They can be suppressed by calculating the partial covariance matrix, that is the part of covariance matrix that shows only the interesting part of correlations.

In contrast to the covariance matrix defined above, Hermitian transposition gets replaced by transposition in the definition.Its diagonal elements may be complex valued; it is a complex symmetric matrix.

These empirical sample covariance matrices are the most straightforward and most often used estimators for the covariance matrices, but other estimators also exist, including regularised or shrinkage estimators, which may have better properties.

The covariance matrix plays a key role in financial economics, especially in portfolio theory and its mutual fund separation theorem and in the capital asset pricing model. The matrix of covariances among various assets' returns is used to determine, under certain assumptions, the relative amounts of different assets that investors should (in a normative analysis) or are predicted to (in a positive analysis) choose to hold in a context of diversification.

The evolution strategy, a particular family of Randomized Search Heuristics, fundamentally relies on a covariance matrix in its mechanism. The characteristic mutation operator draws the update step from a multivariate normal distribution using an evolving covariance matrix. There is a formal proof that the evolution strategy's covariance matrix adapts to the inverse of the Hessian matrix of the search landscape, up to a scalar factor and small random fluctuations (proven for a single-parent strategy and a static model, as the population size increases, relying on the quadratic approximation).[9]Intuitively, this result is supported by the rationale that the optimal covariance distribution can offer mutation steps whose equidensity probability contours match the level sets of the landscape, and so they maximize the progress rate.

Two-dimensional infrared spectroscopy employs correlation analysis to obtain 2D spectra of the condensed phase. There are two versions of this analysis: synchronous and asynchronous. Mathematically, the former is expressed in terms of the sample covariance matrix and the technique is equivalent to covariance mapping.[12]

I would say there is a consensus now to decompose a covariance matrix into a correlation matrix and something else. But there is less consensus on whether the something else should be standard deviations or variances and less consensus on what the prior should be.

i have a question about the noise covariance matrix as it should be computed on some segments of brain signals which do not contain any activity of interest. so when trying to analyze just the resting state eeg data, what is the best option to compute the noise covariance on. as there is no portion of the data more important than the others (also dataset i am working with does not have eeg signal in an empty quiet room)

hello @larsoner thank you for your time and attention, it means a lot.

in fact, it is really hard to come by a clear explanation of noise covariance matrix for clinical applications of resting state eeg. i think noise covariance matrix will have huge impact in source localization.

so is it a good idea to consider some random ad hoc covariance as noise covariance.

I am an amateur any guidance or direction would help me a lot.

thanks a lot in advance.

I'm fairly new to the realm of relational database design, and am trying to store a 9x9 covariance matrix in a table. The rows and columns are x, y, z terms of position, velocity, and acceleration. Like so:

So for instance, the upper-left most element is PosXPosX (shortened to XX above), to the right is PosXPosY, and so on. The matrix is symmetric along the diagonal (i.e. PosXPosY == PosYPosX). It's also possible that I will want to store a 6x6 matrix that includes only position and velocity in this same table.

From my research, I've found a normalized table design of creating a table with fields for row number, column number, and value (How to represent a 2-D data matrix in a database). I can see that the benefit to this is flexibility, since the number of rows and columns can be variable. Is this the best way to proceed, even though I have a set number of rows and columns (9x9 and/or 6x6)? I could also envision creating a table that has fields for each unique row/col combination (PosXPosX, PosXPosY . . . etc). That seems more intuitive to me, but like I said I'm new at this.

My question is: How would you suggest representing my data in a relational database? I've outlined two possible methods but I don't know if either is the best way. "Best" in my case will mean efficiently stored and retrieved. What I'm creating is really a data repository, so the data in the database will not be changing once it is added, only read into numpy arrays or similar.

Some more background:

I'm analyzing test data. I have multiple test runs with different configurations, each having multiple data points that include a lot of different kinds of data. One of the data points I want to store and analyze is covariance. The amount of data I'm dealing with is quite staggering, so I'm hoping that using a database will help me to keep things organized and accessible. The goal is to store all this data, and then write data analysis and visualization tools that will draw from the data. In the case of covariance, I'm calculating things like Mahalanobis Distance, trace, and time propagated eigenvalues. I have many of these tools already, but they currently pull from a lot of different log files and are generally a mess.

As long as your data-set is small and you can assume that the values once read will not be modified by an external application, it might be worth considering using a csv file and just read the data into whatever data structure will be the most useful while you are doing your analysis. This will also let you look at your data slightly easier too because you will only need a text editor or if you wanted a way to view it as a spreadsheet.

Based on what you have said, it seems that the cleanest thing to have one record per cell because this will give you the most flexibility in the future. If you are interested in reading about database design this is a pretty good starting place

where record_id uniquely identifies a record, parent_id is a reference to the owning entity for this matrix, matrix_id uniquely identifies elements in a matrix,x and y would be the coordinates for the record and value is the cell value.

Databases are great at storing staggering amounts of data. Seems like you'll want to use them to quickly, clearly, and readily store and retrieve your information, if not to perform the actual calculations. If so, then you'll want to design your storage for efficient retrieval.

Fields (table columns) for rows and columns would seem to be a must. You would need to be sure never to add values greater than your matrix size (no row/col over 6 or 9). There are tricks you can do within the RDBMS to ensure this never happes, but they can get kind of kludgy.

You will find an additional field to uniquely identify which set of data is being stored essential. Build an index on the table for that column, and retrieving one set of 36 or 81 values from millions will be extremely fast.

I was computing some rather large covariance matrices and noticed that that implementation in Statistics is not multithreaded. TLDR, I want the following to be as fast as possible (ideally, for even larger matrices):

I went down the rabbit hole for some other packages that implement this, but it seems to me that neither of them are making use of threads. The only package I found that tried this was VectorizedStatistics.jl, but I found this to be slower in practice than the implementation from Base.

Ultimately, I would like to use a Cholesky factorization to solve a linear system. Is this problem so ill-conditioned for large matrices that it is pointless to even try to make it more efficient? So far it was good sufficient to add some small diagonal shift to the covariance matrix.

It is somewhat similar to stochastic gradient descend. One tries to optimize a bunch of variational parameters using the quantum geometric tensor, which is essentially a covariance matrix S = - where X is a vector that is estimated using a Markov Chain.

The parameters are optimized by solving a linear system S δα = F

To optimize many parameters, one should have many measurements of X_k so that the covariance matrix is well approximated. However, actually computing the covariance matrix is rather slow.

As you say, I can imagine that practice, this is done very differently. I am absolutely no expert on this to be honest. Perhaps one uses a lazy representation of the covariance matrix instead?

I hope this clarifies things

In particular, if you have an m \times n matrix A, then to compute the covarance matrix you first subtract the column mean to obtain X, which has \Theta(mn) cost; then compute X^* X, which has \Theta(mn^2) cost; then do some normalization, which has \Theta(n^2) cost. So, in cases like yours where m and n are large, the X^* X computation should completely dominate.

You could try different matrix-multiplication libraries, like MKL.jl or Octavian.jl or AppleAccelerate.jl, to see if they are faster than the default OpenBLAS library on your machine. You could also try Float32 precision to see if that is accurate enough for your application.

3a8082e126