¿Por qué la matriz de correlación necesita ser positiva semi-definida y qué significa ser o no positiva semi-definida?

He estado investigando el significado de la propiedad positiva semi-definida de las matrices de correlación o covarianza.

Estoy buscando cualquier información sobre

Definición de semi-definición positiva;
Sus propiedades importantes, implicaciones prácticas;
La consecuencia de tener determinante negativo, el impacto en el análisis multivariante o resultados de la simulación etc.

covariance-matrix eigenvalues determinant correlation-matrix Melón
fuente

¿Quieres entender lo semi-definitud es , o quiere saber por qué matrices de correlación deben ser semi-definido, o si desea saber qué resultados importantes están implícitos en esta propiedad?

whuber

Si las matrices de correlación no son semi-positivas definidas, entonces podría obtener variaciones que fueron negativas.

Edité su pregunta un poco, compruébelo. Además, tenga en cuenta que una matriz con un número par de valores propios negativos seguirá teniendo un determinante positivo.

ttnphns

¡Una matriz de covarianza NO siempre es igual a la matriz de correlación! La covarianza considera variables normalizadas mientras que la matriz de correlación no.

Manoj Kumar

Preguntas relacionadas: ¿Cada matriz de covarianza es positiva definida? considera el caso más amplio de las matrices de covarianza, de las cuales las matrices de correlación son un caso especial; Además, ¿cada matriz de correlación es positiva semi-definida? y ¿Cada matriz de correlación es positiva definida?

Silverfish

Respuestas:

$\sum_i a_i X_i$ $a_i$

var (\sum_{i} a_{i} X_{i}) = \sum_{i} \sum_{j} a_{i} a_{j} cov (X_{i}, X_{j}) = \sum_{i} \sum_{j} a_{i} a_{j} Σ_{i, j},

$\operatorname{var}\left(\sum_i a_i X_i\right) = \sum_i \sum_j a_ia_j \operatorname{cov}(X_i,X_j) = \sum_i \sum_j a_ia_j \Sigma_{i,j},$ we have that the covariance matrix

Σ = [Σ_{i, j}]

$\Sigma = [\Sigma_{i,j}]$ must be positive semidefinite (which is sometimes called nonnegative definite). Recall that a matrix

C

$C$ is called positive semidefinite if and only if

\sum_{i} \sum_{j} a_{i} a_{j} C_{i, j} \geq 0 \forall a_{i}, a_{j} \in R .

$\sum_i \sum_j a_ia_j C_{i,j} \geq 0 \;\; \forall a_i, a_j \in \mathbb R.$

Dilip Sarwate
fuente

Thanks, I removed my downvote but I did not upvote because it does not answer about practical implications. Say I have a matrix that is not positive definite (due for exemple to modifification by 'expert'). What would happen if I use it to calibrate and/or simulate data ? Specifically, is this a real problem when trying to study a big sum and there is only a few negative eigen values ? What would be an efficient algorithm to transform a non positive semi-definite correlation matrix to a positive semi-definite one ? What would be the impact of this algorithm ?

lcrmorin

@Were_cat Thanks for the reversal of the downvote.

Dilip Sarwate

Could you please explain the first equality in the first equation?

Vivek Subramanian

var (X) = cov (X, X)

$\operatorname{var}(X)=\operatorname{cov}(X,X)$

\begin{aligned} cov (\sum_{i} a_{i} X_{i}, Y) & = \sum_{i} a_{i} cov (X_{i}, Y) \\ cov (X, \sum_{i} b_{j} Y_{j},) & = \sum_{j} b_{j} cov (X, Y_{j}) \end{aligned}

$\begin{align}\operatorname{cov}\left(\sum_i a_iX_i,Y\right)&=\sum_i a_i\operatorname{cov}(X_i,Y)\\ \operatorname{cov}\left(X, \sum_i b_jY_j,\right) &=\sum_j b_j \operatorname{cov}(X,Y_j)\end{align}$

Dilip Sarwate

The answer is quite simple.

The correlation matrix is defined thus:

Let $X = [x_1, x_2, ..., x_n]$ be the $m\times n$ data matrix: $m$ observations, $n$ variables.

Define $X_b= [\frac{(x_1-\mu_1 e)}{s_1}, \frac{(x_2-\mu_2 e)}{s_2}, \frac{(x_3-\mu_3 e)}{s_3}, ...]$ as the matrix of normalized data, with $\mu_1$ being mean for the variable 1, $\mu_2$ the mean for variable 2, etc., and $s_1$ the standard deviation of variable 1, etc., and $e$ is a vector of all 1s.

The correlation matrix is then

C = X_{b}^{'} X_{b}

$C=X_b' X_b$

A matrix $A$ is positive semi-definite if there is no vector $z$ such that $z' A z <0$ .

Suppose $C$ is not positive definite. Then there exists a vector w such that $w' C w<0$ .

However $(w' C w)=(w' X_b' X_b w)=(X_b w)'(X_b w) = {z_1^2+z_2^2...}$ , where $z=X_b w$ , and thus $w' C w$ is a sum of squares and therefore cannot be less than zero.

So not only the correlation matrix but any matrix $U$ which can be written in the form $V' V$ is positive semi-definite.

gregor
fuente

This is by far the clearest most concise and useful answer. Thanks !

Yohan Obadia

(Possible looseness in reasoning would be mine. I'm not a mathematician: this is a depiction, not proof, and is from my numeric experimenting, not from books.)

A positive semidefinite (psd) matrix, also called Gramian matrix, is a matrix with no negative eigenvalues. Matrix with negative eigenvalues is not positive semidefinite, or non-Gramian. Both of these can be definite (no zero eigenvalues) or singular (with at least one zero eigenvalue). [Word "Gramian" is used in several different meanings in math, so perhaps should be avoided.]
In statistics, we usually apply these terms to a SSCP-type matrix, also called scalar product matrix. Correlation or covariance matrices are particular cases of such matrix.
Any scalar product matrix is a summary characteristic of some multivariate data (a cloud). For example, given $n$ cases X $p$ variables data, we could compute $p$ X $p$ covariance matrix between the variables or $n$ X $n$ covariance matrix between the cases. When you compute it from real data, the matrix will always be Gramian. You may get non-Gramian (non-psd) matrix if (1) it is similarity matrix measured directly (i.e. not computed from the data) or the similarity measure isn't SSCP-type; (2) the matrix values was incorrectly entered; (3) the matrix is in fact Gramian but is (or so close to be) singular that sometimes the spectral method of computing eigenvalues produces tiny negative ones in place of true zero or tiny positive ones.
An alternative and equivalent summary for the cloud is the matrix of euclidean distances. A scalar product (such as covariance) between a pair of items and the corresponding squared euclidean distance between them are tied by the law of cosines (cosine theorem, look at the picture there): $d_{12}^2 = h_1^2+h_2^2-2s_{12}$ , where the $s$ is the scalar product and the $h$ 's are the distances of the two items from the origin. In case of covariance matrix between variables $X$ and $Y$ this formula looks as $d_{xy}^2 = \sigma_x^2+\sigma_y^2-2cov_{xy}$ .
As interim conclusion: a covariance (or correlation or other scalar product) matrix between some $m$ items is a configuration of points embedded in Euclidean space, so euclidean distances are defined between all these $m$ points.
Now, if [point 5] holds exactly, then the configuration of points is truly euclidean configuration which entails that the scalar product matrix at hand (e.g. the covariance one) is Gramian. Otherwise it is non-Gramian. Thus, to say " $m$ X $m$ covariance matrix is positively semi-definite" is to say "the $m$ points plus the origin fit in Euclidean space perfectly".
What are possible causes or versions of non-Gramian (non-Euclidean) configuration? The answers follow upon contemplating [point 4].
- Cause 1. Evil is among the points themselves: $m$ X $m$ distance matrix isn't fully euclidean. Some of the pairwise distances $d$ are such that they cannot agree with the rest of the points in Euclidean space. See Fig1.
- Cause 2. There is general (matrix-level) mismatch between $h$ 's and $d$ 's. For example, with fixed $d$ 's and some $h$ 's given, the other $h$ 's must only vary within some bounds in order to stay in consent with Euclidean space. See Fig2.
- Cause 3. There is localized (pair-level) mismatch between a $d$ and the pair of corresponding $h$ 's connected to those two points. Namely, the rule of triangular inequality is violated; that rule demands $h_1+h_2 \ge d_{12} \ge |h_1-h_2|$ . See Fig3.
To diagnose the cause, convert the non-Gramian covariance matrix into distance matrix using the above law of cosines. Do double-centering on it. If the resultant matrix has negative eigenvalues, cause 1 is present. Else if any $|cov_{ij}| \gt \sigma_i \sigma_j$ , cause 3 is present. Else cause 2 is present. Sometimes more than one cause get along in one matrix.

Fig1.

Fig1

Fig2.

Fig2

Fig3.

Fig3

ttnphns
fuente

Point 6 needs demonstration: you have shown that a matrix of squared Euclidean distances is p-d, but you assert without proof that to each p-d matrix corresponds a Euclidean configuration of points. Also you haven't connected your definition of p-d ("no negative eigenvalues") to any of your subsequent characterizations. The key idea comes at the end (point 8): a p-d matrix can be used to define a distance. Logically, this is where you should begin the analysis.

whuber

@whuber: Thank you for the critical appraisal. I'm afraid, when it comes to mathematically proving something, I sink. I've reported part of my practical experience (I said that); the answer wasn't really an analytical sequence. Wouldn't you like then to add your own answer that can correct/improve mine? It might turn out a valuable aid. Or, you are free to work on my text to improve it if you find it not downright futile.

ttnphns

P.S. My point 8 implies that since double centering anchors a configuration of points to its centroid, this operation itself does not introduce non-euclidity (it itroduces only singularity because the new point, centre, belongs to the same space). Thence we can check if the initial configuration was euclidean. Is that not correct?

ttnphns