Symmetric and Symmetric Positive Definite Matrices

1. Introduction

Symmetric matrices have some interesting properties - they always have real eigenvalues and their eigenvectors are perpendicular to each other. Symmetric positive definite matrices is a subset of symmetric matrices. We will take a look at how these matrices behave for two dimensions. One can easily extend these ideas to higher dimensions. A matrix M is positive definite if the angle between any vector x and Mx is less than 900 A matrix that is symmetric and positive definite is symmetric positive definite. So what makes it so special? Let's examine the behavior of these matrices in greater detail.

1.1 Two ways of visualizing a matrix

Before we proceed, let's make sure we understand how we visualize a matrix. We take some points on a circle and see how a matrix transforms these points. We can do this in two ways: Way 1: Draw a line from origin to the transformed point Way 2: Draw a line from a point before transformation to a point after transformation. In other words, we draw a line from (x1, y1) to (x2, y2) where (x2, y2) = M * (x1, y1) Points on a circle

Way 1: Lines drawn from origin to transformed points Way 2: Lines drawn from points before and after transformation

We will use the second way to visualize since it makes it possible to see where each point is mapped to. Oh and we will also use colors which depend only on the angle. Thus points at similar angles from x-axis will have the similar colors.

2. Symmetric and symmetric positive definite (SPD) matrices

We will soon look at one important property of positive symmetric definite matrices. But before we do so, let's look at some randomly generated symemtric and positive symmetric definite matrices. 1 0 0 1

Determinant of matrix: 1.000











Matrix =Determinant of matrix: 1.000

Just looking at the visualization above, can you infer any properties about symmetric positive definite matrices?

3. An important property of SPD matrices: xTMx > 0

An important property of symmetric definite matrices is that the angle between input vector and output vector is always less than ninety degrees. This can be written mathematically as: xTMx > 0. In other words, if we draw a line from origin to the input vector and draw another line from origin to the output vector, the angle between these two lines will never exceed ninety degrees. Let's see this in action below. The two lines in bold are lines drawn from origin to input and output vectors. The lines turn red as soon as the angle between them exceeds 90 degrees. You should not see this happen for SPD matrices. For reference, you can see whether or not this property holds true for a randomly generated matrix by clicking on Random Matrix below.

Angle between vectors x and Mx where x is a point on a unit circle.

The lines turn red if the angle exceeds 900. 1 0.20 0.20 1











Matrix =

3.1 Other properties than can be inferred from the property xTMx > 0



a) All eigenvalues are positive This can actually be inferred from the above property. We know that a positive symmetric definite matrix for sure has real eigenvalues since it is symmetric. If it had a negative eigenvalue, it would mean the angle between input and output vectors was 180 degrees. But we just learnt that the angle can never exceed 90 degrees. Thus, the eigenvalues must be positive.

b) Sum of SPD matrices is also SPD This can be seen using the dot product. Suppose there are n SPD matrices M i where i goes from 1 to n. Then for each M i we have:

x . M i x > 0 for every vector x. Therefore Σ i x . M i x > 0

c) All diagonal elements are positive This can be seen by observing how the standard basis vectors are transformed by a SPD matrix. . Let us take a look at third basis vector for a 4D symmetric matrix and its transformation:

x = 0 0 1 0 a b c d x =and Mx = Note that c is the third diagonal element of M. Now we have xTMx = c. But since xTMx > 0 , c > 0.

d) The product MTM and MMT is PSD for any M This is true since for any vector x, we have:

xTMTMx = (MxT)Mx > 0 because the xTx > 0 for any x.

4. Singular 2D matrices

Singular matrices are matrices with zero determinant. Let's see how a singular matrix transforms points on a circle. Click on the (self-explanatory) buttons below to see some pleasing animations: 1 0 0 1





Matrix =

If you are wondering why are we looking at singular matrices, we will look at an interesting property of symmetric matrices as (scaled) sum of singular matrices of a specific type.

4.1 Matrices of the form vvT are symmetric positive semi-definite

Here semi-definite means that the eigenvalues are non-negative (not exactly positive which implies definiteness). Actually since matrices of this form are singular, one of the eigenvalues is always 0.

4.2 Sum of singular matrices of type vvT scaled positively is a symmetric positive semi-definite matrix

Why did we look at singular matrices? Note that matrices of the following form are always singular and symmetric (think why): vvT We can multiply the matrix by a scalar. What it means is that we can multiply the output (i.e. transformed) vector with a scalar to scale it. This can be represented by the following operation (note that the resulting matrix is also singular and symmetric.): d * vvT. We can thus sum up such matrices like so : ‎Σ i d i v i v i T

An interesting observation to make is that if all the scalars are positive, we get a symmetric positive semi-definite matrix. In the visualization below, we take a look at matrices for d 1,2 =1,1 and d 1,2 =1,-1 respectively. 1 0 1 0 1 0 0 0

B = 0 1 0 1 0 0 0 1

A =B = : Rotate A clockwise by 10 degrees

: Rotate B counterclockwise by 10 degrees





Transformations by A and B shown individually

A+B = 1 0 0 1

A+B = Sum of transforms A and B gives a PSD matrix

A-B = 1 0 0 -1

A-B = Not a PSD matrix! : Rotate A clockwise by 10 degrees: Rotate B counterclockwise by 10 degrees Can you prove why is this sum equal to symmetric positive semi-definite matrix for positive values of d? (Hint: mean of dot products which are all positive is also positive)

5. Conclusion