3  Module-2: Matrix Eigen Value Problems

Syllabus: Eigen values and Eigen vectors - Properties of Eigen values - Diagonalization of matrices - Orthogonal transformation and orthogonal matrices - Quadratic forms and their Canonical forms.

3.1 The Big Question: What Does a Matrix Do?

In the first module, we treated matrices as simple containers for the numbers in a linear system. But a matrix is much more than that. A matrix is a transformation. It takes a vector and maps it to a new vector.

When you multiply a vector \(x\) by a matrix \(A\), the resulting vector \(Ax\) is usually stretched and rotated. It points in a new direction.

But for any given matrix, there are a few very special vectors. When you multiply these special vectors by the matrix, they do not change direction. They only get stretched or shrunk. The output vector \(Ax\) is parallel to the input vector \(x\).

These special vectors are the eigenvectors of the matrix, and the scaling factor is the eigenvalue.

Definition: Eigenvalue and Eigenvector For a square matrix \(A\), a non-zero vector \(x\) is an eigenvector if it satisfies the equation: \[ Ax = \lambda x \] where \(\lambda\) is a scalar known as the eigenvalue corresponding to the eigenvector \(x\).

The eigenvectors tell you the “axes” of the transformation, the directions that are preserved. The eigenvalues tell you the scaling factor along those axes. If you know the eigenvalues and eigenvectors of a matrix, you understand its fundamental behavior.

3.2 Finding Eigenvalues and Eigenvectors

How do we find these special numbers \(\lambda\) and vectors \(x\)? We start by rewriting the main equation.

\[ \begin{align*} Ax &= \lambda x \\ Ax - \lambda x &= 0 \\ Ax - \lambda I x &= 0 && \text{(where I is the identity matrix)} \\ (A - \lambda I)x &= 0 \end{align*} \]

Look at that last line! It’s a homogeneous system of equations, just like we saw in Module I. We are looking for a non-zero solution for \(x\). For the system \((A - \lambda I)x = 0\) to have a non-trivial solution, the matrix \((A - \lambda I)\) must be singular.

And what does it mean for a matrix to be singular? Its determinant must be zero.

The Characteristic Equation \[ \det(A - \lambda I) = 0 \]

Solving this equation for \(\lambda\) gives us the eigenvalues. Then, for each eigenvalue, we plug it back into \((A - \lambda I)x = 0\) to find the corresponding eigenvector(s).

Shortcut for finding eigen values of 2x2 Matrices

For any 2x2 matrix \(A = \begin{bmatrix} a & b \\ c & d \end{bmatrix}\), the characteristic equation is always: \[ \lambda^2 - (\text{trace}(A))\lambda + \det(A) = 0 \] Where:

  • The trace is the sum of the diagonal elements: \(\text{trace}(A) = a+d\).

  • The determinant is \(\det(A) = ad-bc\).

This is a beautiful and powerful result! You don’t need to calculate \(\det(A - \lambda I)\) from scratch; just find the trace and determinant and plug them into the quadratic formula.

Shortcut for finding eigen values of 3x3 Matrices

A similar, though more complex, shortcut exists for 3x3 matrices. The characteristic equation is: \[ \lambda^3 - (\text{trace}(A))\lambda^2 + C\lambda - \det(A) = 0 \] Where \(C\) is the sum of the determinants of the 2x2 principal minors (the matrices you get by deleting the \(i\)-th row and \(i\)-th column). \[ C = \det\begin{pmatrix} a_{22} & a_{23} \\ a_{32} & a_{33} \end{pmatrix} + \det\begin{pmatrix} a_{11} & a_{13} \\ a_{31} & a_{33} \end{pmatrix} + \det\begin{pmatrix} a_{11} & a_{12} \\ a_{21} & a_{22} \end{pmatrix} \] While this formula works, solving a cubic equation can be difficult, and this is where numerical methods often become more practical.

3.2.1 Example: A Symbolic Approach

Let’s do this for a simple matrix \(A = \begin{bmatrix} 4 & -2 \\ 1 & 1 \end{bmatrix}\) using SymPy to see all the steps.

Code
import sympy as sp

# Define our matrix A
A = sp.Matrix([
    [4, -2],
    [1,  1]
])

# Create a symbol for lambda
lam = sp.symbols('lambda')

# Create the Identity matrix of the same size as A
I = sp.eye(A.shape[0])  # Corrected

# Form the matrix (A - lambda*I)
char_matrix = A - lam * I

print("The matrix (A - λI):")
sp.pprint(char_matrix)

# Calculate the determinant to get the characteristic polynomial
char_poly = char_matrix.det()
print(f"\nThe characteristic polynomial det(A - λI) is: {char_poly}")

# Solve the characteristic equation det(A - λI) = 0 for the eigenvalues
eigenvalues = sp.solve(char_poly, lam)
print(f"\nThe eigenvalues are: {eigenvalues}")

# SymPy can also do this in one step
print("\n--- Using SymPy's built-in function ---")
sp.pprint(A.eigenvects())
The matrix (A - λI):
⎡4 - λ   -2  ⎤
⎢            ⎥
⎣  1    1 - λ⎦

The characteristic polynomial det(A - λI) is: lambda**2 - 5*lambda + 6

The eigenvalues are: [2, 3]

--- Using SymPy's built-in function ---
⎡⎛      ⎡⎡1⎤⎤⎞  ⎛      ⎡⎡2⎤⎤⎞⎤
⎢⎜2, 1, ⎢⎢ ⎥⎥⎟, ⎜3, 1, ⎢⎢ ⎥⎥⎟⎥
⎣⎝      ⎣⎣1⎦⎦⎠  ⎝      ⎣⎣1⎦⎦⎠⎦

The output shows that for our matrix \(A\), the eigenvalues are \(\lambda_1 = 3\) and \(\lambda_2 = 2\). The corresponding eigenvectors are multiples of \(\begin{bmatrix} 2 \\ 1 \end{bmatrix}\) and \(\begin{bmatrix} 1 \\ 1 \end{bmatrix}\).

3.2.2 Key Properties of Eigenvalues

The shortcuts above hint at a deeper connection. Here are the most important properties of eigenvalues for any \(n \times n\) matrix \(A\).

  • Sum: The sum of the eigenvalues is equal to the trace of the matrix. \[ \sum_{i=1}^{n} \lambda_i = \text{trace}(A) \]
  • Product: The product of the eigenvalues is equal to the determinant of the matrix. \[ \prod_{i=1}^{n} \lambda_i = \det(A) \]
  • Singularity: A matrix is singular (not invertible) if and only if at least one of its eigenvalues is zero. (This follows directly from the product property).
  • Powers: The eigenvalues of \(A^k\) are \(\lambda_1^k, \lambda_2^k, \dots, \lambda_n^k\).
  • Inverse: The eigenvalues of \(A^{-1}\) are \(1/\lambda_1, 1/\lambda_2, \dots, 1/\lambda_n\).
  • Transpose: A matrix and its transpose, \(A\) and \(A^T\), have the same eigenvalues.
  • Triangular Matrices: The eigenvalues of a triangular (or diagonal) matrix are simply the entries on its main diagonal.

Let’s verify a few of these properties with code.

Code
import numpy as np

# A sample 3x3 matrix
A = np.array([
    [2, 1, -1],
    [6, -1, 0],
    [-1, -2, -1]
])

# Get the eigenvalues using NumPy
eigenvalues = np.linalg.eigvals(A)

print(f"The matrix A is:\n{A}")
print(f"\nIts eigenvalues are: {eigenvalues}")

# Verify the Sum property
sum_of_eigenvalues = np.sum(eigenvalues)
trace_of_A = np.trace(A)
print(f"\nSum of eigenvalues: {sum_of_eigenvalues:.4f}")
print(f"Trace of A: {trace_of_A}")
print(f"Are they close? {np.isclose(sum_of_eigenvalues, trace_of_A)}")

# Verify the Product property
product_of_eigenvalues = np.prod(eigenvalues)
determinant_of_A = np.linalg.det(A)
print(f"\nProduct of eigenvalues: {product_of_eigenvalues:.4f}")
print(f"Determinant of A: {determinant_of_A:.4f}")
print(f"Are they close? {np.isclose(product_of_eigenvalues, determinant_of_A)}")
The matrix A is:
[[ 2  1 -1]
 [ 6 -1  0]
 [-1 -2 -1]]

Its eigenvalues are: [ 3.91899444+0.j         -1.95949722+1.23243177j -1.95949722-1.23243177j]

Sum of eigenvalues: 0.0000+0.0000j
Trace of A: 0
Are they close? True

Product of eigenvalues: 21.0000+0.0000j
Determinant of A: 21.0000
Are they close? True

3.2.3 Example: Finding Eigenvectors by Hand

Now that we have the eigenvalues, how do we find the eigenvectors? We solve the system \((A - \lambda I)x = 0\). Let’s do this for our first example, \(A = \begin{bmatrix} 4 & -2 \\ 1 & 1 \end{bmatrix}\), where we found \(\lambda_1 = 3\) and \(\lambda_2 = 2\).

Case 1: Find eigenvector for \(\lambda_1 = 3\).

We need to solve \((A - 3I)x = 0\). \[ (A - 3I) = \begin{bmatrix} 4-3 & -2 \\ 1 & 1-3 \end{bmatrix} = \begin{bmatrix} 1 & -2 \\ 1 & -2 \end{bmatrix} \] So the system is: \[ \begin{bmatrix} 1 & -2 \\ 1 & -2 \end{bmatrix} \begin{bmatrix} x_1 \\ x_2 \end{bmatrix} = \begin{bmatrix} 0 \\ 0 \end{bmatrix} \] Both rows give the same equation: \(x_1 - 2x_2 = 0\), or \(x_1 = 2x_2\). This system has a free variable! If we choose \(x_2 = 1\), then \(x_1 = 2\). So our eigenvector is a multiple of \(v_1 = \begin{bmatrix} 2 \\ 1 \end{bmatrix}\).

Case 2: Find eigenvector for \(\lambda_2 = 2\).

We need to solve \((A - 2I)x = 0\). \[ (A - 2I) = \begin{bmatrix} 4-2 & -2 \\ 1 & 1-2 \end{bmatrix} = \begin{bmatrix} 2 & -2 \\ 1 & -1 \end{bmatrix} \] The system is \(2x_1 - 2x_2 = 0\), or \(x_1 = x_2\). If we choose \(x_2=1\), then \(x_1=1\). So our eigenvector is a multiple of \(v_2 = \begin{bmatrix} 1 \\ 1 \end{bmatrix}\).

These match the results we get from our computer programs perfectly!

3.3 The Geometry of Eigenvectors

Let’s visualize what the matrix \(A\) does to the plane. We will take a circle of vectors, apply the matrix \(A\) to each one, and see where they land. The circle will be transformed into an ellipse.

The key thing to watch for is that the eigenvectors lie along the axes of this new ellipse. They are the only vectors that don’t get rotated off their original line.

Code
import numpy as np
import matplotlib.pyplot as plt

# Define the matrix A
A = np.array([
    [4, -2],
    [1,  1]
])

# Get eigenvalues and eigenvectors
eigvals, eigvecs = np.linalg.eig(A)
v1 = eigvecs[:, 0]
v2 = eigvecs[:, 1]

# Create a unit circle
theta = np.linspace(0, 2*np.pi, 200)
circle_vectors = np.array([np.cos(theta), np.sin(theta)])

# Transform the circle into an ellipse
transformed_vectors = A @ circle_vectors

# --- Plot ---
fig, ax = plt.subplots(figsize=(6, 6))

# Original circle (blue)
ax.plot(circle_vectors[0, :], circle_vectors[1, :], color='blue', label='Original Circle')

# Transformed ellipse (green)
ax.plot(transformed_vectors[0, :], transformed_vectors[1, :], color='green', label='Transformed Ellipse')

# Eigenvectors transformed (red dashed) — scaled by eigenvalues
ax.plot([0, eigvals[0] * v1[0]], [0, eigvals[0] * v1[1]], 'r--', linewidth=2, label=f'Eigenvector λ={eigvals[0]:.2f}')
ax.plot([0, eigvals[1] * v2[0]], [0, eigvals[1] * v2[1]], 'r--', linewidth=2, label=f'Eigenvector λ={eigvals[1]:.2f}')

# Formatting
ax.set_aspect('equal', adjustable='box')
ax.set_xlim(-5, 5)
ax.set_ylim(-5, 5)
ax.set_xlabel('X-axis')
ax.set_ylabel('Y-axis')
ax.set_title('Geometric Action of a Matrix')
ax.legend()
ax.grid(True)

plt.show()
Figure 3.1: The unit circle (blue) is transformed into an ellipse (green). The eigenvectors (red) only stretch, keeping their direction.

3.4 Tutorial 3- Eigen values and Eigen vectors in Engineering Applications

Solve each question using algebraic techniques: form the characteristic polynomial, compute eigenvalues, find eigenvectors (mode shapes/principal directions/steady-state vectors), and verify algebraic and geometric multiplicities.


3.4.1 Questions

  1. MEMS Resonator — Vibration Analysis

A MEMS resonator used in an oscillator circuit has stiffness matrix

\[K = \begin{bmatrix}6 & -2 & 0 \\ -2 & 4 & -2 \\ 0 & -2 & 6\end{bmatrix}.\]

  1. Find the natural frequencies of vibration (eigenvalues of \(K\)).

  2. Determine the mode shapes of vibration (eigenvectors of \(K\)).

  3. Verify algebraic and geometric multiplicities of each eigenfrequency.

  4. Explain the engineering meaning of each eigenvalue as a measure (e.g., squared natural frequency) and comment on stability.


  1. Wireless Channel Reliability

A discrete-time Markov model for a wireless channel has transition matrix

\[P = \begin{bmatrix}0.8 & 0.2 \\ 0.3 & 0.7\end{bmatrix}.\]

  1. Compute the eigenvalues of \(P\).

  2. Find the steady-state reliability vector (the eigenvector corresponding to \(\lambda=1\)) and normalize it to a probability vector.

  3. Verify algebraic and geometric multiplicities.

  4. Interpret the dominant and subdominant eigenvalues with respect to long-term reliability and convergence speed.


  1. Image Compression via PCA

Covariance matrix of a 2-D feature (pixel intensities) is

\[C = \begin{bmatrix}5 & 2 \\ 2 & 2\end{bmatrix}.\]

  1. Find the principal directions (eigenvectors) of \(C\).

  2. Compute the variance explained by each direction (eigenvalues).

  3. Verify algebraic and geometric multiplicities.

  4. Compute the percentage of total variance captured by the largest eigenvalue (compression efficiency).


  1. Quantum 2-Level System (Hamiltonian)

The Hamiltonian of a two-level quantum system is

\[H = \begin{bmatrix}4 & 1 \\ 1 & 4\end{bmatrix}.\]

  1. Find the energy levels (eigenvalues) of \(H\).

  2. Determine the stationary states (normalized eigenvectors).

  3. Verify algebraic and geometric multiplicities.

  4. Express \(H\) using spectral decomposition and state how eigenvalues control state evolution phases.


  1. Control System — State Matrix Modes

A linear time-invariant system has state matrix

\[A = \begin{bmatrix}0 & 1 \\ -5 & -4\end{bmatrix}.\]

  1. Find the modes of system response (eigenvalues of \(A\)).

  2. Determine the response directions (eigenvectors) and indicate whether they are real or complex.

  3. Verify algebraic and geometric multiplicities.

  4. Interpret the real parts of eigenvalues as stability margins and the imaginary parts as oscillation frequency.


3.4.2 Solutions

1. MEMS Resonator — Vibration Modes and Stability

(a) Characteristic polynomial and eigenvalues.

Compute \(\det(K-\lambda I)=0\). The characteristic polynomial simplifies to

\[-\lambda^3 +16\lambda^2 -76\lambda +96 = 0,\]

whose real positive roots are

\[\lambda_1 = 2,\quad \lambda_2 = 6,\quad \lambda_3 = 8.\]

(b) Mode shapes (eigenvectors).

Solve \((K-\lambda I)\mathbf v=0\) for each root:

  • For \(\lambda_1=2\): \(\mathbf v_1 = [1,\,2,\,1]^T\).
  • For \(\lambda_2=6\): \(\mathbf v_2 = [-1,\,0,\,1]^T\).
  • For \(\lambda_3=8\): \(\mathbf v_3 = [1,\,-1,\,1]^T\).

(Students may present normalized forms.)

Python code

Code
import numpy as np

# Stiffness matrix
K = np.array([[6, -2, 0],
              [-2, 4, -2],
              [0, -2, 6]])

# Eigenvalues and eigenvectors
eigvals, eigvecs = np.linalg.eig(K)

print("Eigenvalues:", eigvals)
print("Mode Shapes (eigenvectors):")
print(eigvecs)
Eigenvalues: [2. 6. 8.]
Mode Shapes (eigenvectors):
[[ 4.08248290e-01 -7.07106781e-01  5.77350269e-01]
 [ 8.16496581e-01  4.02240178e-16 -5.77350269e-01]
 [ 4.08248290e-01  7.07106781e-01  5.77350269e-01]]

(c) Multiplicities.

Each eigenvalue appears once in the characteristic polynomial (algebraic multiplicity = 1) and has one independent eigenvector (geometric multiplicity = 1).

(d) Engineering interpretation.

Assuming mass normalization, eigenvalues correspond to squared natural angular frequencies \(\omega^2\). Therefore

\[\omega_1=\sqrt{2},\;\omega_2=\sqrt{6},\;\omega_3=\sqrt{8}.\]

All eigenvalues positive and simple imply distinct, stable oscillatory modes; no zero/negative eigenvalues appear, so no rigid-body mode or instability is present.


2. Wireless Channel Reliability — Long-term Measure & Convergence

(a) Eigenvalues.

Solve \(\det(P-\lambda I)=0\) to get

\[\lambda_1 = 1,\qquad \lambda_2 = \tfrac{1}{2}.\]

(b) Steady-state vector.

Solve \((P-I)\mathbf v=0\). One eigenvector is \([1,1]^T\), normalized to

\[\pi = \begin{bmatrix}1/2\\\\[4pt]1/2\end{bmatrix}.\]

(c) Multiplicities.

Both eigenvalues are simple: algebraic multiplicity = 1 and geometric multiplicity = 1.

(d) Engineering meaning.

\(\lambda_1=1\) yields the steady-state distribution (long-term reliability). The subdominant eigenvalue \(\lambda_2=0.5\) determines exponential convergence speed: transients decay as \((0.5)^t\).

Python code

Code
import numpy as np

A = np.array([[0.8, 0.3],
              [0.2, 0.7]])

eigvals, eigvecs = np.linalg.eig(A)

print("Eigenvalues:", eigvals)
print("Eigenvectors:")
print(eigvecs)

# Check stability
stable = all(abs(eigvals) < 1)
print("System stable?", stable)
Eigenvalues: [1.  0.5]
Eigenvectors:
[[ 0.83205029 -0.70710678]
 [ 0.5547002   0.70710678]]
System stable? False

3. PCA — Principal Directions and Variance Explained

(a) Eigenvalues.

Solve \(\det(C-\lambda I)=0\) to obtain

\[\lambda_1 = 6,\qquad \lambda_2 = 1.\]

(b) Eigenvectors (principal directions).

  • For \(\lambda_1=6\): \(\mathbf u_1=[2,\,1]^T\) (principal direction).
  • For \(\lambda_2=1\): \(\mathbf u_2=[-1,\,2]^T\).

(c) Multiplicities.

Distinct eigenvalues ⇒ algebraic multiplicities = geometric multiplicities = 1.

(d) Compression efficiency.

Total variance \(=6+1=7\). Fraction captured by first PC:

\[\frac{6}{7}\approx 85.71\%.\]

Projecting onto the first principal direction retains ~85.7% of variance.

Python code

Code
import numpy as np

# Covariance matrix from Question 3
C = np.array([[5., 2.],
              [2., 2.]])

# eigendecomposition
eigvals, eigvecs = np.linalg.eig(C)

# sort in descending order of eigenvalue
idx = np.argsort(eigvals)[::-1]
eigvals = eigvals[idx]
eigvecs = eigvecs[:, idx]

# normalize eigenvectors (columns) to unit length
eigvecs_norm = eigvecs / np.linalg.norm(eigvecs, axis=0)

print("Covariance matrix C:\n", C)
print("\nEigenvalues (sorted):", eigvals)
print("Normalized eigenvectors (columns correspond to eigenvalues):\n", eigvecs_norm)

# Variance explained
total_var = eigvals.sum()
fractions = eigvals / total_var
print("\nTotal variance: {:.6f}".format(total_var))
for i,(lam, frac) in enumerate(zip(eigvals, fractions), start=1):
    print(f"PC{i}: eigenvalue = {lam:.6f}  -> variance = {frac*100:.4f}%")

# Algebraic multiplicity (cluster equal eigenvalues within tol)
def algebraic_multiplicities(vals, tol=1e-8):
    vals = list(vals)
    used = [False]*len(vals)
    clusters = []
    for i,v in enumerate(vals):
        if used[i]:
            continue
        cnt = 1
        used[i] = True
        for j in range(i+1, len(vals)):
            if (not used[j]) and (abs(vals[j] - v) < tol):
                cnt += 1
                used[j] = True
        clusters.append((v, cnt))
    return clusters

# Geometric multiplicity = dimension of nullspace of (C - lambda I) = n - rank(C - lambda I)
def geometric_multiplicity(A, lam, tol=1e-10):
    M = A - lam * np.eye(A.shape[0])
    rank = np.linalg.matrix_rank(M, tol=tol)
    return A.shape[0] - rank

print("\nAlgebraic multiplicities (value, count):", algebraic_multiplicities(eigvals))
for lam in np.unique(np.round(eigvals, 12)):
    print(f"Geometric multiplicity of λ={lam}: {geometric_multiplicity(C, lam)}")

# Helper: integer-proportional presentation of eigenvectors (useful for handouts)
def integer_representation(v, tol=1e-8):
    # scale vector so the smallest non-zero absolute entry becomes 1 (or -1), then round
    nonzero = np.abs(v) > tol
    if not nonzero.any():
        return np.zeros_like(v, dtype=int)
    scale = np.min(np.abs(v[nonzero]))
    v_scaled = v / scale
    v_int = np.round(v_scaled).astype(int)
    return v_int

print("\nInteger-proportional eigenvectors (for presentation):")
for i in range(eigvecs_norm.shape[1]):
    v = eigvecs_norm[:, i]
    # fix sign convention for presentation: make first nonzero entry positive
    first_nonzero = np.where(np.abs(v) > 1e-8)[0][0]
    if v[first_nonzero] < 0:
        v = -v
    print(f"λ = {eigvals[i]:.6f} : approx vector ->", integer_representation(v))
Covariance matrix C:
 [[5. 2.]
 [2. 2.]]

Eigenvalues (sorted): [6. 1.]
Normalized eigenvectors (columns correspond to eigenvalues):
 [[ 0.89442719 -0.4472136 ]
 [ 0.4472136   0.89442719]]

Total variance: 7.000000
PC1: eigenvalue = 6.000000  -> variance = 85.7143%
PC2: eigenvalue = 1.000000  -> variance = 14.2857%

Algebraic multiplicities (value, count): [(6.0, 1), (1.0, 1)]
Geometric multiplicity of λ=1.0: 1
Geometric multiplicity of λ=6.0: 1

Integer-proportional eigenvectors (for presentation):
λ = 6.000000 : approx vector -> [2 1]
λ = 1.000000 : approx vector -> [ 1 -2]

4. Quantum 2-Level System — Energy Levels & Stationary States

(a) Eigenvalues (energy levels).

Solve \(\det(H-\lambda I)=0\); roots are

\[\lambda_1 = 3,\quad \lambda_2 = 5.\]

(b) Normalized stationary states.

  • For \(\lambda_1=3\): \(\mathbf p_1 = \tfrac{1}{\sqrt{2}}[-1,\,1]^T\).
  • For \(\lambda_2=5\): \(\mathbf p_2 = \tfrac{1}{\sqrt{2}}[1,\,1]^T\).

(c) Multiplicities.

Both simple (algebraic = geometric = 1).

(d) Spectral decomposition.

\[H = 3\,\mathbf p_1\mathbf p_1^T + 5\,\mathbf p_2\mathbf p_2^T.\]

Time evolution \(e^{-iHt}\) applies phases \(e^{-i\lambda t}\) to each stationary state.

Python code

Code
from scipy.linalg import expm

H = np.array([[4, 1],
              [1, 4]])

eigvals, eigvecs = np.linalg.eig(H)

print("Eigenvalues (energy levels):", eigvals)
print("Eigenvectors (stationary states):\n", eigvecs)

# Verify spectral decomposition
H_recon = eigvecs @ np.diag(eigvals) @ np.linalg.inv(eigvecs)
print("Reconstructed H:\n", np.round(H_recon, 5))

# Time evolution operator at t=1, ħ=1
t = 1.0
U = expm(-1j * H * t)
print("Time evolution operator U(t=1):\n", np.round(U, 4))
Eigenvalues (energy levels): [5. 3.]
Eigenvectors (stationary states):
 [[ 0.70710678 -0.70710678]
 [ 0.70710678  0.70710678]]
Reconstructed H:
 [[4. 1.]
 [1. 4.]]
Time evolution operator U(t=1):
 [[-0.3532+0.4089j  0.6368+0.55j  ]
 [ 0.6368+0.55j   -0.3532+0.4089j]]

  1. Control System — Modes & Stability Margin

(a) Eigenvalues (modes).

Characteristic equation \(\lambda^2 + 4\lambda + 5 = 0\) gives

\[\lambda_{1,2} = -2 \pm i.\]

(b) Eigenvectors.

Corresponding complex eigenvectors can be computed; one representative for \(\lambda=-2+i\) is

\[\mathbf v = \begin{bmatrix}-\tfrac{2}{5}+\tfrac{i}{5}\\\\[4pt] 1 \end{bmatrix}.\]

Real sinusoidal modes can be constructed from the complex eigenpairs.

(c) Multiplicities.

Distinct complex conjugate eigenvalues → algebraic multiplicity = geometric multiplicity = 1 for each.

(d) Interpretation.

Real part \(-2\) indicates exponential decay (stable). Imaginary part \(\pm1\) indicates oscillation frequency 1 rad/s. Decay envelope ~ \(e^{-2t}\).

Python code

Code
# Control system (2x2) analysis: A = [[0,1],[-5,-4]]
import numpy as np
from scipy.linalg import expm

A = np.array([[0., 1.],
              [-5., -4.]])

# Eigen-decomposition
eigvals, eigvecs = np.linalg.eig(A)   # eigvecs columns correspond to eigvals

print("A =\n", A)
print("\nEigenvalues:")
for i, lam in enumerate(eigvals, start=1):
    print(f"  λ{i} = {lam}")

print("\nEigenvectors (columns):\n", eigvecs)

# Real and imaginary parts
print("\nReal parts of eigenvalues:", np.real(eigvals))
print("Imag parts of eigenvalues:", np.imag(eigvals))

# Stability check: asymptotically stable if all real parts < 0
is_stable = np.all(np.real(eigvals) < 0)
print("\nAsymptotically stable?", is_stable)

# Algebraic multiplicities (count equal eigenvalues within tolerance)
def algebraic_multiplicities(vals, tol=1e-8):
    vals_r = np.round(vals.real, 8) + 1j*np.round(vals.imag, 8)
    unique, counts = np.unique(vals_r, return_counts=True)
    return list(zip(unique, counts))

print("\nAlgebraic multiplicities:", algebraic_multiplicities(eigvals))

# Geometric multiplicity: n - rank(A - λI)
def geometric_multiplicity(A, lam, tol=1e-10):
    M = A - lam * np.eye(A.shape[0])
    rank = np.linalg.matrix_rank(M, tol=tol)
    return A.shape[0] - rank

for lam in eigvals:
    gm = geometric_multiplicity(A, lam)
    print(f"Geometric multiplicity of λ={lam}: {gm}")

# Check diagonalizability (geometric multiplicities == algebraic)
# For small matrices we can check if eigvecs are linearly independent:
rank_eigvecs = np.linalg.matrix_rank(eigvecs)
diag_ok = (rank_eigvecs == A.shape[0])
print("\nEigenvector matrix rank:", rank_eigvecs)
print("Matrix diagonalizable over C?", diag_ok)

# Spectral reconstruction: A_recon = V * diag(lambda) * V^{-1}
A_recon = eigvecs @ np.diag(eigvals) @ np.linalg.inv(eigvecs)
print("\nSpectral reconstruction (A_recon):\n", np.round(A_recon, 10))

# If eigenvalues are complex conjugates, form real modal (real-valued 2x2 block) representation
# and compute the real state transition e^{At} via expm
t = 0.5  # example time
Phi = expm(A * t)
print(f"\nState-transition matrix e^{{A t}} for t={t}:\n", np.round(Phi, 6))

# Provide a short interpretation printout
print("\nInterpretation:")
print(" - Eigenvalues provide modes: real part => decay rate, imag part => oscillation frequency.")
print(" - Here, eigenvalues = -2 ± 1j  => decaying oscillation with decay rate 2 and angular freq 1.")
A =
 [[ 0.  1.]
 [-5. -4.]]

Eigenvalues:
  λ1 = (-2.0000000000000004+1.0000000000000004j)
  λ2 = (-2.0000000000000004-1.0000000000000004j)

Eigenvectors (columns):
 [[-0.36514837-0.18257419j -0.36514837+0.18257419j]
 [ 0.91287093+0.j          0.91287093-0.j        ]]

Real parts of eigenvalues: [-2. -2.]
Imag parts of eigenvalues: [ 1. -1.]

Asymptotically stable? True

Algebraic multiplicities: [((-2-1j), 1), ((-2+1j), 1)]
Geometric multiplicity of λ=(-2.0000000000000004+1.0000000000000004j): 1
Geometric multiplicity of λ=(-2.0000000000000004-1.0000000000000004j): 1

Eigenvector matrix rank: 2
Matrix diagonalizable over C? True

Spectral reconstruction (A_recon):
 [[-0.+0.j  1.-0.j]
 [-5.-0.j -4.-0.j]]

State-transition matrix e^{A t} for t=0.5:
 [[ 0.675586  0.176371]
 [-0.881854 -0.029897]]

Interpretation:
 - Eigenvalues provide modes: real part => decay rate, imag part => oscillation frequency.
 - Here, eigenvalues = -2 ± 1j  => decaying oscillation with decay rate 2 and angular freq 1.

Notes for instructors

  • The answers above use convenient integer eigenvectors; students should be awarded full credit for any nonzero scalar multiple or normalized versions.
  • Encourage explicit algebraic steps: expansion of determinants, solving the linear systems for eigenvectors, and normalization when required.

3.5 Diagonalization of a Matrix

This brings us to one of the most powerful ideas in linear algebra: diagonalization. The goal is to write a matrix \(A\) as a product of three simpler matrices.

The Diagonalization Formula \[ A = S \Lambda S^{-1} \] Where: * \(S\) is the matrix whose columns are the eigenvectors of \(A\). * \(\Lambda\) (Lambda) is the diagonal matrix with the eigenvalues on its diagonal.

A matrix can be diagonalized if and only if it has a full set of linearly independent eigenvectors.

Why is this so useful? Consider computing \(A^{100}\). This would be a nightmare. But if \(A\) is diagonalized: \[ A^2 = (S \Lambda S^{-1})(S \Lambda S^{-1}) = S \Lambda (S^{-1}S) \Lambda S^{-1} = S \Lambda^2 S^{-1} \] In general: \[ A^k = S \Lambda^k S^{-1} \] Computing \(\Lambda^k\) is trivial: you just raise the diagonal entries to the \(k\)-th power!

3.5.1 Example: Verifying Diagonalization

Let’s verify \(A = S \Lambda S^{-1}\) and compute \(A^3\) for our matrix.

Code
import numpy as np

# Our matrix A
A = np.array([
    [4, -2],
    [1,  1]
], dtype=float)  # ensure float for safety in inverse

# NumPy's eig function gives both eigenvalues and eigenvectors
eigenvalues, eigenvectors = np.linalg.eig(A)

# S is the matrix of eigenvectors
S = eigenvectors
# Λ is the diagonal matrix of eigenvalues
Lambda = np.diag(eigenvalues)

print("Matrix S (Eigenvectors):")
print(S)
print("\nMatrix Λ (Eigenvalues):")
print(Lambda)

# Calculate S-inverse
S_inv = np.linalg.inv(S)

# Verify A = SΛS⁻¹
A_reconstructed = S @ Lambda @ S_inv
print("\nReconstructed A = SΛS⁻¹:")
print(A_reconstructed)
print("\nIs it close to the original A?", np.allclose(A, A_reconstructed))

# Compute A³ directly
A_cubed_direct = np.linalg.matrix_power(A, 3)

# Compute A³ using diagonalization
Lambda_cubed = np.diag(eigenvalues**3)
A_cubed_diagonal = S @ Lambda_cubed @ S_inv

print("\n--- Computing A³ ---")
print("Direct computation (A³):")
print(A_cubed_direct)
print("\nUsing diagonalization (SΛ³S⁻¹):")
print(A_cubed_diagonal)
print("\nAre the results close?", np.allclose(A_cubed_direct, A_cubed_diagonal))
Matrix S (Eigenvectors):
[[0.89442719 0.70710678]
 [0.4472136  0.70710678]]

Matrix Λ (Eigenvalues):
[[3. 0.]
 [0. 2.]]

Reconstructed A = SΛS⁻¹:
[[ 4. -2.]
 [ 1.  1.]]

Is it close to the original A? True

--- Computing A³ ---
Direct computation (A³):
[[ 46. -38.]
 [ 19. -11.]]

Using diagonalization (SΛ³S⁻¹):
[[ 46. -38.]
 [ 19. -11.]]

Are the results close? True

3.6 Symmetric and Orthogonal Matrices

Things get even nicer for a very special class of matrices: symmetric matrices, where \(A = A^T\).

Symmetric matrices have two incredible properties: 1. All their eigenvalues are real numbers. 2. Their eigenvectors can be chosen to be orthogonal (perpendicular to each other).

If we normalize the eigenvectors (make them unit length), they form an orthonormal set. The matrix \(Q\) whose columns are these orthonormal eigenvectors is an orthogonal matrix.

Orthogonal matrices are wonderful because their inverse is simply their transpose: \(Q^{-1} = Q^T\).

This leads to the Spectral Theorem, which says that any symmetric matrix can be diagonalized as: \[ A = Q \Lambda Q^T \]

3.7 Application: Quadratic Forms

What are these ideas good for? One classic application is understanding the geometry of quadratic forms. A quadratic form is a polynomial where every term has degree two. For example: \[ f(x, y) = 2x^2 + 6xy + 2y^2 \] This equation describes a shape on the plane. But what shape? The \(6xy\) “cross-product” term makes it hard to see because it corresponds to a rotated shape. Our goal is to eliminate this term.

We can write any quadratic form using a symmetric matrix: \[ f(x, y) = \begin{bmatrix} x & y \end{bmatrix} \begin{bmatrix} 2 & 3 \\ 3 & 2 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} = x^T A x \]

The eigenvectors of \(A\) point along the principal axes of the shape, and the eigenvalues tell us the scaling in those directions. By changing to a coordinate system aligned with the eigenvectors, we can describe the shape without a cross-term.

Let’s find the axes of the ellipse given by \(2x^2 + 6xy + 2y^2 = 1\).

Code
import numpy as np
import matplotlib.pyplot as plt

# Symmetric matrix for the quadratic form
A = np.array([
    [3, 1],
    [1, 2]
])

# Eigen-decomposition gives the axes and scaling
eigvals, eigvecs = np.linalg.eig(A)
v1 = eigvecs[:, 0]
v2 = eigvecs[:, 1]

# Generate grid data
x = np.linspace(-1.5, 1.5, 400)
y = np.linspace(-1.5, 1.5, 400)
X, Y = np.meshgrid(x, y)

# Quadratic form value: F(x, y) = [x y] A [x; y]
F = A[0, 0] * X**2 + 2 * A[0, 1] * X * Y + A[1, 1] * Y**2

# Plotting
plt.figure(figsize=(7, 7))

# Draw the level set F = 1 (ellipse)
plt.contour(X, Y, F, levels=[1], colors='blue')

# Draw eigenvectors scaled by semi-axis lengths
axis1 = v1 / np.sqrt(eigvals[0])
axis2 = v2 / np.sqrt(eigvals[1])

plt.quiver(0, 0, axis1[0], axis1[1], angles='xy', scale_units='xy', scale=1, color='red', label=f'Axis 1 (λ={eigvals[0]:.2f})')
plt.quiver(0, 0, -axis1[0], -axis1[1], angles='xy', scale_units='xy', scale=1, color='red')

plt.quiver(0, 0, axis2[0], axis2[1], angles='xy', scale_units='xy', scale=1, color='red', label=f'Axis 2 (λ={eigvals[1]:.2f})')
plt.quiver(0, 0, -axis2[0], -axis2[1], angles='xy', scale_units='xy', scale=1, color='red')

# Formatting
plt.title('Principal Axes of a Quadratic Form')
plt.xlabel('x')
plt.ylabel('y')
plt.grid(True)
plt.axhline(0, color='black', linewidth=0.5)
plt.axvline(0, color='black', linewidth=0.5)
plt.legend()
plt.axis('equal')
plt.show()
Figure 3.2: The ellipse defined by the quadratic form. The eigenvectors (red) point along the major and minor axes.

The plot shows it perfectly. The quadratic form describes a rotated ellipse, and the eigenvectors of the associated symmetric matrix point exactly along its major and minor axes. The signs of the eigenvalues (\(\lambda_1=5, \lambda_2=-1\)) tell us it’s a hyperbola (one positive, one negative), not an ellipse. My description was slightly off, but the math and the plot reveal the truth! This is why we do the computation.

3.8 Tutorial 4: Principal Axis Transformation of Quadratic Forms

  1. Step 1: Write the quadratic form in \(X^TAX = c\) form and compute eigenvalues/eigenvectors of \(A\).
  2. Step 2: Construct the modal matrix \(B\), compute \(B^{-1}\), and form normalized modal matrix \(\hat{B}\).
  3. Step 3: Write orthogonal transformation \(X = \hat{B}Y\).
  4. Step 4: Express the quadratic form in principal axis form with numeric eigenvalues.
  5. Step 5: Express \(Y = B^{-1}X\) with numeric coefficients.

Problem 1: The energy stored in a coupled capacitor system is given by: \[ 5x_1^{2}+4x_1x_2+3x_2^{2}=10 \] Transform to principal-axis form, find the orthogonal transformation \(X=\hat{B}Y\), classify the conic, and give \(Y=B^{-1}X\).

Solution:

Step 1: \(A = \begin{bmatrix}5 & 2\\2 & 3\end{bmatrix}\)

Eigenvalues: \(\lambda_1 = 6.236\), \(\lambda_2 = 1.764\)

Eigenvectors:
\(v_1 = \begin{pmatrix}3.236\\2\end{pmatrix}\), \(v_2 = \begin{pmatrix}-1.236\\2\end{pmatrix}\)

Step 2: Modal matrix \(B = \begin{bmatrix}3.236 & -1.236\\2 & 2\end{bmatrix}\)
Normalized modal matrix:

\[ \hat{B} = \begin{bmatrix}0.853 & -0.525\\0.527 & 0.850\end{bmatrix} \]

Step 3: Orthogonal transformation: \(X = \hat{B}Y\)

Step 4: Principal axis form:

\[ 6.236 y_1^2 + 1.764 y_2^2 = 10 \]

Step 5: Inverse transformation:

\[ Y = B^{-1}X = \begin{bmatrix} 0.359 x_1 + 0.222 x_2\\ -0.251 x_1 + 0.591 x_2 \end{bmatrix} \]

Classification: ellipse


Problem 2: The spread of pixel intensity in an image block is modeled by:

\[ 4x_1^{2}-6x_1x_2+9x_2^{2}=36 \]

Transform to principal axis form, find the orthogonal transformation, classify the conic, and write \(Y=B^{-1}X\).

Solution:

Step 1: \(A = \begin{bmatrix}4 & -3\\-3 & 9\end{bmatrix}\)

Eigenvalues: \(\lambda_1 = 10.405\), \(\lambda_2 = 2.595\)

Eigenvectors: \(v_1 = \begin{pmatrix}1.405\\-3\end{pmatrix}\), \(v_2 = \begin{pmatrix}-6.405\\-3\end{pmatrix}\)

Step 2: Modal matrix \(B = [v_1\ v_2]\)
Normalized modal matrix:

\[ \hat{B} = \begin{bmatrix}0.421 & -0.894\\ -0.907 & -0.447\end{bmatrix} \]

Step 3: \(X = \hat{B}Y\)

Step 4: Principal axis form:

\[ 10.405 y_1^2 + 2.595 y_2^2 = 36 \]

Step 5: Inverse transformation:

\[ Y = B^{-1}X = \begin{bmatrix} 0.404 x_1 - 0.172 x_2\\ 0.865 x_1 + 0.177 x_2 \end{bmatrix} \]

Classification: ellipse


Problem 3: The potential energy of a 3-mass vibrating system is given by:
\[ 6x_1^{2}+4x_1x_2+3x_2^{2}+2x_2x_3+2x_3^{2}=20 \]

Transform to principal axis form, find the orthogonal transformation, classify the quadric surface, and write \(Y=B^{-1}X\).

Solution:

Step 1: \(A = \begin{bmatrix}6 & 2 & 0\\2 & 3 & 1\\0 & 1 & 2\end{bmatrix}\)

Eigenvalues: \(\lambda_1 = 7.541\), \(\lambda_2 = 2.459\), \(\lambda_3 = 1.0\)

Eigenvectors (normalized):
\(v_1 = \begin{pmatrix}0.814\\0.556\\0.166\end{pmatrix}\),
\(v_2 = \begin{pmatrix}-0.576\\0.817\\-0.056\end{pmatrix}\),
\(v_3 = \begin{pmatrix}-0.062\\-0.156\\0.986\end{pmatrix}\)

Step 2: Modal matrix \(B = [v_1\ v_2\ v_3]\)

\[ B = \begin{bmatrix} 0.814 & -0.576 & -0.062\\ 0.556 & 0.817 & -0.156\\ 0.166 & -0.056 & 0.986 \end{bmatrix}, \quad B^{-1} \approx B^T \]

Normalized modal matrix: \(\hat{B} = B\)

Step 3: \(X = \hat{B}Y\)

Step 4: Principal axis form:

\[ 7.541 y_1^2 + 2.459 y_2^2 + 1.0 y_3^2 = 20 \]

Step 5: Inverse transformation:

\[ \begin{bmatrix}y_1\\y_2\\y_3\end{bmatrix} = B^{-1} X \approx \begin{bmatrix} 0.814 & 0.556 & 0.166\\ -0.576 & 0.817 & -0.056\\ -0.062 & -0.156 & 0.986 \end{bmatrix} \begin{bmatrix}x_1\\x_2\\x_3\end{bmatrix} \]

Classification: All positive eigenvalues \(\Rightarrow\) ellipsoid


Problem 4: A quadratic classifier is modeled by: \[ 2x_1^{2}+2x_1x_2+2x_2^{2}=8 \]

Reduce to principal axis form, find the orthogonal transformation, classify the conic, and write \(Y=B^{-1}X\).

Solution:

Step 1: \(A = \begin{bmatrix}2 & 1\\1 & 2\end{bmatrix}\)

Eigenvalues: \(\lambda_1 = 3\), \(\lambda_2 = 1\)

Eigenvectors using formula: \(v_1 = \begin{pmatrix}1\\1\end{pmatrix}\), \(v_2 = \begin{pmatrix}-1\\1\end{pmatrix}\)

Step 2: Modal matrix \(B = [v_1\ v_2]\), normalized:

\[ \hat{B} = \frac{1}{\sqrt{2}}\begin{bmatrix}1 & -1\\1 & 1\end{bmatrix} \]

Step 3: \(X = \hat{B}Y\)

Step 4: Principal axis form:

\[ 3 y_1^2 + 1 y_2^2 = 8 \]

Step 5: Inverse transformation:

\[ Y = B^{-1}X = \frac{1}{2}\begin{bmatrix}1 & 1\\-1 & 1\end{bmatrix}\begin{bmatrix}x_1\\x_2\end{bmatrix} = \begin{bmatrix}\frac{x_1 + x_2}{2}\\ \frac{-x_1 + x_2}{2}\end{bmatrix} \]

Classification: ellipse


Problem 5: A Lyapunov function candidate is: \[ 7x_1^{2}-4x_1x_2+4x_2^{2}+2x_2x_3+3x_3^{2}=15 \]

Transform to principal axis form, find the orthogonal transformation, determine stability, and write \(Y=B^{-1}X\).

Solution:

Step 1: \(A = \begin{bmatrix}7 & -2 & 0\\-2 & 4 & 1\\0 & 1 & 3\end{bmatrix}\)

Eigenvalues: \(\lambda_1 = 8.061\), \(\lambda_2 = 3.0\), \(\lambda_3 = 2.939\)

Eigenvectors (normalized):
\(v_1 = \begin{pmatrix}0.872\\-0.441\\0.207\end{pmatrix}\),
\(v_2 = \begin{pmatrix}-0.308\\0.886\\-0.345\end{pmatrix}\),
\(v_3 = \begin{pmatrix}0.379\\0.153\\0.912\end{pmatrix}\)

Step 2: Modal matrix \(B = [v_1\ v_2\ v_3]\), \(\hat{B} = B\), \(B^{-1} \approx B^T\)

Step 3: \(X = \hat{B}Y\)

Step 4: Principal axis form:

\[ 8.061 y_1^2 + 3.0 y_2^2 + 2.939 y_3^2 = 15 \]

Step 5: Inverse transformation:

\[ Y = B^{-1} X \approx \begin{bmatrix} 0.872 & -0.441 & 0.207\\ -0.308 & 0.886 & -0.345\\ 0.379 & 0.153 & 0.912 \end{bmatrix} \begin{bmatrix}x_1\\x_2\\x_3\end{bmatrix} \]

Classification: All eigenvalues positive \(\Rightarrow\) stable system


Interpretation of the sign of eigenvalues and the nature of normalized modal matrix in terms of stability

The orthogonal modal transformation \(X=\hat{B}Y\) rotates (and possibly reflects) the coordinate axes to align with the eigen-directions of \(A\). This eliminates cross-terms (interaction terms) and diagonalizes the quadratic form. Because \(\hat{B}\) is orthonormal, it preserves quadratic quantities such as energy and definiteness: eigenvalue signs therefore directly determine the type of conic/quadric and stability. The transformed coordinates \(y_i\) are independent modal contributions scaled by the eigenvalues.

3.9 Module II Summary

  • The core idea of this module is the equation \(Ax = \lambda x\).
  • Eigenvectors (\(x\)) are the special directions for a matrix that are only scaled, not rotated. Eigenvalues (\(\lambda\)) are the scaling factors.
  • We find eigenvalues by solving the characteristic equation \(\det(A - \lambda I) = 0\).
  • Diagonalization (\(A = S \Lambda S^{-1}\)) is a powerful tool that simplifies matrix powers and reveals the true nature of a transformation.
  • Symmetric matrices are the nicest of all, with real eigenvalues and orthogonal eigenvectors, leading to the decomposition \(A = Q \Lambda Q^T\).
  • This theory has direct applications in geometry, such as finding the principal axes of shapes defined by quadratic forms.