Meetup summary

2025-09-05 - Linear algebra basics - part 3 (diagonalization)

The following videos will give you a good intuition/understanding about the intention behind a change of basis and eigendecomposition. We only briefly touched on the change-of-basis interpretation of matrix multiplication in the last session, so this is a good refresher and backs it with nice visualizations.

Agenda:

This will be the last session on linear algebra basics (assuming we get through everything). The general theme is diagonalization and eigendecomposition. This is a very useful and widely applicable operation that helps with all of the following (not exhaustive):

  • Gain insight into the “core” impact of effect of left-multiplication by a given matrix.
  • Quantify numerical stability of multiplication by a given matrix or its inverse
  • Distinguish saddle points from local optima for high-dimensional differentiable functions.
  • Directly compute fixpoints of (finite) Markov chains.
  • Compute arbitrary complex functions of matrices (given an appropriate series expansion).
  • Analyze various optimization routines through the lens of linear algebra to understand what’s happening at a deeper level.

Note that we will not cover the above use cases above (but might touch on some of them). Instead, we will directly cover the following:

  • Eigendecomposition:
    • The eigenvector/eigenvalue conditions
    • How to compute them
    • Eigenvector renormalization
  • Linear forms
  • Quadratic forms
  • Symmetric/Hermitian matrices:
    • Review of definitions
    • Canonicalization of quadratic forms
    • As a special case of eigendecomposition
  • Spectral theorem (just the most salient results and their derivations):
    • The eigenvalues of a Hermitian are real.
    • Eigenvectors of distinct eigenvalues are orthogonal.
    • A Hermitian matrix can always be decomposed into a full basis of mutually-orthogonal eigenvectors (even with algebraic multiplicity).
  • Positive (negative) (semi)definite matrices:
    • Quadratic form definition
    • Eigenvalue definition
    • Equivalence between the above
    • Why is this useful?

Notes

We got through the agende through the basic spectral theorem statement and proved some special properties of Hermitian matrices, but did not prove why a full eigenbasis necessarily exists. We also discussed quadratic forms and positive definiteness but did not get into the eigenvalue definition or show why this is equivalent. We’re putting the linear algebra path on pause due to low turnout/interest and will resume next time to pick up where we left off on probability. (We now have much—but not all—of what we need to get into random vectors now.)

tags: