Linear Algebra Fundamentals
What Linear Algebra Actually Is
Linear algebra is the math of flat things. Lines, planes, flat surfaces. That's it. Everything in this field deals with how points, lines, and planes behave in n-dimensional space.
You need this if you're doing machine learning, computer graphics, data science, engineering, physics, or anything that involves manipulating data at scale. If someone told you linear algebra is optional in tech, they were wrong.
Vectors: The Building Blocks
A vector is an ordered list of numbers. That's all. Think of it as a point in space with a direction.
Example: [3, 4] points 3 units right and 4 units up. The magnitude (length) is 5. You get that from the Pythagorean theorem.
Vector Operations
- Addition: Add corresponding components. [1,2] + [3,4] = [4,6]
- Scalar multiplication: Multiply every component by a number. 3 × [1,2] = [3,6]
- Dot product: Multiply corresponding components and sum. [1,2] · [3,4] = 1×3 + 2×4 = 11
- Cross product: Only works in 3D. Gives you a vector perpendicular to both inputs
The dot product tells you if vectors point in similar directions. Cross product tells you if they point in opposite directions.
Matrices: Tables That Do Math
A matrix is a rectangular grid of numbers. A 2×3 matrix has 2 rows and 3 columns. Matrices represent linear transformations and systems of equations.
Matrix Types You Should Know
- Square matrix: Same rows and columns (n×n)
- Identity matrix: 1s on diagonal, 0s everywhere else. Multiplying by identity does nothing
- Diagonal matrix: Only diagonal has non-zero values
- Symmetric matrix: Mirrored across diagonal (A = A^T)
- Orthogonal matrix: Columns are perpendicular unit vectors. Transpose equals inverse
Matrix Multiplication: The Weird Part
Matrix multiplication is not element-by-element. It's row times column.
To get element (i,j) in the result, take row i from the first matrix and column j from the second, then compute the dot product.
This matters because:
- AB does NOT equal BA in general
- Matrix multiplication is associative: (AB)C = A(BC)
- It's distributive: A(B+C) = AB + AC
When you multiply a vector by a matrix, you're transforming that vector. Rotations, scaling, shearing—all done with matrix multiplication.
Linear Transformations
A linear transformation preserves two things:
- Lines stay lines (they don't curve)
- Origin stays fixed
Every linear transformation in n-dimensional space can be represented by an n×n matrix. That's the whole point of matrices— they're compact representations of transformations.
Common transformations:
- Scaling: Multiply by diagonal matrix
- Rotation: Multiply by rotation matrix
- Shearing: Off-diagonal entries shift one axis based on another
- Reflection: Flip across an axis or plane
Solving Systems of Linear Equations
Linear algebra's practical heart. You have multiple equations with multiple unknowns. Represent it as Ax = b, where A is the coefficient matrix, x is the unknown vector, and b is the result vector.
Methods to solve:
- Gaussian elimination: Row reduce to row echelon form
- LU decomposition: Break A into lower and upper triangular matrices
- Matrix inverse: x = A⁻¹b (only if A is invertible)
- Cramer's rule: Works but computationally stupid for large systems
If the system has no solution, the equations are inconsistent. If it has infinite solutions, the equations are dependent.
Determinants: The Weird Number
The determinant is a single number computed from a square matrix. It tells you:
- If det = 0: Matrix is singular, not invertible
- If det ≠ 0: Matrix is invertible
- Absolute value of det: How much areas scale during transformation
For a 2×2 matrix [[a,b],[c,d]], the determinant is ad - bc.
For larger matrices, you compute recursively using minors and cofactors. Don't memorize the formula. Know that it exists and what it represents.
Eigenvalues and Eigenvectors
This is where most people check out. Don't.
An eigenvector is a vector that doesn't change direction when you multiply it by a matrix. It only scales or shrinks.
An eigenvalue is the factor by which the eigenvector stretches or compresses.
Ax = λx
- A is the matrix
- x is the eigenvector
- λ (lambda) is the eigenvalue
To find eigenvalues, solve det(A - λI) = 0. The determinant gives you a polynomial. Find the roots. Those roots are your eigenvalues.
Where this matters:
- PCA (Principal Component Analysis) uses eigenvalues to find the most important directions in data
- Google's PageRank uses eigenvalues to rank web pages
- Quantum mechanics is built on eigenvalue equations
- Vibration analysis uses eigenvalues to find resonant frequencies
Key Formulas Reference
| Concept | Formula | What It Tells You |
|---|---|---|
| Vector magnitude | ||v|| = √(v₁² + v₂² + ...) | Length of vector |
| Dot product | a · b = Σ aᵢbᵢ | Similarity of direction |
| Cross product | a × b = |a||b|sin(θ) | Area of parallelogram |
| 2×2 Determinant | ad - bc | Singularity check |
| Eigenvalue equation | Ax = λx | Invariant directions |
| Matrix inverse condition | det(A) ≠ 0 | System solvable |
Practical Applications
Machine Learning: Neural networks are giant matrix multiplications. Weights are matrices. Training is finding the right matrices through gradient descent.
Computer Graphics: Every transformation in 3D rendering—rotation, translation, scaling—is a matrix operation. Your GPU is a linear algebra machine.
Data Science: Dimensionality reduction with PCA. Recommender systems use matrix factorization. Clustering uses distance metrics derived from linear algebra.
Search Engines: PageRank is an eigenvalue problem. The stationary distribution of web page visits is the dominant eigenvector of the link matrix.
Economics: Input-output models (Leontief matrices) predict how changes in one industry affect others.
Getting Started
Here's what you actually need to do:
- Master vectors first. Addition, scalar multiplication, dot product, cross product. These are the atoms.
- Learn matrix operations. Multiplication, transpose, inverse. Practice until multiplication is automatic.
- Understand transformations. See how specific matrices create specific transformations. Visualize them.
- Solve systems by hand. Do Gaussian elimination on paper for 3×3 systems. Understand what row operations do.
- Then tackle eigenvalues. Start with 2×2 matrices. Find the characteristic polynomial. Solve for λ.
For practice: 3Blue1Brown's Essence of Linear Algebra on YouTube. It's the best visual introduction that exists. After that, Gilbert Strang's textbook or his MIT OpenCourseWare lectures.
Don't try to memorize everything. Understand the core concepts: vectors are points, matrices are transformations, eigenvalues are invariant directions. The rest is detail.