Ordinary least squares

Lore & Background

In ordinary least squares, the data consists of n observations, each including a scalar response and a column vector of p regressors. The response variable is modeled as a linear function of the regressors plus an unobserved random error term. This model can be written in matrix notation as y = Xβ + ε, where y and ε are n×1 vectors and X is an n×p matrix of regressors, also called the design matrix. Typically, a constant term is included in the regressors by setting one regressor to 1 for all observations, and its coefficient is called the intercept.

Reader's Guide

Ordinary least squares is significant because it provides a simple, closed-form estimator for linear regression parameters. Geometrically, it minimizes the sum of squared distances, parallel to the dependent variable axis, between data points and the regression surface. The OLS estimator is consistent for level-one fixed effects when regressors are exogenous and have perfect colinearity (rank condition). Under the Gauss–Markov theorem, OLS is optimal in the class of linear unbiased estimators when errors are homoscedastic and serially uncorrelated, providing minimum-variance mean-unbiased estimation. If errors are normally distributed with zero mean, OLS becomes the maximum likelihood estimator, outperforming any non-linear unbiased estimator. Its legacy lies in being a foundational tool for statistical modeling and data analysis.

Did You Know?

• OLS minimizes the sum of the squares of the differences between observed dependent variable values and the output of the linear function of the independent variable.
• The OLS estimator is not consistent under perfect collinearity; perfect collinearity makes OLS estimation impossible because the design matrix is singular.
• By the Gauss–Markov theorem, OLS is optimal in the class of linear unbiased estimators when errors are homoscedastic and serially uncorrelated.
• Under the additional assumption that errors are normally distributed with zero mean, OLS is the maximum likelihood estimator.

The Core Principle and Geometric Vision

OLS works by selecting the unknown parameters of a linear regression model through the principle of least squares. Concretely, the method minimizes the sum of squared gaps between what the observed dependent variable actually shows in the dataset and what the linear function of the independent variable predicts. Geometrically, you can picture this as measuring the squared distances—drawn parallel to the axis of the dependent variable—between each data point and its corresponding location on the regression surface. The smaller those gaps, the tighter the model hugs the data. In the simplest case, where only a single regressor sits on the right-hand side of the equation, the resulting estimator takes a particularly clean and compact formula. Some authors even treat OLS as synonymous with linear regression itself, underscoring how central this technique is to the broader statistical landscape.

The Architecture of the Linear Model

The setup begins with n observations, each pairing a scalar response value with a column vector containing p parameters, or regressors. The model posits that the response is a linear combination of those regressors weighted by unknown coefficients, plus an error term. In vector notation, each observation's response equals the transpose of its regressor vector multiplied by the coefficient vector, plus a random disturbance. That disturbance captures all the influences on the response that originate from sources outside the explanatory variables included in the model. The coefficient vector is a p-by-1 array of the unknowns the analyst seeks to estimate. This structure is what makes OLS applicable: the relationship between inputs and output is assumed to be additive and linear, with randomness entering only through the error component.

Conditions for Consistency and Optimality

OLS earns its reputation through a cascade of statistical guarantees, each tied to specific assumptions. When regressors are exogenous and satisfy the rank condition (no perfect colinearity), the estimator is consistent for the level-one fixed effects. If the regressors also possess finite fourth moments, the variance estimate of the residuals is consistent. The Gauss-Markov theorem then steps in: provided the errors are homoscedastic and serially uncorrelated, OLS is the best linear unbiased estimator in its class. Under the further condition that errors have finite variances, the method delivers minimum-variance mean-unbiased estimation. These layered conditions mean that OLS is not universally optimal; its strength is conditional on the data behaving in specific, well-defined ways.

The Normality Upgrade and Maximum Likelihood

A particularly powerful extension emerges when one adds the assumption that the errors follow a normal distribution with zero mean. Under this stronger condition, the OLS estimator is no longer merely the best among linear unbiased estimators—it becomes the maximum likelihood estimator. This is a significant upgrade because maximum likelihood estimators are not confined to the linear class; they can be compared against any non-linear unbiased estimator, and OLS still outperforms them. In other words, the normality assumption elevates OLS from a within-class champion to a broader optimality winner. This result ties the practical, computationally simple least-squares procedure to the deeper theoretical machinery of likelihood-based inference, giving analysts a principled justification for choosing OLS when the normality assumption is defensible.

Gallery