Non-linear least squares

Lore & Background

Non-linear least squares is the form of least squares analysis used to fit a set of m observations with a model that is non-linear in n unknown parameters (m ≥ n). It is used in some forms of nonlinear regression. The basis of the method is to approximate the model by a linear one and to refine the parameters by successive iterations. There are many similarities to linear least squares, but also some significant differences. In economic theory, the non-linear least squares method is applied in (i) the probit regression, (ii) threshold regression, (iii) smooth regression, (iv) logistic link regression, (v) Box–Cox transformed regressors.

Reader's Guide

Non-linear least squares is a fundamental technique in statistical modeling when the relationship between variables is not linear. Its significance lies in its ability to handle complex models where parameters enter nonlinearly, such as in economic applications like probit and threshold regression. The method works by iteratively linearizing the model using a first-order Taylor expansion and solving for parameter updates, a process that continues until convergence. This approach allows analysts to fit models that would otherwise have no closed-form solution, making it indispensable for empirical research in economics and other fields. The technique's legacy is its extension of least squares principles to a much broader class of problems, enabling more realistic and flexible data analysis.

Did You Know?

• Non-linear least squares is used in some forms of nonlinear regression.
• The method approximates the model by a linear one and refines parameters by successive iterations.
• In economic theory, it is applied in probit regression, threshold regression, smooth regression, logistic link regression, and Box–Cox transformed regressors.
• The gradient equations in nonlinear least squares generally do not have a closed solution.

The Iterative Engine — How Nonlinear Least Squares Works

The fundamental goal of least squares is to find the parameter values that make a model's predictions align as closely as possible with observed data, measured by the aggregate squared gap between actual and predicted values. When the model's dependence on its unknown parameters is purely linear, a direct algebraic formula delivers the answer in one step. Nonlinear least squares, however, confronts models where parameters enter in more complex ways—exponentials, ratios, products—and no single closed-form expression exists. Instead, the solver engages in a cycle of iterative refinement: at each step it locally approximates the curved problem with a straight-line surrogate, solves that simpler linear sub-problem, and then re-evaluates the approximation around the improved estimate. This means the heavy lifting at every iteration mirrors the linear case, even though the overall trajectory is nonlinear. Polynomial regression sits at the boundary, describing how variance in a predicted quantity decomposes into the fitted curve and its deviations. Under certain distributional assumptions—normal, Poisson, binomial, exponential—the standardized least-squares solution coincides exactly with the maximum-likelihood estimate, and the procedure can also be re-derived as a method-of-moments estimator.

A Century of Convergence — The Historical Buildup

The least-squares principle did not emerge from a single eureka moment but from a slow accumulation of ideas across the eighteenth century. Newton, in unpublished work from 1671 and again in 1700, first noted that pooling multiple observations reduces error rather than amplifying it; Roger Cotes gave the idea its earliest formal statement in 1722. The method of averages—combining repeated measurements taken under identical conditions—was practiced by Newton during his equinox studies, by Tobias Mayer in his lunar libration work of 1750, and by Laplace in his 1788 analysis of Jupiter and Saturn. Extending the idea to observations gathered under differing conditions, Boscovich applied a least-absolute-deviation approach to the shape of the Earth in 1757, a problem Laplace revisited in 1789 and 1799. Laplace also attempted to ground the criterion in probability, modeling errors with what we now call the Laplace distribution, though his resulting estimator was the posterior median rather than the arithmetic mean. It was not until Legendre's 1805 publication that the method received its first clear, concise algebraic exposition, and within a decade it had become standard practice in French, Italian, and Prussian astronomy and geodesy.

The Ceres Rescue — Least Squares Proves Its Worth

On the first day of 1801, Italian astronomer Giuseppe Piazzi spotted a new object in the sky and tracked it for roughly forty days before solar glare swallowed it from view. The astronomical community urgently needed to know where the body—later named Ceres—would reappear, but doing so without resorting to Kepler's notoriously intricate nonlinear equations of planetary motion seemed nearly impossible. The solution that ultimately succeeded belonged to a twenty-four-year-old Carl Friedrich Gauss, who applied his least-squares machinery to Piazzi's sparse positional data. His prediction was the only one that allowed Hungarian astronomer Franz Xaver von Zach to relocate the lost asteroid. This practical triumph gave the method an undeniable credibility that pure theory could not. Gauss had published his orbital-calculation work in 1809, claiming prior possession of the technique since 1795, which ignited a priority dispute with Legendre. Yet Gauss's contribution went well beyond Legendre's algebraic recipe: he wove the method into the fabric of probability theory, linked it to the normal distribution, and demonstrated that the arithmetic mean is the optimal estimate of a location parameter under those assumptions.

From Ad Hoc Fitting to Optimal Estimation — Theoretical Foundations

The theoretical scaffolding that elevates least squares from a useful trick to a provably optimal procedure was assembled over several decades. In 1810, Laplace, after establishing the central limit theorem, used it to supply a large-sample justification for both the least-squares criterion and the normal error distribution, effectively completing the program he had outlined earlier. In 1822 Gauss articulated what is now recognized as the Gauss-Markov theorem: in a linear model whose errors are uncorrelated, normally distributed, have zero mean, and share a common variance, the least-squares estimator is the best linear unbiased estimator of the coefficients. An extended form of this result broadens the conditions under which optimality holds. Independently, the American mathematician Robert Adrain formulated the least-squares idea in 1808, underscoring that the principle was arriving from multiple intellectual traditions simultaneously. Over the following two centuries, statisticians and error-theory researchers discovered numerous alternative implementations and generalizations, and the method's connection to the exponential family of distributions—where standardized least-squares and maximum-likelihood estimates coincide under mild regularity conditions—further cemented its place as a cornerstone of statistical inference.