The following theorem provides all the solutions (optimal set) of a least-squares problem.
Theorem: optimal set of ordinary least-squares (Proof).
The optimal set of the OLS problem
can be expressed as where is the pseudo-inverse of , and is the minimum-norm point in the optimal set. If is full column rank, the solution is unique, and equal to In general, the particular solution is the minimum-norm solution to the least-squares problem.
We consider the situation where with
the data matrix (known), with full column rank (hence ).
is the measurement (known).
is the vector to be estimated (unknown).
is a measurement noise or error (unknown).
We can use OLS to provide an estimate of . The idea is to seek the smallest vector such that the above equation becomes feasible, that is,
This leads to the OLS problem:
Since is full column rank, its SVD can be expressed as
where contains the singular values of , with .
Since is full column rank, the solution to the OLS problem is unique, and can be written as a linear function of the measurement vector : with the pseudo-inverse of . Again, since is full column rank,
The OLS formulation provides an estimate of the input such that the residual error vector is minimized in norm. We are interested in analyzing the impact of perturbations in the vector , on the resulting solution . We begin by analyzing the absolute errors in the estimate, and then turn to the analysis of relative errors.
Let us assume a simple model of potential perturbations: we assume that belongs to a unit ball: , where is given. We will assume for simplicity; the analysis is easily extended to any .
In the above we have exploited the fact that is a left inverse of , that is, .
The set of possible errors on the solution is then given by which is an ellipsoid centered at zero, with principal axes given by the singular values of . This ellipsoid can be interpreted as an ellipsoid of confidence for the estimate , with size and shape determined by the matrix .
We can draw several conclusions from this analysis:
The largest absolute error in the solution that can result from a unit-norm, additive perturbation on is of the order of , where is the smallest singular value of .
The largest relative error is , the condition number of .
We now return to the case of an OLS with full column rank matrix .
Let us assume that belongs to a unit ball: . The set of resulting errors on the solution is then
which is an ellipsoid centered at zero, with principal axes given by the singular values of . This ellipsoid can be interpreted as an ellipsoid of confidence for the estimate , with size and shape determined by the matrix .
It can be shown that the OLS estimator is optimal in the sense that it provides the ‘‘smallest’’ ellipsoid of confidence among all unbiased linear estimators. Specifically:
This optimality of the LS estimator is referred to as the BLUE (Best Linear Unbiased Estimator) property.