Symmetric matrices and quadratic functions
Second-order approximation of non-linear functions
Special symmetric matrices
A square matrix is symmetric if it is equal to its transpose. That is,
The set of symmetric matrices is denoted . This set is a subspace of .
Gram matrix of data points.
A function is said to be a quadratic function if it can be expressed as
for numbers , , and , . A quadratic function is thus an affine combination of the 's and all the ‘‘cross-products’’ . We observe that the coefficient of is .
The function is said to be a quadratic form if there are no linear or constant terms in it: , .
Note that the Hessian (matrix of second-derivatives) of a quadratic function is constant.
There is a natural relationship between symmetric matrices and quadratic functions. Indeed, any quadratic function can be written as
for an appropriate symmetric matrix , vector and scalar . Here, is the coefficient of in ; for , is the coefficient of the term in ; is that of ; and is the constant term, . If is a quadratic form, then , , and we can write where now .
We have seen here how linear functions arise when one seeks a simple, linear approximation to a more complicated non-linear function. Likewise, quadratic functions arise naturally when one seeks to approximate a given non-quadratic function by a quadratic one.
If is a twice-differentiable function of a single variable, then the second order approximation (or, second-order Taylor expansion) of at a point is of the form
where is the first derivative, and the second derivative, of at . We observe that the quadratic approximation has the same value, derivative, and second-derivative as , at .
Example: The figure shows a second-order approximation of the univariate function , with values
at the point (in red).
In multiple dimensions, we have a similar result. Let us approximate a twice-differentiable function by a quadratic function , so that and coincide up and including to the second derivatives.
The function must be of the form
where , , and . Our condition that coincides with up and including to the second derivatives shows that we must have
where is the Hessian, and the gradient, of at .
Solving for we obtain the following result:
Second-order expansion of a function. The second-order approximation ofa twice-differentiable function at a point is of the form
Perhaps the simplest special case of symmetric matrices is the class of diagonal matrices, which are non-zero only on their diagonal.
If , we denote by , or for short, the (symmetric) diagonal matrix with on its diagonal. Diagonal matrices correspond to quadratic functions of the form
Such functions do not have any ‘‘cross-terms’’ of the form with .
Another important class of symmetric matrices is that of the form , where . The matrix has elements , and is symmetric. Such matrices are called symmetric dyads. (If , then the dyad is said to be normalized.)
Symmetric dyads corresponds to quadratic functions that are simply squared linear forms: .
Example:A squared linear form.