Simple ordinary least squares regression (SOLSR) means the following. Given data , , find a line in represented by that fits the data in the following sense. The loss of each data point to the line is

for every ,

so we find that minimizes the loss function

See here for a closed-form of the minimizer . Instead of SOLSR, one can consider the distance between a data point and a line in as the loss. Notice that is not the distance from to unless . Then the new least squares problem can be formulated as follows.

A general line in can be expressed as where . Thus the distance between and this line is

for every .

Hence we want to find that minimizes the loss function

.

Notice that, not only SOLSR, this problem also has a lot of real-life applications. It turns out there is still a closed-form for , and we will derive it. There is a high chance that the answer can be found somewhere, but we could not find it so far. Also it is a good exercise for Hong Kong students who know Additional Maths, although the subject disappeared.

Setting the partial derivative to be zero one has

which implies

.

Using the formulae and , one has

where

.

Plugging in the above expression of into the above formula, we reach

where

.

It implies the formula

,

which concludes the result:

* Theorem 1.* The minimizer of the loss function satisfies

and .

In particular the point lies on the best fitted line .

We leave to the readers to work on the high-dimensional cases, and the case using weighted data.