Linear regression

The minimum value of the sum $$E(m,\,b) \ = \ (p_1 - y_1)^2 + (p_2 - y_2)^2 + \ldots + (p_n - y_n)^2\qquad \qquad \qquad \qquad\qquad$$ $$\qquad \qquad \qquad \ = \ ( mx_1 + b - y_1)^2 + ( mx_2 + b - y_2)^2 + \ldots + ( mx_n + b - y_n)^2$$ as $m,\ b$ vary is called the least squares error. For the minimizing values of $m$ and $b$, the corresponding line $y = mx + b$ is called the least squares line or the regression line.

In the following video, we derive the equations for a least squares line and work an example.

For your convenience, we repeat the calculation in text form:

To determine the critical point of $E(m,\, b)$ we compute the components of the gradient and set them equal to zero: $$\frac{\partial E}{\partial m} \ = \ 2\left(x_1(m x_1 + b-y_1) + x_2(mx_2 + b- y_2) + \ldots + x_n(mx_n + b - y_n)\right) \ = \ 0\,,$$ $$\frac{\partial E}{\partial b} \ = \ 2\left((m x_1 + b-y_1) + (mx_2 + b- y_2) + \ldots + (mx_n + b - y_n)\right) \ = \ 0\,,$$ After rearranging terms, we get \begin{eqnarray*}0 \ = \ \frac{\partial E}{\partial m} &=& 2\Big(\big(x_1^2 + x_2^2 + \ldots + x_n^2\big)m + \big(x_1 + x_2 + \ldots + x_n\big)b -\big(x_1y_1 + x_2y_2 + \ldots + x_n y_n\big)\Big), \cr \cr 0 \ = \ \frac{\partial E}{\partial m} &=& 2\Big(\big(x_1 + x_2 + \ldots + x_n\big)m + \big(1 + 1 + \ldots + 1\big)b -\big(y_1 + y_2 + \ldots + y_n\big)\Big)\,.\end{eqnarray*} That's two linear equations in $m$ and $b$, and solving two linear equations in two unknowns isn't hard. In fact, most spreadsheet programs or computer algebra systems have a built-in algorithm for calculating $m$ and $b$ to determine the regression line for a given data set. Here's how they do it.

Dividing the two equations by $2$ and rearranging terms gives the system of equations \begin{eqnarray*} \left ( \sum_{i=1}^n x_i^2 \right ) m + & \left (\sum_{i=1}^n x_i \right ) b & \ = \ \sum_{i=1}^n x_i y_i, \cr \left ( \sum_{i=1}^n x_i \right ) m + & nb & \ = \ \sum_{i=1}^n y_i. \end{eqnarray*} Instead of looking at sums, it's convenient to look at averages, which we denote with angle brackets. Let $$\langle x\rangle \ = \ \frac{1}{n} \sum_{i=1}^n x_i, \qquad \langle x^2 \rangle \ = \ \frac{1}{n}\sum_{i=1}^n x_i^2, \qquad \langle y \rangle \ = \ \frac{1}{n}\sum_{i=1}^n y_i, \qquad \langle xy \rangle \ = \ \frac{1}{n}\sum_{i=1}^n x_i y_i. $$ After dividing by $n$, our equations become \begin{eqnarray*} \langle x^2 \rangle m + & \langle x \rangle b \ = \ & \langle xy \rangle, \cr \langle x \rangle m + & \phantom{\langle x \rangle} b \ = \ & \langle y \rangle. \end{eqnarray*} The first equation minus $\langle x \rangle$ times the second equation gives $$\left ( \langle x^2 \rangle - \langle x \rangle^2 \right ) m \ = \ \langle xy \rangle - \langle x \rangle \langle y \rangle,$$ which we can solve for $m$. Plugging that back into the second equation gives $b$. The results are

The equations of the least squares line through the points $(x_1,y_1)$, $(x_2,y_2), \ldots, (x_n,y_n)$ is $y=mx+b$, where \begin{eqnarray*} m \ = \ & \displaystyle{ \frac{\langle xy \rangle - \langle x \rangle \langle y \rangle} {\langle x^2 \rangle - \langle x \rangle^2}} & \ = \ \frac{\displaystyle{n \left (\sum_{i=1}^n x_iy_i\right ) - \left ( \sum_{i=1}^n x_i\right ) \left ( \sum_{i=1}^n y_i\right )}} {\displaystyle{n\left ( \sum_{i=1}^n x_i^2\right ) - \left (\sum_{i=1}^n x_i \right )^2}}, \cr \cr b \ = \ & \frac{\displaystyle{\langle x^2 \rangle\langle y \rangle - \langle x \rangle \langle xy \rangle}} {\displaystyle{\langle x^2 \rangle - \langle x \rangle^2}} & \ = \ \frac{\displaystyle{\left ( \sum_{i=1}^n x_i^2\right ) \left ( \sum_{i=1}^n y_i \right ) - \left ( \sum_{i=1}^n x_i\right ) \left (\sum_{i=1}^n x_i y_i \right )}} {\displaystyle{n\left (\sum_{i=1}^n x_i^2 \right ) - \left (\sum_{i=1}^n x_i \right )^2}}. \end{eqnarray*}