APPLICATIONS of LEAST SQUARES APPROXIMATION

Regression Line: there are many other applications of optimization. For example, 'fitting' a curve to data is often important for modelling and prediction. To the left below, a linear fit seems appropriate for the given data, while a quadratic fit seems more appropriate for the data to the right.

But what does 'best' fit mean and how might we determine that best fit? This is an optimization problem which for the linear case can be formulated as minimizing a function of two variables.

Suppose an experiment is conducted at times $x_1,\, x_2, \, \dots \, x_n$ yielding observed values $y_1,\, y_2, \, \dots \, y_n$ at these respective times. If the points $(x_j,\, y_j),\ 1 \le j \le n,$ are then plotted and they look like ones the to left above, one might conclude that the experiment can be described mathematically by a Linear Model indicated by the line drawn through the data. Using $y = mx + b$ for the equation of the model line, we get predicted values $p_1,\, p_2, \, \dots \, p_n$ at $x_1,\, x_2, \, \dots \, x_n$ by setting $p_j = mx_j + b$. The difference $p_j - y_j = mx_j + b - y_j$ between the predicted and observed values is a measure of the error at the $j^{\hbox{th}}$-observation: it measures how far above or below the model line the observed value $y_j$ lies. We want to minimize the total error over all observations.

The minimum value of the sum $$E(m,\,b) \ = \ (p_1 - y_1)^2 + (p_2 - y_2)^2 + \ldots + (p_n - y_n)^2\qquad \qquad \qquad \qquad\qquad$$ $$\qquad \qquad \qquad \ = \ ( mx_1 + b - y_1)^2 + ( mx_2 + b - y_2)^2 + \ldots + ( mx_n + b - y_n)^2$$ as $m,\ b$ vary is called the Least Squares Error.
For the minimizing values of $m$ and $b$, the corresponding line $y = mx + b$ is called the Least Squares Line or the Regression Line.

Taking squares $(p_j - y_j)^2$ avoids positive and negative errors canceling each other out. Other choices like $|p_j - y_j|$ could be used, but the fact that we'll want to differentiate to determine $m$ and $b$ means the calculus will be a lot simpler if we don't use absolute values!!

To determine the critical point of $E(m,\, b)$ differentiate partially: $$\frac{\partial E}{\partial m} \ = \ 2\left(x_1(m x_1 + b-y_1) + x_2(mx_2 + b- y_2) + \ldots + x_n(mx_n + b - y_n)\right) \ = \ 0\,,$$ $$\frac{\partial E}{\partial b} \ = \ 2\left((m x_1 + b-y_1) + (mx_2 + b- y_2) + \ldots + (mx_n + b - y_n)\right) \ = \ 0\,,$$ giving a pair a simultaneous linear equations in $m$ and $b$. Now solve for $m$ and $b$. To do it by hand, as in homework assignments, one hopes that $n$ is small - say $n = 3$ - but solving the simultaneous equations is a problem in linear algebra, and most spread-sheet programs or computer algebra systems have a built-in algorithm for calculating $m$ and $b$ to determine the regression line for a given data set. The mathematics explains where that algorithm originated!

Problem: can you see how to modify this argument for a linear model to determine the Least Squares Parabola providing a best quadratic fit for data as shown to the right above? its surface area is given by the integral $$ \int\int_D\ \left\|\frac{\partial {\mathbf\Phi} }{\partial u} \times \frac{\partial {\mathbf\Phi} }{\partial v} \right\|\, du dv\,.$$ The case of the paraboloid

studied previously is typical.

As an illustration of the surface area formula, let's prove another remarkable result of Archimedes - obtained, remember, over 1,600 years before calculus was invented!

Recall first some known results:

the surface area of a sphere of radius $R$ is $4\pi R^2$,

the surface area of right circular cylinder, open top and bottom, of radius $R$ and height $h$ is $2\pi R h$.

But if the cylinder exactly circumscribes the sphere as shown to the right, i.e, the cylinder and the sphere have the same radius and height, then $h = 2R$. In this case, both the sphere and circumscribing cylinder have equal surface area $4\pi R^2$. Even though this result is clear on the basis of computation, does it seem obvious geometrically?

Now suppose the complete sphere is replaced by a sphere 'topped' at the North and South Poles, i.e., the portion of a sphere between two lines of latitude, as shown in blue to the right below where the 'cylinder' slider also produces the green cylinder exactly circumscribing this capped sphere. Drag the animation around to convince yourself of this!

Problem: do the blue capped sphere and the exactly circumscribing green cylinder always have equal surface area?

Solution: the surface area of the cylinder is easy to determine - it's just $2\pi R h$ when the topped sphere has radius $R$ and height $h$. Determining the surface area of the topped sphere, however, needs the surface area integral formula: parametrize the sphere by $${\mathbf\Phi} (\theta, \, \phi) = R \cos \theta \sin \phi \, {\bf i} + R \sin \theta \sin \phi \, {\bf j} + R \cos \phi\, {\bf k}\,.$$ The scalar surface element is then $$ \left\|\frac{\partial {\mathbf\Phi}}{\partial \theta} \times \frac{\partial {\mathbf\Phi}}{\partial \phi}\right\|\,d\theta d \phi\ = \ R^2 \sin \phi\, d\theta d \phi\,.$$ Now slice the sphere by the equatorial plane and a horizontal plane of height $h$ above this equatorial plane. To use the surface area integral we need to determine the limits of integration.

The portion of the upper hemisphere centered at the origin of radius $R$ cut off by the plane $z = h$ is shown in cross-section to the right. Its $$\hbox{surface area} = R^2 \int_0^{2\pi}\Big(\int_{\alpha}^{\pi/2} \sin \phi\, d\phi \Big)d\theta$$ $$\qquad = \ 2\pi R^2 \cos \alpha \ = \ 2\pi R h\,,$$ which is exactly the same as the surface area of the circumscribing cylinder of radius $R$ and height $h$!!

Scalar Surface Integrals: the surface area formula is just the special case $f(x,\,y,\,z) = 1$ of the integral of a scalar function over a parametrized surface $S$.

Definition: the surface integral of a scalar valued continuous function $z = f(x,\,y,\,z)$ over a surface $S$ parametrized by ${\mathbf \Phi} : D \,\subseteq \, {\mathbb R}^2 \to {\mathbb R}^3$ is given by $$\int\int_S \ f\, dS \ = \ \int\int_{D}\ f\big({\mathbf \Phi} (u,\,v)\big)\, \left \|\frac{\partial {\mathbf \Phi}}{\partial u} \times \frac{\partial {\mathbf \Phi}}{\partial v}\right \|\, du dv\,.$$

Notice that the change of variable formula for double integrals was just the 'flat earth' version of a surface integral with distorting function a mapping ${\mathbf \Phi} : D \subseteq {\mathbb R}^2 \to {\mathbb R}^2$ having range in ${\mathbb R}^2$, not in ${\mathbb R}^3$.

In practice, the surface area element for special categories of surfaces is often needed.

I. Graph of a function $z = f(x,\,y)$: when ${\mathbf \Phi}(x,\,y) = x\,{\bf i} +y\,{\bf j} +f(x,\,y)\,{\bf k}$ is the parametrization of the graph of $z = f(x,\,y)$ by rectangular coordinates $x$ and $y$, then for a fixed point $P = {\mathbf \Phi}(a,\,b) = (a,\,b,\, f(a,\,b))$ on the surface, the curves on this surface through $P$ corresponding to $x \to {\mathbf \Phi}(x,\, b)$ and $y \to {\mathbf \Phi}(a,\, y)$ have respective tangent vectors $${\mathbf \Phi}_x(a) \ = \ \frac{\partial {\mathbf \Phi}}{\partial x}(a,\,b) \ = \ {\bf i} + \frac{\partial f}{\partial x}(a,\,b)\, {\bf k}, \qquad {\mathbf \Phi}_y(b) \ = \ \frac{\partial {\mathbf \Phi}}{\partial y}(a,\,b) \ = \ {\bf j} + \frac{\partial f}{\partial y}(a,\,b)\, {\bf k}. $$ Thus the normal to the surface at $P(a,\,b)$ is $${\bf n} (a,\,b) \ = \ {\mathbf \Phi}_x(a) \times {\mathbf \Phi}_y(b) \ = \ \left| \begin{array}{ccc} {\bf i} & \ \ {\bf j}& {\bf k}\\ \\ 1 &\ \ 0 & f_x(a,\,b)\\ \\ 0 & \ \ 1 &f_y(a,\,b) \end{array} \right| \ = \ -f_x(a,\,b)\, {\bf i} -f_y(a,\,b)\, {\bf j} + {\bf k}\,. $$ Thus the surface area element for the graph of $z = f(x,\,y)$, $$ dS \ = \ \| {\bf T}_x \times {\bf T}_y\| \ = \ \sqrt {(f_x)^2 + (f_x)^2 + 1} \ = \ \sqrt {\|(\nabla f)(x,\,y)\|^2 + 1}\,;$$ this graph will have $$ \hbox{surface area} \ = \ \int\int_D \ \sqrt {\|(\nabla f)(x,\,y)\|^2 + 1}\, dxdy$$ over a region $D$ in the $xy$-plane.

Example 1: evaluate the integral $$ I \ = \ \int\int_S \ 2xyz \, dS$$ when $S$ is the part of the plane $$ 2z \ = \ 2 - x +2y\,,$$ above the rectangle $[0,\,1] \times [0,\,1]$ in the $xy$-plane.

Solution: the surface area element for the plane $$z \ = \ 1 - \frac{x}{2} +y$$ is given by $$dS \ = \ \sqrt{z_x^2 + z_y^2 + 1} \ = \ \sqrt{\frac{1}{4} + 1 + 1} \ = \ \frac{3}{2}\,.$$

On the other hand, on $S$, $$2xyz \ = \ 2xy\left(1 - \frac{x}{2} +y\right) \qquad \qquad $$ $$\qquad \qquad = \ 2xy - x^2y + 2xy^2\,.$$ Thus $$I \ = \ \frac{3}{2} \int_0^1\left(\int_0^1\, (2xy - x^2y + 2xy^2)dy\right)dx$$ $$= \ \frac{3}{2} \int_0^1\, \left(x - \frac{1}{2}x^2 + \frac{2}{3}x\right)\, dx\,. $$ Consequently, $$I \ = \ \frac{3}{2}\left(\frac{1}{2} - \frac{1}{6} +\frac{1}{3}\right) \ = \ 1\,.$$

II. Surface of revolution: if $y = f(x)$ is a positive function on $[a,\, b]$, then rotating the graph of $f$ around the $x$-axis produces a surface of revolution $S$ whose cross-section by a plane $x = c$ perpendicular to the $x$-axis is a circle in the plane $x = c$ with radius $f(c)$ and center at $(c,\,0,\,0)$. Since the equation of this circle is the space curve $(c,\, f(c)\cos \psi,\, f(c)\sin \psi),\ 0 \le \psi \le 2\pi$, varying $c$ parametrizes $S$ by $${\mathbf \Phi} (x,\, \psi) \ = \ \big(x,\, f(x)\cos \psi,\, f(x)\sin \psi)\,, \quad a \le x\le b, \ \ 0 \le \psi \le 2\pi\,.$$ Thus $${\mathbf \Phi}_x \ = \ {\bf i} + f'(x)\cos \psi\, {\bf j} + f'(x) \sin \psi\,{\bf k}, \qquad {\mathbf \Phi}_{\psi} \ = \ - (f(x) \sin \psi)\, {\bf j} + (f(x) \sin \psi)\, {\bf k}\,, $$ and so the normal to the surface is $${\mathbf \Phi}_x\times {\mathbf \Phi}_{\psi} \ = \ \left| \begin{array}{ccc} \ \ {\bf i}& {\bf j} & {\bf k}\\ \\ \ \ 1 & f'(x)\cos \psi & f'(x)\sin \psi \\ \\ \ \ 0 & -f(x)\sin \psi & f(x)\cos \psi \end{array} \right| \ = \ f(x)\big(f'(x)\, {\bf i} -\cos \psi\, {\bf j} - \sin \psi\, {\bf k}\big)\,. $$ Thus yet another application of the Pythgorean trig identity gives $$ dS \ = \ | {\mathbf \Phi}_x \times {\mathbf \Phi}_{\psi} | \ = \ |f(x)|\big(1 + |f'(x)|^2 \big)^{1/2}dx d\psi\,.$$ Consequently, $S$ has $$\hbox{surface area} \ = \ \int_0^{2 \pi}\int_a^b \, |f(x)|\big(1 + |f'(x)|^2 \big)^{1/2}dx d\psi\ = \ 2\pi\int_a^b \, |f(x)|\big(1 + |f'(x)|^2 \big)^{1/2}dx \,.$$

Example 2: use the fact that a sphere of radius $R$ is created by rotating the graph of $$ z \ = \ \sqrt{R^2 - x^2}\,, \quad |x| \le R\,,$$ about the $x$-axis to determine the surface area of a sphere of radius $R$.

Solution: when $z = \sqrt{R^2 - x^2}$ is rotated about the $x$-axis,

$$dS = \sqrt{R^2 - x^2} \left( 1 + \left(\frac{x}{\sqrt{R^2 - x^2}}\right)^2\right)^{1/2}dx d\psi.$$ After simplification, this becomes $dS = R\,dxd\psi$. Thus a sphere of radius $R$ has $$\hbox{surface area} \ = \ \int_0^{2 \pi}\int_{-R}^R\, R\,dxd\psi\ = \ 4\pi R^2\,.$$

Can you evaluate this last double integral? The following standard integral might help:

Recall: $$\int\, \sec^3u\, du \ = \ \frac{1}{2} \left (\ln \left| \sec u + \tan u\right| + \sec u \tan u \right) + C$$