Multiple Linear Regression Jan 14 2019

Today

• Matrix warmup
• Multiple Linear Regression
• Matrix setup

Matrix warmup

See handout

Simple linear regression

Recall in simple linear regression:

Have $n$ observations of a response $y_i$, and a single explanatory variable, $x_i$.

The response is related to the explanatory variable by: $$y_i={\beta }_0+{\beta }_1{x}_i+{ϵ}_{i}i=1,\dots ,n$$

where ${ϵ}_i$ are independent and identically distributed with expected value 0, and variance ${\sigma }^2$.

Multiple linear regression

Now we have more than one explanatory variable.

Have $n$ observations of a response, $y_i$ and a set of explanatory variables, $(x_{i1},x_{i2},\dots ,x_{i(p-1)})$.

The response is related to the explanatory variables by: $$y_i={\beta }_0+{\beta }_1{x}_{i1}+{\beta }_2{x}_{i2}+\dots +{\beta }_{p-1}{x}_{i(p-1)}+{ϵ}_{i}i=1,\dots ,n$$

where ${ϵ}_i$ are independent and identically distributed with expected value 0, and variance ${\sigma }^2$.

Example: Galápagos Islands

Faraway 2.6

Measurements on 30 Galápagos Islands are made.

First 5 islands:

	Species	Area	Elevation	Nearest	Scruz	Adjacent
Baltra	58	25.09	346	0.6	0.6	1.84
Bartolome	31	1.24	109	0.6	26.3	572.3
Caldwell	3	0.21	114	2.8	58.7	0.78
Champion	25	0.1	46	1.9	47.4	0.18
Coamano	2	0.05	77	1.9	1.9	903.8

Variable Descriptions

?gala

A possible model

$$\begin{array}{cc}{\text{Species}}_i& ={\beta }_0+{\beta }_1{\text{Area}}_i+{\beta }_2{\text{Elevation}}_i+{\beta }_3{\text{Nearest}}_i+\\ & {\beta }_4{\text{Scruz}}_i+{\beta }_5{\text{Adjacent}}_i+{ϵ}_{i}i=1,\dots ,n\end{array}$$

E.g. $i=1$, Baltra: $$58={\beta }_0+{\beta }_{1}25.09+{\beta }_{2}346+{\beta }_{3}0.6+{\beta }_{4}0.4+{\beta }_{5}1.84+{ϵ}_1$$

Your turn:

• What does $i$ index?
• What is the value of $n$?
• What is the value of $p$?

General matrix form

$$\begin{array}{cc}\left(\begin{array}{c}{y}_1\\ y_2\\ ⋮\\ y_n\end{array}\right)& =\left(\begin{array}{ccccc}1& x_{11}& x_{12}& \dots & x_{1(p-1)}\\ 1& x_{21}& x_{22}& \dots & x_{2(p-1)}\\ ⋮& ⋮& ⋮& ⋮& ⋮\\ 1& x_{n1}& x_{n2}& \dots & x_{n(p-1)}\end{array}\right)\left(\begin{array}{c}{\beta }_0\\ {\beta }_1\\ ⋮\\ {\beta }_{p-1}\end{array}\right)+\left(\begin{array}{c}{ϵ}_1\\ {ϵ}_2\\ ⋮\\ {ϵ}_n\end{array}\right)\\ y& =X\beta +ϵ\end{array}$$ where $$\begin{array}{cc}{y}_{n\times 1}& =(y_1,y_2,\dots ,y_n{)}^T\\ {ϵ}_{n\times 1}& =({ϵ}_1,{ϵ}_2,\dots ,{ϵ}_n{)}^T\\ {\beta }_{p\times 1}& =({\beta }_0,{\beta }_1,\dots ,{\beta }_{p-1}{)}^T\\ X_{n\times p}& =\left(\begin{array}{ccccc}1& x_{11}& x_{12}& \dots & x_{1(p-1)}\\ 1& x_{21}& x_{22}& \dots & x_{2(p-1)}\\ ⋮& ⋮& ⋮& ⋮& ⋮\\ 1& x_{n1}& x_{n2}& \dots & x_{n(p-1)}\end{array}\right)\end{array}$$

Galápagos: Matrix form

$$y_{30\times 1}=\left(\begin{array}{c}58\\ 31\\ 3\\ 25\\ 2\\ ⋮\end{array}\right),X_{30\times 6}=\left(\begin{array}{cccccc}1& 25.09& 346& 0.6& 0.6& 1.84\\ 1& 1.24& 109& 0.6& 26.3& 572.33\\ 1& 0.21& 114& 2.8& 58.7& 0.78\\ 1& 0.1& 46& 1.9& 47.4& 0.18\\ 1& 0.05& 77& 1.9& 1.9& 903.82\\ ⋮& ⋮& ⋮& ⋮& ⋮& ⋮\end{array}\right)$$

$${\beta }_{6\times 1}=\left(\begin{array}{c}{\beta }_0\\ {\beta }_1\\ {\beta }_2\\ {\beta }_3\\ {\beta }_4\\ {\beta }_5\end{array}\right),{ϵ}_{30\times 1}=\left(\begin{array}{c}{ϵ}_1\\ {ϵ}_2\\ {ϵ}_3\\ {ϵ}_4\\ {ϵ}_5\\ ⋮\end{array}\right)$$

Your Turn

Write out the design matrix, $X$, for the following models, using the data for the first five islands:

$$\begin{array}{cc}{\text{Species}}_i& ={\beta }_0+{\beta }_1{\text{Area}}_i+{\beta }_2{\text{Nearest}}_i+{ϵ}_i\\ {\text{Species}}_i& ={\beta }_1{\text{Area}}_i+{\beta }_2{\text{Area}}_i^2+{ϵ}_i\\ {\text{Species}}_i& ={\beta }_0+{\beta }_1{1}_{\{{\text{Area}}_i>1\}}+{ϵ}_i\end{array}$$ where $1_{\{.\}}$ is an indicator variable that takes the value 1, when the condition in the argument is true, and 0 otherwise.

Fitted values and residuals

If we had an estimate for the $\beta$ vector, $$\hat{\beta }={({\hat{\beta }}_0,{\hat{\beta }}_1,\dots ,{\hat{\beta }}_{p-1})}^T$$

Then we can define fitted value and residual vectors: $$\begin{array}{cc}\hat{y}& =(\widehat{y_1},\dots ,\widehat{y_n}{)}^T=X\hat{\beta }\\ e& =\widehat{ϵ}=(e_1,\dots ,e_n{)}^T=y-X\hat{\beta }\end{array}$$

Questions to answer this week:

• How will we find $\hat{\beta }$?
• What properties do the estimates have?