Least squares estimates of the regression parameters Jan 16 2019

Warmup

Recall from last time we can set up a multiple linear regression model in the matrix form: $$y=X\beta +ϵ$$

Give the name and dimensions of each term

Today’s goal

Derive the form of the estimates for the parameter vector $\beta$.

Least Squares

Just like in simple linear regression, we’ll estimate $\beta$ by least squares. In simple linear regression this involved finding $\widehat{{\beta }_0}$ and $\widehat{{\beta }_1}$ to minimise the sum of squared residuals: $$\text{sum of squared residuals SLR}=\sum _{i=1}^n{e}_i^2=\sum _{i=1}^n{(y_i-({\hat{\beta }}_0+{\hat{\beta }}_1{x}_i\left)\right)}^2$$

Your turn: What procedure do you use to minimise a function? E.g. if $f\left(x\right)$ is a function of a single real value $x$, how do you find the $x$ that minimises $f\left(x\right)$?

(2 min discussion)

For multiple linear regression the least squares estimate of the $\beta$ is the vector $\hat{\beta }$ that minimizes the sum of squared residuals: $$\text{sum of squared residuals MLR}=\sum _{i=1}^n{e}_i^2=||e|{|}^2={(y-X\hat{\beta })}^T(y-X\hat{\beta })$$

Your turn Expand the matrix product on the right into four terms. Be careful with the order of matrix multiplication and recall ${\left(uX\right)}^T=X^T{u}^T$.

$$\sum _{i=1}^n{e}_i^2=||e|{|}^2={(y-X\hat{\beta })}^T(y-X\hat{\beta })$$

Consider the terms: $$-{\hat{\beta }}^T{X}^{T}y\text{and}-y^{T}X\hat{\beta }$$

Argue that these can be combined into the single term $$-2{\hat{\beta }}^T{X}^{T}y$$

(Hint: consider the dimensions of these terms)

Finding the minimum

Now our objective is to find $\hat{\beta }$ that minimises: $$y^{T}y-2{\hat{\beta }}^T{X}^{T}y+{\hat{\beta }}^T{X}^{T}X\hat{\beta }$$

The usual procedure would be to take derivative with respect to $\hat{\beta }$, set to zero and solve for $\hat{\beta }$. Except $\hat{\beta }$ is a vector! We need to use vector calculus.

Vector calculus

You should be familiar with the usual differentiation rules for scalar $a$ and $x$:

• $\frac{\partial }{\partial x}a=0$
• $\frac{\partial }{\partial x}ax=a$
• $\frac{\partial }{\partial x}a{x}^2=2ax$

There are analogs when we want to take derivative with respect to a vector $x$:

• $\frac{\partial }{\partial x}a=0$, where $a$ is a scalar
• $\frac{\partial }{\partial x}{x}^{T}u=u$, where $u$ is a vector
• $\frac{\partial }{\partial x}{x}^{T}Ax=(A+A^T)x$, where $A$ is a matrix

Use the rules above to take the derivative of the sum of squared residuals with respect to the vector $\hat{\beta }$

$$\begin{array}{cc}\frac{\partial }{\partial \hat{\beta }}(y^{T}y-2{\hat{\beta }}^T{X}^{T}y+{\hat{\beta }}^T{X}^{T}X\hat{\beta })& =\end{array}$$

Normal Equations

Setting the above derivative to zero leads to the Normal Equations. The least squares estimates satisfy: $$X^{T}y=X^{T}X\hat{\beta }$$

If $X^{T}X$ is invertible, the least squares estimates are (fill me in): $$\hat{\beta }={\left(X^{T}X\right)}^{-1}{X}^{T}y$$

If $X$ has rank $p$ then $X^{T}X$ will be invertible.

Fitted Values and Residuals

Plug in the least squares estimate for $\hat{\beta }$ to find the fitted values and residuals $$\begin{array}{c}\hat{y}=X\hat{\beta }=\\ \widehat{ϵ}=e=y-X\hat{\beta }=\end{array}$$

Hat matrix

The hat matrix is: $$H=X{\left(X^{T}X\right)}^{-1}{X}^T$$ named because it puts “hats” on the response, i.e. multiplying the response by the hat matrix gives the fitted values: $$Hy=\hat{y}$$

Your Turn: Show $(I-H)X=00$

Other properties of $H$

• $H$ is symmetric, so is $(I-H)$
• $H$ is idempotent ($H^2=H$), and so is $I-H$
• $X$ is invariant under $H$ (i.e. $HX=X$)
• $(I-H)H=H(I-H)=0$

You can use these results to argue that the residuals are orthogonal to the columns of $X$, i.e. show $e^{T}X=00$ $$\begin{array}{cc}{e}^{T}X& =\left(\right(I-H)Y{)}^{T}X\text{plug in form for residuals}\\ & =Y^T(I-H{)}^{T}X\text{distribute transpose}\\ & =Y^T(I-H)X\text{symmetry}\\ & =Y^{T}0\text{from above}\\ & =0\end{array}$$

Next time

What are the properties of the least squares estimates?