Summary of last week

For the linear regression model $$Y=X\beta +ϵ$$ where $\text{E}\left(ϵ\right)=0_{n\times 1}$, $\text{Var}\left(ϵ\right)={\sigma }^2{I}_n$, and the matrix $X_{n\times p}$ is fixed with rank $p$.

The least squares estimates are $$\hat{\beta }=(X^{T}X{)}^{-1}{X}^{T}Y$$

Furthermore, the least squares estimates are BLUE, and $$\text{E}\left(\hat{\beta }\right)=\beta ,\text{Var}\left(\hat{\beta }\right)={\sigma }^2(X^{T}X{)}^{-1}$$

We have not used any Normality assumptions to show these properties.

Today

• Verify:

  $$\text{E}\left({\hat{\sigma }}^2\right)=\text{E}\left(\frac{1}{n-p}\sum _{i=1}^n{e}_i^2\right)={\sigma }^2$$

• Add Normal assumption to get inference on regression coefficents.

Go over the estimation of $\sigma$

Strategy: Write $e_i^2$ as a linear combination of uncorrelated variables, ${ϵ}_i$.

Write correlated residuals as combination of uncorrelated errors

Claim:

$$||e|{|}^2={ϵ}^T(I-H)ϵ$$

Your turn at home:

1. Show $(I-H)ϵ=e$. Hint: substitute $ϵ=Y-X\beta$, expand and use properties of $H$.

2. Show $||e|{|}^2=e^{T}e={ϵ}^T(I-H)ϵ$. Hint: substitute in $e=(I-H)ϵ$ from above and use properties of $(I-H)$.

Find expected value of $||e|{|}^2$ in terms of $\text{trace}(I-H)$

Show $\text{E}\left({ϵ}^T\right(I-H\left)ϵ\right)={\sigma }^2\text{trace}(I-H)$

Hint $$x^{T}Ax=\sum _{i=1}^n\sum _{j=1}^n{x}_i{x}_j{A}_{ij}$$ where $$x={(x_1,x_2,\dots ,x_n)}^T,A={\left(\begin{array}{ccc}{A}_{11}& A_{12}& \dots \\ A_{21}& A_{22}& \dots \\ ⋮& & \end{array}\right)}_{n\times n}$$

Find expected value of $||e|{|}^2$ in terms of $\text{trace}(I-H)$

$$\text{E}\left({ϵ}^T\right(I-H\left)ϵ\right)=$$

Find $\text{trace}(I-H)$

Show $$\text{trace}(I-H)=n-p$$

Hint: $$\begin{array}{cc}\text{trace}(A+B)& =\text{trace}\left(A\right)+\text{trace}\left(B\right)\\ \text{trace}\left(AB\right)& =\text{trace}\left(BA\right)\end{array}$$

$$\text{trace}(I-H)=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa$$

Put it all together

$$\text{E}\left({\hat{\sigma }}^2\right)=$$

Inference on the regression coefficients