Time Series • Time Series Regression Model
Time Series / Least Squares Estimation in Linear Regression Models

Least Squares Estimation in Linear Regression Models

Notes 5 Time Series Regression Model
Quick concept

Least squares chooses the coefficients of a linear regression so that the total of squared residuals is as small as possible. A residual is the gap between an observed y and the value predicted by the fitted line. Ordinary least squares (OLS) uses equal weight for every observation.

Notes

Least Squares Estimation in Linear Regression Models

Definition

Least squares chooses the coefficients of a linear regression so that the total of squared residuals is as small as possible. A residual is the gap between an observed y and the value predicted by the fitted line. Ordinary least squares (OLS) uses equal weight for every observation.

Example

Four constructed pairs (x, y): (1, 3), (2, 5), (3, 6), (4, 8). The OLS line computed from these points is ŷ = 1.5 + 1.6x (calculation on the next page). At x = 2 the line predicts 4.7, while the observed y is 5, so the residual is 0.3.

The Linear Model

A simple linear regression is written

y = β0 + β1x + ε

  • y — observed dependent value
  • x — explanatory value
  • β0 — intercept (mean of y when x = 0, if that is meaningful)
  • β1 — slope (change in the mean of y for a one-unit change in x)
  • ε — random error (unobservable shock around the line)

After fitting, the estimated intercept and slope are written β̂0 and β̂1. The fitted (predicted) value is ŷ = β̂0 + β̂1x. The residual is e = y − ŷ. The residual is the sample leftover; the error ε is the theoretical term in the model. They are related ideas, not identical names.

Least Squares Objective

Observed value ↓ Predicted value ↓ Residual (y − ŷ) ↓ Square residual ↓ Minimize total squared residuals Σ ei2

OLS chooses β̂0 and β̂1 so that Σ ei2 is minimized. Squaring treats over- and under-prediction equally and penalises large misses more than small ones. Other methods (GLS, WLS) change the weights; they still start from this residual idea.

Exam-Oriented Key Points

  1. OLS estimates are chosen to minimize the sum of squared residuals.
  2. Residual ei = yi − ŷi; error ε is the unobservable model shock.
  3. β0 is the intercept; β1 is the slope.
  4. Fitting a line is not the same as checking whether the line is adequate.

A Worked Four-Point OLS Example

Data and Formulas

Use the constructed pairs (1, 3), (2, 5), (3, 6), (4, 8). n = 4, Σx = 10, Σy = 22, x̄ = 2.5, ȳ = 5.5, Σxy = 63, Σx2 = 30.

Slope: β̂1 = (n Σxy − (Σx)(Σy)) / (n Σx2 − (Σx)2) = (4×63 − 10×22) / (4×30 − 102) = (252 − 220) / (120 − 100) = 32 / 20 = 1.6.

Intercept: β̂0 = ȳ − β̂1 x̄ = 5.5 − 1.6×2.5 = 1.5.

Interpretation: each extra unit of x is associated with a 1.6 unit rise in fitted y, on this tiny teaching set. The intercept 1.5 is the fitted value at x = 0; here x = 0 is outside the observed range, so treat 1.5 as an algebraic intercept, not a real-world claim.

Small Educational Python Example

The same four constructed points. Expected display: intercept near 1.5 and slope near 1.6, plus a scatter with the fitted line. Do not treat software rounding as a new exam number.

# Import libraries import numpy as np import pandas as pd import matplotlib.pyplot as plt import statsmodels.api as sm # Load / prepare a constructed four-point series df = pd.DataFrame({ "x": [1, 2, 3, 4], "y": [3, 5, 6, 8] }) X = sm.add_constant(df["x"]) # intercept column # Fit OLS fit = sm.OLS(df["y"], X).fit() print(fit.params) # Plot points and fitted line plt.scatter(df["x"], df["y"], label="observed") plt.plot(df["x"], fit.fittedvalues, label="OLS line") plt.legend() plt.title("Constructed OLS example") plt.tight_layout() plt.show()

Exam-Oriented Key Points

  1. Show the data before quoting a fitted intercept and slope.
  2. β̂0 = ȳ − β̂1 x̄ for simple OLS.
  3. A fitted line describes association in the sample; it is not proof of cause.