Time Series • Time Series Regression Model
Time Series / Variable Selection Methods in Regression

Variable Selection Methods in Regression

Notes 5 Time Series Regression Model
Quick concept

Variable selection is the process of choosing which predictors to keep in a regression. The aims are to include variables that matter for the problem, to drop variables that add noise or duplication, to reduce overfitting, and to keep the equation readable. More variables do not always improve the model.

Notes

Variable Selection Methods in Regression

Definition

Variable selection is the process of choosing which predictors to keep in a regression. The aims are to include variables that matter for the problem, to drop variables that add noise or duplication, to reduce overfitting, and to keep the equation readable. More variables do not always improve the model.

Example

Daily demand might be considered with temperature, a holiday indicator, an advertising spend series, and a linear time index. Temperature and holidays may earn a place; a second temperature copy, or a weakly measured extra series, may not.

Why Selection Is Needed

  • Relevance: keep predictors with a clear story and a stable association.
  • Unnecessary variables: extra x terms use up degrees of freedom and can inflate coefficient SEs.
  • Overfitting: an equation that hugs the training sample may forecast later dates poorly.
  • Interpretability: a short, justified list is easier to explain in an exam and in practice.

Basic Search Methods

Method Idea
Forward selection Start with no predictors; add the most useful remaining x one at a time
Backward elimination Start with all candidates; drop the least useful x one at a time
Stepwise selection Allow both adding and dropping as the search proceeds

These searches are helpers, not proofs. Different paths can keep different sets. Always re-check residuals and the story of the variables after the search.

Simple Comparison Criteria

Ordinary R2 never falls when a variable is added, so it is a poor “more is better” score. Adjusted R2 penalises extra predictors in a simple way. AIC and BIC are information criteria: smaller is preferred on the usual scale; BIC penalises extra variables more heavily than AIC. Use them to compare a few candidate equations, not as a machine-learning tournament.

Exam-Oriented Key Points

  1. Variable selection balances fit, leftover diagnostics, and a short justified predictor list.
  2. Forward, backward and stepwise are search habits, not unique true models.
  3. Adjusted R2, AIC and BIC can compare candidates; ordinary R2 always rises with extra x.
  4. More variables do not always improve the model, especially for later-date forecasts.