Time Series • Statistics Background for Forecasting
Time Series / Evaluating and Monitoring Forecasting Model Performance

Evaluating and Monitoring Forecasting Model Performance

Notes 3 Statistics Background for Forecasting
Quick concept

Evaluation means measuring how close forecasts are to actual values, preferably on dates not used for fitting. Monitoring means repeating that comparison as new actual observations arrive, so that a method which has become poor can be updated.

Notes

Evaluating and Monitoring Forecasting Model Performance

Definition

Evaluation means measuring how close forecasts are to actual values, preferably on dates not used for fitting. Monitoring means repeating that comparison as new actual observations arrive, so that a method which has become poor can be updated.

Example

A shop forecasts October sales as 50 packets. Actual October sales are 52. The forecast error is 2 packets. Repeating this for later months shows whether the method stays useful.

Why Evaluation Is Needed

A method can follow the training period closely and still fail on later dates. Judging a forecast only by how well it copies the fitted sample is not enough. Evaluation asks: how large are the misses on new time points?

Actual, Forecast and Error

Let Yt be the actual value and Ft be the forecast value for the same date.

Forecast error et = Yt − Ft

A positive error means the actual value was higher than the forecast. A negative error means the actual value was lower. The sign shows direction; size is judged with summary measures.

Small Illustrative Example

Three hold-out months:

Month Actual Y Forecast F Error Y − F
1 50 49 1
2 52 51 1
3 48 50 −2

Error Measures, Training vs Test Data, and Monitoring

Common Evaluation Measures

Measure Idea Usefulness Limitation
MAE Mean of absolute errors Easy to interpret in original units Treats every miss equally; does not emphasise large errors
MSE Mean of squared errors Penalises large misses more strongly Units are squared; harder to read in the original scale
RMSE Square root of MSE Large misses matter, and the unit matches the data Still sensitive to a few very large errors
MAPE Mean of absolute errors as a percentage of actual values Useful for comparing series of different sizes Unstable if actual values are near zero

For the three-month example: MAE = (|1| + |1| + |−2|) / 3 = 4 / 3. MSE = (1² + 1² + (−2)²) / 3 = 6 / 3 = 2. RMSE = √2. MAPE uses |error| / actual × 100 for each month, then the mean of those percentages. Do not memorise extra decimal places from software; be able to compute the three errors and the idea of each measure.

Training Data vs Evaluation Data

Fit the method on training (earlier) dates. Compute MAE, RMSE or MAPE on test (later) dates whenever the series is long enough. Reporting only training error can make a method look better than it is for real forecasting.

Monitoring Over Time

Forecast ↓ Compare with Actual ↓ Calculate Error ↓ Evaluate ↓ Monitor Over Time ↓ Improve Model if Needed

After the forecast period, new actual values become available. Compare them with the forecasts. If errors become systematically larger, or if the plot of the series changes behaviour, the method may have deteriorated. Then collect newer data, inspect the graph again, and reconsider the method. Monitoring is part of the forecasting process, not an optional extra.

Small Educational Python Example

# Import libraries import numpy as np actual = np.array([50, 52, 48]) forecast = np.array([49, 51, 50]) error = actual - forecast mae = np.mean(np.abs(error)) mse = np.mean(error ** 2) rmse = np.sqrt(mse) mape = np.mean(np.abs(error / actual)) * 100 print("Errors:", error) print("MAE:", mae) print("MSE:", mse) print("RMSE:", rmse) print("MAPE (%):", mape)

Errors should be 1, 1 and −2. MAE should be 4/3. MSE should be 2. RMSE should be the square root of 2. MAPE depends on the three percentage misses and should be described as a percentage, not as an invented extra case study.

Unit 2 Quick Revision

Topic Short definition / exam point
Graphical Displays Visual presentation of the series. Inspect a graph before choosing a method.
Time Series Plots Time on the x-axis, value on the y-axis, chronological line plot.
Smoothing Reduces short-term noise (for example a moving average) but can hide sudden changes.
Numerical Description Mean, median, spread and extremes summarise values; they do not replace a time plot.
Transformations and Adjustments Logs change scale; differencing is a basic level adjustment. Choose from the data.
Modelling Workflow Objective → data → plot → structure → optional transform → fit → evaluate → forecast → monitor.
Evaluation and Monitoring Error = actual − forecast. Use MAE/RMSE/MAPE on later dates and keep watching performance.
  1. Always inspect a time series graph before selecting a forecasting method.
  2. Smoothing reveals the general path; it is not a complete model.
  3. Training data fit the method; test data evaluate it.
  4. A forecast is an estimate and must be monitored against later actual values.