Quick concept
Hold out the last 4 months of a 20-month series, produce a trailing-mean forecast, and compute MAE, RMSE and MAPE. Explain each metric.
Practical / Solution
P2.10 Evaluate Forecast Performance
Problem Statement
Hold out the last 4 months of a 20-month series, produce a trailing-mean forecast, and compute MAE, RMSE and MAPE. Explain each metric.
Learning Outcomes
- Use a hold-out window.
- Compute MAE, RMSE and MAPE and interpret them.
Theory
MAE is the average absolute error (same units as the data). RMSE penalises large errors more. MAPE is a percentage error and is unstable if actuals are near zero. Metrics should match the decision.
Dataset / Data Source
Constructed 20 monthly values. Teaching data.
Kaggle-style Workflow
- Inspect series
- Train/test split in time
- Fit a simple method on train only
- Forecast the hold-out
- Compute error metrics
- Plot actual vs predicted
Analysis / Program
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
idx = pd.date_range("2024-01-01", periods=20, freq="MS")
y = pd.Series(np.linspace(30, 50, 20) + np.sin(np.arange(20)), index=idx)
train, test = y.iloc[:-4], y.iloc[-4:]
pred = pd.Series(train.tail(4).mean(), index=test.index)
err = test - pred
mae = err.abs().mean()
rmse = np.sqrt((err ** 2).mean())
mape = (err.abs() / test.abs()).mean() * 100
print("MAE:", round(mae, 3))
print("RMSE:", round(rmse, 3))
print("MAPE %:", round(mape, 2))
test.plot(label="actual")
pred.plot(label="forecast", style="--")
plt.legend()
plt.tight_layout()
plt.show()
Expected Output
Printed MAE, RMSE and MAPE for this constructed hold-out, plus an actual-vs-forecast plot. Do not memorise the printed numbers as official answers.
Result / Interpretation
MAE is easiest to explain in original units. RMSE is larger if one month is badly wrong. MAPE is useful here because values are well above zero. All three are computed on months not used to form the trailing mean.
Note
Forecast accuracy should be evaluated on data not used for fitting whenever possible.