Part V · Chapter 16

Forecast Evaluation

Bias, RMSE, MAE, MAPE, and Theil's U₂ each emphasize different aspects of forecast error. Pick the metric that matches your loss function.

Learning objectives

Compute bias, MAE, MSE, RMSE, MAPE, Theil U₂.

Understand which metric penalizes which errors.

Recognize zero-bias but high-error situations.

Compare model vs naive benchmark with Theil U₂.

Forecast Error Dashboard

Forecast bias 0.0 Error variance σ_e 1.0 n 200 Apply "bias cancellation" trap

Seed:

Live metrics

Bias: —

MAE: —

RMSE: —

MAPE: — %

Theil U₂: —

📐 Metric formulas

\[ \text{Bias}=\bar e,\; \text{MAE}=\overline{|e_t|},\; \text{MSE}=\overline{e_t^2},\; \text{RMSE}=\sqrt{\text{MSE}} \]

\[ \text{MAPE}=100\,\overline{|e_t/y_t|},\quad U_2=\sqrt{\frac{\sum(\hat y_t - y_t)^2}{\sum(y_{t-1}-y_t)^2}} \]

RMSE penalizes large errors quadratically

MAE linear penalty, robust

MAPE scale-free; unstable near zero

U₂ < 1 beats naive RW benchmark

🔍 What to look for

Switching on "bias cancellation" creates equal positive & negative errors → bias near 0 but RMSE/MAE large.
If U₂ > 1, your model is worse than "tomorrow = today".
MAPE explodes when y_t is near zero — careful with returns/growth rates.

⚠️ Pro Tip: What to Avoid

Student says

"Bias is near zero, so the forecast is accurate."

Why this is wrong

Zero bias means errors cancel ON AVERAGE. The forecast can still be wildly off period-by-period (high RMSE/MAE). Bias is necessary but not sufficient.

Correct interpretation

Report bias + RMSE + Theil U₂ jointly. Low bias + low RMSE + U₂ < 1 is the win condition.

📝 Mini-quiz

📋 Key Takeaways

Metric	Use when
RMSE	Large errors are costly
MAE	All errors equally costly
MAPE	Scale-free comparison (NO zeros)
Theil U₂	Compare vs naive RW benchmark
Bias	Detect systematic over/under-prediction