L22 Measuring Forecast Accuracy

  • Forecasting from an ARMA model

    • \(\lim_{ n \to \infty } \hat{Y}_{n}(l) =\mu\) (disadvantage, it converges to mean)
    • \(\lim_{ n \to \infty } V(e_{n}(l))=\gamma_{0} \lt \infty\) (converge to actual variance)
    • \(\implies\) Use ARMA or ARIMA model only for short-term forecasts

  • Prediction Intervals?

    • 95% PI for \(Y_{n+l}\) is \(\hat{Y}_{n}(l) \pm 1.96 \sqrt{ \sum_{i=0}^{l-1} \psi^{2}_{i} }\)
    • For 1-step ahead forecast would be \(\hat{Y}_{n}(l) \pm 1.96a_{n}\)

  • Advantages of having Long Realizations (more data)

    • Correlation structure get accurate standard errors
    • Better Prediction intervals
    • Likelihood function better behaved
    • Withholding recent data (train-test split)
    • Check model stability by dividing data into parts (analyze separately)

  • Advantages of Parsimonious Models

    • Fewer numerical problems in estimation
    • Easier to understand the model
    • Fewer parameters, less sensitive to deviations between parameters and estimates.
    • Applied more generally to similar processes (complex is niche)
    • Rapid real-time computations
    • If realization is large → parsimony not so important.

  • Notation

    • \(a_{t}\) is actual value
    • \(f_{t}\) is forecast value

  • Measures of Forecast accuracy

    • MSE (Squared)
      • \(MSE = \dfrac{1}{n}\sum_{t=1}^n (a_{t}- f_{t})^{2}\)
    • ME
    • MPE (Percentage)
      • \(MPE = \dfrac{1}{n}\sum_{t=1}^{n} \dfrac{a_{t}-f_{t}}{a_{t}} \times 100\)
    • MAE (Absolute)
      • \(MAE = \dfrac{1}{n}\sum_{t=1}^{n} \lvert a_{t}-f_{t}\rvert \times 100\)
    • MAPE (Absolute Percentage)
      • \(MAPE = \dfrac{1}{n}\sum_{t=1}^{n} \dfrac{\lvert a_{t}-f_{t} \rvert}{a_{t}} \times 100\)

  • Naive Forecast (GUESSING)

    • Last period actuals are used as this period's forecast without adjusting them (no causal factors applied)
    • Use only for comparing with forecast generated by other sophisticated models.

  • Theil's \(U\) Statistics

    • \(U_{1} = \dfrac{\sqrt{ \sum_{t=1}^n (a_{t}- f_{t})^{2} }}{\sqrt{ \sum_{t=1}^n a_{t}^{2} } + \sqrt{ \sum_{t=1}^n f_{t}^{2} }}\)
      • Closer to 0 \(\implies\) Better forecasting accuracy
    • \(U_{2}= \sqrt{ \dfrac{\sum_{t=1}^{n-1} \left(\dfrac{f_{t+1}-a_{t+1}}{a_{t}}\right)}{\sum_{t=1}^{n-1} \left(\dfrac{a_{t+1}-a_{t}}{a_{t}}\right)} }\)
      • \(=1\) \(\implies\) same as Naive
      • \(\lt 1\) \(\implies\) better than Naive
      • \(\gt 1\) \(\implies\) worse than Naive