Measuring Forecast Accuracy

Evaluating the performance of a model is essential for determining its reliability in real-world applications. This involves analyzing the behavior of forecasts, the impact of data volume, and specific mathematical metrics.

1. Behavior of ARMA Forecasts

Long-term forecasts from stationary ARMA models exhibit specific convergence properties:
* Mean Convergence: \(\lim_{l \to \infty} \hat{Y}_{n}(l) = \mu\). The forecast eventually reverts to the process mean, which is why ARMA/ARIMA models are generally preferred for short-term forecasts.
* Variance Convergence: \(\lim_{l \to \infty} V(e_{n}(l)) = \gamma_{0} < \infty\). The forecast error variance eventually converges to the total variance of the process.

2. Prediction Intervals (PI)

A Prediction Interval provides a range within which the actual future value is expected to fall with a certain probability (e.g., 95%):
* General \(l\)-step Ahead PI: \(\hat{Y}_{n}(l) \pm 1.96 \sqrt{ \sum_{i=0}^{l-1} \psi^{2}_{i} }\).
* 1-step Ahead PI: \(\hat{Y}_{n}(1) \pm 1.96 \sigma_{a}\).

3. Model and Data Strategy

The quality of a forecast is heavily influenced by the amount of data available and the complexity of the model used.

Advantages of Long Realizations (Large Datasets)

• Statistical Rigor: Provides more accurate standard errors for the correlation structure and produces better prediction intervals.
• Estimation Stability: The likelihood function is better behaved, making parameter estimation more reliable.
• Validation: Allows for withholding recent data for a train-test split and checking model stability by analyzing separate parts of the data.

Advantages of Parsimonious Models

Parsimony—the principle of using the simplest model that fits the data—offers several benefits:
* Numerical Efficiency: Results in fewer numerical problems during estimation and allows for rapid real-time computations.
* Interpretability: Simplifies the understanding of the model's dynamics.
* Generalization: Fewer parameters mean the model is less sensitive to estimation deviations and can be applied more generally to similar processes.
* Note: If the realization is very large, parsimony becomes less critical.

4. Measures of Forecast Accuracy

To quantify accuracy, we compare actual values (\(a_{t}\)) against forecasted values (\(f_{t}\)).

Metric	Formula	Description
MSE	\(MSE = \dfrac{1}{n}\sum_{t=1}^n (a_{t}- f_{t})^{2}\)	Mean Squared Error; penalizes larger errors more heavily.
MAE	\(MAE = \dfrac{1}{n}\sum_{t=1}^{n} \lvert a_{t}-f_{t}\rvert\)	Mean Absolute Error; gives the average absolute deviation.
MAPE	\(MAPE = \dfrac{1}{n}\sum_{t=1}^{n} \dfrac{\lvert a_{t}-f_{t} \rvert}{a_{t}} \times 100\)	Mean Absolute Percentage Error; provides error relative to the scale of data.
MPE	\(MPE = \dfrac{1}{n}\sum_{t=1}^{n} \dfrac{a_{t}-f_{t}}{a_{t}} \times 100\)	Mean Percentage Error; helps detect systematic bias.

Naive Forecast (The Baseline)

A Naive Forecast uses the last observed value as the forecast for the next period without any adjustments. It serves strictly as a benchmark to compare against sophisticated models.

Theil’s \(U\) Statistics

These statistics help evaluate model performance relative to a zero-error or naive benchmark:
* \(U_{1}\): Ranges between 0 and 1; a value closer to 0 indicates higher forecasting accuracy.
* \(U_{2}\): Compares the model to a Naive Forecast:
* \(U_{2} = 1\): Same accuracy as Naive.
* \(U_{2} < 1\): Better than Naive.
* \(U_{2} > 1\): Worse than Naive.

---