Skip to content

Machine Learning in TS

Machine Learning (ML) serves as an extension of traditional time series analysis, offering flexibility and the ability to capture complex patterns that linear models often miss.

Black-Box Models

ML often relates to black-box models. We don't always know exactly what goes into the decision-making process inside the model, unlike the explicit coefficients in an ARIMA or GARCH model.

1. Classical Machine Learning Models

These models rely on structured feature engineering to transform the sequential nature of TS into a tabular format suitable for standard algorithms.

  • Linear Regression: The basic benchmark for trend analysis.
  • Support Vector Machines (SVMs): Useful for both TS classification and regression tasks, especially with non-linear relationships.
  • Random Forests and Gradient Boosting: Tree-based ensembles that excel at handling non-linearities and interactions between lagged variables.

2. Neural Networks (Deep Learning)

Recurrent Neural Networks (RNNs)

  • LSTMs (Long Short-Term Memory): Designed to handle long-term dependencies in sequential data, overcoming the "vanishing gradient" problem.

  • GRUs (Gated Recurrent Units): A simpler alternative to LSTMs with comparable performance and fewer parameters.

Specialized Architectures

  • Convolutional Neural Networks (CNNs): Useful for extracting local patterns from small batches of TS data, often used in combination with RNNs.

  • Transformers:

    • Attention Mechanism: Highly effective for long sequence modelling, allowing the model to "focus" on relevant past time steps.

    • Time Series Transformers (TST): Specific adaptations of the transformer architecture for TS forecasting and classification.

    • Autoencoders: Primarily used for anomaly detection and feature extraction by learning a compressed representation of the series.

Hybrid Models

  • ARIMA + Neural Networks: ARIMA models the linear mean relationship, while the NN captures the complex residual patterns.
  • CNN-RNN Hybrids: CNNs extract features from the input, while RNNs handle the temporal (TS) modelling.
  • DeepAR: A probabilistic forecasting model developed by Amazon that uses RNNs to predict value distributions.

3. Probabilistic & Bayesian Models

These are essential because they provide uncertainty estimates in forecasts.

  • Gaussian Process: Non-parametric models effective for smaller datasets.
  • Bayesian Structural Time Series: Useful for modelling trends, seasonality, and conducting causal inference.

4. Specialized Models for TS

  • Facebook Prophet: Designed specifically for business forecasting; handles seasonality, holidays, and trends automatically.

  • N-BEATS:

    • Neural Basis Expansion for interpreTable forecaSting.
    • Decomposes the signal into interpretable components like trend and seasonality blocks.

  • TFT (Temporal Fusion Transformer): Combines the attention mechanism with features that provide interpretable forecasting results.

5. Ensemble Learning

  • Bagging and Boosting: Aggregating predictions from multiple models to reduce variance or bias.
  • Stacking: Layering models where the predictions of one model serve as input to another (the meta-model).