Toward Rigorous Zero-Shot and Few-Shot Benchmarking of Time-Series Foundation Models Under Domain Shift: A Leakage-Aware Benchmark Specification, Governance Framework, and Executable Pilot Instantiation

Ibezimako Chiazagomekpere

doi:10.58776/ijitcsa.v4i2.254

Authors

Ibezimako Chiazagomekpere The Akenten Appiah-Menka University of Skills Training and Entrepreneurial Development

DOI:

https://doi.org/10.58776/ijitcsa.v4i2.254

Keywords:

time-series foundation models, domain shift, zero-shot forecasting, few-shot adaptation, benchmark design, robustness, reproducibility, leakage-aware evaluation

Abstract

Time-series foundation models (TSFMs) are increasingly promoted as reusable forecasting systems that can generalize across domains with zero-shot or few-shot adaptation. That claim is scientifically consequential, but current evaluation practice remains under-specified where it matters most target-domain separation, contamination control, adaptation budget definition, shift severity characterization, and aggregation across heterogeneous deployment conditions. This paper reconstructs TSFM benchmarking as a methodological problem rather than a leaderboard problem. We formalize zero-shot and few-shot forecasting under domain shift as conditional risk estimation over governed target distributions; develop a forecasting-specific taxonomy of shift covering temporal regime, entity, resolution, schema, horizon, observation-quality, intervention, and label-formation change; and propose a six-layer benchmark architecture spanning model governance, dataset governance, deterministic shift generation, evaluation tracks, metric tensors, and reporting bundles. The contribution is primarily conceptual, but to avoid a purely rhetorical framework, we also provide an executable pilot instantiation on a public electricity-transformer forecasting setting. Because large-scale TSFM execution was not conducted in this package, the pilot uses lightweight surrogate forecasters to validate the benchmark machinery itself rather than to claim new TSFM state of the art. Even this limited pilot shows that in-domain and cross-domain rankings can diverge sharply, that adaptation gains must be interpreted jointly with cost, and that robustness to observation degradation and calibration cannot be inferred from average point error alone. The paper therefore advances a benchmark doctrine: credible TSFM claims require leakage-aware governance, severity-conditioned analysis, explicit adaptation accounting, and multi-objective reporting that aligns evidence with generalization claims.

References

R. Bommasani et al., “On the opportunities and risks of foundation models,” arXiv preprint arXiv:2108.07258, 2021, doi: 10.48550/arXiv.2108.07258.

Y. Liang et al., “Foundation models for time series analysis: A tutorial and survey,” in Proceedings of the 30th ACM SIGKDD conference on knowledge discovery and data mining, 2024, pp. 6555–6565. doi: 10.1145/3637528.3671451.

Q. Ma, J. Liu, et al., “A survey on time-series pre-trained models,” IEEE Transactions on Knowledge and Data Engineering, vol. 36, pp. 7536–7555, 2024, doi: 10.1109/TKDE.2024.3475809.

S. R. K. Kottapalli et al., “Foundation models for time series: A survey,” arXiv preprint arXiv:2504.04011, 2025, doi: 10.48550/arXiv.2504.04011.

B. Lim and S. Zohren, “Time-series forecasting with deep learning: A survey,” Philosophical Transactions of the Royal Society A, vol. 379, no. 2194, p. 20200209, 2021, doi: 10.1098/rsta.2020.0209.

B. Cohen et al., “This time is different: An observability perspective on time series foundation models,” arXiv preprint arXiv:2505.14766, 2025, doi: 10.48550/arXiv.2505.14766.

G. Pucher et al., “Evaluating zero-shot foundation models for time series forecasting in clinical settings: A simulation study with electronic health records,” Studies in Health Technology and Informatics, vol. 329, pp. 820–824, 2025, doi: 10.3233/SHTI250954.

A. Das, W. Kong, R. Sen, and Y. Zhou, “A decoder-only foundation model for time-series forecasting,” arXiv preprint arXiv:2310.10688, 2024, doi: 10.48550/arXiv.2310.10688.

A. F. Ansari et al., “Chronos: Learning the language of time series,” arXiv preprint arXiv:2403.07815, 2024, doi: 10.48550/arXiv.2403.07815.

M. Goswami et al., “MOMENT: A family of open time-series foundation models,” arXiv preprint arXiv:2402.03885, 2024, doi: 10.48550/arXiv.2402.03885.

G. Woo, C. Liu, A. Kumar, C. Xiong, S. Savarese, and D. Sahoo, “Unified training of universal time series forecasting transformers,” in Proceedings of the 41st international conference on machine learning, 2024. doi: 10.48550/arXiv.2402.02592.

X. Liu, G. Woo, T. Aksu, C. Liu, S. Savarese, and D. Sahoo, “Moirai-MoE: Empowering time series foundation models with sparse mixture of experts,” arXiv preprint arXiv:2410.10469, 2024, doi: 10.48550/arXiv.2410.10469.

K. Rasul et al., “Lag-llama: Towards foundation models for probabilistic time series forecasting,” arXiv preprint arXiv:2310.08278, 2024, doi: 10.48550/arXiv.2310.08278.

V. Ekambaram et al., “Tiny time mixers (TTMs): Fast pre-trained models for enhanced zero/few-shot forecasting of multivariate time series,” arXiv preprint arXiv:2401.03955, 2024, doi: 10.48550/arXiv.2401.03955.

H. Hewamalage, K. Ackermann, and C. Bergmeir, “Forecast evaluation for data scientists: Common pitfalls and best practices,” Data Mining and Knowledge Discovery, vol. 37, no. 2, pp. 788–832, 2023, doi: 10.1007/s10618-022-00894-5.

Z. Li et al., “TSFM-bench: A comprehensive and unified benchmark of foundation models for time series forecasting,” in Proceedings of the 31st ACM SIGKDD conference on knowledge discovery and data mining, 2025, pp. 5595–5606. doi: 10.1145/3711896.3737442.

M. Meyer, S. Kaltenpoth, K. Zalipski, and O. Müller, “Time series foundation models: Benchmarking challenges and requirements,” arXiv preprint arXiv:2510.13654, 2025, doi: 10.48550/arXiv.2510.13654.

T. Aksu et al., “GIFT-eval: A benchmark for general time series forecasting model evaluation,” arXiv preprint arXiv:2410.10393, 2024, doi: 10.48550/arXiv.2410.10393.

O. Shchur et al., “Fev-bench: A realistic benchmark for time series forecasting,” arXiv preprint arXiv:2509.26468, 2025, doi: 10.48550/arXiv.2509.26468.

Z. Li et al., “FoundTS: Comprehensive and unified benchmarking of foundation models for time series forecasting,” in arXiv preprint arXiv:2410.11802, 2024. doi: 10.48550/arXiv.2410.11802.

R. J. Hyndman and A. B. Koehler, “Another look at measures of forecast accuracy,” International Journal of Forecasting, vol. 22, no. 4, pp. 679–688, 2006, doi: 10.1016/j.ijforecast.2006.03.001.

S. Makridakis, E. Spiliotis, and V. Assimakopoulos, “The M4 competition: 100,000 time series and 61 forecasting methods,” International Journal of Forecasting, vol. 36, no. 1, pp. 54–74, 2020, doi: 10.1016/j.ijforecast.2019.04.014.

X. Song, L. Deng, H. Wang, et al., “Deep learning-based time series forecasting,” Artificial Intelligence Review, vol. 58, p. 23, 2025, doi: 10.1007/s10462-024-10989-8.

H. Zhou et al., “Informer: Beyond efficient transformer for long sequence time-series forecasting,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 12, pp. 11106–11115, 2021, doi: 10.1609/aaai.v35i12.17325.

H. Wu, J. Xu, J. Wang, and M. Long, “Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting,” in Advances in neural information processing systems, 2021, pp. 22419–22430. Available: https://proceedings.neurips.cc/paper_files/paper/2021/hash/bcc0d400288793e8bdcd7c19a8ac0c2b-Abstract.html

T. Zhou, Z. Ma, Q. Wen, X. Wang, L. Sun, and R. Jin, “FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting,” in Proceedings of the 39th international conference on machine learning, 2022, pp. 27268–27286. Available: https://proceedings.mlr.press/v162/zhou22g.html

Y. Nie, N. H. Nguyen, P. Sinthong, and J. Kalagnanam, “A time series is worth 64 words: Long-term forecasting with transformers,” in International conference on learning representations, 2023. doi: 10.48550/arXiv.2211.14730.

M. Jin, S. Wang, L. Ma, Z. Chu, J. Y. Zhang, et al., “Time-LLM: Time series forecasting by reprogramming large language models,” arXiv preprint arXiv:2310.01728, 2023, doi: 10.48550/arXiv.2310.01728.

N. Karaouli, D. Coquenet, E. Fromont, M. Mermillod, and M. Reyboz, “How foundational are foundation models for time series forecasting?” arXiv preprint arXiv:2510.00742, 2025, doi: 10.48550/arXiv.2510.00742.

J. Zhang, Z. Zhang, S. Zheng, X. Wen, J. Li, and J. Bian, “Are time-series foundation models deployment-ready? A systematic study of adversarial robustness across domains,” arXiv preprint arXiv:2505.19397, 2025, doi: 10.48550/arXiv.2505.19397.

P. W. Koh et al., “WILDS: A benchmark of in-the-wild distribution shifts,” in Proceedings of the 38th international conference on machine learning, 2021, pp. 5637–5664. Available: https://proceedings.mlr.press/v139/koh21a.html

R. Bommasani, P. Liang, and T. Lee, “Holistic evaluation of language models,” Annals of the New York Academy of Sciences, vol. 1525, no. 1, pp. 140–146, 2023, doi: 10.1111/nyas.15007.

T. Gneiting and A. E. Raftery, “Strictly proper scoring rules, prediction, and estimation,” Journal of the American Statistical Association, vol. 102, no. 477, pp. 359–378, 2007, doi: 10.1198/016214506000001437.

T. Gneiting, F. Balabdaoui, and A. E. Raftery, “Probabilistic forecasts, calibration and sharpness,” Journal of the Royal Statistical Society: Series B, vol. 69, no. 2, pp. 243–268, 2007, doi: 10.1111/j.1467-9868.2007.00587.x.

A. N. Angelopoulos and S. Bates, “A gentle introduction to conformal prediction and distribution-free uncertainty quantification,” Annual Review of Statistics and Its Application, vol. 10, pp. 1–28, 2023, doi: 10.1146/annurev-statistics-033121-015250.