Controversies in Zero-Shot Forecasting with Time Series Foundation Models and a Three-Layer Framework for Trustworthy Evaluation

Main Article Content

Huiyun Yang
School of Information Science and Technology (School of Artificial Intelligence), Beijing Forestry University, Beijing 100083, China

DOI: https://doi.org/10.70267/cai.v3n6.0120

Keywords

time series foundation models, zero-shot forecasting, benchmark evaluation, data leakage, evaluation protocols

Abstract

Time series foundation models (TSFMs) aim to forecast new datasets without target-specific parameter updates. Assessing this capability requires scrutiny of how evaluation evidence is generated and interpreted. We present a semi-systematic review of 80 core papers on the evaluation of TSFMs and large language model (LLM)-based forecasting methods. The review comprises 68 papers from the initial search and 12 from a targeted supplementary search within the same January 2022 to August 2026 publication window. We organize the evidence around benchmark integrity, ranking robustness, and domain transferability. In the reviewed literature, empirical audits of overlap between TSFM pretraining corpora and test benchmarks remain limited. Protocol choices, including normalization, forecast horizons, information availability, and metric aggregation, can alter model comparisons. Domain studies report both competitive performance and limitations, depending on task formulation, the relevance of pretraining data, sampling granularity, and data quality. These findings warrant conditional interpretations of zero-shot results rather than general claims of success or failure. We propose a five-part checklist for trustworthy evaluation and identify research priorities in corpus auditing, statistical comparison, domain-specific calibration, broader empirical coverage, and model selection. Reliable deployment requires evidence relevant to the intended task and operating conditions.

Abstract 10 | PDF Downloads 2

References

  • [1]Ansari A F, Stella L, Turkmen C, et al. Chronos: Learning the language of time series. Transactions on Machine Learning Research, 2024.
  • [2]Das A, Kong W, Sen R, Zhou Y. A decoder-only foundation model for time-series forecasting. In: Proceedings of the 41st International Conference on Machine Learning (ICML). PMLR, 2024.
  • [3]Woo G, Liu C, Kumar A, Xiong C, Savarese S, Sahoo D. Unified training of universal time series forecasting transformers. In: Proceedings of the 41st International Conference on Machine Learning (ICML). PMLR, 2024: 53140-53164.
  • [4]Goswami M, Szafer K, Choudhry A, et al. MOMENT: A family of open time-series foundation models. In: Proceedings of the 41st International Conference on Machine Learning (ICML). PMLR, 2024.
  • [5]Rasul K, Ashok A, Williams A R, et al. Lag-Llama: Towards foundation models for probabilistic time series forecasting. arXiv preprint arXiv:2310.08278, 2023.
  • [6]Jin M, Wang S, Ma L, et al. Time-LLM: Time series forecasting by reprogramming large language models. In: Proceedings of the 12th International Conference on Learning Representations (ICLR), 2024.
  • [7]Ahmed I, Krompaß D, Feng C, et al. A comparative study on how data normalization affects zero-shot generalization in time series foundation models. In: Proceedings of ICASSP. IEEE, 2026.
  • [8]Zhang Y, Zeng Q, Zhang Y, Xu Z, Zheng M, Gao C, Jiang M, Zheng Z. Are time series foundation models ready for zero-shot forecasting? In: 1st ICML Workshop on Foundation Models for Structured Data, 2025 (OpenReview: kQDzqIkXLO).
  • [9]Zhang X, Sridharan S, Zahrin N H B, Venkataraman N, Goh S H. Benchmarking a time-series foundation model (TimeGPT) for real-world forecasting applications. Machine Learning with Applications, 2026, 23: 100801. https://doi.org/10.1016/j.mlwa.2025.100801
  • [10]Tan M, Merrill M, Gupta V, Hartvigsen T, Althoff T. Are language models actually useful for time series forecasting? In: Advances in Neural Information Processing Systems 37 (NeurIPS), 2024: 60162-60191.
  • [11]Liang Y, Wen H, Nie Y, et al. Foundation models for time series analysis: A tutorial and survey. In: Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2024: 6555-6565.
  • [12]Jin M, Kong Y, Liang Y, et al. Large models for time series and spatio-temporal data: A survey and outlook. ACM Computing Surveys, 2026.
  • [13]Wang Y, Wu H, Dong J, Liu Y, Long M, Wang J. Deep time series models: A comprehensive survey and benchmark. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2026, 48: 11642-11662.
  • [14]Shao Z, Yu C, et al. Large language models for time series forecasting in decision intelligence. Journal of Command and Control, 2025.
  • [15]Luo Y, Sun J, Tao L. Research progress on large models for time series and spatiotemporal data analysis. Science & Technology Review, 2025, 43(18): 48-56.
  • [16]Snyder H. Literature review as a research methodology: An overview and guidelines. Journal of Business Research, 2019, 104: 333-339.
  • [17]Aksu T, Woo G, Liu J, et al. GIFT-Eval: A benchmark for general time series forecasting model evaluation. arXiv preprint arXiv:2410.10393, 2024.
  • [18]Kapoor S, Narayanan A. Leakage and the reproducibility crisis in machine-learning-based science. Patterns, 2023, 4(9): 100804.
  • [19]Li Z, Qiu X, Chen P, et al. TSFM-Bench: A comprehensive and unified benchmark of foundation models for time series forecasting. In: Proceedings of ACM SIGKDD, 2025.
  • [20]Meyer M, Kaltenpoth S, Zalipski K, Müller O. Time series foundation models: Benchmarking challenges and requirements. arXiv:2510.13654, 2025. https://doi.org/10.48550/arXiv.2510.13654
  • [21]Montet C, et al. Benchmarking foundation models for time series forecasting: Zero-shot, few-shot, and full-shot. MDPI CS&Math Forum, 2025, 11(1): 32.
  • [22]Qiao Z, et al. It's TIME: Towards the next generation of time series forecasting benchmarks. arXiv preprint arXiv:2602.12147, 2026.
  • [23]Ashok A, et al. Beyond naive prompting: Strategies for improved zero-shot context-aided forecasting. OpenReview, 2025.
  • [24]Hewamalage H, Ackermann K, Bergmeir C. Forecast evaluation for data scientists: Common pitfalls and best practices. Data Mining and Knowledge Discovery, 2023, 37: 1688-1742.
  • [25]Adachi N, Fujisawa K, Osborne M A. Fixing the pitfalls of probabilistic time-series forecasting evaluation. arXiv preprint arXiv:2503.06079, 2025.
  • [26]Adler D, et al. Beyond accuracy: Are time series foundation models well-calibrated? In: Proceedings of ICLR, 2026.
  • [27]Guibert Q, et al. Benchmarking time series foundation models on their accuracy and energy. In: Swiss AI Days, PMLR 309, 2026: 46-55.
  • [28]Simeone A. Time series foundation models for energy load forecasting on consumer hardware: A multi-dimensional zero-shot benchmark. arXiv preprint arXiv:2602.10848, 2026.
  • [29]Gupta S. Evaluating time series foundation models on noisy periodic time series. arXiv preprint arXiv:2501.00889, 2025.
  • [30]Bao Y, et al. Universal redundancies in time series foundation models. arXiv preprint arXiv:2602.01605, 2026.
  • [31]Zhang W, Chen X, Li X, Chen K, Guan W, Nie L. Unified transferability metrics for time series foundation models. In: Advances in Neural Information Processing Systems 38 (NeurIPS), 2025.
  • [32]Cohen B, Khwaja E, Doubli Y, Lemaachi S, et al. This time is different: An observability perspective on time series foundation models. arXiv preprint arXiv:2505.14766, 2025.
  • [33]Chuang Y N, et al. LTSM-Bundle: A toolbox and benchmark on large language models for time series. In: Proceedings of ACM SIGKDD, 2025.
  • [34]Fons E, et al. Evaluating large language models on time series feature understanding: A comprehensive taxonomy and benchmark. In: Proceedings of EMNLP, 2024.
  • [35]Chen J, et al. MTBench: A multimodal time series benchmark for temporal reasoning and question answering. arXiv preprint arXiv:2503.16858, 2025.
  • [36]Weng Z, et al. TemporalBench: Evaluating LLM-based agents on contextual and event-informed time series tasks. In: Proceedings of ACM SIGKDD, 2026.
  • [37]Xie Q, et al. FinBen: A holistic financial benchmark for large language models. In: NeurIPS Datasets and Benchmarks, 2024.
  • [38]Liu H, Zhao Z, Wang J, Kamarthi H, Prakash B A. Evaluating System 1 vs. 2 reasoning approaches for zero-shot time series forecasting: A benchmark and insights. arXiv preprint arXiv:2503.01895, 2025.
  • [39]Lu J. Time series foundation models in finance: Pretraining corpora, architectures, financial benchmarks, and risk-aware evaluation. In: Proceedings of the 2025 2nd International Conference on Digital Economy and Computer Science. ACM, 2025: 144-151.
  • [40]Balloccu S, Schmidtova P, Lango M, Dusek O. Leak, cheat, repeat: Data contamination and evaluation malpractices in closed-source LLMs. In: Proceedings of EACL, 2024: 67-93.
  • [41]Dong Y, et al. Generalization or memorization: Data contamination and trustworthy evaluation. In: Findings of ACL, 2024.
  • [42]Wen S, Feofanov V, Zhang Y. Measuring pre-training data quality without labels for time series foundation models. arXiv preprint arXiv:2412.06368, 2024.
  • [43]Liu Z, Aksu T, et al. Empowering time series analysis with synthetic data: A survey and outlook in the era of foundation models. arXiv preprint arXiv:2503.11411, 2025.
  • [44]McGreivy N, Hakim A. Weak baselines and reporting biases lead to overoptimism in machine learning for fluid-related partial differential equations. Nature Machine Intelligence, 2024, 6(10): 1256-1269.
  • [45]Saqur R, et al. Seeking SOTA: Time-series forecasting must adopt taxonomy-specific evaluation. arXiv preprint arXiv:2603.15506, 2026.
  • [46]Chakraborty S, Delibasoglu I, Heintz F. Scaling transformers for time series forecasting: Do pretrained large models outperform small-scale alternatives? Artificial Intelligence Review, 2026, 59(2).
  • [47]Gruver N, Finzi M, Qiu S, Wilson A G. Large language models are zero-shot time series forecasters. In: Advances in Neural Information Processing Systems 36 (NeurIPS), 2023.
  • [48]Merrill M A, Tan M, Gupta V, Hartvigsen T, Althoff T. Language models still struggle to zero-shot reason about time series. In: Findings of EMNLP, 2024.
  • [49]Jin M, et al. Position: What can large language models tell us about time series analysis? arXiv preprint arXiv:2402.02713, 2024.
  • [50]Cao D, Wang Y. An evaluation of standard statistical models and LLMs on time series forecasting. In: Proceedings of IEEE FMLDS, 2024.
  • [51]Hassani H, Silva E S. Large language models as benchmarks in forecasting practice. International Journal of Applied Forecasting, 2024.
  • [52]Wang J, Cheng M, Wang D, Tao X, Liu Q, Chen E. Can slow-thinking LLMs reason over time? Empirical studies in time series forecasting. arXiv preprint arXiv:2505.24511, 2025.
  • [53]Shao Z, et al. Exploring progress in multivariate time series forecasting: Comprehensive benchmarking and heterogeneity analysis. IEEE Transactions on Knowledge and Data Engineering, 2025, 37(1): 291-305.
  • [54]Zhu L, Carpentier M, Verbeke W. When foundation models are one-liners: Limitations and future directions for time series anomaly detection. In: Proceedings of ICLR, 2026.
  • [55]Kamarthi H, Prakash B A. Large pre-trained time series models for cross-domain time series analysis tasks. In: Advances in Neural Information Processing Systems 37 (NeurIPS), 2024: 56190-56214.
  • [56]Meyer M, et al. Benchmarking time series foundation models for short-term household electricity load forecasting. IEEE Access, 2025.
  • [57]Lin N, Yun D, Xia W, Palensky P, Vergara P P. Comparative analysis of zero-shot capability of time series foundation models in short-term load prediction. In: 2025 IEEE Kiel PowerTech. IEEE, 2025: 1-6.
  • [58]Marchesi G, Ballarino A, Brusaferri A. Assessing time series foundation models for probabilistic electricity price forecasting. Energies, 2025, 18(23): 6269.
  • [59]Hornek T, Sartipi A, Tchappi I, Fridgen G. Benchmarking pre-trained time series models for electricity price forecasting. In: 16th International Conference on the European Energy Market (EEM). IEEE, 2025: 1-7.
  • [60]Duan Z, Bian C, Yang S. Zero-shot forecasting of volatile wind power against data missing with large language model through attentive residual prompt tuning. Renewable Energy, 2026, 257: 124808. https://doi.org/10.1016/j.renene.2025.124808
  • [61]Zhang Z, Liu D, et al. A review of power system forecasting techniques based on large language models. Power Generation Technology, 2025, 46(3): 438-453.
  • [62]Pucher G, Dada A, Nensa F, Schuler M, Reinhardt C, Kleesiek J, Sauer C. Evaluating zero-shot foundation models for time series forecasting in clinical settings: A simulation study with electronic health records. Studies in Health Technology and Informatics, 2025, 329: 820-824.
  • [63]Ali M, Moore P W, Barkouki T, Brattain L J. Zero-shot forecasting for ECG time series data using generative foundation models. In: 2024 IEEE 20th International Conference on Body Sensor Networks (BSN). IEEE, 2024: 1–4. https://doi.org/10.1109/BSN63547.2024.10780646
  • [64]Chu R Y K, Szeto K C, Chung P H. Multivariate influenza forecasting using time-series foundation models: A comparative study with classical statistical models using 11 years of surveillance data in Hong Kong. Scientific Reports, 2026. https://doi.org/10.1038/s41598-026-64335-9
  • [65]Toner W, Lee T L, Joosen A, Singh R, Asenov M. Performance of zero-shot time series foundation models on cloud data. In: I Can't Believe It's Not Better: Challenges in Applied Deep Learning at ICLR 2025 Workshops. PMLR, 2025, 296: 1–12. https://proceedings.mlr.press/v296/toner25a.html
  • [66]Woo G, et al. Pushing the limits of pre-training for time series forecasting in the CloudOps domain. arXiv preprint arXiv:2310.05063, 2023.
  • [67]Jetwiriyanon J, Susnjak T, Ranathunga S. Generalisation bounds of zero-shot economic forecasting using time series foundation models. Machine Learning and Knowledge Extraction (MAKE), 2025, 7(4): 135.
  • [68]Xiao W Y L, Hou C, Han S. A financial multivariate time series forecasting framework integrating large language models: Experimental evaluation. Journal of Beijing Normal University (Natural Science), 2025, 61(6): 797-804.
  • [69]Puvvada S, Chaudhuri A. Critical evaluation of time series foundation models in demand forecasting. In: TSFALM Workshop (OpenReview: TS42sRKINd), 2024.
  • [70]Sun A Y, Sun A A. Zero-shot forecasting of streamflow using time series foundation models: Are we there yet? Machine Learning: Earth, 2026, 2: 010501. https://doi.org/10.1088/3049-4753/ae4982
  • [71]Huang Y, et al. Evaluating the generalizability of foundation models for extreme environmental events: Case study of California wildfire PM2.5. arXiv preprint arXiv:2607.07951, 2026.
  • [72]Cao X, Liang P, Huang W, Zhang Z, Gong Y. Forecast evaluation of large artificial intelligence models for the 2024 Meiyu season in the middle and lower reaches of the Yangtze River. Journal of Tropical Meteorology, 2024, 40(6): 1045-1062.
  • [73]Meyer M, et al. TS-Arena: A live forecast pre-registration platform for leakage-free evaluation. In: Proceedings of ACM SIGKDD, 2026.
  • [74]Luo W, Wang H. A review of large language model evaluation. Journal of Chinese Information Processing, 2024, 38(1): 1-23.
  • [75]Zhang T, Ouyang B, Shu Y, Chen X, Guo C, Yang B. SwiftTS: A swift selection framework for time series pre-trained models via multi-task meta-learning. In: Proceedings of ICLR, 2026.
  • [76]Abdallah Z, et al. Evaluation-free time series forecasting model selection via meta-learning. ACM Transactions on Knowledge Discovery from Data, 2025, 19(3): 1-41.
  • [77]Zhang J, Wen X, Zhang Z, Zheng S, Li J, Bian J. ProbTS: Benchmarking point and distributional forecasting across diverse prediction horizons. Advances in Neural Information Processing Systems 37, Datasets and Benchmarks Track, 2024. https://doi.org/10.52202/079017-1523
  • [78]Shchur O, Ansari A F, Turkmen C, et al. fev-bench: A realistic benchmark for time series forecasting. arXiv:2509.26468, 2025. https://doi.org/10.48550/arXiv.2509.26468
  • [79]Williams A R, Ashok A, Marcotte É, et al. Context is Key: A benchmark for forecasting with essential textual information. In: Proceedings of ICML. PMLR, 2025, 267: 66887–66944. https://proceedings.mlr.press/v267/williams25a.html
  • [80]Ghoroghchian N, Zhang H, Han S, Labach A, Stein G. Foundations without fundamentals: Zero-shot blind spots in time series FMs. In: 2nd ICML Workshop on Foundation Models for Structured Data, 2026. https://openreview.net/forum?id=iIRdd86Xkr
  • [81]O'Donnell L, Kolbeinsson A, Kolbeinsson B, Deisenroth M P. Towards benchmarking time series foundation models on native scientific data. In: 2nd ICML Workshop on Foundation Models for Structured Data, 2026. https://openreview.net/forum?id=NtU2gLl4hN
  • [82]Syed M A B, Ahamed M Y, Wasi A T. Position: Time-series foundation models require explicit domain-level benchmarks. In: Proceedings of ICML. PMLR, 2026, 306: 172221–172247. https://proceedings.mlr.press/v306/syed26a.html
  • [83]Li L, Sleem L, Wang Y, Xu Y, Gentile N, State R. Uncovering zero-shot generalization gaps in time-series foundation models using real-world videos. In: AI4TS Workshop at AAAI, 2026. arXiv:2509.26347. https://doi.org/10.48550/arXiv.2509.26347
  • [84]Lee J, Seo J, Lee S, et al. FinVerse: Financial time-series benchmark. arXiv:2608.03259v1, 2026. https://arxiv.org/abs/2608.03259v1
  • [85]Rahimikia E, Ni H, Wang W. Re(Visiting) time series foundation models in finance. arXiv:2511.18578, 2025. https://doi.org/10.48550/arXiv.2511.18578
  • [86]Zhang J, Zhang Z, Zheng S, Wen X, Li J, Bian J. Are time-series foundation models deployment-ready? A systematic study of adversarial robustness across domains. arXiv:2505.19397, 2025. https://doi.org/10.48550/arXiv.2505.19397
  • [87]Jiang L L, Cheong W S, Ng J S L. Multi-step ahead carbon credit price forecasting using time series foundation models. Applied Soft Computing, 2026, 198: 115290. https://doi.org/10.1016/j.asoc.2026.115290
  • [88]Li Y, Zhao Q, Wang M. Zero-shot traffic flow prediction with foundation models: A comparison with deep learning approaches. Scientific Reports, 2026, 16: 20472. https://doi.org/10.1038/s41598-026-50464-8

Article Sidebar

Full Text
PDF
Information
Copy Publication Information
Published
Nov 2, 2026
Issue
Vol. 3 No. 6 (2026)
Section
Research Articles
Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.

How to Cite

Yang, H. (2026). Controversies in Zero-Shot Forecasting with Time Series Foundation Models and a Three-Layer Framework for Trustworthy Evaluation. Computers and Artificial Intelligence, 3(6), 1-20. https://doi.org/10.70267/cai.v3n6.0120