Bitcoin is an unusually attractive benchmark for forecasting models: the market trades continuously, historical data is widely available, volatility is high, and researchers can define many horizons. Those same qualities also make it easy to produce misleading results. A model can appear accurate because of leakage, a favorable test window, an unrealistic trading rule, or a comparison against weak baselines. This article lays out a reproducible benchmark for Chronos-2, TimesFM-3, Moirai 2.0, and simpler methods, with the goal of measuring genuine forecasting value rather than creating a spectacular equity curve.
Begin with a precise question
“Can AI predict Bitcoin?” is not a testable research question. The target must specify venue, data type, sampling frequency, horizon, and output. One defensible first experiment uses a liquid BTC/USD spot market, hourly bars, and forecasts one, six, and twenty-four hours ahead. The output can include future log return, realized volatility, and quantiles of the future price distribution. These targets represent different levels of difficulty and different operational uses.
Next-period return is noisy but directly connected to trading. Realized volatility is more persistent and useful for position sizing, options, and risk. Price-level error often looks impressive because price is highly autocorrelated, yet a naive “next price equals current price” forecast can be difficult to beat. Reporting all three targets prevents the study from selecting whichever metric flatters a model.
Build an immutable data layer
Choose a primary exchange and download trades or bars with explicit UTC timestamps. Preserve the untouched source, then create a versioned analytical dataset. Document how duplicate timestamps, missing bars, exchange outages, and abnormal prints are handled. If multiple venues are combined, do not average them casually: venue composition changes through time, and historical availability can introduce survivorship bias. A benchmark should be reproducible from a manifest that identifies source, retrieval date, checksum, and transformations.
Features must be causal. At an hourly forecast origin, the last bar should be included only after it is complete. Rolling volatility, moving averages, and volume statistics must use past observations. Funding rates, open interest, stablecoin flows, or macro variables need publication-time alignment rather than period labels. A value described as belonging to Monday may not have been known until Tuesday.
Use walk-forward evaluation
A random train-test split is inappropriate because it lets the model learn from future regimes and then predict the past. Use expanding or rolling windows. At each evaluation block, all fitting, scaling, calibration, and model selection occur using earlier data. Then freeze the pipeline and forecast the next block. Advance the origin and repeat. This creates a sequence of genuinely out-of-sample forecasts across bull, bear, quiet, and stressed periods.
Foundation models add a special challenge: their pretraining data may contain parts of Bitcoin history. Zero-shot does not automatically mean uncontaminated. Record the model release, known training cutoff, and evaluation dates. A clean “post-release” segment is valuable, while older history can still be reported as a potentially contaminated diagnostic. The GIFT-Eval project demonstrates the importance of labeling test-data leakage explicitly.
Define the model roster fairly
Test Chronos-2, TimesFM-3, and Moirai 2.0 with published default settings first. Include seasonal naive, random walk, exponentially weighted volatility, linear autoregression, and a tuned gradient-boosted tree. These baselines are not decorative. If a large pretrained model cannot beat a simple rule consistently, that is a useful result. The benchmark should also include the same context lengths and forecast information wherever architectures permit.
A second track can allow adaptation, but every model receives the same budget: a fixed number of validation evaluations, fixed hardware time, and the same historical window. Otherwise a heavily tuned challenger is being compared with an untouched baseline. Keep a complete experiment ledger so researchers cannot quietly discard unsuccessful variants.
Metrics for forecast quality
| Target | Primary metrics | Why it matters |
|---|---|---|
| Price level | MAE, MASE | Checks scale-aware path accuracy |
| Return | MAE, rank correlation, sign probability loss | Measures weak directional information |
| Volatility | QLIKE, MAE | Connects to sizing and derivatives |
| Quantiles | Pinball loss, interval coverage | Tests distribution quality |
| Trading rule | Net Sharpe, drawdown, turnover | Tests economic relevance after costs |
Aggregate metrics should be accompanied by time slices. An average can conceal a model that performs well only during trends and fails during reversals. Report results by volatility regime, trend state, calendar year, and horizon. Confidence intervals can be estimated using blocks that respect temporal dependence. Pairwise model comparisons should use the same forecast origins.
Translate forecasts into a minimal strategy
The first economic test should be intentionally simple. Convert the predicted return distribution into a position scaled by expected return and capped by predicted risk. Introduce a no-trade band so tiny forecasts do not generate excessive turnover. Execute with a one-bar delay unless the forecast was demonstrably available before the bar boundary. Apply fees, bid-ask spread, estimated impact, funding, and borrow where relevant.
Do not optimize a complex strategy separately for each model. That tests the researcher’s ability to overfit, not forecast quality. Use one declared mapping or a very small family selected on validation data. Report gross and net performance together. If gross alpha disappears under plausible costs, the forecast might still help risk management, but it should not be advertised as a trading edge.
Stress tests that expose fragile models
Shift the forecast origin by a few minutes, vary the context length, and test adjacent venues. Resample to four-hour and daily bars. Perturb inputs within realistic data-error ranges. Remove one covariate at a time. A robust model should degrade gradually rather than collapse when a specific exchange, start date, or normalization is changed. Theme changes should not matter unless chart images are an input.
Regime stress is equally important. Evaluate sudden volatility expansions, liquidity deterioration, persistent trends, and range-bound periods separately. Models trained on smooth sequences may produce intervals that are dangerously narrow during shocks. Coverage plots reveal whether a nominal 90 percent interval actually contains outcomes near 90 percent of the time.
What a useful result page should show
Publish the dataset period, forecast schedule, model versions, hardware, inference latency, and preprocessing. Show a table of primary metrics, a cumulative error comparison, interval-calibration plot, and net strategy curve with drawdown. Include failures: invalid outputs, memory limits, and missing forecasts. A polished chart without an experiment manifest cannot be independently assessed.
The benchmark should also expose forecast files or at least a representative sample. That allows others to test alternative economic mappings without rerunning expensive models. Code that reproduces the split and metrics is more valuable than a notebook containing manual steps. Amazon’s Chronos repository, Google’s TimesFM repository, and Salesforce’s Uni2TS repository are appropriate implementation references.
Interpreting the winner
One model may win return ranking while another produces better calibrated volatility intervals. That is not a contradiction. Forecasting systems have multiple roles. A return model can rank opportunities, a volatility model can size them, and a probabilistic model can control tail exposure. An ensemble may outperform every member, but ensemble weights must be learned on past validation data.
Statistical significance is not permanence. Crypto market structure evolves as participants, regulation, leverage, and venues change. Treat the benchmark as a dated measurement, not a universal declaration. Schedule reevaluation and monitor live forecast calibration. When performance decays, compare against the same naive baselines before retraining.
Conclusion
A credible Bitcoin benchmark is less about selecting an exciting model and more about constraining the experiment. Immutable data, causal features, walk-forward splits, strong baselines, honest costs, and post-release evaluation turn a demonstration into research. Chronos-2, TimesFM-3, and Moirai 2.0 deserve careful testing, but none receives a free pass because it is a foundation model. If Quantin publishes the full protocol alongside the results, even a negative finding becomes valuable: it tells practitioners where general forecasting intelligence does—and does not—survive contact with a real market.