Forecasting electricity prices in a volatile market
A case study · Australian National Electricity Market
Mohammed Osman Gani · Zhipeng He · Chun Ouyang · Sara Khalifa
Energy Transition Centre & School of Information Systems, Queensland University of Technology
Act I · Volatility
NEM prices at a glance
A week of half-hourly Regional Reference Prices in Queensland sets out three structural conditions that any forecaster has to handle. Each appears in a different hour of the day, and each carries different downstream stakes.
↗ Midday negatives (solar oversupply)
↗ Evening spike (ramp + peak demand)
↗ Direction-flip hours
Act I · Volatility
One spike, two views
On a single evening, the settled price climbed past $1,100, sixteen times the week's typical level. AEMO's predispatch forecast, published 24 hours earlier, placed that same half-hour near the market price cap, an over-forecast of roughly an order of magnitude, and held that level after the settled price had already fallen back. The failure is one of magnitude and persistence rather than detection.
The size of the vertical gap on that hour is the forecast value at stake for every battery, retailer, and trader acting on the official forward path.
Act II · Diagnosis
Where models actually fail
A single average error number hides where forecast errors actually live. Decomposed across the 48 half-hour intervals of the day, three structural failure windows appear — and every model studied falls into all three.
Evening volatility window
Midday negative-price window
Direction-flip hours
- LSTM
- CNN-LSTM
- Transformer
- TimeMixer
- TimeXer
Act II · Diagnosis
The leaderboard surprise
Selecting a Region, Horizon, and Metric changes the ranking of the models. The benchmark's principal results show: the standard deep-learning baselines (LSTM, CNN-LSTM and Transformer) rank among the top three in nearly every setting, and none generalises across the market. Architectural sophistication alone therefore does not transfer to this market, which points toward volatility-aware modelling rather than further architectural sophistication.
The three marked models are the architectures EPF studies already rely on. Switch region, horizon, or metric: they hold the top ranks nearly every time, and the newer benchmark models re-shuffle below them. Test-split values.
- #1TransformerTransformer · standard in EPF54.94A$/MWh
- #2LSTMRNN · standard in EPF57.88A$/MWh
- #3MambaSSM58.70A$/MWh
- #4CNN-LSTMHybrid · standard in EPF59.03A$/MWh
- #5TimeXerTransformer65.85A$/MWh
- #6TimesNetCNN75.25A$/MWh
- #7iTransformerTransformer75.47A$/MWh
- #8DLinearLinear77.64A$/MWh
- #9TimeMixerMLP88.81A$/MWh
Act II · Diagnosis
The extreme prices are where it collapses
Restricted to the spike and crash prices, day-ahead MAE inflates roughly five-fold for every baseline and recent time-series model. Neither regime is solved by newer architectures.
Day-ahead (24 h) horizon.
Bridge
Spikes and crashes break things differently
The two regimes are not similar. Spike prices run past $15,000 while crashes never fall below −$200 and each arrives by a different mechanism. Spikes are scarcity: evening demand climbs just as solar falls away. Crashes are oversupply: midday demand sinks toward a solar supply that holds firm. A correction designed for one direction is not the correction needed for the other.
Demand climbs as solar falls away → price spikes
Scarcity — the demand–solar gap blows out into the evening peak.
Demand sinks toward steady solar → price goes negative
Oversupply — solar covers most of a shrinking midday load.
Act III · Treatment
A gated tail-residual network
The treatment augments the base forecasts rather than replacing it. A convolutional (CNN) encoder extracts local temporal patterns from the raw input window, and an adapter conditions a FiLM layer on the base model’s own representation so that the resulting features are sensitive to tail behaviour. Two non-negative residual heads, one for upward and one for downward deviations, propose asymmetric corrections, and a horizon-wise soft gate determines at each forecast step how much of each correction to apply. Training proceeds in two stages: the base model is fitted first and then frozen, so the correction module learns only the tail residuals and normal-regime forecasts remain essentially unchanged from the base forecast.
Act III · Treatment
Asymmetric correction
The window below covers a single 24-hour day-ahead horizon that contains both failure modes: a midday crash into negative prices and an evening spike. During normal hours the gate holds p(normal) close to one, and the corrected output remains close to the base forecast. As the spike develops, p(spike) rises and the spike head, which is non-negative by construction, adds upward magnitude; around the midday crash, p(crash) rises and the crash head returns the forecast below zero. In both events the corrected forecast recovers a substantial share of the base model's error, although a residual gap to the actual price remains.
Horizon-wise gate — regime weight per step
Applied correctionper step
Hour 18.0 · base 513 → GTR 1419 (actual 1787) · 0.83 · 0.02
Act III · Treatment
It works
Across every base model studied and across QLD and NSW, the tail-residual treatment concentrates its gain on the spike and crash regimes. Crash-regime error falls by 12% to 52% depending on the base model, and the weaker the base model on the tails, the more it gains. Normal-regime accuracy is not sacrificed: it improves too, by a smaller margin. Every comparison is significant at p < 0.001 under a Diebold–Mariano test with small-sample correction. Toggle base and region to inspect.
Note: spike = actual price at or above the 95th percentile, crash = at or below the 5th, the rest normal. Bar values are mean absolute error in A$/MWh on the day-ahead horizon; lower is better.
Open threads
Where the work goes next
- Multivariate inputs. Demand, renewable output, and interconnector flows likely carry the structural signals current price-only models cannot recover from a univariate history.
- Drift under regime change. The 2023–2025 data window already covers post-COVID and post-5-min-settlement; the NEM keeps evolving, and any deployed model needs an explicit drift response.
- Probabilistic forecasting. A point forecast conveys nothing about its own uncertainty, particularly in the spike and crash regimes; predictive distributions or quantiles for each half-hour would allow downstream decisions to account for tail risk explicitly.
The forecasting feeds AI-driven battery orchestration; tail accuracy is what unlocks the operational value that project delivers.
Code released alongside publication. ← Research · Contact the group