Kalshi Weather Edge, called BT until September 2026, tested whether a 31-member GFS ensemble, read as a probability distribution, beats Kalshi's price on daily high-temperature contracts after fees. It ran unattended for four months, settled 7,440 signals against the official NWS climate record, and answered no. The same pipeline now carries a market maker for the other side of those books, built and gated but not yet run.
What it does
Every scan window, four per day, for five cities (New York, Chicago Midway, Denver, Miami, Los Angeles):
- Pull the 31-member GFS ensemble from Open-Meteo for the contract day.
- Fetch the live market list from Kalshi and parse each bracket into a temperature interval.
- Compute the model probability as the fraction of ensemble members landing in the bracket.
- Compute the fee-adjusted edge against the market price.
- Persist every bracket evaluation as a signal, edge or not, so calibration sees the full distribution.
- After settlement, reconcile each signal against the NWS Daily Climate Report and record the outcome.
- Report reliability curves, Brier score, and a paper simulation. Trading is gated on this report.
Step 7 is the design. Live trading was never allowed to start on a backtest; it had to pass calibration on settled signals first. It did not pass.
The result
Over 83 contract days and 2,472 distinct contracts, a bracket resolved yes 16.7% of the time. The model was overconfident at both ends.
| Predicted bucket | n | Mean predicted | Realized hit rate |
|---|---|---|---|
| 0.00 – 0.17 | 5,223 | 0.02 | 0.14 |
| 0.17 – 0.33 | 661 | 0.26 | 0.21 |
| 0.33 – 0.50 | 548 | 0.42 | 0.26 |
| 0.50 – 0.67 | 406 | 0.58 | 0.24 |
| 0.67 – 0.83 | 272 | 0.74 | 0.19 |
| 0.83 – 1.00 | 330 | 0.92 | 0.24 |
Brackets the model priced near zero hit 14% of the time; brackets it priced above 83% hit 24%. Reading ensemble-member frequency directly as probability ignores that the members share systematic bias on a given day, so the spread understates true uncertainty.
Isotonic calibration, out of fold with one contract day held out at a time, fixes the reliability curve and adds no skill. On the 2,777 signals with a two-sided order book:
| Probabilities | Brier |
|---|---|
| Raw model | 0.251 |
| Calibrated model, out of fold | 0.204 |
| Kalshi market price | 0.120 |
Calibration makes the model honest by squeezing every forecast into roughly 0.13 to 0.46, so it barely separates brackets. The market price beats both versions by a wide margin. There is no information here the market does not already have. The paper simulation says the same thing in dollars: yes-side trades at the executable ask, one contract per signal, the taker fee deducted.
| Probabilities | Edge threshold | Trades | Net return per $1 staked |
|---|---|---|---|
| Raw | 0.05 | 740 | −$0.087 |
| Raw | 0.10 | 617 | −$0.112 |
| Raw | 0.20 | 467 | −$0.104 |
| Calibrated | 0.05 | 1,115 | −$0.420 |
| Calibrated | 0.10 | 740 | −$0.273 |
Negative everywhere. The calibrated model does worse because it lifts long shots just far enough to clear the threshold.
A model that says 74% and wins 19% is not edge. It is a bug with a P&L attached.
Three bugs before the number was real
The first version of this analysis, on 1,299 Chicago-only outcomes, showed a positive simulated return. Every cent of it was an artifact.
- Wrong weather station. The settlement fetcher passed the NWS forecast-office code where the endpoint expects the climate-station code. Four cities never settled at all, and Chicago settled against the wrong office's report. 246 of 1,299 stored outcomes were wrong.
- Phantom market prices. The stored price was the raw yes ask. On an empty book Kalshi returns 100, so 4,663 of 7,440 signals were recorded as the market saying 100% when the market said nothing. A bot trading against that price looks brilliant on paper and gets no fills.
- Retired forecast model. Open-Meteo dropped the model id the blend adapter used; the blend had been silently failing and falling back to GFS only.
The fixes: a station-keyed fetch, a backfill that resettles every signal from the IEM archive of NWS climate reports (validated against Kalshi's official results on 2,724 overlapping contracts, with zero disagreements), mid-price capture with a book-validity flag, and the new model id.
What the pipeline got right
The settlement-source guard worked. In August Kalshi changed the resolution text on every series to cite a different source. The bot checks the market's rules text against the expected station on every startup, detected the mismatch, and aborted rather than settle against an unverified source. It stayed halted for six weeks until I updated the expected tokens. The guard was right; the missing alarm on the abort loop was a monitoring gap.
Four months unattended. 337 scheduled scans, heartbeat monitoring, log rotation, clock-drift checks and rate-limit backoff, on a laptop.
Auditable state. Every evaluation is a row in SQLite and an event in an append-only journal. All of the analysis above was reconstructed from that data after the fact, including catching the bugs.
Math, briefly
Kalshi's taker fee is quadratic in price and peaks at $0.0175 per contract at even odds. Edge deducts it before anything else.
fee = 0.07 * P * (1 - P)
edge = p_model - p_market - fee(p_market)
Sizing is fractional Kelly with hard caps the strategy cannot raise.
kelly_fraction = f * (edge / (price * (1 - price)))
position_usd = bankroll * min(kelly_fraction, max_position_pct)
What would change the answer
Not more of the same data. The ensemble alone is dominated by the price. Things that might contain information the market lacks: a multi-model blend with per-station bias correction, intraday observations close to settlement, or station-specific downscaling. Each is a new hypothesis, not a tuning of this one, and the pipeline can test any of them the same way.
A second pipeline adapts the same shape to sports moneylines using de-vigged sharp consensus as the model. It is built and tested, ran for a single afternoon, and has no results to report.
The other side of the book
If the forecast has no edge, the price still might: someone is being paid to provide liquidity. Across 785 historical snapshots with all six brackets quoted, the median sum of the six asks was 1.06. That six percent overround is what makers earn, and maker orders on Kalshi pay no fee. So the next experiment quotes the brackets instead of forecasting them.
Fair value for one bracket comes from its siblings, with no forecast input at all, and at most one missing leg is imputed:
fair_k = clip(100 - sum(mid_j for j != k), 1, 99)
bid = fair_k - 2 ask = fair_k + 2 one contract per side
A tick runs every two minutes under launchd or systemd, and there is no daemon: every order carries a six-minute expiry on the exchange, so if the ticks stop the quotes die on their own. Hard caps live in the risk gate, not the quoter: five dollars of worst-case loss per event, twenty-five in total, three contracts per leg, and the same kill file as the scanner.
Two gates stand between this code and money. The first is plumbing, about two weeks on the demo exchange: clean reconciliation every tick, no orphan orders, caps never breached, expiry verified by killing a tick. It has no profit criterion. The second is the verdict, about four weeks live: at least a hundred fills with positive mean markout at both horizons and positive realized P&L per fill, or the experiment is inconclusive.
As of 24 September 2026 the quoter is merged, off by default, and has placed zero orders on demo or live. There are no results, and there will be none on this page until the gates produce them.