Calibrating a forecast against a market price

A probability forecast is only useful if its numbers mean what they say: of all the times it says 30%, the event should happen about 30% of the time. That property is calibration. A market price is the most demanding benchmark to calibrate against, because it is someone else's forecast with money behind it.

Calibration and sharpness

A forecast has two properties worth separating. Calibration is whether stated probabilities match observed frequencies. Sharpness is how far the forecast moves away from the base rate when the evidence allows it. A forecast that always says the base rate is perfectly calibrated and useless. A forecast that is confident and wrong is worse than useless. The aim is the sharpest forecast that stays calibrated.

Proper scoring rules reward both at once. The Brier score, the mean squared difference between the forecast and the outcome, and the log score, the logarithm of the probability the forecast gave to what actually happened, are both optimised in expectation by reporting the probability you really believe. The log score punishes a confident miss severely, which suits trading, where a confident forecast usually means a large position.

A reliability diagram makes calibration visible: forecasts are grouped into bins, and each bin's average forecast is plotted against how often the event happened. A calibrated model sits on the diagonal. The shape of the miss is informative in itself. A model whose points fall into an S around the diagonal is too timid; one whose points cross it the other way is overconfident.

Reading the market's probability

A price is not directly a probability. Where there is a spread, the midpoint is a better estimate than either side, and where the book is thin even the midpoint is noisy. Where a market lists several outcomes that cannot happen together, their prices often add up to more than one, and the excess has to be removed before the prices can be read as probabilities. Fees matter as well: a contract bought at 0.60 with a fee on top is a bet at a higher effective price.

Markets are not perfectly calibrated either. The best documented pattern, seen in horse racing, sports betting and some prediction markets, is the favourite-longshot bias: outcomes that are very unlikely tend to be priced above their true frequency, and heavy favourites a little below. Its size varies by market and over time, which is why it is measured rather than assumed.

The test that matters

The question that matters is whether the model beats the market price at the moment a trader would act on it. That means scoring the model and the market on the same events, at the same timestamps, with the model allowed only the information that existed at that moment. A test that lets anything from later leak in, a final statistic, a revised figure, a feature computed over the whole day, measures hindsight.

Then the difference has to pay for itself. A forecast that beats the market by less than the spread and the fees is a better forecast and not a trade. The comparison worth making is expected profit after costs, at the prices that were really available, at the size that could really have been filled.

Walked forward, judged by the day

Evaluation runs forward in time. The model is fitted on the past and scored on the next period, then the window moves on, so that every score comes from data the model had not seen. Events on the same day are rarely independent: one news story, one weather system, one slate of games moves many of them together. Treating each event as an independent sample overstates the confidence in any result. Resampling whole days, rather than single events, gives an honest interval on the gain, and a model whose gain disappears when its best day is left out has not shown much.

The market as a prior

The market price is also information, often the best single forecast available. A model built from scratch that ignores the price usually does worse than one that starts from the market's probability and adjusts it where it has a specific reason to know better: fresher data, a better reading of the settlement source, a pattern the market has been shown to misprice. The adjustment should be large where that reason applies and close to zero everywhere else. A model that disagrees with the market everywhere is usually wrong everywhere.