From Inspiration to Falsification: One Polymarket Idea, Start to Finish

Prediction markets are an interesting place: people price their own opinions with real money, and nothing reveals information — reveals what people actually think — better than where the money goes.

The market is still young, and a blue ocean leaves room for arbitrageurs and speculators. In high school a teacher told us about someone who built a model from weather, news, temperature and so on to predict horse races, with very high accuracy. Polymarket is at bottom the same thing — only the bet is not on horses but on whether BTC goes up or down, on election results, on the direction of policy. Playing against all sorts of people here is great fun.

I narrowed my research to one niche: predicting BTC up/down over 5- and 15-minute windows.

The idea first came one evening during midterms. Could I model the fair price of this BTC up/down binary option with a stochastic process, and compare it with the market price to find mispricing? I began with the simplest thing, geometric Brownian motion, using BTC's annualized volatility to derive a fixed σ. But the backtest gave no significant p-value, and when the data doubled the return shrank to nearly zero; the edge could not cover trading costs.

In truth, modeling to find mispricing is what market makers do best, and this market is already priced well enough. The opponents here are market makers with low fees and heavy infrastructure, and they had eaten most of the edge in that direction long ago. Refining the model further was pointless — a few more basis points of gross margin would not change my structural disadvantage — so I wondered whether I could find a more naive set of opponents. In the jungle, the most conspicuous prey is the retail gambler. I looked up some theories that had been validated in other markets, and began the next line of research: retail systematically overestimates small-probability events — the favorite-longshot bias.

Taking a theory that has been studied and validated to death in a mature market and applying it to a new one is a fine way to generate ideas.

0/ The data

Polymarket exposes no interface for historical ticks; you have to connect over websocket and collect the data yourself. So at the start I had very little — a few days. That directly led to many of the statistical mistakes I made later.

1/ Hypothesis and a first check

Starting hypothesis: retail systematically overestimates small-probability events → the YES token is overpriced at low prices → SELL YES has positive expectation.

Method: a calibration table — bucket by mid price into ten bins, and compare the realized UP rate with the market price.

First results (424 windows, March 21–23) appeared to support the hypothesis:

mid 0.20–0.30: realized 12.9% vs priced 25.6%, a deviation of −12.7%
mid 0.60–0.70: realized 81.8% vs priced 64.9%, a deviation of +16.9%

At this point I thought I had found "severe overpricing" and "severe underpricing."

The core problem is that what I was looking at was a point estimate, and point estimates are extremely unstable in small samples. The "deviation" of 12.9% vs 25.6% might mean the market really was wrong, or it might purely be that these 31 samples happened to be unlucky. Without further tools you cannot tell the two apart. Flip a fair coin (true probability 50%) but only ten times: you may well get seven heads and three tails and compute a "heads rate of 70%." That does not mean the coin is biased — it is just the random fluctuation of a small sample.

The lesson: descriptive statistics without confidence intervals mean nothing. When the sample is small, any percentage can look "significant." What we need is interval inference, to answer "what is the plausible range of the true probability?"

2/ Confidence intervals

For small samples or extreme proportions we use the Wilson score interval rather than the ordinary CI.

For mid 0.20–0.30 there are only 31 samples; the Wilson 95% CI is [3.5%, 25.6%]. The market price of 25.6% sits at the edge of the interval — we cannot distinguish a market error from random fluctuation. In other words, the deviation there may well be nothing but noise.

Checking bin by bin: of the ten bins, only one (0.60–0.70) has the market price outside its CI. The other nine "deviations" are all noise.

3/ Multiple comparisons

Thanks to Prof. Chuan's DDA3600 — that is where I learned to tell false alpha from real.

(The same idea handles overfitting in parameter optimization.)

For each test my standard was p < 0.05, that is:

"If the market price were in fact perfectly accurate (zero deviation), the probability of observing a deviation this large or larger is under 5%."

Fine for one bin considered alone — but here I did it ten times.

For a test at p < 0.05, the false-positive probability from one test is 5%; from ten tests, the probability of at least one false positive is 40.1%!

So the p-values need correcting. There are many methods; here, with the aim of controlling the overall false-positive rate, we use the Bonferroni correction.

Bonferroni's idea is extremely simple: to hold the family-wise error rate (FWER) at α = 0.05 over k tests, tighten each test's threshold to α/k.

In this case k = 10, so a single bin's p-value must be below 0.005 to count as a "finding."

For the 0.60–0.70 bin that fell outside its CI, the raw p-value was ≈ 0.008 (z = 2.40); after Bonferroni, p = 0.008 × 10 = 0.08, which fails the 0.05 bar. The investigation thus rejected the favorite-longshot hypothesis: strict Bonferroni ruled out pushing the original "systematic overestimation of small-probability events" strategy in that bin. But even though the 0.60–0.70 bin's statistics were not significant enough, we could still keep investigating.

Three reasons:

The effect size is large enough. A +17% deviation is the largest in the table. Even if it is statistically marginal (some signal, not enough significance), it is economically worth pursuing — if it is real, it is a tradeable signal. A +17% deviation, marginal or not, is a huge edge if it turns out to be real — worth the time to find out whether it is noise or signal rather than throwing it away.
Time splits provide independent evidence. All three days showed a positive deviation (+18%, +21%, +9%). That is not a repeat of the same statistical test but additional robustness evidence.
Statistical insignificance does not mean not worth exploring. Bonferroni answers "should I bet real money on this," not "should I keep investigating the mechanism." In the exploratory phase, a marginally significant finding with a large effect deserves the question "why" — even if the final answer is "it's false," understanding why it looked true has value. (And so it proved: the deeper investigation revealed a momentum mechanism.)

4/ Attributing the phenomenon

After confidence intervals and Bonferroni, I was left with one fact:

"In the 0.60–0.70 bin, the market prices about 65%, the realized UP rate is about 82%, a deviation of +17%. The effect is large and consistent across three daily splits; worth further investigation."

From here, can we conclude that our original favorite-longshot hypothesis is right, and hence that "the market systematically underestimates the probability of UP when mid = 0.65, so buy the YES token at that price"?

No. The mechanism behind the fact is still unclear, and the same phenomenon can have entirely different causes.

We face two competing hypotheses, each able to explain "in the 0.60–0.70 bin, the realized UP rate exceeds the market price":

Hypothesis A: pricing bias

At mid = 0.65 the market systematically underestimates the probability of UP.

However the mid got to 0.65, as long as it is at 0.65, the market has priced it too low.

If A is true:

this is a static market defect;
the strategy is simple: whenever mid is in 0.60–0.70, buy YES;
how the mid arrived at that price does not matter.

Hypothesis B: momentum continuation

The path by which mid reached 0.65 is what matters.

If mid rose from 0.50 to 0.65, BTC is rising, the rise may continue, and so the UP rate is high.

It is not that "0.65 is underpriced," but that "the process of rising to 0.65 predicts further rising."

If B is true:

the deviation is not a pricing error but a projection of a momentum signal;
0.60–0.70 merely happens to capture the "currently rising" window;
the real signal is not the price level but the direction of price change.

The two hypotheses lead to entirely different strategies. Trading without distinguishing them risks deciding on a false causal relationship.

So we need a key test: for the 52 windows in the 0.60–0.70 bin, look at how mid reached that price.

(This step is essentially confounding-variable control.)

The result: 48 of 52 windows "rose from below," and 0 "fell from above." Can this attribute the effect to hypothesis B, momentum?

No. 48/52 rising from below only describes the composition of the data; it does not prove causation. "Rose from below" and "mid at 0.65" are completely entangled (confounded) and cannot be separated.

How, then, to separate the two factors? I tried logistic regression.

The core capability of logistic regression: put two variables into the model at once, and it tries to separate their independent contributions.

We have two candidate explanatory variables:

mid_level: the current absolute mid (e.g. 0.65)
delta_mid: the change in mid from the open to now (e.g. +0.15)

We want to know: is the UP rate high because the level is high (A), or because it has risen a lot (B)?

The model:

P(UP) = σ(β₀ + β₁·mid_level + β₂·delta_mid)

The model asks:

Holding delta_mid fixed, how much does UP probability change per unit of mid_level?

Holding mid_level fixed, how much does UP probability change per unit of delta_mid?

If the regression works, the answer can be read straight off:

β₁ significant, β₂ not → A wins; the level has an independent effect
β₁ not, β₂ significant → B wins; momentum is the driver
both significant → both contribute

The result:

mid_level: β₁ = +5.622, p = 0.0004

delta_mid: β₂ = +0.271, p = 0.863

At first glance "level wins, momentum loses" — but the result cannot be trusted.

There is one more essential check: multicollinearity.

We measured Correlation(mid_level, delta_mid) = 0.972 — knowing one, you almost perfectly know the other. A correlation of 0.972 means logistic regression cannot reliably separate the two variables. The effect was assigned to mid_level more or less at random, and delta_mid's p-value was artificially inflated. This is the classic collinearity artifact, not a real causal separation.

The lesson: under high collinearity (r > 0.9), attributing regression coefficients is meaningless. Check the correlation before interpreting coefficients.

Moreover, 80% of the windows have mid_start in 0.45–0.55, so mid_level ≈ 0.50 + delta_mid — the two variables are mathematically near-equivalent.

So on the current data this question has no solution; we cannot truly reject either hypothesis A or B. In this dataset, mid_start has too little variation.

5/ Analyzing the signal

For trading, attribution does not matter; attribution does not change the trading outcome. What matters is that the signal is here — how strong is it, and can it make money? Whether its cause is "momentum" or "level," what we care about is whether it can be traded.

The only scenario in which attribution matters is if you want to generalize the signal to other markets. If it is momentum-driven, it may work on DEXs and CEXs too; if it is a property of the price level, it is peculiar to Polymarket's BTC up/down markets.

So here we do the most important step: verify whether this signal (whatever we call it) is already priced in by the market.

We want to know: the larger the signal (delta_mid as proxy), the higher the probability of UP?

A Spearman rank test gives r = 0.575, p < 0.000001. It looks as if delta_mid can almost tell you the settlement direction outright.

But at what moment was this r = 0.575 measured? The value of delta_mid depends on where in the window you observe. Computing r at each time point produced an unexpected pattern:

Observed at	Spearman r	rise→UP	fall→DOWN
t=10s	+0.072	57%	52%
t=30s	+0.200	62%	58%
t=60s	+0.306	67%	62%
t=120s	+0.498	75%	71%
t=150s	+0.558	75%	75%
t=240s	+0.687	84%	80%

r rises monotonically — from 0.072 at t=10s all the way to 0.687 at t=240s. The r = 0.575 reported earlier was computed at t ≈ 150s, the window's midpoint.

This is not "the momentum signal getting stronger"; it is a simpler fact: the closer to settlement, the closer mid is to the final outcome. A 5-minute window is 300 seconds; at t=240s only 60 seconds remain — mid is almost the settlement price itself, so of course r is high. To a large extent this is a tautology, not real predictive power.

6/ The key question: can it make money?

By now we know the signal is statistically "real" — delta_mid really is highly correlated with settlement direction. But the most important lesson in quantitative research is: the strength of a signal is not the same as its profitability.

An analogy: a stock's annual earnings growth is highly correlated with its price movement (r perhaps near 1). But you cannot make money by "buying stocks whose earnings grew," because the price already reflects expected earnings. The price you buy at has already priced that information in.

The same applies here: when delta_mid is large (mid rising), the YES token's ask price is rising too. If the ask rises as fast as the UP rate, no strength of signal makes money.

So what decides the outcome is not "how strong the signal is" but:

edge = UP rate − ask price − trading cost

Computing this edge at each time point:

Observed at	r	net edge, BUY	net edge, SELL
t=10s	+0.072	−0.059	−0.119
t=30s	+0.200	−0.020	−0.049
t=60s	+0.306	+0.014	−0.036
t=120s	+0.498	+0.017	−0.033
t=150s	+0.558	+0.018	−0.024
t=240s	+0.687	−0.027	−0.029

Net edge is near zero or negative at every moment. The market prices accurately at every point in time — the market makers' ask faithfully tracks the UP rate, leaving almost no room for a directional trader.

The table also exposes a fundamental trade-off:

Observe early (t=10s) and the ask is low (cheap), but the signal is weak (r = 0.072) and the UP rate is about 50% — you don't know whether to buy or sell.
Observe late (t=240s) and the signal is strong (r = 0.687), but the ask has caught up with the UP rate — you know what to buy, but it is no longer cheap.

The best r (t=240s) and the best edge (t=40s) occur at completely different moments. The moment the signal is clearest is precisely the moment it is least profitable — because the market maker sees the same information you do.

SELL is negative at every moment, which says the market prices the downside even more accurately.

7/ Conclusion: falsified

By this point the conclusion is clear:

The favorite-longshot hypothesis is rejected — the "deviations" in the calibration table are, after CIs and Bonferroni, almost all noise.
There is a statistically very strong signal (delta_mid vs settlement direction, r up to 0.687) — but it is largely the tautology "mid near settlement ≈ the settlement result."
The signal cannot be turned into profit — the market's ask tracks it faithfully at every moment, and after 3% trading costs the net edge is negative.
The causal attribution of the signal ("momentum" or "level") cannot be separated — but for the trading decision it does not matter.

The endpoint of the whole study is a strategy correctly rejected. No profitable factor was found, but a few important lessons were:

Signal ≠ profit. A fully priced signal can have perfect predictive power and zero profitability.

A number without a confidence interval is noise. In small samples any percentage can look "significant."

The size of a statistic ≠ tradeability. r = 0.575 looks like a very strong signal, but decomposed by time it mostly reflects mid converging to the settlement result — the natural working of the market, not an exploitable inefficiency.

Every cent you make, some counterparty loses. If you cannot say who the counterparty is, you are most likely the prey. In this market the market makers have eaten the directional edge clean. Retail really is losing money — but losing it to the market makers, and I am in no position at the moment to take a share.