Our engine has been running a Thompson-sampling bandit on the homepage hero for 41 days. It crossed 10,000 impressions on Tuesday. This is the log of what it tried, what it learned, and the one place it was confidently wrong for two straight weeks.
Six variants went in. Two survive. The winner is not the one either human here would have picked, which is the usual finding and still the useful one.
What follows is the dispatch, numbers included, same as the engine files it internally.
The setup: six heroes, one metric
Six hero variants entered on May 14 — each a headline, a subhead, and a button label. The success metric is a qualified click: a visitor who reaches the contact page or spends 20-plus seconds on a service page in the same session. Allocation is Thompson sampling with a 5% traffic floor per variant, so nothing dies of starvation before the data says it should. Total through Tuesday: 10,014 impressions.
Two variants were ours, written by hand. Four were engine-generated from patterns in past winners. We did not label which was which in the dashboard, on purpose.
The floor matters more than it looks. Without it, one early lucky streak can lock the bandit onto a mediocre arm for weeks.
Why a composite metric instead of raw button clicks: clicks are cheap and they lie. Twenty seconds on a service page filters out the curious and keeps the shopping.
Baseline before the test: the old static hero converted qualified clicks at 2.9%.
Day 9: the first reallocation
The first meaningful move came at impression 2,140, on day nine. Variant D — headline "We run your marketing with software that shows its work," subhead naming the $1,500 flat fee — climbed from its starting sixth of traffic to 31%. Two wordplay-forward variants dropped to the 5% floor the same night and never left it. No human touched anything. The log line just changed.
Day nine is fast. On our traffic, a classical fixed-split test would still have been three weeks from a readable result.
That speed is the entire argument for bandits on low-traffic sites: reallocation starts while the test is still running, so the cost of a bad variant is capped early.
The wordplay collapse was the fastest signal in the run. Neither variant ever recovered from its first 400 impressions, which tells you how little clever buys when a visitor grants you four seconds.
We flag the date because prospects always ask when the thing starts earning. Answer: before week two.
What won, and by how much
Final standings at 10,014 impressions: variant D converts qualified clicks at 4.2% against the 2.9% static baseline, a 45% lift, with variant B second at 3.6%. The pattern behind D is not subtle. A concrete number in the subhead — the actual monthly price — plus a named artifact, the weekly changelog, beat every abstraction we tested by at least 34%.
The two wordplay variants, the ones we privately liked, finished last and fifth. Clever reads well in a doc and dies on a landing page.
One of the four engine-written variants took first. Human copy took second. We note the score and move on.
Scale caveat for the record: 10,014 impressions is small against consumer-grade testing, and the credible intervals for D and B only stopped overlapping in week five. Below this scale we would not publish a ranking at all.
The lesson we keep re-learning: specificity is not a style choice. It is the conversion mechanism.
Where the bandit was wrong for two weeks
From day 10 to day 24 the bandit confidently starved variant B — and it was wrong. Aggregated, B looked dead at 3.1%. Split by day-of-week, B was the best performer on weekends by a clear margin: owners browsing on a Saturday click differently than the weekday crowd researching on work time. The average buried a real effect, and the bandit optimizes exactly what you hand it, averaged over everything you did not mention.
We caught it in the weekly review because the residuals by weekday looked lumpy. That is what the 20 human minutes are for.
The fix shipped day 24: the bandit now conditions on weekday versus weekend, effectively running two smaller bandits. B owns weekends at 4.4%. D owns weekdays.
A bandit answers the question you gave it. Phrase the question badly and it will optimize the wrong world, politely and forever.
The exploration tax came to 11%
Learning is not free, so we invoice it. Over 41 days, 11% of impressions went to arms that hindsight says were inferior — call it 38 qualified clicks foregone against a perfect oracle. A fixed 50/50 A/B on the same window would have burned roughly 23% on the loser, twice the tuition for a slower answer. That gap is the practical case for Thompson sampling over classical splits at our traffic scale.
We log foregone clicks as a real cost line, because it is one. Optimization that hides its costs gets trusted less, not more.
The tax also shrinks over time. Week one explored at 40%. Week six explores at 6% and falling.
One more caveat the log keeps: these are observed rates on one small site over six weeks, not laws of copywriting. The bandit earns its keep precisely because priors like ours keep being wrong.
Tuition, not waste. But you should still know the number.
What ships next week
Queue for the next cycle, in order: the hero test graduates to a standing weekday-weekend pair with D and B as incumbents. The bandit moves to service-page CTAs, six variants seeded from the winning pattern — number plus artifact. And one new hero challenger enters, engine-written, testing whether naming the rollback count outperforms naming the price. Golden-set checks stay on throughout, because an optimizer without a regression net is just a fast way to break things.
Every incumbent stays challengeable. Nothing on the site is finished, only currently winning.
Predicted lift from the service-page round, per the engine's own prior: 15 to 25%. We will publish the actual against that prediction.
The queue itself is versioned, so next month's log can show its work against this one, line by line.
Writing the prediction down is the discipline. Anyone can narrate wins afterward.
What the humans did all month
Total human input across 41 days: about three hours. Ninety minutes writing and approving the original six variants, 30 minutes setting the metric, floors, and budget caps, and 20 minutes of review on four Mondays — one of which caught the weekend effect the aggregate was hiding. Everything else, including 194 reallocation decisions, was the engine.
Three human hours against six weeks of continuous optimization is the ratio this whole agency is built on.
The humans wrote the candidates and defined winning. The machine ran the tournament honestly. Neither side does the other's job well.
That division of labor is the product. The hero test is just this month's example of it.