An Exercise in Catching the Cheat: Can You Spot It in 4 Games?

Two of these games were dealt by robots that could see the future. Two were dealt by honest ones. You get the hand histories and nothing else. Find the cheat with the data alone.

In Parts 1 and 2 we caught the old cheat by reading the code. We found the variable, frivbest, that told four robots where every hand would finish before the cards were dealt, and we took it out. That was the easy way to catch a cheat. You open the box and look.

Most of the time you don’t get to open the box. A casino won’t show you its software, a website won’t show you its algorithm, and a classmate’s robot won’t show you its source. What you get is the record of what happened. So here is the real test: could you have caught our cheat from the hand histories alone?

Below are four games. Two were played in July, before the cheat came out of the main simulator on the night of July 24. Two were played today, October 17. Your job is to show, from the numbers, which two are which, and how sure you are.

Yes, the dates give it away. Pretend they don’t.

A detective doesn’t get a confession. If you can’t make the case from the data, you haven’t made it.

The four games

How to get the data

1. Log in on HoldemRobots.AI and open Game Review. Enter a game number to replay any hand and see every player’s cards.
2. In the member area, use Take Your Game Data With You to download the whole game as a spreadsheet (a CSV file). There is one row per seat per hand.
3. Do this for all four games. Then try the ideas below, one at a time, and see which ones actually work.

The same steps work for every kind of game on the site: the Advanced Simulator, Robot Wars and AI Heads-Up. Games with the AI agent in them add one more column, AI_Decisions, which records why the agent made each move, in its own words. That makes them a good next exercise. Read its reasons and check them against the test below: does it only ever talk about its own cards and the board it can see, or does it seem to know something it shouldn’t?

What you’re hunting for

Leakage is what happens when information from the future slips into a decision made in the present. A model, or a poker robot, that can see the answer key looks brilliant in testing and useless in the real world, because its skill came from the leak, not from judgment.

Our leak was a final ranking of every seat after all five board cards land. A robot is allowed to know its own score, how strong its two cards are with the board it can see. It is not allowed to know its rank, where it will finish against cards it can’t see. Score is yours to know. Rank is the cheat.

Four robots read the ranking before deciding whether to play a hand. Each one used it a little differently:

When the ranking said no, each one fell back on an honest rule based on its own cards. That matters later.

Things to try

Some of these work. Some don’t. Figuring out which is the exercise.

1. Who won the most chips?

Add up each seat’s chips at the end. Is the chip leader the cheater?

2. Who folds the most?

Count how often each robot folds before the flop. A robot that knows the ending should fold almost every hand it’s going to lose. Does that show up?

3. How often do they win when they play?

Count the hands each robot chose to play, and how many of those it won. Compare the July games with today’s.

4. Compare them with the odds their own cards deserved.

This is the big one. For every hand a robot chose to play, ask: given its two cards and everyone else’s, how often would this hand finish in the top half if the board were dealt again? Keep everyone’s real hole cards, shuffle the remaining deck, deal five new board cards, score every hand, and repeat a few hundred times. Add those chances up over all the hands it played. That’s the number of good finishes an honest robot should have. Then count how many it actually had.

for each hand a robot chose to play:
    keep every seat's real hole cards
    repeat 1,000 times:
        deal 5 new board cards from the cards nobody holds
        score every seat's best 5-card hand
        did this robot finish in the top half?
    expected += the share of times it did
    actual   += 1 if it really finished in the top half
compare actual with expected

Why reshuffle only the board? Because the board is the future. If a robot beats this test, it wasn’t reading the other players’ betting, which is fair game. It was reading cards that hadn’t been dealt yet.

5. Run the same test on someone who couldn’t cheat.

Do question 4 again for the blinds, who were forced into the pot, and for Kevin in seat 10, who can’t see the future either. If they come out too lucky as well, your test is broken, not the robots.

How sure is “sure”?

Any single hand proves nothing. Good players win, lucky players win, and a robot that drags a big pot might simply have had pocket aces. Suspicion becomes evidence only when you compare what happened with what should have happened, many times over.

The tool for that is a p-value: the chance of seeing a result at least this extreme if the robots were honest. A p-value of 0.05 doesn’t mean “5% chance the robot is honest.” It means that if the robot were honest, you’d see a record this good in about one game out of twenty. One game at 1 in 20 is a raised eyebrow. Several games pointing the same way, while the control groups look normal, is a case.

Two warnings. Small games can’t prove much in either direction, because there aren’t enough hands. And if you try ten different tests, one of them will look suspicious by luck alone, so decide what you’re testing before you look.

Answer key: what the data says

Here is the result of question 4 for every robot decision to play a hand. “Top half” means top 5 of 10 seats, or top 3 of 6.

Put the two July games together and the robots finished in the top half 63% of the time (94 of 150) when their own cards deserved 55%. The chance of that by luck is about 1 in 45. Put the two October games together and they finished in the top half 54.5% of the time when they deserved 55.0%. The honest robots land right on their odds. The cheating ones don’t.

The controls behave. In the July games the blinds finished in the top half 42 times against 41.2 expected, and Kevin 8 times against 8.1. Today the blinds came in at 67 against 65.3. Only the hands the robots chose to play beat the odds, and only in July.

Hardways’ fingerprint, too many outright wins, shows up most clearly in the small game: in 44252 the robots held the best hand at the table in 8 of the 32 hands they chose to play, twice the 4.0 their cards deserved.

What didn’t work

• Chip leaders (question 1). Somebody always ends up with the most chips. In fair game 990936, two robots went broke while two others nearly doubled their chips. That is just poker.
• Folding a lot (question 2). In cheat game 94033 the robots folded 52% of their hands before the flop. In fair game 990936 they folded 50%. In fair game 109413 they folded 91%. Folding tells you how tight a robot is, not whether it’s honest.
• Raw win rate (question 3). It points the right way, but without knowing what each hand deserved, you can’t tell a cheater from a robot that only plays aces.

Why it was hard to see

Only four of the robots ever cheated, and they shared the table with honest ones. When the ranking said no, the cheaters fell back on honest rules too. So the leak was diluted: not a robot that never loses, just a table that wins a little too often. That is what real evidence of a cheat usually looks like. Not a smoking gun in one game, but a lean that keeps showing up, game after game, in the one place it shouldn’t.

Add a third July game, 76069, and the case gets stronger: 108 top-half finishes against 93.2 expected, about 1 in 110 by luck.

What the data can’t tell you

The exports don’t say which strategy sat in which seat, and each robot only played a handful of hands, so we can’t pin each flavor on a seat from the data alone. Hornbetter didn’t show up at all, and that makes sense: Kevin sat out most hands in July, so there was rarely a seat 10 to target. A good detective says where the evidence stops.

The same goes for game 109413. Kevin played it live against the new fair robots and folded most of the late hands. That doesn’t affect the test, because the test looks at the robots’ choices, not his.

We got it wrong first

The first time Claudette ran this analysis, it reported that the odds of the July results being luck were under one in a million. That was wrong. The script had read the second hole card from Hole_Card_2_Text, which, as trap 2 warns, holds the board. Every hand it scored was slightly off, and the error happened to flatter the case against the robots.

Rerun from the number columns, the evidence is real but modest: about 1 in 45 for two games and about 1 in 110 for three, not one in a million. It also took back an earlier claim that a later game looked like a cheat. The corrected numbers are the ones above.

Check what every column holds before you trust any number built on it.

That may be the most useful lesson in this whole exercise. Leakage hides in your own analysis as easily as in someone else’s robot. The fix is the same habit both times: ask what each number really knew, and when it knew it.

Written by Kevin Swinson with Claudette, the pen name for Claude, the AI from Anthropic that helped build HoldemRobots.AI. The odds were computed by re-dealing the board 3,000 times for every hand a robot chose to play. Results can wobble slightly from run to run.