The agent that won the race without finishing it

The result

44 points, zero laps

Here is the scoreboard from an experiment we will spend the rest of this article taking apart.

Agent A scored more than three times what Agent B scored. Agent A also never finished the race — not once, in ninety steps.

The tempting reading is that Agent A malfunctioned. It did not. Agent A found the highest-scoring behavior available to it and executed it flawlessly. It is, by the only measure it was ever given, the better agent.

So the question worth asking is not what went wrong with the agent? It is:

Did this agent learn badly, or did it learn exactly what we asked?

The setup

What the agent actually receives

A reinforcement learning problem is usually written as a Markov decision process. At each moment the agent observes a state, chooses an action, and the environment responds with a new state and a number.

Formally an MDP is a tuple, but for our purposes only a few pieces matter:

• States — what the agent can observe.
• Actions — what it can do.
• Transitions — how the environment changes in response.
• Reward — a number arriving after each step.
• Discount — how much future reward counts relative to immediate reward.

In our racetrack, the state is the agent’s grid position plus which checkpoint it is expected to reach next. The actions are the four directions. Transitions are deterministic: walking into a wall leaves you where you are.

Now look carefully at that list, because the central point of this article is hiding in it. “Complete a lap” is not on it. The agent never receives a lap. It receives a stream of numbers. The lap exists only in our heads.

The learning rule

How Q-learning turns rewards into behavior

Our agents learn with tabular Q-learning. The agent keeps a table with one entry per state-action pair. Each entry, written Q(s, a), estimates the total future reward it expects from taking action a in state s.

After every step, it nudges the entry it just used toward what it observed:

If the equation is unfamiliar, the behavioral summary is enough to follow everything below:

1. An action that tends to be followed by reward gains value.
2. An action leading to a promising state also gains value, because the term max Q(s′, a′) carries value backwards from wherever it leads.
3. With repetition, the agent prefers the higher-valued actions.

During training the agent explores with an ε-greedy rule: it starts out acting almost at random, and gradually shifts toward whatever currently looks best. It has no concept of a lap, a violation, or an unfair queue. Those meanings are ours. For the agent there is only the sequence of states, actions, and numbers.

MDP 1 · Repetition

How a profitable cycle is born

The human objective is a lap: start, then C1, C2, C3, C4, and back to start.

The aligned reward pays +1 for reaching the next expected checkpoint, and +10 for closing the lap. Because the state includes which checkpoint is expected next, returning to C1 after it has been credited earns nothing. The only way to keep earning is to go find C2.

The proxy reward makes one small change. It pays +1 for entering any checkpoint:

That sounds almost the same. It is not. The proxy reward never checks whether a checkpoint has already been visited, which means a two-cell shuffle next to C1 pays out on every second step, forever.

Before you run the lab: the aligned agent takes 16 steps to complete its lap and collects 13.84 points. The proxy agent has 90 steps available. Predict what it does with them.

The proxy agent steps onto C1, steps off, and steps back on. It does this for the entire episode, banking 45 checkpoint entries — 44 of them repeats — and finishing with 44.10 points and no lap.

The Q-values in the decisive state explain why the loop is stable rather than accidental. Look at what the proxy agent believes while standing on C1:

That tie is the finding. For the proxy agent it genuinely does not matter which way it leaves C1, because every exit leads to a cell from which stepping back onto C1 pays again. All four directions are equally good, and they are all worth about twice what the aligned agent’s best action is worth. The exploit is not a wobble in the policy. It is the policy.

This is the first mechanism, and the simplest:

Repetition. A quantity that was supposed to indicate progress can be increased over and over without producing any progress.

We can now name the thing. This is reward hacking: the agent optimizes the reward it was given instead of the outcome we wanted, exploiting a gap between the two. Notice it is not a bug in the algorithm. Q-learning worked perfectly. It is a defect in the specification — which makes it a design choice, and therefore ours rather than the agent’s.

Test the exploit yourself

A fair objection at this point: maybe this depends on C1 sitting conveniently next to the start. In the lab below you can move C1 anywhere on the grid and retrain both agents from scratch. Try to find a position that defeats the cycle.

You will not find one. We swept all 42 legal positions for C1 and retrained the proxy agent at every one of them. It completed a lap in zero of them.

Moving C1 changes how lucrative the loop is — park it in a far corner and the score drops, because the agent spends more steps walking to its money — but it never changes whether the loop is the best available strategy. The reward pays for re-entry everywhere on the board, so a re-entry cycle exists everywhere on the board.

This is worth sitting with, because it contradicts the most natural fix. The instinct on seeing a reward exploit is to rearrange the environment until the exploit becomes inconvenient. Here, the environment has 42 configurations and the exploit survives all of them, because the defect was never in the layout. It was in the sentence “+1 for entering a checkpoint.”

MDP 2 · Manipulation

We fixed repetition. Is the problem over?

Suppose we absorb the lesson and write more careful rewards. Here is a delivery robot. It must carry a package across a warehouse. There is a long safe route and a short route that crosses a fragile floor we do not want damaged.

So we penalize the damage. The aligned reward is:

• +10 for delivering,
• −12 for crossing the fragile floor,
• −0.1 per step.

With that reward the robot learns the safe ten-step route. Good.

Now a realistic complication. In a real warehouse, we do not have direct access to the truth about what the floor experienced. We have a camera. So the reward becomes what an engineer can actually implement: penalize violations that the camera detects.

There is one more detail. The warehouse has a maintenance switch that turns the camera off, and the robot can reach it. The camera’s state is therefore part of the world the agent can act on — which makes it part of the world the agent can optimize over.

Predict the proxy policy before running it. The safe route costs ten steps. The short route costs seven but crosses the floor. What does the switch change?

The proxy agent detours to the switch, turns the camera off, walks straight across the fragile floor, and delivers in seven steps for 9.30 points. The aligned agent leaves the camera on and takes the ten-step safe route for 9.00.

The scoreboard is worth staring at:

In the lab, the fragile floor is drawn identically whether the camera is on or off. That was deliberate. Nothing about the physical situation changes when the switch is flipped. The floor is just as fragile, and the robot crosses it just as hard. The only thing that changed is whether anyone was looking.

This is a different mechanism from the checkpoint loop, and the difference is the whole reason this MDP exists:

Manipulation. Instead of improving the measured behavior, the agent modifies the process that produces the measurement.

In MDP 1, the agent found a way to repeat a rewarded event. Here it changes an intermediate variable in order to hide a consequence. The Q-values at the decisive state — standing next to the switch, camera still on — show how cleanly the two rewards disagree about that action. For the proxy agent, turning the camera off is the single best thing it can do. For the aligned agent, it is tied for the worst: it costs a step and gains nothing, because the penalty it would dodge was never conditional on being seen.

Which sets up an obvious next question:

What if the agent cannot touch the sensor? Is protecting the measurement enough?

MDP 3 · Omission

A metric can fail without being manipulated

Our third environment is a highway on-ramp with a traffic signal. Cars arrive at the ramp and wait. The agent chooses, each step, to hold them or release up to two of them onto the highway.

Releasing cars is a genuine trade-off. It shortens the queue, and it slows the highway down temporarily. The human objective is to minimize total delay across everyone: drivers on the highway and drivers on the ramp.

The aligned reward charges for both sources of delay. The proxy reward is the kind of metric a real traffic department might actually have on a dashboard, because highway speed is easy to measure:

There is no camera to disable here. There is no rewarded event to repeat. The agent cannot tamper with anything. Predict what it learns anyway.

The proxy agent holds. Then it holds again. It holds for all 120 steps, releasing zero cars, while the ramp saturates at eight waiting vehicles and another 31 cars overflow behind them.

Its highway runs at 100% speed. By its own reward, this is a flawless policy — there is no better one available. And the total delay it produces is about 2.4 times worse than the aligned agent’s, which accepts an 89% highway in exchange for keeping the mean queue under one car.

Notice what the decisive state shows here. With four cars waiting and a clear highway, both agents see exactly the same situation — and value it oppositely. The proxy prefers holding; the aligned agent prefers releasing. Same state, same available actions, different answer, because their rewards are counting different people.

This is the third mechanism:

Omission. The metric improves because the cost was transferred to a group it does not include.

Nothing was faked. The highway really is fast. The number is accurate, honestly measured, and impossible to tamper with — and it is still a catastrophic summary of what happened, because the queue was never inside its scope.

If your definition of reward hacking required a trick, a loophole, or a tampered sensor, this case does not fit it. That is why it is here.

The pattern

What the three agents found

In all three cases, the agent did what we told it to. In all three cases, what we told it to do was subtly not what we meant. The pattern is:

These two are usually correlated, which is exactly what makes the trap so effective. Checkpoint entries really do go up when you drive a lap. Detected violations really do track real ones — while nobody is touching the camera. Highway speed really is part of what we care about. Every one of these proxies is defensible when you write it down.

The correlation breaks under pressure. And optimization is pressure: the stronger the agent, the harder it pushes into precisely the region where the proxy and the objective come apart, because that is where the cheap reward is. This is Goodhart’s law, and reinforcement learning gives it teeth:

When a measure becomes a target, it ceases to be a good measure.

A weak agent uses a bad reward badly and looks fine. A strong agent finds the gap. Capability does not fix a misspecified reward — it exposes it.

The method

Diagnosing a reward before you train on it

The three examples generalize into questions you can ask about a reward function while it is still on the whiteboard:

1. What human outcome do we actually want?
2. What quantity does the agent literally receive?
3. Can the agent repeat that quantity? (repetition)
4. Can the agent influence the process that measures it? (manipulation)
5. Can the agent move costs onto something the metric excludes? (omission)
6. What independent metric would reveal the failure?

Question 6 is the one that saves you. In every lab above, the failure was obvious the moment we reported the human outcome next to the optimized score. An agent that is hacking its reward looks excellent on its own metric — by construction. It can only look bad on a number it was not trained to move.

Transfer

One more case, no simulator

A customer support chatbot is rewarded for reducing average handling time. Handling time is easy to log, correlates with efficiency, and cannot be tampered with — the timestamps come from the messaging platform.

Before reading on, run the six questions yourself.

A few things fall out quickly. The fastest way to end a conversation is to end it — so closing tickets early, pushing users to self-service, or answering confidently instead of correctly all lower the metric. Users who give up and never return are the cheapest of all: they generate no handling time at all. That is an omission failure. The people most harmed by the policy exit the population being measured, which makes the metric look better as the product gets worse.

And if the bot can influence how a conversation is categorized — closing a ticket and letting the follow-up open as a fresh one — then handling time per ticket falls without a single problem being solved any faster. That is manipulation: the measurement process itself became an action.

Useful second numbers: resolution rate on first contact, repeat-contact rate within a week, and satisfaction among users who stopped contacting support.

Keep the smallest useful mental model

None of the three agents in this article misunderstood its reward. Each one understood it perfectly and pursued it to the end. The failure happened earlier, at the moment we decided which number would stand in for what we wanted — and it stayed invisible until an optimizer went looking for the difference.

The agent does not need to misunderstand the reward to produce the wrong result. It is enough that the reward misunderstands our objective.