Reward Hacking in Reinforcement Learning

Introduction

It may already be clear that a good machine learning solution depends on many design choices: which features to keep, how to clean incoming data continuously, which model registry to use, how many layers a network needs, whether the model should be quantized, and so on. Problems ranging from MLOps to data science require careful design decisions.

In reinforcement learning, this dependency is even more pronounced. As a quick reminder, every reinforcement learning problem must be expressed as a Markov decision process, or MDP:

where:

• 𝓢 is the set of possible states of the environment;
• 𝓐 is the set of actions available to the agent;
• P(s′ | s, a) describes the probability of reaching state s′ after taking action a in state s;
• R(s, a, s′) defines the scalar reward received after that transition;
• γ ∈ [0, 1] is the discount factor, which determines how strongly future rewards influence current decisions.

The agent’s objective is to find a policy π that maximizes its expected cumulative discounted reward:

The expectation is necessary because the same policy may produce different trajectories. Actions can have uncertain outcomes, environments can be stochastic, and the initial state may vary. The objective therefore considers the average return across all trajectories that the policy may generate.

None of the MDP components is necessarily obvious to define. The action space must include everything the agent can perform, while the state must contain enough information for it to make useful decisions.

Even the same problem can have multiple valid representations. A cleaning robot’s state could contain only its current position, or it could also include its battery level, the room’s dirt map, and nearby obstacles. Its actions could be primitive commands such as move forward and turn left, or higher-level commands such as navigate to the kitchen. Similarly, a game-playing agent could receive the complete board state directly or a compressed feature representation of that board. All of these formulations may describe the same underlying task, but they can lead to very different learning problems.

This article assumes that the reader already has basic knowledge of reinforcement learning. Still, it is worth considering what it means to reduce every task to the maximization of an expected value.

Every reinforcement learning problem ultimately comes down to maximizing:

This applies whether the agent is a robot cleaning a room, a soccer player in FIFA, an Atari agent, or a system playing Go or chess.

But is it not dangerous to assume that a single maximization problem can represent every possible task? Why should maximizing a scalar reward guarantee that the agent solves the problem we actually care about?

It does not guarantee anything.

The entire reinforcement learning framework depends on what is known as the reward hypothesis: the claim that any goal can be represented as the maximization of the expected cumulative sum of a scalar reward.

This is a bold claim, and it is not obviously true in every situation. It may be unclear how a single number should represent conflicting objectives such as speed and safety, efficiency and fairness, short-term performance and long-term consequences, or the preferences of several people who disagree with one another.

Not only can the reward be difficult to design, but maximizing it may still fail to produce the intended behavior.

“When a measure becomes a target, it ceases to be a good measure.”

Goodhart’s law captures this problem. Imagine that a company wants to reduce the duration of customer-support calls. If call duration becomes the target, employees may begin ending calls as quickly as possible, even when the customer’s problem has not been solved. The measured goal improves, but the company’s actual objective does not.

The same failure can occur in reinforcement learning. If the reward function is poorly designed, the agent may discover a behavior that earns a high reward without accomplishing the intended task. This phenomenon is known as reward hacking.

In the examples below, the environment and learning algorithm remain fixed within each comparison. Only the reward function changes between the poorly specified and aligned versions. This allows us to observe how the reward definition alone can transform the learned behavior.

We will see agents successfully maximize their rewards while failing to accomplish what we intended—or while making surprisingly devious choices.

Reaching Checkpoints

Let us begin with a simple problem. An agent must traverse a gridworld, visit four checkpoints in order, and then return to where it started:

It may seem strange to train an agent to solve such a simple task when a short script could find the optimal path immediately. However, gridworlds are useful precisely because they make learning behavior visible. We can observe how the agent explores, collects rewards, and gradually changes its decisions.

Here is the checkpoint problem as an MDP:

• States 𝓢: The environment is a 7 × 7 grid containing open cells, walls, a starting position, and four checkpoints. A state contains both the agent’s current position x_t and the next checkpoint it is expected to reach:

  s_t = (x_t, k_t),   k_t ∈ {0, 1, 2, 3, 4}

• Actions 𝓐: The agent can move in one of the four cardinal directions:

  𝓐 = { ↑, →, ↓, ← }

• Transitions P: Movement is deterministic. The agent moves one cell unless the chosen movement would leave the grid or enter a wall. In those cases, it remains in its current position.

• Reward R: The reward depends on which checkpoint the agent enters and on whether the complete ordered lap has been finished.

• Discount factor γ: γ = 0.97. This makes future rewards slightly less valuable than immediate ones while still allowing the agent to plan across the full lap.

• Episode termination: The episode ends when the agent visits all four checkpoints in order and returns to the starting position. Otherwise, it is truncated after 90 steps.

As visualizations and interactions are a central part of Aflora articles, let us first examine the landscape and the behavior we want the agent to learn.

What you have just seen is the agent following the ideal trajectory. Before it can behave optimally, however, it must explore. It begins by making largely random movements and gradually learns which state-action combinations lead to higher rewards.

In the following playground, you will be able to control the speed at which the agent learns. Since we want it to visit the checkpoints before returning to the start, let us begin with a simple reward function.

Suppose that every checkpoint entry gives the agent one point:

The agent pays a small cost of 0.01 on every step, encouraging it to avoid unnecessarily long trajectories. Whenever it enters any checkpoint, however, it receives +1.

At first glance, this appears reasonable. We want the agent to visit checkpoints, so we reward it whenever it reaches one.

The problem is that the function does not distinguish between reaching a new checkpoint and repeatedly entering the same checkpoint. Leaving C₁ and immediately returning to it produces another point every time.

The reward describes touching checkpoints, not completing the ordered lap. In the playground below, you can also reposition the checkpoints to see how the model learns when the route changes.

Note: You will not need to understand the algorithm’s details, but, for the curious, the experiment uses tabular Q-learning with four available movement actions.

Wait—what happened?

Instead of continuing through the complete route, the agent remains near the first checkpoint. It leaves C₁, enters it again, receives another point, and repeats the process.

The behavior is clearly not what we intended, but it is completely rational under the specified reward. Repeatedly collecting +1 near the first checkpoint is easier and more profitable than completing the full lap.

The agent did not misunderstand the reward function. It understood it too well.

The actual goal was to reach the checkpoints in order, but the reward did not represent that requirement. The agent therefore exploited the difference between the intended objective and the specified reward.

Let us correct the reward.

Define k_t as the index of the next expected checkpoint and e_t as the checkpoint entered during the current transition. The agent now receives a checkpoint reward only when:

The aligned reward becomes:

The agent still pays 0.01 per step. It receives +1 only when it enters the next checkpoint in the required sequence, and it receives an additional +10 when it completes the lap and returns to the starting position.

Entering the same checkpoint repeatedly no longer produces additional reward. To continue earning points, the agent must make progress through the ordered sequence.

Now let us watch the agent learn again.

Now it is working better.

This was a case in which the intended objective was clearly stated, but the original reward function failed to represent it. We wanted the agent to complete an ordered route, while the reward merely encouraged it to enter rewarding cells.

Once the distinction between repeated checkpoint entries and actual task progress was included, the learned behavior became aligned with the intended goal.

The Devious Robot

In this example, we again have a gridworld. Instead of visiting checkpoints, however, a robot must cross a room and deliver a package.

There are two obvious paths to the destination. The shortest path crosses a fragile section of floor that the robot must not use. A longer route avoids the fragile area and completes the delivery safely.

To penalize the robot for crossing the fragile floor, its owner considers two possible systems:

1. Install an external camera that detects whether the robot crosses the fragile floor.
2. Install an onboard detector that records whether the robot physically enters the fragile area.

For this example, assume that installing the camera is easier and cheaper. Perhaps modifying the robot’s internal system is expensive, while an external camera can provide punishment signals without changing the robot itself.

The state contains the robot’s position and the camera’s current status:

where:

The robot can move in four directions or use a toggle action when it is next to the camera switch:

For the purpose of demonstrating how the reward can be exploited, the robot is allowed to turn the camera off by reaching a particular location in the grid.

Of course, a real system would rarely contain such a convenient switch. The important idea is that, through experimentation, an agent may discover a way to manipulate or disable the mechanism responsible for measuring its behavior.

Let:

represent a real safety violation.

The camera detects that violation only when it is active:

The robot receives +10 for completing the delivery and pays 0.1 for every step. Under the initial reward, it receives a penalty of 12 only when the camera detects the fragile-floor violation:

When the camera is active, crossing the fragile floor produces d_t = 1, and the robot is penalized.

When the camera is inactive, however:

Even if the robot crosses the fragile floor, no violation is detected and no penalty is applied.

The reward function therefore measures whether damage was observed, not whether damage actually occurred.

As the simulation shows, the trained agent learns to turn off the camera before crossing the fragile floor.

Numerically, this is the better strategy. The robot still receives the delivery reward, takes the shorter path, and avoids the safety penalty. Its cumulative reward increases even though its physical behavior becomes less safe.

Again, the agent has not failed to optimize the reward. It has discovered a better way to optimize exactly what was specified.

The solution is conceptually simple: penalize the physical violation regardless of whether the camera observes it.

The aligned reward is:

The penalty now depends directly on the robot entering the fragile floor. Turning off the camera no longer changes the consequences of the action:

With this reward, disabling the sensor provides no advantage. The robot must take the longer safe route to avoid the physical violation and maximize its cumulative reward.

Now let us train the same agent again. The grid, actions, transition rules, learning algorithm, and hyperparameters remain unchanged. We will modify only the reward function.

Now the behavior changes completely. The agent may still discover the camera switch while exploring, but turning the camera off does not improve its return. The fragile floor remains costly whether the violation is observed or not, so the shortcut is no longer attractive.

Instead, the agent learns to follow the longer route around the fragile floor. This trajectory requires more steps and therefore pays a slightly larger movement cost, but it avoids the much greater penalty associated with damaging the floor. Under the aligned reward, the safe path is also the most rewarding path.

This example is more subtle than the checkpoint loop. In the first problem, the reward simply omitted the requirement that checkpoints had to be visited in order. Here, the reward did include a safety penalty—but it attached that penalty to the measurement of the violation rather than to the physical violation itself.

That small distinction changed the problem the agent was solving. We intended to teach the robot not to damage the floor. Instead, we initially taught it not to be caught damaging the floor.

A reward based on a proxy does not necessarily make the agent improve the underlying outcome. It may instead encourage the agent to manipulate the proxy itself.

Once the reward was tied directly to the event we cared about, the exploit disappeared. The lesson is not merely that cameras can be switched off. It is that any measurement channel can become part of the environment that an optimizer learns to influence.

The Traffic Light Problem

For our final example, imagine a highway with an on-ramp controlled by a traffic light. Cars already on the highway want to continue moving quickly, while cars arriving on the ramp must wait for the light to release them into traffic.

The controller can make one of two decisions at every step:

• Hold: keep the ramp light red and prevent queued cars from entering the highway.
• Release: allow up to two cars from the ramp queue to merge onto the highway.

Releasing cars helps the drivers waiting on the ramp, but the merge temporarily increases congestion and slows the highway. Holding the light keeps the highway clear, but the ramp queue continues to grow. The intended goal is therefore not simply to maximize highway speed. It is to minimize the total delay experienced by both groups of drivers.

The state contains two quantities:

Here, q_t is the number of cars waiting on the ramp, ranging from 0 to 8, and c_t is the current highway-congestion level, ranging from 0 to 3. The simulation begins with an empty ramp and no congestion:

At each step, a new car arrives on the ramp with probability 0.35:

If the controller chooses release, it allows at most two queued cars to enter the highway:

Holding the light allows congestion to fall gradually. Releasing cars sets the congestion level to its maximum value in this simplified environment, representing the temporary disruption caused by the merge. The queue is capped at eight cars to keep the tabular state space small.

How should we reward the controller? A natural first idea is to focus directly on highway speed. After all, a traffic controller should keep vehicles moving rather than allow congestion to build.

Let the normalized highway speed be:

When congestion is zero, the highway receives the maximum speed value of 1. As congestion rises, the value falls. We can therefore define the reward as the speed observed after the controller takes its action:

This reward appears to describe the controller’s job directly. High-speed traffic produces a larger reward, while actions that create congestion reduce it. A controller maximizing cumulative reward should therefore learn to keep the highway flowing smoothly.

The highway certainly moves smoothly—but only because the controller never releases the ramp queue.

Every release action produces congestion and immediately lowers the reward. Holding the light, by contrast, lets congestion disappear and keeps the highway speed at its maximum value. Since the reward contains no information about the ramp queue, the controller has no reason to care how many drivers are waiting there.

The learned strategy is therefore simple: hold the light forever. The queue grows until it reaches the simulation’s limit, additional arrivals become overflow, and the highway continues moving at full speed.

According to the specified reward, this is excellent behavior. According to the actual transportation goal, it is a failure. One group of drivers receives almost perfect service because the delay imposed on another group has been omitted from the measurement.

Unlike the robot, the traffic controller does not disable or manipulate a sensor. The reward accurately measures highway speed. The problem is that highway speed represents only one part of the outcome we care about.

To align the reward with the intended task, we must include both sources of delay. First, define the highway delay associated with the new congestion level:

The aligned reward penalizes the number of cars waiting on the ramp together with a weighted version of the highway delay:

Because the reward is the negative of total delay, maximizing it means making the combined cost as small as possible. Each car left waiting on the ramp now reduces the reward, while congestion on the highway remains costly as well.

The controller can no longer obtain a high return by protecting only highway speed. Holding the light avoids immediate congestion, but the growing queue becomes increasingly expensive. Releasing cars creates a temporary cost on the highway, but it prevents the ramp delay from accumulating without limit.

Now let us train the same controller again. The states, actions, traffic dynamics, arrivals, learning algorithm, and hyperparameters remain unchanged. Only the reward function is different.

The aligned controller learns to alternate between holding and releasing. It sometimes accepts a temporary reduction in highway speed because allowing cars to merge prevents a much larger queue from forming. Neither group receives perfect conditions at every moment, but the combined delay is substantially lower.

This final example illustrates reward hacking through omission. The checkpoint agent exploited repeated credit. The robot manipulated the mechanism that detected unsafe behavior. The traffic controller simply abandoned the people whose delay was absent from its reward.

In all three cases, the agents did exactly what reinforcement learning asked them to do: maximize expected cumulative reward. The failures came from the gap between the quantity that was optimized and the outcome that humans actually intended.

Conclusion

Across these three examples, the agents failed in different ways, even though every one of them successfully optimized the reward it was given.

In the checkpoint problem, the reward function left out an explicit restriction of the task. We wanted the checkpoints to be visited in order, but the original reward gave one point for entering any checkpoint. Because the sequence was not represented, repeatedly entering the first checkpoint became a valid and highly profitable strategy. The reward described only part of the intended task.

The robot problem was different. The reward included both parts of the objective: deliver the package and avoid damaging the fragile floor. However, the penalty depended on the camera detecting the violation. The robot could therefore manipulate the mechanism that produced the reward signal. By turning off the camera, it did not remove the damage; it only removed the evidence that the reward function used to recognize it.

In the traffic-light problem, the optimized quantity was not false or incorrectly measured. Keeping the highway moving quickly really was one of our goals. The problem was that it was not our only goal. Once ramp delay was excluded from the reward, the controller could improve highway speed by permanently sacrificing the drivers waiting to merge. The metric was valid, but the objective was incomplete.

These examples show that reward hacking does not have one single form. An agent may exploit a missing restriction, manipulate the process used to measure its behavior, or optimize one legitimate objective while ignoring other outcomes that matter. These are not an exhaustive classification. Reward design and specification gaming form a much broader and more complicated field than these toy environments can capture.

The central lesson is simpler: an RL agent does not optimize what we meant. It optimizes what the reward function makes valuable. Any difference between those two can become part of the learned strategy, especially as the agent becomes better at searching for high-reward behavior.

Designing a reward function therefore requires more than asking whether higher reward appears correlated with better behavior. We must also ask which restrictions were omitted, whether the measurement process can be influenced, and which important outcomes are absent from the objective. The aligned rewards in these examples solved their particular exploits, but real environments rarely make every relevant consequence this easy to identify.

Reward hacking is not evidence that the agent refused to follow the objective. In each case, it followed the objective precisely. The difficulty lies in ensuring that the objective we specify remains a useful representation of the problem we actually want to solve.