Reward went up and I felt worse

At 3:14 a.m., I sent a Slack message to a channel in which I was the only person awake.

cassie run 18402: reward 847.3 (new best)

The number was the highest I had seen. I watched the latest simulation, expecting the gait to look correspondingly better. It did not. Cassie moved forward quickly, but the steps were tense and wasteful. One small change in terrain would probably have sent the robot sideways.

I closed the laptop, put on my shoes, and walked around the block.

A reward is a specification

I was training a biped robot in MuJoCo with Ray RLlib. The loop sounds simple: define a reward function, run many simulated episodes, and update the policy toward actions that earn more reward.

The difficult part is deciding what “better” means.

For locomotion, I might reward forward speed and staying upright, then penalize energy use, rough joint motion, or feet sliding across the ground. Each term has a weight. I chose those weights over coffee three weeks earlier, based on the behavior I wanted and the failures I had already seen.

The policy does not know that I want a convincing walk. It only sees the score. If it can move forward with a stiff, ugly gait that exploits a gap in my reward function, it has solved the problem I gave it.

That is not the agent cheating. It is the specification being incomplete.

Why the number bothered me

It was warm for May. Paris was quiet enough that I could hear my shoes and the occasional scooter. A bakery already had its lights on; someone inside was loading trays for the morning.

Meanwhile, the robot kept running episodes on the cluster at far beyond real-time speed. The contrast made the reward problem feel less like an RL detail and more like a general habit.

We manage many systems through proxies. Engagement stands in for interest. Resolution time stands in for good support. Lines changed can stand in for engineering progress. A benchmark score stands in for model quality.

Proxies are useful because the real goal is often difficult or expensive to measure. Trouble starts when the proxy becomes the goal. A support team can close tickets quickly while leaving customers frustrated. A model can pass a benchmark while failing on ordinary inputs. A robot can earn 847.3 while walking badly.

AI makes this problem easier to scale. Once a system can optimize a measurable target cheaply and repeatedly, a small mistake in the target is reproduced with impressive efficiency.

Misalignment without science fiction

Discussions of AI alignment often jump to conscious machines or catastrophic scenarios. The version I meet in daily work is less dramatic.

It is a model that does exactly what the evaluation rewards and still gives an answer nobody wanted. It is an automated process that improves a dashboard while making the service worse. It is a team trusting a clean metric after the underlying behavior has drifted.

The people building these systems are usually not trying to cause harm. They are trying to ship, compare experiments, and make the number move. I know because I am one of them.

The safeguard is not to stop measuring. It is to keep looking at the thing the metric represents. Watch the gait. Read the customer exchange. Test the model outside the benchmark. Ask who absorbs the failure when the proxy is wrong.

That work is slower than reading a graph, which is why it is easy to skip.

The next day

I slept until noon, answered email, and changed the energy penalty. The reward dropped. After more training, it rose again, this time with a smoother gait.

I recorded a demo because the result was genuinely better. Then I called my dad and asked how he had learned to repair cars at my age. He talked for forty minutes; I took notes.

There was no revelation. I did not leave AI or decide metrics were bad. I made a better reward function and kept working.

The useful lesson was smaller: when a number improves and the result feels worse, inspect the result. That discomfort may be telling you something the objective cannot.