Practical tips to avoid Agentic Cheating

For the last ~8 months I’ve been building an “Engine” to replicate (bit-for-bit) the calculation logic of Microsoft Excel. Since this problem has a ground truth (what Excel computes as outputs for a given set of inputs), the problem lends itself to automation - agents can generate workbooks, diff the Engine behavior against Excel, and then “Fix Agents” can propose changes to the Engine code to bring it closer to alignment with Excel (learn more about how I’m building that here). When I originally allowed Fix Agents to submit their code changes without my review, they cheated with enough frequency that I had to throw out broad swathes of the implementation, and at one point, had to start the core engine from scratch. After that, I review every PR, and whenever I find a cheating Fix Agent, I read through its transcript to figure out what happened, and then tweak my scaffolding and instructions to try to make it less likely.

By “cheating”, I mean everything from hard-coding if (input = valueUnderTest) { return ExpectedResult } (example below), to building API calls to run Excel within the Engine code, to much more subtle forms of cheating where the agent constructs rube-goldberg machines of massive complexity that solve exactly the set of cases they were presented with, but fail to generalize - essentially overfitting to the scenarios they are given.

Reading the transcripts from agents that cheat is fascinating and fun - it helped me understand both why LLMs might cheat, and helped me imagine the ways I ended up scaffolding against those weaknesses. Over time, I’ve been able to ferret out most of the cheating behaviors with various forms of prompt engineering and process, which has been a pleasant surprise. Here are some generic tips.

1 - Explicitly allow failure

Agents tend to cheat only after they have made many legitimate attempts at a problem, and appear (in their transcripts) to express frustration about the paucity of their tools or the impossibility of the task they’ve been given. I’ve found that if you give the agent, in its initial task instructions, the option to declare that the problem cannot be solved within reason, the Agent will often exercise that option rather than trying to cheat.

2 - Hold out all evaluation criteria

When my systems find a difference between Excel’s calculation logic and my code, they automatically dispatch a Fix Agent to try to diagnose and fix it. Originally, I sent the full failing test case to the Fix Agent. This allowed the agent to overfit or cheat to meet the expected behavior of the failing case. To get around this, I built scaffolding which generates several thousand alternative cases of input/output values (through Excel), and gives the agent only about half of these in the prompt to fix. This serves two functions - it allows the agent to see many more cases for how a calculation with a given structure ought to behave, and it also detects cheating: when the agent says it is “done”, the proposed diff gets routed to my evaluation infrastructure which tests the proposed solution against the full set of cases. When an agent passes all of their cases and fails a significant number of the held out cases, it likely cheated in some way, and that gets flagged to me for review.

3 - Cross model agentic review

Much has been written about how effective cross-model review is (including by me), so I’ll not spill more ink. However, for cheating specifically, there are two useful lessons. First, giving the Review Agent specific instructions on what the overarching goal of the project is and the specific context of what a “cheating” might look like allows them to spot cheating with much greater accuracy. Second, warning the Fix Agent that they’re going to be reviewed by another agent appears to induce compliance - if they know their work will be scrutinized, they seem less inclined to find shortcuts.

4 - Intelligence Escalation Ladder

When one of the above steps fails, it can be an indication of cheating, but also could just be an incomplete or partial implementation, or an indication that the problem is too hard for that agent. In these cases I employ an “Intelligence Escalation Ladder”. When a Fix Agent fails to achieve a task (due to cheating, failing evaluations, a bad review, or it gives up), I have the Fix Agent write up a doc describing what it tried and where it left the work, and then I pass the task up to a more capable/intelligent/effortful model, including information about the Fix Agent’s path (but critically, not any of the cases/info about the anti-cheat, if that is what triggered the escalation). This approach works pretty well because the highest tier models are wildly capacious, and the headstart from a previous fixing session saves precious tokens that make using top-tier intelligence more cost efficient.

All of these are bandaids. I am profoundly worried that if we are only able to solve alignment through bandaids, these will fail in important cases. I hope the AI labs throw all the resources they can at the fundamental challenge of alignment. But in the interim, we’re not rudderless - AI consistently falls into patterns of cheating that can be minimized by good scaffolding. It’s possible to make immense and reliable automations, despite the unreliability of the individual parts.