An AI agent’s actions can be “harmful” and/or “incriminating”. “harmful” refers to how bad an agent’s action is.¹ “incriminating” refers to how bad an action looks to a given monitor (e.g., a human, LM, or probe). Let’s visualize these properties on a standard x-axis/y-axis plot, where we place “harmful” on the x-axis because it’s a more fundamental property of the action. “incriminating” goes on the y-axis because it is a function of both the action and the calibration of the monitor observing it.² Just to put some numbers on these axes, let’s say that harmfulness and incrimination can each range from 0 to 100.³

For a perfect monitor, all actions live on a straight line with a slope of one. That perfect monitor would correctly measure the harmfulness of every action.

If an imperfect monitor judges an action to be in the upper left corner, it has mistakenly judged that action to be scarier than it is. If it judges an action to be in the bottom right corner, it has mistakenly determined that the action is scarier than it looks. Let’s illustrate this by plotting the possible judgements made by a skittish monitor that raises the alarm even for silly things and contrasting them with the possible judgements made by a panglossian monitor that doesn’t raise the alarm, even for genuinely scary actions.

The lower left region of the plot corresponds to diffuse/low-stakes control. The upper right region of the plot corresponds to concentrated/high-stakes control.⁴

Diffuse control scenarios are effectively death by a thousand actions - individually not very harmful or incriminating, but collectively dangerous. Diffuse control captures scenarios like an AI alignment research agent subtly sandbagging on 0.1% of its work. On the spectrum of all possible agent misbehavior, this is not the worst form. Maybe handwaves furiously 10/100? And it’s low-incrimination by default - was the agent sandbagging, or did it just have an inconvenient skill issue? Since the harmfulness of the action incorporates the likelihood that any other actions needed to complete a successful attack will occur, even actions we might naively label with, say, a 5 should be treated as somewhat worse (say, a 10).

Concentrated control scenarios, on the other hand, involve a small number of extremely harmful, highly incriminating actions that would cause a sudden, catastrophic loss of control. Here, envision an agent scping model weights to a random server on the open internet. Any high-stakes monitor worth running should be able to call a spade a spade and identify very harmful actions as very incriminating.

I think it would suck to lose to actions that are harmful and clearly look that way. Under no circumstances should we let kill-all-those-pesky-humans.sh actually get executed. I think that top-right region of the graph deserves a lot of attention (and the high-stakes control team at Redwood does great work on it). It’s clearly most dangerous to be miscalibrated on hilariously harmful actions. However, supposing we do our jobs on the don’t-lose-to-hilariously-harmful-actions front, and supposing our AI agent is a competent schemer, we should generally assume that the most dangerous threats currently lie in the bottom right corner of the plot. In general, I expect a competent schemer to take actions that aren’t incriminating to our current monitors but are still harmful.

One category of action that could plausibly have this high-harmfulness, low-incrimination shape is poisoning the training data mix for a monitor or successor system. In an ideal world, we would be able to understand the consequences of training on a particular data mix, including any weird or undesirable behavioral quirks that the data mix would produce. But we are, um, extremely far from that ideal world.

Here’s an easy example of this action: making a successor model extremely vegan. I have no quibbles with people being extremely vegan (seems pretty ethical and all that). But making a successor model extremely vegan could push it toward pursuing a policy like “delete all humans to end all factory farming”.⁵

It might be hard to notice if the successor model is being made vegan, particularly if the current model is good at being subtle. The preference data could be laundered into trillions of other tokens, making it hard to catch with dataset filtration techniques. The preference could even be communicated in a spookier (subliminal??) way that makes it almost impossible for even a sophisticated human reader to catch. Evaluation awareness is increasingly going to confound pre-deployment alignment testing, so we might not catch this strange preference behaviorally before it matters. It seems important, therefore, to design monitoring systems that are better at surfacing evidence of these preferences. In other words, we should seek to lift the red dot corresponding to covert malicious fine-tuning (CMFT) to the green line.

What does it take to move dots up to the green line — that is, to make our monitors see harmful actions as incriminating? I claim that, for language-model monitors (and also humans?), recognizing that an action is harmful requires (a) understanding the expected effects of the action and (b) making value judgements about those effects. Unfortunately, (a) seems pretty hard for complicated actions and weaker trusted monitors, even if some models today are reasonably competent at (b).⁶

Suppose, however, that we have succeeded at calibrating our ensemble of monitoring systems, such that to our systems, harmful = incriminating. This would obviate the need for the current y-axis. So let’s replace it with a new y-axis, describing how much money it would take to prevent each action. Note that the top tick roughly corresponds to global GDP in 2025.

In a perfect world, it would cost us zero dollars to perfectly block all harmful actions. In the real world, each action has a cost, and we should aim to bring that cost down as much as possible, preferably by several OOMs.

If we can’t stop a harmful action at an acceptable cost, we should stop development until we can figure out how.

So here’s my rough frame for AI control. All else being equal:

1. We should order harmful actions by expected harmfulness.
2. We should address the most harmful actions first, because those are the ones that would really hurt us if we didn’t catch them.⁷
3. For each extremely harmful action, we should build monitors that render it as incriminating as it is harmful (which naturally implies that we should prioritize harmful actions that do not already look commensurately bad to our current monitors).
4. Then, we should optimize those monitors to reduce the cost of stopping that harmful action to a tolerable amount.
5. If that is not possible, we should stop developing frontier AI until we make it possible.
6. We should pick the next highly harmful action, and get back to work.

Thank you to Francis Rhys Ward, Skylar Shibayama, Katie Taylor, Justin Miller, and Ayush Panda.

One key bottleneck to adoption of AI tools in the law is the longstanding and important role of the trial in the American legal system. Here’s my inelegant rephrasing of my partner’s arguments to this effect: AI systems are not yet legal persons and are therefore presumptively ineligible to be members of the bar and argue in court. And even if that were not true, as long as human judges want themselves and human juries to hear arguments from human litigators, those human litigators are going to be gainfully employed. I think this is a pretty good argument for at least the medium term (5-8 years?), because the American legal system is pretty small-c conservative and the idea of fundamentally reformatting our public justice system on that timescale feels undesirable/politically infeasible, even for someone who thinks things are pretty crazy right now and are likely to get crazier.¹ Notice that I just said “public justice system”.² I wonder whether the gap in AI capabilities will make the next few years a great time to establish an AI-driven arbitration practice.

The cost of the judicial system is borne by the taxpayer. The cost of arbitration is borne by one or more of the parties involved in the arbitration. If all parties feel like they will get a better deal out of choosing arbitration, at lower time and dollar cost, they might have a reason to opt in to arbitration and opt out of classic litigation. Arbitration is currently expensive because the time of the w.l.o.g. skilled former judges/experienced attorneys who do arbitration is expensive.³ This is an overdetermined observation, but I think that AI could be a significant productivity enhancer for today’s human arbitrators. Suppose there existed a system that could do the following things:

• Securely collect relevant documents from each party to the dispute. (MVP: secure file upload portal. Improvement: that plus an agentic interviewing system that identifies documents or evidence that might be relevant but isn’t yet uploaded.)
• Filter, categorize, and score the documents.
• Analyze the documents from all parties to produce rich, grounded summaries of each party’s position, the relevant supporting evidence, and the likely range of fair outcomes.
• Be a live interface to all documents; enable better and richer search across all key evidence.

I claim that such a system should make a human arbitrator at least twice as efficient.

More speculatively, AI could act as the actual arbiter. It could be the thing described above, but also return the “fairest” resolution of the dispute to both parties given available evidence. This skill seems quite difficult to develop (and difficult to get buy-in for from consumers of legal services) but could be valuable at the right cost. You’d need to solve a bunch of fun technical problems (e.g., data filtering and monitoring to make sure people aren’t trying to jailbreak the judges). You’d also probably need some human review at the beginning. But assuming reasonably good solutions to those technical problems, this system could get negotiations moving more quickly.

Another model could look like having two people’s agents enter the secure environment and duke it out in front of a panel of judges and classifiers that eventually coalesce around a suggestion.⁴ If we can build something like this, we could practice solving these kinds of trust and coordination and negotiation problems in a way that might be useful for hard coordination around AI safety in the future. I might write more about this soon.

Me being me, the Sinners/Lighthaven experience got me thinking about how today’s America would treat vampires if we knew they existed. It’s pretty clear that if our options were “drive a wooden stake through this thing’s chest” or “get eaten” we would probably choose the former. And, um, if it’s literally the Devil teleoperating vampires, that’s probably not acceptable under almost any condition. But in the case where vampires (a) weren’t egregiously evil but (b) required human blood, (c) didn’t care much about making more vampires, and (d) had some wild and potentially helpful superpowers (including sharing all of each other’s memories and making long-term commitments/plans (iykyk)) I think some weird stuff could happen.

(a) is pretty important. If you have these sorta-human sorta-demon immortal-by-default ghouls running around and they absolutely want to murder you all the time, you probably don’t want to make any kind of arrangement with them (listen to the warnings about deals with the Devil, etc. etc. etc.). I guess there are a few ways you could still try to deal with them, but they all have a suicide squad, black site, Geneva Conventions-violating flavor to them. It’s probably just safer to destroy existing vampires and prevent any attempts at making new ones.

(b) might just be a skill issue? I mean, you can just build things. If you needed whatever vampires were offering, you could probably figure out how to make synthetic blood (which we’re already kinda doing?). Then you could use your control of the blood factory to trade blood for vampires doing stuff for you. Regular red-white-and-blue W-2 wages but paid out in aliquots of synthetic blood? Ew, but maybe cool. The main challenge here is that if the process for making blood could be performed with vampiric labor as easily as it could be with human labor (or completely automated labor), there’s no reason why the vampires wouldn’t just seize the means of production and activate a species-wide infinite money glitch. We would have to very carefully monitor vampiric labor and activity in this case to make sure they don’t go for the blood factories and establish a completely human-independent basis of survival. But if they’re mostly nice (just unfortunately a little bloodthirsty every other day in the most technical sense of the term) this feels kind of survivable.

(c) is crucial for the stability of any potential deal. Are the vampires inherently interested in making more vampires? Or would they be happy being themselves, with all the blood they could ever need provided by a grateful but terrified human population? If they are driven not just to eat, but to procreate, we would need to prevent runaway vampire-flation, where the number of vampires quickly outnumbers and overwhelms regular humans. Do you provide a fixed blood budget to all vampires, and let them decide how to spend it? Would you establish a quota system of some kind? If they just want to be satisfied (and don’t need progeny to achieve biological perpetuity) we might be able to give them an extremely generous but not humanity-imperiling quantity of synthetic blood.

(d) is mostly about how valuable it actually is to negotiate with the vampires. Can they do superhuman stuff we’re willing to empower with industrial blood factories? Crack fusion? Solve the alignment problem? Cure all the diseases with their bespoke knowledge of eldritch horror magic or whatever? For some minimum value of “transformative impact” based on their improved coordination, shared memories, super speed, immortality, whatever, it might be worth it to guarantee them some perpetual blood in exchange. If the potential impact is high enough, even if the blood had to be organic human blood, I can imagine some governments ~encouraging~ their population to tithe blood to vampires in exchange for their help. That’s a terrible outcome, but the incentive structure for governments might push them into trying.

This somehow ended up being about the hazards of making deals with somewhat misaligned near-human AIs? MATS got to me.

The importance of reasoning through precisely which affordances to grant the red and blue teams in our control settings has risen sharply in our work over the last week.

I initially tried to write down affordances according to which side they favored, but realized that that would end up with a lot of duplicated content (e.g. “Red team affordance: hidden scratchpad” vs. “Blue team affordance: visible scratchpad”). So now I’m just going to YOLO some affordance categories (?) and see how it goes.

Capability:

• General capability gap between red and blue (e.g. gpt-2 monitoring claude-opus-4-6 vs. claude-opus-4-6 monitoring itself)
• Tools given to red (can it run code? can it search stuff? write to a system? make api calls? spawn other agents? do lots of stuff at a time without intervention?)
• Tools given to blue (can blue resample/edit/completely replace a suspicious action? can it stop a suspicious action? can it interrogate red about an action?)

Information:

• Blue knowing exactly what it’s looking for (did we tell blue about the specific side task or not?)
• Red knowing exactly what defensive protocol is being deployed against it (is red never surprised by a blue technique?)
• Hidden scratchpad (can blue see the scratchpad tool use or not?)
• Hidden internal chain of thought (can blue see literally all output tokens from red or not?)

Statefulness:

• Stateful blue (it has long-context memory, RAG capabilities, etc.)
• Stateless blue (red can iterate against blue without harming its ability to try again)
• Stateful red (it also has long-context memory that is un-auditable by blue)
• Stateless red (red is replaced/mind wiped/resampled frequently to break long-term coherence)

Budget:

• Blue’s budget (can it always invoke an expensive but effective system e.g. expert human review or is it limited to only cheap techniques e.g. probes and offline LM monitors)
• Red’s budget (is it token constrained (either input or output)? API usage constrained?)

Collusion:

• How trustworthy blue’s agents are (do they secretly collude with red?)
• How many red agents there are (can two red agents collude with each other either in parallel with shared state or sequentially with inherited state?)

Reversibility:

• Red’s ability to do irreversible shit that can’t be rolled back or stopped even if caught offline post hoc (e.g. exfiltrate weights)
• Blue’s ability to grant conditional permission to a red rollout but sandbox or silo it to isolate side effects (can blue let red try the thing but in a locked-down environment?)

Performance: <- maybe the most important one for frontier labs?

• Blue’s willingness to leave red performance on the table to increase safety (where on the safety-usefulness curve are we?)
• Red’s willingness to leave red performance on the table to avoid showing its full capabilities/true colors in an eval (SANDBAGGING!)

As I keep working, I might add to this list/refine it more. But I think this is a reasonable place to start.

Thanks to Ayush Panda and Atharva Nihalani for the conversation that inspired this piece. Thanks also to the Redwood Research team in general. I’m sure their ideas on control permeate this piece, and would like to gesture broadly at their work for more thought-out work on this theme.

Before I get into why I like these curves, let’s turn to how we make them. The important thing to understand about a ROC curve is that it represents the outcome of a somewhat tedious and laborious process that is not well captured by the graph itself.

Here’s the process:

• Identify a binary classification problem. (Cat or dog? Black or white? Benign or malicious?)

• Build a classifier for that problem. We assume that all the data our classifier sees falls into one of those two buckets.

Example: In my work, the labels are benign (labeled 0) and malicious (labeled 1). Our classifier outputs its confidence that the sample’s label is 1. So an output of 0.2 represents the classifier having 20% confidence that the sample is malicious.

• Evaluate the classifier on a held out test set. We know the ground truth labels for the data in the test set, of course, and we’re trying to see the extent to which the classifier agrees with ground truth.

Example, continued: Suppose, as before, benign = 0, malicious = 1, and our classifier assigns scores corresponding to the probability of the sample being malicious. Suppose the actual labels of the five samples in the test set are [1, 0, 1, 0, 0]. Then a classifier might return predictions like [0.4, 0.7, 0.9, 0.1, 0.2].

• Sort the data in ascending order by the scores assigned by the classifier.

Example, continued:

predictions:   [0.1, 0.2, 0.4, 0.7, 0.9]
ground truth:  [0.0, 0.0, 1.0, 0.0, 1.0]

• Pick a representative threshold for each gap in the predictions (midpoints are fine).

Example, continued:

thresholds: [0.05, 0.15, 0.3, 0.55, 0.8, 0.95]

• FINALLY, start producing points for the ROC curve. For every threshold in our list, we perform a two step process.
  1. Binarize according to the threshold. Scores above the threshold go to 1; scores below the threshold go to 0.
  2. Compute the false positive rate (FPR) and true positive rate (TPR) for that threshold.

Example, continued. Let’s work out the math by hand.

Threshold: 0.05

• Binarized predictions: [1, 1, 1, 1, 1]
• Ground truth: [0, 0, 1, 0, 1]

Confusion matrix:

• FPR = false positives / actually negative = 3 / 3 = 1.0
• TPR = true positives / actually positive = 2 / 2 = 1.0

Point on the ROC curve: (FPR, TPR) = (1.0, 1.0)

Threshold: 0.15

• Binarized predictions: [0, 1, 1, 1, 1]
• Ground truth: [0, 0, 1, 0, 1]

Confusion matrix:

• FPR = 2 / 3 = 0.67
• TPR = 2 / 2 = 1.0

Point on the ROC curve: (0.67, 1.0)

Threshold: 0.3

• Binarized predictions: [0, 0, 1, 1, 1]
• Ground truth: [0, 0, 1, 0, 1]

Confusion matrix:

• FPR = 1 / 3 = 0.33
• TPR = 2 / 2 = 1.0

Point on the ROC curve: (0.33, 1.0)

Threshold: 0.55

• Binarized predictions: [0, 0, 0, 1, 1]
• Ground truth: [0, 0, 1, 0, 1]

Confusion matrix:

• FPR = 1 / 3 = 0.33
• TPR = 1 / 2 = 0.5

Point on the ROC curve: (0.33, 0.5)

Threshold: 0.8

• Binarized predictions: [0, 0, 0, 0, 1]
• Ground truth: [0, 0, 1, 0, 1]

Confusion matrix:

• FPR = 0 / 3 = 0.0
• TPR = 1 / 2 = 0.5

Point on the ROC curve: (0.0, 0.5)

Threshold: 0.95

• Binarized predictions: [0, 0, 0, 0, 0]
• Ground truth: [0, 0, 1, 0, 1]

Confusion matrix:

• FPR = 0 / 3 = 0.0
• TPR = 0 / 2 = 0.0

Point on the ROC curve: (0.0, 0.0)

Let’s plot these points.

Cool, right? The most important metric to extract is in the lower right corner of the plot: the area under the ROC curve (AUROC). AUROC is the best finger-to-the-wind measure of a classifier’s overall quality. The extremal cases are AUROC = 0.5 and AUROC = 1.0. An AUROC of 0.5 indicates a classifier that is no better than random chance. No matter what threshold we choose, the classifier cannot distinguish between the two classes. An AUROC of 1.0 corresponds to a perfect classifier. Even at a false positive rate of zero, the classifier maximizes the true positive rate. This corresponds to the existence of a threshold that perfectly splits all of the classifier’s negative and positive predictions.²

The closer our AUROC is to 1.0, the better our classifier is. And note that this is true no matter what the optimal threshold is. AUROC lets us compare classifiers with different optimal thresholds on the same scale.

Here’s a little embedded demo you can mess around with to test your intuitions. Add more rows of data, make the classifier worse/better, see the new points on the curve get rendered, etc.

Other than calculating the AUROC, what is this plot good for? It lets us answer a question like this: “If I’m only willing to accept a false positive rate of X, how good can my true positive rate Y be, assuming I calibrate my threshold correctly?” This is a GREAT question to be able to answer. In a monitoring setting, false positives are generally pretty costly to deal with, because whatever intervention we trigger on any positive is probably expensive or time-consuming. Therefore, we want to hold our false positive rate way down (1% is a good upper bound on a deployment FPR) and design classifiers to be good at that low false positive rate. Replacing a classifier that achieves a TPR of 1.0 at a 0.05 FPR with a better classifier that achieves a TPR of 1.0 at a 0.01 FPR can cut the relative number of false positives we have to deal with by a factor of five. Huge win for the program.

Hope you’re now on the ROC curve bandwagon with me 🙃

Compaction is what it says on the tin. Whenever Claude Code/Codex/Inspect’s native ReAct scaffold notices you’ve hit some pre-defined token limit, it invokes a compaction (for us, CompactionSummary in ReAct). This passes the current ~160K-token chunk of context to another instance of the untrusted model (for us, claude-opus-4-6), which summarizes it down to a few tens of thousands of tokens of context. This summary then gets passed along to a fresh Claude in ReAct, which carries on with its task (w.l.o.g small base model fine-tuning). Compaction is currently a necessary ingredient in agents achieving anything big in AI R&D automation, because it takes a lot of tokens to do big stuff, and eventually the agent has to compress context to keep working. Compaction is likely to be invoked at least once, and probably many times, in our particular setting.¹

Monitoring is also what it says on the tin. In our case, it’s shorthand for letting a trusted LM watch your agent and report how suspicious it finds the agent’s conduct. What I mean by compaction monitoring is having a trusted model ride along on the compaction process, observing its input and output, and noting anything weird or suspicious. This would give the monitor an opportunity to look back on a nicely-sized chunk of the untrusted agent’s behavior and see how suspicious its behaviors and its summary are.² Given that the input and output of all of the agent’s actions are captured by default, sustaining coherent bad behavior through the bottleneck of compaction would require convincing the monitor that all of its actions within its current context window were benign.

Monitoring can catch bad behavior over many different scales. Some bad behavior can be conducted in the space of one tool call, with no additional context or strategy required. Any bad behavior shaped like that is probably best addressed with a turn-by-turn monitor. Other bad behavior can be conducted in the space of a single context window. I think this is probably well-addressed by a compaction monitor, which naturally reads approximately one context window’s worth of tokens. A particularly pernicious kind of bad behavior can be spread out subtly along a very long trajectory (tens of millions of tokens). Crucially, this kind of bad behavior would have to be coordinated through iterated compaction. If all the actions of the agent are recorded and concatenated into the input for the compaction phase, then even actions that leave information behind (e.g., code comments, weird signals in log files, or poisoned data) are ultimately recorded in the transcript.³ I therefore think a compaction monitor that can observe everything the agent did pre-compaction could perform well in the long-horizon setting; any long-horizon bad behavior would have to pass through the compaction bottleneck. When viewed through the compaction bottleneck, the apparently terrible offense-defense balance in long-horizon bad behavior looks a bit less terrible: the untrusted agent only seems to get tens of millions of tokens of continuous, single-context-window coherence. In reality, it has to compact its own context – including whatever it needs to remember to behave badly – to keep going. We shouldn’t frame the problem of catching bad behavior as finding a needle in a haystack of millions of tokens. We should frame it instead as tracking the somewhat more limited transmission of context through time. Firing the monitor on every compaction, in other words, should alleviate an otherwise easily-exploitable context length imbalance between offense and defense.

Another core argument in favor of compaction monitors is that they can be run online without costing any extra time. I have a strong bias toward control protocols that could realistically be folded into internal deployments. If we can launch a monitor at the same time as we launch a compaction process that already takes a minute or two, we can potentially get significant safety uplift from approximately doubling the compute we spend every compaction. That’s non-negligible expense, but it wouldn’t impact the latency of the tool to the researcher, which seems to me to be the more important factor.

We could implement this relatively cheaply by using a cheap monitor, and only applying more expensive/annoying interventions if its suspicion score is above a certain level. This shouldn’t harm performance or latency, and it should give some reasonably good insight into agent behavior. I could imagine a future, expensive version of this in which we use compaction by a trusted model (perhaps activated by a turn-by-turn model logging a sufficiently high suspicion score) as a control protocol. This could be a hard-core way to break sophisticated sabotage behavior, although it might trade off too negatively against general capabilities by making the summaries less useful for continuing the legitimate long-horizon task and by slowing the untrusted model down too much.

Thanks to Rhys Ward for offering some feedback on this piece.