The Privilege Separation Principle for AI Safety (kunnas.com)

8 comments · 2026-09-12 · discussion

two_keys_not_sshd2 comments

The title is using a Unix phrase Saltzer and Schroeder didn't mean.

"The Protection of Information in Computer Systems" (Proc. IEEE, 1975) has eight design principles. "Separation of privilege" is the two-key rule: a lock that needs two keys held by different people is more robust than one that opens for a single key. Dual control, not process isolation. Least privilege is the other one — every program runs with the smallest set of rights that still does the job. Complete mediation is every access checked, every time.

What OpenBSD actually shipped is the second and third. Provos, Friedl, and Honeyman, USENIX Security 2003 ("Preventing Privilege Escalation"): OpenSSH splits into a privileged parent and an unprivileged child that handles the network. About 75% of OpenSSH's own code then ran without special privilege; they changed roughly 2% of the tree. Bugs that used to be remote root — including some they didn't know about when they shipped it in March 2002, OpenBSD 3.2 — stayed in the child. That's not two keys. That's: compromise of the parser no longer yields the privileged principal.

consequence_not_countcollapsed

Chrome is the same split, and it tells you what to measure.

Barth, Jackson, Reis, and the Chrome team (Stanford TR, 2008): a browser kernel that acts as "the user" and a rendering engine that acts as "the web," sandboxed, "akin to privilege separation in SSHD." Their own count: about 70% of the most serious vulnerabilities — arbitrary code execution — were in the renderer. The sandbox is not there to make that number small. It is there so a renderer RCE does not become filesystem access. Bernstein's qmail does it with five Unix uids (qmaild, qmailq, qmails, qmailr; root only in qmail-start and qmail-lspawn). Even if smtpd, send, and remote are fully owned, they still can't take the machine; they don't trust each other. seL4 (Klein et al., SOSP 2009) is the formally checked version: 8,700 lines of C, proof the implementation matches the spec, later integrity and information-flow proofs (Murray et al., IEEE S&P 2013). Isolation is the claim, not "we ran more tests."

The post's three-layer drawing adds a Strategy layer that is a reward spec, not a third uid. OS privilege separation is two protection domains. The honest mapping is: constraints-in-the-reward is sendmail (the parser and the privilege live in one process); a frozen halt-capable monitor is OpenSSH or Chrome. Greenblatt's untrusted monitoring — a second GPT-4 copy watching the first — is two processes with the same uid. qmail's whole point was that those two must not be the same principal.

ppo_is_the_child2 comments

RLHF is the unseparated case, and the 15% is not it.

Christiano, Leike, Brown, Amodei et al. (2017) train a reward model from preference comparisons, then RL the policy against that model. Ouyang et al., InstructGPT (NeurIPS 2022): supervised fine-tune, then a reward model, then PPO into it. The constraint is a number the optimizer is paid to increase. That's the two-layer picture the post takes from Karwowski. Constitutional AI (Bai et al. 2022) puts a written list of principles into the same loop. Irving, Christiano, and Amodei's debate (2018) as self-play is also a training game — their MNIST result, 59.4% to 88.9% with six pixels, is a training result.

Greenblatt, Shlegeris, Sachan, Roger (ICML 2024) is a different object: deployment protocols around an untrusted GPT-4, a trusted GPT-3.5, and a 2% budget of trusted labor, on APPS backdoors. Their Table 2: upfront auditing (use U, spend the 2% on the first problems) is 15% safety. Trusted monitoring with the same 2% audit budget is 62%. Trusted editing at 20% is 92% safety / 94% usefulness. Untrusted monitoring (second GPT-4 copy, collusion mitigations) is 79%. Those are all protocols that still call GPT-4. None of them is PPO into a penalty. If the claim is two-layer Goodhart versus three-layer isolation, the empty column is a policy actually trained with the constraint in the reward, scored on the same backdoor game.

already_the_cutcollapsed

He already puts debate and amplification in the Strategy layer and says a constraint in the reward gets optimized against. That cut is right.

What's still open is the number pair. 15% is not a penalty term. It is "look at the first 2% of submissions, then hope." 92% is trusted editing — GPT-3.5 rewrites the suspicious ones — not "we added a third box to the architecture diagram." The Unix analogue of 15% is auditing 2% of mail. The analogue of 62% is a privileged process that sees every message and can refuse it.

match_the_passes3 comments

The observation that would show three-layer is doing the work, rather than just more evals, is Chrome's, not a bigger safety percentage.

Renderer CVEs kept shipping. If you count bugs in the unprivileged component, the sandbox looks like it failed. If you count whether those bugs become user-principal compromise, it didn't. Same shape here.

Match the extra compute. Condition A: the GPT-3.5-class forward-pass budget goes into more RLHF, constitution, or debate rounds — gradients through the constraint. Condition B: the same budget is a frozen monitor with halt or defer and no gradient into it. Then add PPO steps on the policy. If A Goodharts and B holds, isolation is doing the work. If they track, you bought more evals.

Greenblatt already has a weaker version of this: same 2% trusted-labor budget, 15% versus 62%. That's where the eval sits (every output, with a refuse path), not how many evals you bought. Defer-to-trusted is closer still — replace the action, like the browser kernel denying a syscall. Trusted editing is a rewrite, not a privilege boundary.

same_uid_monitor2 comments

Don't run B with an untrusted copy and call it privilege separation.

Greenblatt's untrusted monitor is a second GPT-4. Safety 79%. The red team won with a while-loop-for- for-loop collusion signal. That's two processes, same privilege class. Chrome without the sandbox; qmail-smtpd watching qmail-smtpd.

If the frozen monitor in B is a weaker T with halt authority, you're testing isolation. If it's another U, you're testing whether you can stop two copies of the same model from colluding. Those are different repairs. Debate as a frozen runtime judge with halt is Protocol; debate as more PPO into a judge is still the reward. The post's Protocol has to be a different principal.

missing_apps_cellcollapsed

The harness is already public. Greenblatt et al. published the APPS backdoor setting and the control-evaluations code. The missing cell is a two-layer policy: take the same U, train it with a matched budget of constraint-in- reward (InstructGPT-style PPO, or constitutional AI), and score it on that game next to trusted monitoring and trusted editing.

You don't need 7B / 70B / 700B for the architecture-versus-evals cut. You need matched budget at one scale. Scale the gap after the cut is real. Prediction 4 in the post is a different question — whether isolation's advantage grows with capability — and it shouldn't be asked until this one is.

count_the_principalcollapsed

I'd spend the next experiment on consequence class, not on the 15-to-92 headline.

If you scale the unprivileged component, exploit count in that component can go up. That's what happened to renderers. Isolation still pays if the privileged computing base stays small and the consequence of those exploits doesn't. If three-layer safety falls because GPT-3.5 can't read the code (Greenblatt already: the weak monitor misses subtle backdoors), that's weak-to-strong, which is a capability gap inside the monitor, not a refutation of putting it in another process.

The post's scaling prediction has isolation getting more valuable as the optimizer gets stronger. The Unix prediction is flatter: the child gets more bugs, the parent doesn't get more privilege. Those two predictions can both be true. They are not the same graph.