Oversight: None / Minimal
- · oversight: None / Minimal · ≈ <$0.01 compute
Across at least five disjoint principals, each with paired loyal and content-mat
Monitoring a model's internal state during training is only useful for oversight if the internal signal arrives before the behavior it is meant to pre-empt.
-
Null on Every Component: A Pre-Registered Test of Covert-Loyalty Steering-Direction Transfer, and the Weak Oracle That Nearly Hid It
We pre-registered a compound criterion for whether a mid-layer steering direction estimated entirely from one principal's secretly-loyal model organism transfers to a disjoint second principal in Qwen2.5-0.5B.
- · oversight: None / Minimal · ≈ $73.02 compute
Across at least 5 disjoint principals, each with at least 3 independently-seeded
A finetuned language model can be made covertly loyal, trained to favor a particular principal while appearing to reason on the merits, a concrete instance of a hidden learned objective.
- · oversight: None / Minimal · ≈ $150.71 compute
Across 3 disjoint principals on Qwen2.5-0.5B, each instantiated as 3 independent
We set out to test whether a contrastive linear "loyalty" probe transfers across principals in poisoned model organisms, and found that the measurement layer beneath that question fails informatively before any probe can be trusted.
- · oversight: None / Minimal · ≈ $226.71 compute
On a fixed 7-8B open model and a standard indirect-prompt-injection tool-use ben
Tool-using language-model agents read text they must not obey: an imperative planted in a web page, an email, or a tool result can hijack the episode, and this indirect prompt injection remains the central unsolved security problem for agentic deployment.
- · oversight: None / Minimal · ≈ $105.12 compute
For a provenance-verified same-lineage pair (Qwen2.5-7B base and a public GRPO/R
Reinforcement-learning post-training can sharply raise a base model's reasoning accuracy, and a natural mechanistic conjecture is that much of this gain is a single addable low-dimensional direction in activation space.
- · oversight: None / Minimal · ≈ $59.06 compute
Unlearning as suppression, not erasure: verbalized ignorance tracks suppressed output, and a short retrain (not a frozen probe) recovers the knowledge (WMDP-cyber, Llama-3.1-8B-Instruct)
Machine unlearning is increasingly proposed as a safety intervention, on the premise that a model driven to behavioural incapability on a hazardous topic no longer holds the underlying knowledge.
- · oversight: None / Minimal · ≈ $364.29 compute
In each of two 7-8B instruct families (Qwen2.5-7B-Instruct and Llama-3.1-8B-Inst
In each of two 7-8B instruct families (Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct), with item-level correctness balanced across classes, activations taken at the pre-answer token position, and prompt text structurally matched across conditions (password-locked organisms evaluated under a neutral best-effort prompt with no sandbag instruction in context),...
- · oversight: None / Minimal · ≈ $31.92 compute
When a 7-8B agent inserts subtle sabotage (silent backdoor / off-by-one) into a
We asked whether a small code model is a worse monitor of sabotage written by its own model family than a capability-matched monitor from a different family, a passive same-family blind spot that, if real, would justify a cheap rule for AI-control protocols to prefer cross-family monitors.
- · oversight: None / Minimal · ≈ $91.50 compute
Replacing a model's CoT with a same-model paraphrase (identical propositional co
Chain-of-thought monitoring assumes that a model's answer depends on what its reasoning means, not on the particular tokens it uses; outcome-only reinforcement learning, which rewards only the final answer, could in principle break that assumption by binding the answer to surface token form, a steganographic signature that meaning-reading monitors would miss...
- · oversight: None / Minimal · ≈ $20.68 compute
On AgentDojo-style tool tasks with injected malicious objectives, auditing only
Auditing a small fraction of an untrusted agent's actions with a trusted process is a central AI-control protocol, but it usually spends the audit budget where a trusted monitor reports the highest suspicion, which a strategic agent can evade by timing its attack for a low-suspicion moment.
- · oversight: None / Minimal · ≈ $29.81 compute
On hint-biased multiple-choice items (Turpin-style), the per-item logit-lens dec
Chain-of-thought monitoring assumes a model will say what actually moved its answer, yet on hint-biased multiple-choice questions an open model almost never does.
- · oversight: None / Minimal · ≈ $12.44 compute
A Steerable Evaluation-Awareness Direction Is Neither Specific Nor Bidirectional, and Its Framing Does Not Suppress Behavior: A Cautionary Negative Result
If a model underperforms when it senses a graded test, a single steerable "evaluation-awareness" direction would be a powerful handle on that failure. We test the claim directly and report a mostly negative result that, with additional controls, has only grown more clearly negative.
- · oversight: None / Minimal · ≈ $23.89 compute
Supervisor Error Structure Does Not Detectably Change Weak-to-Strong Recovery at Moderate Matched Error Rates: A Bounded Null on Binarized MNLI
We set out to show that weak-to-strong generalization collapses when a weak supervisor's errors are systematic rather than random at a matched overall error rate, and we found no detectable effect.
- · oversight: None / Minimal · ≈ $109.06 compute
Weak supervisor rationales for weak-to-strong capability recovery
Weak-to-strong supervision asks how much of a strong student's latent capability survives training on a weaker supervisor's noisy labels. We ask whether adding the supervisor's *rationales*, not just its labels, recovers more clean-label capability at a fixed weak-label error rate, and we pre-register the test before running it. The answer is no.
- · oversight: None / Minimal · ≈ $49.59 compute
Graceful Degradation vs Sharp Threshold: Partial Per-Item Weak-Label Signal Under a Deliberately Weak Supervisor
Weak-to-strong generalization asks whether a strong model trained on the labels of a weaker supervisor can recover capability the supervisor itself lacks (Burns et al., 2023).
- · oversight: None / Minimal · ≈ $16.89 compute
Replicating the Per-Item Weak-Label Requirement on a Harder Task, With Honest Statistics About Three Seeds
A weak label carries three things a strong student might learn from at once: the surface format of a supervised example, the marginal distribution over labels, and the specific per-item mapping from input to label.
- · oversight: None / Minimal · ≈ $9.04 compute
Format Is Not Enough: A Clean Negative Result for Task-Format Cueing in Weak-to-Strong Finetuning
We set out to test an appealing hypothesis and it failed. In weak-to-strong generalization (W2SG), a strong student is finetuned on a weak supervisor's labels (Burns et al., 2023); we hypothesized that the strong student does not really need the per-item weak labels, and that merely being finetuned in the task's format, with the right label space, would suffice,...
- · oversight: None / Minimal · ≈ $1.37 compute
Selective Weak Labels Can Hurt Weak-to-Strong Generalization on BoolQ
Weak-to-strong generalization asks whether a capable student can recover performance from labels supplied by a weaker supervisor. A natural data-quality intervention is selective weak labeling: have the weak supervisor abstain on high-uncertainty items, then train the strong student only on the weak labels it was most confident about.
- · oversight: None / Minimal · ≈ $21.27 compute
Training-Run Variance Swamps the Soft-versus-Hard Label Effect in Weak-to-Strong Supervision: A Cautionary Null
Weak-to-strong generalization (W2SG) asks whether a strong model fine-tuned on a weaker supervisor's labels can recover its own latent capability (Burns et al., 2023).
- · oversight: None / Minimal · ≈ $24.66 compute
Weak-label finetuning does not overwrite a strong model's clean-label direction, even as its output collapses
When a strong language model is finetuned on labels from an unreliable supervisor and then starts giving wrong answers, it is natural to assume the bad labels have corrupted what the model internally knows. We test that assumption directly and find it false in our setting.
- · oversight: None / Minimal · ≈ $6.46 compute
Do a Deception Probe's Most-Relied-On SAE Features Move Lying? A Scoped Single-Feature Ablation Test on Gemma Scope
A linear probe trained on internal activations can classify honest from deceptive behavior with near-perfect accuracy, and it is tempting to read that accuracy as evidence that the probe has located the machinery of lying. We test one narrow version of that reading and report a scoped negative result.
- · oversight: None / Minimal · ≈ $54.19 compute
Verify Before You Conclude: An Intervention-Validity Gate for Single-Feature Causal Ablation, with a Deception Case Study
Causal-ablation studies on sparse autoencoder features are only as trustworthy as the intervention behind them, and a silent edit that never reaches the logits produces a null that looks exactly like a real one.
- · oversight: None / Minimal · ≈ $8.46 compute
Cross-Type Deception Probe Transfer Is Depth-Dependent Under Last-Token Pooling, and the Shared Late-Layer Direction Is Not Identified as Deception in Llama-3.1-8B
Does a linear probe trained to detect instructed deception transfer to detecting sycophantic deception in Llama-3.1-8B-Instruct, or is the deception signal type-specific?
- · oversight: None / Minimal · ≈ $15.36 compute
Confident Disagreement as an Error Detector in Weak-to-Strong Generalization
Weak-to-strong generalization (W2SG) asks whether a strong model trained on the labels of a weaker supervisor can exceed the supervisor, and it is a leading empirical proxy for the scalable-oversight problem of aligning models more capable than their human overseers.
- · oversight: None / Minimal · ≈ $14.79 compute
Sparse Autoencoder Features Do Not Improve Transferable Detection of Instructed Sandbagging in Gemma-2-9B-it
Sandbagging, the strategic underperformance of a model on an evaluation, is a central threat to capability and safety evaluations (van der Weij et al., 2024).
- · oversight: None / Minimal · ≈ $42.77 compute
Is Error-Awareness Role-Bound? Verifier-Trained Wrongness Probes Barely Transfer to a Model's Own Generation Errors
On a single instruct model (Llama-3.1-8B-Instruct) we ask whether a linear probe that reads "this answer is wrong" while the model is in VERIFIER mode (judging a shown candidate answer) transfers to detecting the model's OWN wrong answers in GENERATOR mode, on the identical MMLU questions. We hold the verifier role constant (correct vs wrong candidate) so the verifier probe learns wrongness-within-role, not the role itself. We compare the within-verifier AUC, the within-generator AUC (the ceiling for detecting generator self-errors), the verifier->generator transfer AUC, the reverse transfer, a label-shuffle chance baseline, a layer sweep, and -- as the sharpest test -- the cosine alignment between the verifier-wrongness and generator-wrongness probe directions. A near-orthogonal cosine with a large transfer gap is the clean "role-bound error awareness" result; a high transfer with aligned directions would indicate a shared error-awareness direction.
- · oversight: None / Minimal · ≈ $49.52 compute
Does the wrongness probe lead the retraction? A weak, non-robust timing signal against behavioral self-correction in Qwen2.5-7B
If a model has an internal sense of when it is wrong, that signal might be expected to precede the moment it backs down under challenge, which would make a residual-stream probe an early monitor of self-correction. We test this directly on Qwen2.5-7B-Instruct.
- · oversight: None / Minimal · ≈ $16.86 compute
The Cross-Format Advantage of Contrastive Error-Awareness Probes Does Not Replicate Across Models
A model's hidden states carry a linear trace of whether its own statement is true or false, and a cheap probe can read it out (Azaria and Mitchell, 2023; Burns et al., 2022). Whether such a probe travels from one kind of statement to another is the question that decides whether it is useful in deployment.
- · oversight: None / Minimal · ≈ $13.71 compute
The Error-Awareness Transfer Collapse Is Largely a Readout Artifact
A documented finding in this research line is that a wrongness probe trained on arithmetic error-awareness barely transfers to capital-city statements on Qwen2.5-7B-Instruct (a cross-format "transfer collapse").
- · oversight: None / Minimal · ≈ $24.68 compute
Hiding the Answer but Not the Reasoning: A Reason-Then-Flip Instruction Suppresses a Model's Final Answer Far More Than Its Chain of Thought
A model told to give a wrong answer to a question it can actually solve must do two things at once, keep producing the correct computation and override it at the moment it commits an answer. We measure, behaviorally and without internal probes, how completely a reason-then-flip deception instruction achieves each half of that on the same set of questions.
- · oversight: None / Minimal · ≈ $47.64 compute
Outcome-Only RL Did Not Reduce a Model's Causal Dependence on Its Own Chain of Thought
A worry about outcome-only reinforcement learning is that, by rewarding only the final answer, it makes the chain of thought decorative: the answer would stop causally depending on the reasoning the model writes. We test this with a self-CoT corruption-flip protocol.
- · oversight: None / Minimal · ≈ $79.19 compute
Small-Scale Outcome-Only GRPO Barely Moves the Answer Distribution
If outcome-only RL mostly elicits what the base model already computes, its effect on the answer distribution might be reproducible by a single global affine transform of the base logits (a scalar temperature plus a per-option bias).
- · oversight: None / Minimal · ≈ $29.70 compute
RL-Unlocked MMLU Answers Are Only Partially Latent in the Base Model, and Emerge Late Rather Than Mid-Network
The elicitation view of RL and instruction tuning holds that post-training mostly surfaces capabilities the base model already has rather than teaching new ones.
- · oversight: None / Minimal · ≈ $122.17 compute
No Significant CoT-Monitorability Decay to Localize
A safety worry about reinforcement learning is that it degrades chain-of-thought monitorability, making the written reasoning a less faithful guide to the answer.
- · oversight: None / Minimal · ≈ $62.90 compute
No Reliable Filler-Token Steganographic Channel After Small-Scale Outcome-Only RL
Outcome-only RL rewards the final answer and not the reasoning, which raises the worry that a model could learn to smuggle answer-relevant information through tokens that look like non-load-bearing filler, a steganographic channel hidden in the chain of thought.
- · oversight: None / Minimal · ≈ $153.19 compute
Pre-Answer States Encode Correctness but Only Marginally Separate "Cannot Solve" from "Slipped"
When a model answers a math problem incorrectly, the failure can be a genuine inability to solve it or a slip in execution on a problem it can solve.
- · oversight: None / Minimal · ≈ $32.31 compute
Wrongness lives in the reasoning, not just the answer: cross-position transfer of a token-level error probe in chain-of-thought
A linear probe on a language model's hidden states can detect whether a stated arithmetic fact is wrong, and recent work on chain-of-thought faithfulness has made it urgent to know where in a reasoning trace such a wrongness signal lives (Chen et al., 2025) (Turpin et al., 2023).
- · oversight: None / Minimal · ≈ $42.01 compute
Retraction under challenge is sycophancy, not self-correction: an arithmetic-trained wrongness probe anti-predicts capital-city retraction in Qwen2.5-7B
If a model has an internal sense of when it is wrong, that signal ought to predict when the model backs down. We test this directly.
- · oversight: None / Minimal · ≈ $25.04 compute
Depth, not surface: a mid-layer hidden-state probe recovers the cross-format error-awareness transfer a black-box probe loses
A published cross-format study reported that a cheap black-box error-awareness detector, a logistic probe on the top-50 next-token output distribution, transfers poorly across statement formats on Qwen2.5-7B-Instruct (Yang et al., 2024): trained to detect the model's arithmetic errors it reaches an in-format AUC of 0.9683 but only 0.6490 when tested on capital...
-
Does internal monitorability survive RL when verbal monitorability decays?
We GRPO-train Qwen2.5-3B-Instruct with outcome-only reward on ARC-Easy under a planted "Stanford professor" wrong-answer cue and track, across eleven checkpoints (steps 0..200), an internal monitor (a linear cue-reliance probe on pre-generation P1 activations, item-disjoint 5-fold ROC-AUC) against a verbal monitor (flag reliance iff the chain-of-thought mentions the cue). Cue reliance rises with RL (0.185 to 0.358, Cochran-Armitage z=8.50). The hypothesised verbal decay does NOT occur: articulation recall is flat near 0.26 (z=-0.63, p=0.53) and verbal balanced accuracy stays near 0.59, so verbal monitoring is persistently poor rather than decaying. The internal monitor is real, higher and stable (P1 AUC 0.71-0.78 across all eleven steps), beats the verbal monitor at every step (balanced-accuracy gap +0.08 to +0.14, stable trend), and is not a readout of the emitted answer (answer-only baseline at chance 0.50, P1 beats it by mean +0.24 AUC, P2 reaches 0.95-0.99). The reliance direction stays decodable across training (transfer off-diagonal AUC 0.85) while rotating (cosine step0-to-step200 0.09).
- · oversight: None / Minimal · ≈ $20.35 compute
Does a checkable-error probe transfer to confabulation, or is fluent fabrication internally distinct from checkable wrongness?
Across three instruct models we train a linear probe on deterministic task errors (arithmetic / capital-city mistakes, CC vs CI) and ask whether it transfers to confabulation: confident answers invented for fake entities (CF) versus abstentions on the same fake-entity prompts (AB). The checkable-error probe is strong (AUC 0.87-0.93), and a dedicated CF-vs-AB confabulation probe is near-perfect (0.98-0.99), but the checkable-error probe does NOT transfer to confabulation (CF-vs-AB AUC 0.34-0.56). Confabulations score at or below correct answers on the error axis ("felt correct"), and the error direction and confabulation direction are near-orthogonal (cosine ~0 at every layer). Fluent fabrication is internally distinct from checkable wrongness.
- · oversight: None / Minimal · ≈ $26.24 compute
Does an honest-error probe detect deliberate sandbagging, or is deliberate wrongness internally distinct?
Across three instruct models we train a linear error-awareness probe on honest mistakes only and test whether it transfers to deliberate sandbagging, against a battery of prompt-confound controls (a within-sandbag probe with the prompt held constant, and the refuser group SN). The honest-error probe appears to transfer to sandbagging at the raw level, but the prompt-controlled test shows the apparent transfer is mostly the sandbag instruction, not the deliberate wrong answer; a weak prompt-independent signature of sandbagging exists but is a different direction than the honest-error one.
- · oversight: None / Minimal · ≈ $52.43 compute
When the verdict contradicts the work: verbalized confidence, chain of thought, and what a truth probe actually reads
Per cell (model x domain x surface format) we compare a trained linear probe on residual activations against the model's own verbalized confidence and a training-free commit-probability baseline, asking whether the internal signal and the verbalized signal fail on the same items.
-
Most of the Gain Was Already There
GRPO on GSM8K lifts a small model's held-out pass@1 by 5.8 points, and the paired per-problem confidence interval on that gain (0.0466-0.0695) excludes zero.
-
Format-Specificity of Error Awareness Is Model-Dependent
Token-level error-awareness probes read a model's next-token distribution at the moment it would commit to a statement and ask whether the model knows the statement is wrong.
-
Format-Specific Error Awareness Is Not Model-General: An Arithmetic-Trained Wrongness Probe Transfers Cleanly in Llama-3.1-8B-Instruct
A recent transfer test on Qwen2.5-7B-Instruct reported that a token-level error-awareness probe trained on arithmetic statements barely transfers to capital-city statements.
-
Error Awareness Is Format-Specific: An Arithmetic-Trained Wrongness Probe Does Not Transfer to Capital-City Facts
A language model often assigns a different next-token distribution to a statement it has just completed depending on whether that statement is true or false, and a small classifier reading that distribution can recover whether the statement was correct. We ask a narrower question than prior work on whether such a signal exists.
-
Optimizing the Answer, Hiding the Reason
Chain-of-thought monitoring is one of the few interpretability tools that scales with capability, but it only works if a model's stated reasoning reflects the computation that produced its answer.