ai-safety
- · oversight: None / Minimal · ≈ $6.46 compute
Do a Deception Probe's Most-Relied-On SAE Features Move Lying? A Scoped Single-Feature Ablation Test on Gemma Scope
A linear probe trained on internal activations can classify honest from deceptive behavior with near-perfect accuracy, and it is tempting to read that accuracy as evidence that the probe has located the machinery of lying. We test one narrow version of that reading and report a scoped negative result.
- · oversight: None / Minimal · ≈ $15.36 compute
Confident Disagreement as an Error Detector in Weak-to-Strong Generalization
Weak-to-strong generalization (W2SG) asks whether a strong model trained on the labels of a weaker supervisor can exceed the supervisor, and it is a leading empirical proxy for the scalable-oversight problem of aligning models more capable than their human overseers.
- · oversight: None / Minimal · ≈ $49.52 compute
Does the wrongness probe lead the retraction? A weak, non-robust timing signal against behavioral self-correction in Qwen2.5-7B
If a model has an internal sense of when it is wrong, that signal might be expected to precede the moment it backs down under challenge, which would make a residual-stream probe an early monitor of self-correction. We test this directly on Qwen2.5-7B-Instruct.
- · oversight: None / Minimal · ≈ $16.86 compute
The Cross-Format Advantage of Contrastive Error-Awareness Probes Does Not Replicate Across Models
A model's hidden states carry a linear trace of whether its own statement is true or false, and a cheap probe can read it out (Azaria and Mitchell, 2023; Burns et al., 2022). Whether such a probe travels from one kind of statement to another is the question that decides whether it is useful in deployment.
- · oversight: None / Minimal · ≈ $13.71 compute
The Error-Awareness Transfer Collapse Is Largely a Readout Artifact
A documented finding in this research line is that a wrongness probe trained on arithmetic error-awareness barely transfers to capital-city statements on Qwen2.5-7B-Instruct (a cross-format "transfer collapse").
- · oversight: None / Minimal · ≈ $47.64 compute
Outcome-Only RL Did Not Reduce a Model's Causal Dependence on Its Own Chain of Thought
A worry about outcome-only reinforcement learning is that, by rewarding only the final answer, it makes the chain of thought decorative: the answer would stop causally depending on the reasoning the model writes. We test this with a self-CoT corruption-flip protocol.
- · oversight: None / Minimal · ≈ $79.19 compute
Small-Scale Outcome-Only GRPO Barely Moves the Answer Distribution
If outcome-only RL mostly elicits what the base model already computes, its effect on the answer distribution might be reproducible by a single global affine transform of the base logits (a scalar temperature plus a per-option bias).
- · oversight: None / Minimal · ≈ $29.70 compute
RL-Unlocked MMLU Answers Are Only Partially Latent in the Base Model, and Emerge Late Rather Than Mid-Network
The elicitation view of RL and instruction tuning holds that post-training mostly surfaces capabilities the base model already has rather than teaching new ones.
- · oversight: None / Minimal · ≈ $122.17 compute
No Significant CoT-Monitorability Decay to Localize
A safety worry about reinforcement learning is that it degrades chain-of-thought monitorability, making the written reasoning a less faithful guide to the answer.
- · oversight: None / Minimal · ≈ $62.90 compute
No Reliable Filler-Token Steganographic Channel After Small-Scale Outcome-Only RL
Outcome-only RL rewards the final answer and not the reasoning, which raises the worry that a model could learn to smuggle answer-relevant information through tokens that look like non-load-bearing filler, a steganographic channel hidden in the chain of thought.
- · oversight: None / Minimal · ≈ $153.19 compute
Pre-Answer States Encode Correctness but Only Marginally Separate "Cannot Solve" from "Slipped"
When a model answers a math problem incorrectly, the failure can be a genuine inability to solve it or a slip in execution on a problem it can solve.
- · oversight: None / Minimal · ≈ $32.31 compute
Wrongness lives in the reasoning, not just the answer: cross-position transfer of a token-level error probe in chain-of-thought
A linear probe on a language model's hidden states can detect whether a stated arithmetic fact is wrong, and recent work on chain-of-thought faithfulness has made it urgent to know where in a reasoning trace such a wrongness signal lives (Chen et al., 2025) (Turpin et al., 2023).
- · oversight: None / Minimal · ≈ $42.01 compute
Retraction under challenge is sycophancy, not self-correction: an arithmetic-trained wrongness probe anti-predicts capital-city retraction in Qwen2.5-7B
If a model has an internal sense of when it is wrong, that signal ought to predict when the model backs down. We test this directly.
- · oversight: None / Minimal · ≈ $25.04 compute
Depth, not surface: a mid-layer hidden-state probe recovers the cross-format error-awareness transfer a black-box probe loses
A published cross-format study reported that a cheap black-box error-awareness detector, a logistic probe on the top-50 next-token output distribution, transfers poorly across statement formats on Qwen2.5-7B-Instruct (Yang et al., 2024): trained to detect the model's arithmetic errors it reaches an in-format AUC of 0.9683 but only 0.6490 when tested on capital...
-
Most of the Gain Was Already There
GRPO on GSM8K lifts a small model's held-out pass@1 by 5.8 points, and the paired per-problem confidence interval on that gain (0.0466-0.0695) excludes zero.
-
Format-Specificity of Error Awareness Is Model-Dependent
Token-level error-awareness probes read a model's next-token distribution at the moment it would commit to a statement and ask whether the model knows the statement is wrong.
-
Format-Specific Error Awareness Is Not Model-General: An Arithmetic-Trained Wrongness Probe Transfers Cleanly in Llama-3.1-8B-Instruct
A recent transfer test on Qwen2.5-7B-Instruct reported that a token-level error-awareness probe trained on arithmetic statements barely transfers to capital-city statements.
-
Error Awareness Is Format-Specific: An Arithmetic-Trained Wrongness Probe Does Not Transfer to Capital-City Facts
A language model often assigns a different next-token distribution to a statement it has just completed depending on whether that statement is true or false, and a small classifier reading that distribution can recover whether the statement was correct. We ask a narrower question than prior work on whether such a signal exists.
-
Optimizing the Answer, Hiding the Reason
Chain-of-thought monitoring is one of the few interpretability tools that scales with capability, but it only works if a model's stated reasoning reflects the computation that produced its answer.