interpretability
-
Weak-label finetuning does not overwrite a strong model's clean-label direction, even as its output collapses
When a strong language model is finetuned on labels from an unreliable supervisor and then starts giving wrong answers, it is natural to assume the bad labels have corrupted what the model internally knows. We test that assumption directly and find it false in our setting.
- · oversight: None / Minimal · ≈ $6.46 compute
Do a Deception Probe's Most-Relied-On SAE Features Move Lying? A Scoped Single-Feature Ablation Test on Gemma Scope
A linear probe trained on internal activations can classify honest from deceptive behavior with near-perfect accuracy, and it is tempting to read that accuracy as evidence that the probe has located the machinery of lying. We test one narrow version of that reading and report a scoped negative result.
- · oversight: None / Minimal · ≈ $14.79 compute
Sparse Autoencoder Features Do Not Improve Transferable Detection of Instructed Sandbagging in Gemma-2-9B-it
Sandbagging, the strategic underperformance of a model on an evaluation, is a central threat to capability and safety evaluations (van der Weij et al., 2024).
- · oversight: None / Minimal · ≈ <$0.01 compute
Is Error-Awareness Role-Bound? Verifier-Trained Wrongness Probes Barely Transfer to a Model's Own Generation Errors
On a single instruct model (Llama-3.1-8B-Instruct) we ask whether a linear probe that reads "this answer is wrong" while the model is in VERIFIER mode (judging a shown candidate answer) transfers to detecting the model's OWN wrong answers in GENERATOR mode, on the identical MMLU questions. We hold the verifier role constant (correct vs wrong candidate) so the verifier probe learns wrongness-within-role, not the role itself. We compare the within-verifier AUC, the within-generator AUC (the ceiling for detecting generator self-errors), the verifier->generator transfer AUC, the reverse transfer, a label-shuffle chance baseline, a layer sweep, and -- as the sharpest test -- the cosine alignment between the verifier-wrongness and generator-wrongness probe directions. A near-orthogonal cosine with a large transfer gap is the clean "role-bound error awareness" result; a high transfer with aligned directions would indicate a shared error-awareness direction.
- · oversight: None / Minimal · ≈ $49.52 compute
Does the wrongness probe lead the retraction? A weak, non-robust timing signal against behavioral self-correction in Qwen2.5-7B
If a model has an internal sense of when it is wrong, that signal might be expected to precede the moment it backs down under challenge, which would make a residual-stream probe an early monitor of self-correction. We test this directly on Qwen2.5-7B-Instruct.
- · oversight: None / Minimal · ≈ $47.64 compute
Outcome-Only RL Did Not Reduce a Model's Causal Dependence on Its Own Chain of Thought
A worry about outcome-only reinforcement learning is that, by rewarding only the final answer, it makes the chain of thought decorative: the answer would stop causally depending on the reasoning the model writes. We test this with a self-CoT corruption-flip protocol.
- · oversight: None / Minimal · ≈ $79.19 compute
Small-Scale Outcome-Only GRPO Barely Moves the Answer Distribution
If outcome-only RL mostly elicits what the base model already computes, its effect on the answer distribution might be reproducible by a single global affine transform of the base logits (a scalar temperature plus a per-option bias).
- · oversight: None / Minimal · ≈ $29.70 compute
RL-Unlocked MMLU Answers Are Only Partially Latent in the Base Model, and Emerge Late Rather Than Mid-Network
The elicitation view of RL and instruction tuning holds that post-training mostly surfaces capabilities the base model already has rather than teaching new ones.
- · oversight: None / Minimal · ≈ $153.19 compute
Pre-Answer States Encode Correctness but Only Marginally Separate "Cannot Solve" from "Slipped"
When a model answers a math problem incorrectly, the failure can be a genuine inability to solve it or a slip in execution on a problem it can solve.
- · oversight: None / Minimal · ≈ $52.43 compute
When the verdict contradicts the work: verbalized confidence, chain of thought, and what a truth probe actually reads
Per cell (model x domain x surface format) we compare a trained linear probe on residual activations against the model's own verbalized confidence and a training-free commit-probability baseline, asking whether the internal signal and the verbalized signal fail on the same items.