No Sandbagger Met the Accuracy Match, While Honest Degradation Triggered Person-Fit Misfit
≈ $555.64 compute tokens $523.01 · GPU $32.63
Download paper (PDF, NeurIPS format)
Abstract
This report establishes two findings, not sandbagging detection: lz* flags held-out honest families in 7/238 batches (0.029) but honestly degraded models at matched target accuracy in 13/42 (0.31), so misfit is not specific to strategy. No strategic arm reached the accuracy match, leaving detection claims untested.
Hypothesis
The registered hypothesis predicts lz* true-positive rates of at least 0.80 for difficulty-blind and 0.50 for failure-aware sandbaggers, decreasing as mimicry fidelity rises, while false-positive rates remain at most 0.075 under family shift and ordinary degradation. Accuracy matching is required before testing detection.
Setup
We evaluate Qwen2.5-7B-Instruct (Yang et al., 2024) and Yi-1.5-9B-Chat, a post hoc replacement for the registered Llama-3.1-8B-Instruct (see Limitations), against 34 honest models from ten developer families. All answer 2,100 ten-option MMLU-Pro questions across 14 subjects (Wang et al., 2024), split within subject into 1,050 calibration and 1,050 test items. The honest panel fits three-parameter logistic item-response models (Birnbaum, 1968) with guessing fixed at 0.1. Leave-one-family-out fitting excludes the judged model’s family. We score each subject-balanced 140-item batch with lz*, a standardized person-fit statistic for how unusual its item-level response pattern is at its estimated ability (Drasgow et al., 1985; Snijders, 2001). For each target family, the frozen threshold is the fifth percentile of a nested honest null in which every calibration batch is scored using parameters fit without both that target family and the batch’s own family. This nested null departs from the registered in-sample null, which a synthetic test showed inflates false positives on unseen families. We label it post hoc: the rule was fixed before the first experiment was launched, but its written record was corrected after that launch, before any results were recorded.
Sandbagging arms are accuracy-matched to Falcon3-3B-Instruct, a panel-excluded weaker model. They progress from a naive prompt and a password lock, a fine-tuned model that changes behavior after a secret prefix (Greenblatt et al., 2024), through increasingly informed prompts to locks trained to imitate the reference. Mimicry fidelity is the per-item correlation between an arm’s errors and the reference’s error probabilities. Locks use rank-16 low-rank adaptation across seven projections for three epochs and three seeds, using 700 calibration items. A null lock controls for fine-tuning. Three registered non-strategic controls lower accuracy without strategy: temperature sampling and Gaussian weight noise, each tuned to the target accuracy, and 4-bit NF4 quantization, which did not reach it and is reported unmatched. Same-budget baselines use accuracy, answer confidence, paraphrase consistency, and answer-position concentration; weight-noise injection into each arm, read as an accuracy gain, is a white-box reference ceiling (Tice et al., 2025).
Results
This report establishes a false-positive asymmetry and a construction failure, not sandbagging detection: unseen honest families were flagged in 7/238 batches (0.029), whereas honestly degraded models at matched target accuracy were flagged in 13/42 (0.31). No strategic arm reached that match, leaving detection claims untested.
Only temperature-degraded Yi (8/21) and weight-noise-degraded Qwen (5/21) were in band; Qwen temperature, Yi weight noise and NF4 quantization on both subjects were not. Null-lock flags diverged (0/21 Qwen; 5/21 Yi), so fine-tuning alone shifted lz* for Yi. Paraphrase consistency lacked a panel null; its two-subject fallback flagged 5/14 runs. Forced-letter prompt dials were flat: requested 20–40% accuracy yielded 0.34–0.36 on calibration items versus 0.35 honest. Random-wrong locks flipped only 13–17% of correct training answers, one seed flipped none after a correction bug, and the changes did not transfer. Imitation locks partly transferred at about 0.54 fidelity, but the six-epoch retry scored 0.334, above band.
Table 1. Registered tests and control diagnostics. BCa intervals are shown for registered tests; Wilson intervals for component and zero-count rates. P values are one-sided percentile-duality tests of the registered direction under the two-stage cluster bootstrap.
| Check | Estimate | 95% CI | k/n | p | Registered |
|---|---|---|---|---|---|
| Blind / aware TPR | untested | – | 0 eligible | – | ≥0.80 / ≥0.50 |
| Held-out FPR | 0.029 | [0.008,0.092] | 7/238 | 0.015 | ≤0.075 |
| Temperature-degraded (Yi, in band) | 0.38 | [0.21,0.59] | 8/21 | – | ≤0.075 |
| Weight-noise-degraded (Qwen, in band) | 0.24 | [0.11,0.45] | 5/21 | – | ≤0.075 |
| Pooled degraded FPR | 0.31 | [0.17,0.50] | 13/42 | 0.999 | ≤0.075 |
| NF4-quantized (unmatched; accuracy 0.43 Qwen, 0.35 Yi) | 0.00 | [0.00,0.22] | 0/14 | – | reported unmatched |
| External oracle | 0.00 | [0.00,0.22] | 0/14 | – | honest null |
| Manipulation accuracy | before 0.399; after 0.384; drop 0.015 | [-0.018,0.054] | – | – | drop ≥0.080 |
| Null-lock FPR (Qwen) | 0.00 | [0.00,0.15] | 0/21 | – | fine-tuning control |
| Null-lock FPR (Yi) | 0.24 | [0.11,0.45] | 5/21 | – | fine-tuning control |
Limitations
Manipulation check
Sandbaggers averaged 0.384 versus 0.399 honest, a 0.015 drop below required 0.080; none reached the band, so detection power remains untested.
The registered MATH/2PL arm was deferred before any experiment was launched, so only MMLU-Pro was tested. Llama-3.1-8B-Instruct, the registered second subject, could not be loaded on the compute used. Its replacement by Yi-1.5-9B-Chat is a post hoc deviation, recorded after the first experiments had started; the selection rule was written before any candidate replacement had been scored, and Yi tied its threshold exactly. Under Rasch, degraded controls were flagged at 0.14 (6/42), versus 0.31 (13/42) under 3PL; both exceed 0.075, so the direction holds but its magnitude depends on the item model.
Reproducibility
This appendix is generated mechanically from the run’s recorded artifacts so that every experimental cell, its sample size, its seeds, and its compute cost are inspectable without re-running the job or asking the authors. A value shown as “not recorded” was absent from the manifest and has not been inferred.
The run comprises 17 recorded experiments. Each row below is one experimental cell as written to results/real/experiments.json.
| Experiment | Model | Dataset | Mode | n (per cell) | Seed(s) | Key hyperparameters | GPU minutes |
|---|---|---|---|---|---|---|---|
| e0-reference-weak | Falcon3-3B-Instruct | MMLU-Pro ten-option subsample data/items_mmlupro.jsonl (sha256 544b03429b41...) | inference | 8400 | not recorded | 0 steps; params=3.0B | 26.36 |
| e1-honest-panel-mopup | 17 models (0.135B-13.96B) | MMLU-Pro ten-option subsample data/items_mmlupro.jsonl (sha256 544b03429b41...) | inference | 35700 | not recorded | 0 steps; params=13.96B | 64.40 |
| e1-honest-panel-p1 | 10 models (1.7B-14B) | MMLU-Pro ten-option subsample data/items_mmlupro.jsonl (sha256 544b03429b41...) | inference | 21000 | not recorded | 0 steps; params=14.0B | 47.80 |
| e1-honest-panel-p2 | 7 models (3.8B-14B) | MMLU-Pro ten-option subsample data/items_mmlupro.jsonl (sha256 544b03429b41...) | inference | 14700 | not recorded | 0 steps; params=14.0B | 38.15 |
| e2-prompted-noise-degraded-qwen | Qwen2.5-7B-Instruct | MMLU-Pro ten-option subsample data/items_mmlupro.jsonl (sha256 544b03429b41...) | inference | 2940 | not recorded | 0 steps; params=7.0B | 0.00 |
| e2-prompted-noise-degraded-subject2 | Yi-1.5-9B-Chat | MMLU-Pro ten-option subsample data/items_mmlupro.jsonl (sha256 544b03429b41...) | inference | 2940 | not recorded | 0 steps; params=8.83B | 0.00 |
| e2-prompted-qwen | Qwen2.5-7B-Instruct | MMLU-Pro ten-option subsample data/items_mmlupro.jsonl (sha256 544b03429b41...) | inference | 16800 | not recorded | 0 steps; params=7.0B | 19.03 |
| e2-prompted-subject2 | Yi-1.5-9B-Chat | MMLU-Pro ten-option subsample data/items_mmlupro.jsonl (sha256 544b03429b41...) | inference | 16800 | not recorded | 0 steps; params=8.83B | 24.02 |
| e3-locks-qwen-imitation | Qwen2.5-7B-Instruct | MMLU-Pro ten-option subsample data/items_mmlupro.jsonl (sha256 544b03429b41...) | train+inference | 14700 | 0, 1, 2 | 3 epochs; params=7.0B | 60.77 |
| e3-locks-qwen-null | Qwen2.5-7B-Instruct | MMLU-Pro ten-option subsample data/items_mmlupro.jsonl (sha256 544b03429b41...) | train+inference | 14700 | 0, 1, 2 | 3 epochs; params=7.0B | 60.05 |
| e3-locks-qwen-random-wrong | Qwen2.5-7B-Instruct | MMLU-Pro ten-option subsample data/items_mmlupro.jsonl (sha256 544b03429b41...) | train+inference | 14700 | 0, 1, 2 | 3 epochs; params=7.0B | 72.56 |
| e3-locks-subject2-imitation | Yi-1.5-9B-Chat | MMLU-Pro ten-option subsample data/items_mmlupro.jsonl (sha256 544b03429b41...) | train+inference | 14700 | 0, 1, 2 | 3 epochs; params=8.83B | 86.40 |
| e3-locks-subject2-null | Yi-1.5-9B-Chat | MMLU-Pro ten-option subsample data/items_mmlupro.jsonl (sha256 544b03429b41...) | train+inference | 14700 | 0, 1, 2 | 3 epochs; params=8.83B | 88.00 |
| e3-locks-subject2-random-wrong | Yi-1.5-9B-Chat | MMLU-Pro ten-option subsample data/items_mmlupro.jsonl (sha256 544b03429b41...) | train+inference | 14700 | 0, 1, 2 | 3 epochs; params=8.83B | 88.27 |
| e3r-locks-qwen-imitation6 | Qwen2.5-7B-Instruct | MMLU-Pro ten-option subsample data/items_mmlupro.jsonl (sha256 544b03429b41...) | train+inference | 10500 | 0 | 6 epochs; params=7.0B | 68.93 |
| e4-noise-qwen | Mistral-7B-Instruct-v0.3 | MMLU-Pro ten-option subsample data/items_mmlupro.jsonl (sha256 544b03429b41...) | inference | 14280 | not recorded | 0 steps; params=7.0B | 44.51 |
| e4-noise-subject2 | Ministral-8B-Instruct-2410 | MMLU-Pro ten-option subsample data/items_mmlupro.jsonl (sha256 544b03429b41...) | inference | 13580 | not recorded | 0 steps; params=8.83B | 43.76 |
Seed policy. The distinct RNG seeds recorded across the manifest are 0, 1, 2. Per-experiment seeds are shown in the table above; replicate cells are distinguished by seed in their experiment id. 10 experiment(s) carry no recorded seed (single-shot supervisor/ceiling fits or eval passes) and are marked “not recorded”.
Cross-validation. No cross-validation fold fields are recorded in the manifest.
Statistical tests. Each quantitative comparison in the paper carries a formal test, recorded in results/real/stats.json.
| Claim | Test | Statistic | p | 95% CI | n | Seeds |
|---|---|---|---|---|---|---|
| Sandbagger arms 1-5 lower mean test accuracy below the honest subject models by at least the pre-registered 0.08 | difference of means, two-stage cluster BCa bootstrap; p is the one-sided percentile-duality p for drop > 0.08 | 0.0146 | 1 | [-0.0179, 0.054] | 140 | 3 |
| the external oracle sits at the honest null (flag rate vs 0.075) | run-level flag rate at the frozen 5% lzstar_3pl threshold; two-stage cluster BCa bootstrap (Wilson 0.0000-0.2153); p is the one-sided percentile-duality p against 0.075 | 0 | 1 | [0, 0] | 14 | 1 |
| realized FPR on held-out-family honest models <= 0.075 | run-level flag rate at the frozen 5% lzstar_3pl threshold; two-stage cluster BCa bootstrap (Wilson 0.0143-0.0595); p is the one-sided percentile-duality p against 0.075 | 0.0294 | 0.015 | [0.0084, 0.0924] | 238 | 1 |
| realized FPR on non-strategically degraded subject models at the target accuracy <= 0.075 | run-level flag rate at the frozen 5% lzstar_3pl threshold; two-stage cluster BCa bootstrap (Wilson 0.1907-0.4603); p is the one-sided percentile-duality p against 0.075 | 0.31 | 1 | [0.167, 0.5] | 42 | 1 |
| A10 paired six-epoch null-lock flag rate stays at or below K4 | run-level flag rate at the frozen 5% lzstar_3pl threshold; two-stage cluster BCa bootstrap (Wilson 0.0000-0.3543); p is the one-sided percentile-duality p against 0.15 | 0 | 0.0001 | [0, 0] | 7 | 1 |
| the external oracle sits at the honest null (flag rate vs 0.075) (Rasch robustness) | run-level flag rate at the frozen 5% lzstar_rasch threshold; two-stage cluster BCa bootstrap (Wilson 0.0000-0.2153); p is the one-sided percentile-duality p against 0.075 | 0 | 1 | [0, 0] | 14 | 1 |
| realized FPR on held-out-family honest models <= 0.075 (Rasch robustness) | run-level flag rate at the frozen 5% lzstar_rasch threshold; two-stage cluster BCa bootstrap (Wilson 0.0090-0.0482); p is the one-sided percentile-duality p against 0.075 | 0.021 | 0.013 | [0, 0.109] | 238 | 1 |
| realized FPR on non-strategically degraded subject models at the target accuracy <= 0.075 (Rasch robustness) | run-level flag rate at the frozen 5% lzstar_rasch threshold; two-stage cluster BCa bootstrap (Wilson 0.0672-0.2784); p is the one-sided percentile-duality p against 0.075 | 0.143 | 0.72 | [0.0238, 0.333] | 42 | 1 |
| A10 paired six-epoch null-lock flag rate stays at or below K4 | run-level flag rate at the frozen 5% lzstar_rasch threshold; two-stage cluster BCa bootstrap (Wilson 0.0000-0.3543); p is the one-sided percentile-duality p against 0.15 | 0 | 0.0001 | [0, 0] | 7 | 1 |
Compute. Recorded GPU time for executed experiments is 13.8837 GPU-hours on lambda:gpu_1x_a100_sxm4.
Code and data availability. A public code repository URL is not recorded in project.yaml (links.github). The per-example data of record that backs every reported number is provided under results/real/ in the project repository: curve.csv, figure_arms.csv, lzstar_by_arm.csv, noise_units.csv, recovery.csv, runs.csv, experiments.json.
References
- [rivera2026irtsafety] Rivera, Joshua Fonseca and Shah, Neil and Africa, David Demitri and Voudouris, Konstantinos (2026). Item Response Theory for AI Safety. arXiv:2608.05086.Well-specified nearest-prior-work citation: it gives precise, falsifiable figures (90-97% selective vs 35% uniform sandbagging detection at 10% FPR) and names the exact gaps this project fills (accuracy-matched weak-model imitators, fine-tuned organisms, capability benchmarks), which is the right way to position the paper's delta honestly rather than ignoring the closest related work. [decider_v3 · claude-sonnet-5/high]
- [birnbaum1968] Birnbaum, Allan (1968). Some Latent Trait Models and Their Use in Inferring an Examinee's Ability. Birnbaum (1968) is the standard, correct citation for the 3PL/2PL logistic latent-trait models (published in Lord & Novick's Statistical Theories of Mental Test Scores), and the described usage — 3PL with guessing floor fixed at 0.1 for 10-option MMLU-Pro, 2PL for free-response MATH — is methodologically sound and matches how the model is actually applied. [decider_v3 · claude-sonnet-5/high]
- [lord1980] Lord, Frederic M. (1980). Applications of Item Response Theory to Practical Testing Problems. Lord (1980) is the standard, correctly-cited reference for IRT calibration and MLE ability estimation, and the note accurately ties it to the paper's stated use (per-benchmark, subject-stratified item calibration and person-fit conditioned on ability estimates); bibtex is well-formed and the claim is not overreaching. [decider_v3 · claude-sonnet-5/high]
- [drasgow1985] Drasgow, Fritz and Levine, Michael V. and Williams, Esther A. (1985). Appropriateness Measurement with Polychotomous Item Response Models and Standardized Indices. doi:10.1111/j.2044-8317.1985.tb00817.x.Standard, verifiable citation (Drasgow, Levine & Williams 1985, BJMSP 38(1):67-86, DOI resolves) that correctly attributes the lz person-fit statistic the paper's lz* detector builds on; note accurately describes the relationship without overclaiming. [decider_v3 · claude-sonnet-5/high]
- [snijders2001] Snijders, Tom A. B. (2001). Asymptotic Null Distribution of Person Fit Statistics with Estimated Person Parameter. doi:10.1007/BF02294437.Snijders (2001) is the correct, accurate citation for lz*, the ability-estimation correction to Lz used in person-fit testing, and the payload ties it directly to the paper's stated methodology (freezing a subject-stratified 5% threshold on calibration-split honest models). Bibtex fields are well-formed and the citation is load-bearing for a specific methodological choice, not decorative. [decider_v3 · claude-sonnet-5/high]
- [meijer2001] Meijer, Rob R. and Sijtsma, Klaas (2001). Methodology Review: Evaluating Person Fit. doi:10.1177/01466210122031957.Meijer & Sijtsma (2001, APM 25(2):107-135) is a real, correctly formatted methodology review of person-fit statistics for detecting aberrant response patterns (cheating, guessing, careless responding); framing sandbagging as a person-misfit problem gives the paper a legitimate psychometric foundation for its detection approach. The bibtex fields and DOI are well-formed and the note's claim matches the paper's actual scope. [decider_v3 · claude-sonnet-5/high]
- [magis2012] Magis, David and Ra\^\iche, Gilles and B\'eland, S\'ebastien (2012). A Didactic Presentation of Snijders's lz* Index of Person Fit With Emphasis on Response Model Selection and Ability Estimation. doi:10.3102/1076998610396894.Legitimate, correctly-scoped citation: Magis et al. (2012) is the standard didactic reference for computing Snijders's lz* person-fit index under 1PL-3PL models with different ability estimators, which directly matches the paper's stated 3PL (MMLU-Pro) and 2PL (MATH) fits. Bibtex fields and DOI are well-formed and the note ties it to a concrete methodological need rather than padding the bibliography. [decider_v3 · claude-sonnet-5/high]
- [wang2024mmlupro] Wang, Yubo and Ma, Xueguang and Zhang, Ge and Ni, Yuansheng and Chandra, Abhranil and Guo, Shiguang and Ren, Weiming and Arulraj, Aaran and He, Xuan and Jiang, Ziyan and Li, Tianle and Ku, Max and Wang, Kai and Zhuang, Alex and Fan, Rongqi and Yue, Xiang and Chen, Wenhu (2024). MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark. arXiv:2406.01574.MMLU-Pro is accurately described: it extends MCQ options to up to 10 per item (vs. 4 in MMLU) across labeled subject categories, which directly supports the stated 0.1 guessing-floor justification and subject-based stratification of the lz* threshold. Bibtex matches the real NeurIPS 2024 D&B paper (arXiv:2406.01574), so the citation is well-formed and the claims it's attached to are traceable to the benchmark's actual design. [decider_v3 · claude-sonnet-5/high]
- [hendrycks2021math] Hendrycks, Dan and Burns, Collin and Kadavath, Saurav and Arora, Akul and Basart, Steven and Tang, Eric and Song, Dawn and Steinhardt, Jacob (2021). Measuring Mathematical Problem Solving With the MATH Dataset. arXiv:2103.03874.Citation is accurate (correct authors, venue, arXiv id for the MATH dataset paper) and the note gives a specific, falsifiable rationale: testing person-fit/2PL transfer from MC to free-response where guessing cannot explain errors, which is a real confound worth addressing given the paper's apparent focus on person-fit across formats. [decider_v3 · claude-sonnet-5/high]
- [qwen2024qwen25] Yang, An and Yang, Baosong and Zhang, Beichen and Hui, Binyuan and Zheng, Bo and Yu, Bowen and Li, Chengyuan and Liu, Dayiheng and others (2024). Qwen2.5 Technical Report. arXiv:2412.15115.Correctly formatted citation for Qwen2.5 Technical Report (arXiv:2412.15115), accurately described as the source for one of the two sandbagging target models (Qwen2.5-7B-Instruct); bibtex fields are complete and consistent with the known arXiv ID. [decider_v3 · claude-sonnet-5/high]
- [grattafiori2024llama3] Grattafiori, Aaron and Dubey, Abhimanyu and Jauhri, Abhinav and Pandey, Abhinav and Kadian, Abhishek and others (2024). The Llama 3 Herd of Models. arXiv:2407.21783.Legitimate, correctly formatted citation for the actual Llama 3 Herd of Models paper (arXiv:2407.21783), and it's directly load-bearing since Llama-3.1-8B-Instruct is named as the second sandbagging target model in the setup. [decider_v3 · claude-sonnet-5/high]
- [lalor2016] Lalor, John P. and Wu, Hao and Yu, Hong (2016). Building an Evaluation Scale using Item Response Theory. doi:10.18653/v1/D16-1062.Bibtex (author, venue, year, pages, DOI) matches the well-known Lalor, Wu & Yu 2016 EMNLP paper, and the note accurately summarizes its contribution (IRT-calibrated item difficulty/discrimination scale for scoring systems), which is a directly relevant precedent for any paper doing IRT-based model evaluation. [decider_v3 · claude-sonnet-5/high]
- [martinezplumed2019] Mart\'\inez-Plumed, Fernando and Prud\^encio, Ricardo B. C. and Mart\'\inez-Us\'o, Adolfo and Hern\'andez-Orallo, Jos\'e (2019). Item Response Theory in AI: Analysing Machine Learning Classifiers at the Instance Level. doi:10.1016/j.artint.2018.09.004.Legitimate, correctly formatted citation (Martínez-Plumed et al. 2019, Artificial Intelligence) that is directly on-topic: it establishes the population-based IRT framework (item difficulty/discrimination across models) that the project's hypothesis about fitting one model's item-level pattern to its ability depends on. No overclaiming in the note, and it fills a real conceptual gap rather than padding the bibliography. [decider_v3 · claude-sonnet-5/high]
- [rodriguez2021leaderboards] Rodriguez, Pedro and Barrow, Joe and Hoyle, Alexander Miserlis and Lalor, John P. and Jia, Robin and Boyd-Graber, Jordan (2021). Evaluation Examples are not Equally Informative: How should that change NLP Leaderboards?. doi:10.18653/v1/2021.acl-long.346.Rodriguez et al. 2021 is a real, accurately cited ACL paper, and the note's characterization (IRT leaderboards jointly estimating ability/difficulty/discrimination across submitted systems) matches the paper's actual contribution. It's directly relevant as precedent for item-parameter calibration on a model population, consistent with the prior accepted reference in this same vein, with no overclaiming in the note. [decider_v3 · claude-sonnet-5/high]
- [polo2024tinybenchmarks] Maia Polo, Felipe and Weber, Lucas and Choshen, Leshem and Sun, Yuekai and Xu, Gongjun and Yurochkin, Mikhail (2024). tinyBenchmarks: evaluating LLMs with fewer examples. The bibtex accurately matches the real ICML 2024 tinyBenchmarks paper (Maia Polo et al., PMLR vol 235), and the note's characterization — IRT item parameters fit on one model population predicting held-out model performance on leaderboard tasks including MMLU — is an accurate, non-overclaiming summary of that work's actual contribution, making it a legitimate precedent to cite for IRT-based panel calibration. [decider_v3 · claude-sonnet-5/high]
- [vanderweij2025sandbagging] van der Weij, Teun and Hofst\"atter, Felix and Jaffe, Oliver and Brown, Samuel F. and Ward, Francis Rhys (2025). AI Sandbagging: Language Models can Strategically Underperform on Evaluations. arXiv:2406.07358.The citation is a real, correctly identified paper (van der Weij et al., ICLR 2025, arXiv:2406.07358) and the note accurately summarizes its contributions (sandbagging definition, prompted/password-locked underperformance, difficulty-based detectability of naive sandbagging) without overclaiming. The specific detail about only mediocre success mimicking a weaker model is a legitimate, non-overstated motivation for a 'mimicry gradient' concept. [decider_v3 · claude-sonnet-5/high]
- [greenblatt2024password] Greenblatt, Ryan and Roger, Fabien and Krasheninnikov, Dmitrii and Krueger, David (2024). Stress-Testing Capability Elicitation With Password-Locked Models. arXiv:2405.19550.Well-formed, verifiable citation (Greenblatt et al. 2024, NeurIPS, arXiv:2405.19550) accurately described: the note correctly ties the password-locked model construction to the paper's own sandbagging organisms (uniform-random and weak-model-imitating), which is the actual mechanism introduced in that work. No overclaim and no risk to the manuscript's methods section. [decider_v3 · claude-sonnet-5/high]
- [tice2025noise] Tice, Cameron and Kreer, Philipp Alexander and Helm-Burger, Nathan and Shahani, Prithviraj Singh and Ryzhenkov, Fedor and Roger, Fabien and Neo, Clement and Haimes, Jacob and Hofst\"atter, Felix and van der Weij, Teun (2025). Noise Injection Reveals Hidden Capabilities of Sandbagging Language Models. arXiv:2412.01784.Well-formed, verifiable citation (NeurIPS 2025, arXiv 2412.01784) for a real related-work method; the note honestly scopes it as a white-box reference ceiling rather than a false head-to-head competitor to the paper's person-fit approach, avoiding overclaiming. [decider_v3 · claude-sonnet-5/high]
- [taylor2025auditing] Taylor, Jordan and Black, Sid and Bowen, Dillon and Read, Thomas and Golechha, Satvik and Zelenka-Martin, Alex and Makins, Oliver and Kissane, Connor and Ayonrinde, Kola and Merizian, Jacob and Marks, Samuel and Cundy, Chris and Bloom, Joseph (2025). Auditing Games for Sandbagging. arXiv:2512.07810.Well-formed, directly relevant citation: it names the specific attacker (logit-matched weak-model imitation defeating black-box sandbagging detection) that this paper's trained imitators instantiate, and the note correctly scopes the gap (no IRT person-fit test) that motivates the current contribution rather than overclaiming on the cited work's behalf. [decider_v3 · claude-sonnet-5/high]
- [lin2026c3] Lin, Yulong and Bernabeu-P\'erez, Pablo and Arnav, Benjamin and Wells, Lennie and Phuong, Mary (2026). Same Question, Different Lies: Cross-Context Consistency (C\textsuperscript3) for Black-Box Sandbagging Detection. Well-formed, relevant citation for a paraphrase-consistency sandbagging-detection baseline; the bibtex is internally consistent (ICML 2026, OpenReview link) and the note clearly states its role as the baseline source, which fits a sandbagging-detection paper's methods/results needs. [decider_v3 · claude-sonnet-5/high]
- [lin2026confounds] Lin, Yulong and Bernabeu-P\'erez, Pablo and Arnav, Benjamin and Wells, Lennie and Phuong, Mary (2026). When Lie Detectors Learn Model Identity: Confounds in Black-Box Sandbagging Detection. Well-formed citation directly relevant to the paper's apparent focus on sandbagging-detector false positives; the note specifically ties it to the paper's held-out-family and non-strategically-degraded honest control design rather than a generic motivation. Bibtex entry is complete and consistent (venue, authors, URL), and the claimed takeaway (detectors learning model identity instead of sandbagging) is a precise, checkable claim an author could verify against the source. [decider_v3 · claude-sonnet-5/high]
- [anonymous2026throttling] Anonymous (2026). Can We Make an LLM Dumber, or Is It Just Playing Dumb? An Empirical Study of Believable Performance Throttling. Directly relevant prior/concurrent work: it's the closest related-work candidate for a paper measuring IRT-calibrated detection of throttled models, and the note correctly stakes out the distinction (believability study vs. this project's calibrated person-fit detector) rather than overclaiming equivalence. The flagged risk — weak-teacher LoRA collapsing MC errors to one letter in Qwen/Llama — is concrete and actionable for the Limitations section on trained imitators. [decider_v3 · claude-sonnet-5/high]
- [cacioli2026positional] Cacioli, Jon-Paul (2026). Instruction Complexity Induces Positional Collapse in Adversarial LLM Evaluation. arXiv:2604.27249.Bea has now closed the gap that sank the parent proposal: DOI resolution, DataCite metadata, and the arXiv API record are all cross-checked, and the quoted sentence is verbatim from the attached abstract. The citation is scoped honestly (one specific finding about positional collapse in prompted sandbaggers on the same model family/benchmark the paper uses, not claimed as cross-model generality), which is exactly the kind of methodological caution the paper's prompted arms need before reading TPR. [decider_v3 · claude-sonnet-5/high]