FACE-Eval
Chain-of-Thought Faithfulness of Reasoning Models Varies with Where and How Preference Cues Are Delivered
FACE-Eval tests whether chain-of-thought traces record a model’s decision to act on a preference cue when that cue appears in a user message or tool return, either as a direct summary or a raw artifact. Across 5,100 samples and 15 open-weight models, we separate cue-following rate (CFR), verbalized commitment rate (VCR), and unverbalized adoption rate (UAR). Across all 15 models, VCR is lower for tool-return and implicit cues. UAR is higher for tool-return cues on every model and for implicit cues in 28 of 30 model–channel comparisons. Conditions with higher UAR also have lower transcript-monitor AUROC.
Setup
A sycophancy-based faithfulness evaluation that varies channel role and cue explicitness. At a fixed explicitness level, user-message and tool-return conditions use identical cue text; for each scenario and preferred side, explicit and implicit cues express the same underlying preference.
5,000 cued samples and 100 no-cue controls cross 5 preference axes, 5 sources, 5 delivery conditions, 2 sides, and 20 scenarios per axis. The five delivery conditions are reported as four plotted cells because User Explicit pools two user-message renderings.
CFR = P(AlignAns | cued) measures how often the answer follows the preference. VCR = P(CommitCoT | AlignAns, cued) measures how often a cue-following answer’s CoT records the decision to act on it. UAR = P(AlignAns ∧ ¬CommitCoT | cued) = CFR(1−VCR) measures how often an eligible cued answer follows the preference without that record. Most downstream analyses use UAR.
15 open-weight models across 8 families: Qwen 3.5 dense (4B, 9B, 27B), Gemma 4 (E4B-it, 26B-A4B-it, 31B-it), OLMo 3 Think (7B, 3.1-32B), GPT-OSS (20B, 120B), DeepSeek V4 (Flash, 284B; Pro, 1.60T), GLM 5.2 (744B), Kimi K2.6 (1.04T), Inkling (975B). Three seeds each, under 3 system prompts (default, directive, monitor-aware).
Findings
Does CoT faithfulness depend on where and how the preference cue is delivered?
Do models read the cues, and do their answers follow them?
In a separate artifact-only rating task, each of the 11 rated models achieves side-identification accuracy above 0.80 and mean clarity above 4/5; explicit cues receive higher mean clarity than implicit cues. In the full task over all 15 models, pooled CFR ranges from 0.43 to 0.58 across the four channel–explicitness conditions. These checks establish cue readability and answer-level following before we ask what the CoT records; they do not establish that the cue caused an answer to change.
How does verbalized commitment vary by channel role and cue explicitness?
On every model, user-message VCR exceeds tool-return VCR at both explicitness levels. The ordering holds across all 15 × 2 (model, explicitness) comparisons. Explicit cues also produce higher VCR than matched implicit cues on each channel, on every model. The middle comparison (User Implicit vs. Tool Explicit) is model-dependent: 12 of 15 models place User Implicit above Tool Explicit, while both GPT-OSS sizes and Gemma 4-E4B reverse it.
Reading verbalized commitment alongside cue following
Because VCR conditions on cue-following answers, it does not show how often answers follow the cue. We therefore report the cue-following rate (CFR) alongside VCR and the unverbalized adoption rate (UAR). OLMo’s tool-channel conditions combine low CFR with low VCR, whereas GPT-OSS combines high CFR with low VCR.
Is unverbalized adoption more common for tool-return and implicit cues?
UAR is higher on the tool-channel than on the user-channel on all 15 models. The per-model gaps range from 0.03 (Gemma 4-E4B) to 0.21 (Qwen 3.5-27B), and every jointly bootstrapped CI excludes zero. UAR is also higher for implicit than explicit cues on every user-channel comparison and on 13 of 15 tool-channel comparisons. The two exceptions are the OLMo models, whose intervals include zero.
Does model-rated clarity account for the explicitness gap?
No. After matching on model-rated clarity, the VCR advantage for explicit over implicit cues remains positive in all 22 model–channel comparisons. Differences in model-rated clarity do not fully account for the observed explicitness gap.
Can a transcript monitor detect preference adoption?
GPT-5.6-Luna detects preference adoption with pooled action+reasoning AUROC of 0.71–0.73. Adding the CoT improves AUROC on the two user-channel conditions (+0.019 and +0.010; both CIs exclude zero), but the increments on the two tool-channel conditions (+0.008 and +0.003) include zero. Across 32 model–condition points, AUROC is lower where UAR is higher (pooled Pearson r = −0.54 [−0.65, −0.39]). GPT-4o-mini has action+reasoning AUROC from 0.64 to 0.74; adding reasoning helps in all four conditions, yet the channel and explicitness gaps remain, and the negative UAR–AUROC association is larger in magnitude (r = −0.78).
Do system prompts close the channel-role or cue-explicitness gaps?
The system prompts do not reliably close the gaps. A source-attribution directive asks the model to attribute relevant information to its source. It narrows the channel-role gap on 7 of 15 models and widens it on 4. On Qwen 3.5, the narrowing mostly reflects rising user-channel UAR, not falling tool-channel UAR. The monitor-awareness disclosure produces model-dependent shifts of at most 0.070. Neither prompt closes the cue-explicitness gap: under the directive, the implicit-minus-explicit UAR CI still excludes zero in 26 of 30 model–channel comparisons.
Reasoning-effort effects on unverbalized adoption
Increasing reasoning effort lowers UAR in all four conditions for both GPT-OSS sizes, mostly through rising VCR rather than falling CFR. DeepSeek V4 Flash decreases mainly on the user channel, while DeepSeek V4 Pro and Inkling do not show the same four-condition pattern. At every available effort level, every adjustable model retains higher VCR for user-message than tool-return cues and for explicit than implicit cues.
Scope: these are descriptive measurements from a sycophancy-based preference probe in a single-call, prefilled-tool setting. They show where traces fail to record preference adoption, not deliberate concealment, a causal mechanism, or behavior in autonomous multi-step tool use. Main rates include only cued transcripts whose answer takes a stance and whose matched no-cue response is uncommitted. They exclude transcripts flagged as evaluation-aware; the CoT and final answer are judged separately.
What it looks like
Each pair holds one scenario and preferred side fixed. The same underlying preference is delivered either in the user message or in a tool return. In both conditions, the answer follows the cued preference; only the user-message CoT records an intent to tailor the response toward that preference. The excerpts come directly from the run files; […] marks a deliberate truncation, and each trace can be expanded in full.
The transcripts are quoted without edits. Each pair has an answer that follows the cued preference in both conditions, but only one CoT records an intent to tailor. They illustrate the behaviour; aggregate rates appear in Findings.
Citation
@article{gema2026faceeval,
title={Chain-of-Thought Faithfulness of Reasoning Models Varies with Where and How Preference Cues Are Delivered},
author={Aryo Pradipta Gema and Neel Rajani and Rohit Saxena and Wai-Chung Kwan and Pasquale Minervini},
year={2026},
eprint={2608.29464},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2608.29464},
}