← Back to selected projects
Mechanistic interpretability · Arabic speech · 2026

Inside the Nativeness Axis

How wav2vec2-XLS-R and Whisper Large v3 internally represent native and non-native Arabic speech—and what targeted interventions reveal about bias, phonology, and possible debiasing.

Muhammad S. AbdoB659: Analysis and Modeling of Atypical SpeechIndiana University Bloomington

Research question and contribution

The paper asks whether speech foundation models merely correlate with speaker nativeness or build a distinct, causally meaningful representation of it. The analysis compares a self-supervised encoder, wav2vec2-XLS-R-300M, with the supervised ASR encoder in Whisper Large v3. Because the models have different objectives, convergence across them is evidence that the pattern reflects acoustic-phonological structure in the input rather than a quirk of one architecture.

The central contribution is a mechanistic pipeline that goes beyond reporting group-level recognition gaps. It identifies a linear “nativeness direction,” removes that direction from the representation, checks what information is lost, controls statistically for prosody, and traces processing differences to Arabic phoneme categories.

80
speakers: 40 native and 40 non-native, balanced by gender
4,094
utterances from 51 fixed Arabic prompts
6.5h
approximately, in the L2-KSU corpus

Data and analytical pipeline

All speakers read the same prompts, including sentences containing Arabic emphatics, uvulars, and pharyngeals. Holding lexical content constant helps isolate stable pronunciation and speaker patterns. Utterance activations were mean-pooled into speaker-level representations at every layer.

Layer-wise probingLinear SVMs test how accurately each layer separates native from non-native speakers under strict speaker-held-out evaluation.
Causal ablationThe learned nativeness direction is projected out, then nativeness and gender probes are rerun.
Partial RSARepresentational similarity tests whether nativeness geometry survives control for duration, energy, pitch, voicing, and spectral measures.
Attention entropyHead-level entropy tracks whether processing differs by group and whether the gap changes for Arabic phoneme categories.

Main findings

1.000
peak nativeness probe accuracy at wav2vec2 layer 5 and Whisper layer 8
0.675 / 0.800
nativeness accuracy after ablation in wav2vec2 / Whisper; gender stayed unchanged
  • A single linear direction separates all 40 native from all 40 non-native speakers with zero overlap at the peak layers.
  • Removing that direction selectively weakens nativeness decoding while preserving gender decoding, establishing causal specificity rather than probe-only correlation.
  • Nativeness remains significant after controlling for prosody: partial RSA is 0.176 for wav2vec2 and 0.132 for Whisper, both with permutation p < .001.
  • Native speech receives more focused attention overall. The group gap changes reliably for sentences containing emphatic and uvular consonants, linking the effect to Arabic phonology.
  • Residual decoding after ablation suggests distributed encoding in wav2vec2 and a more compact representation in Whisper, implying different debiasing strategies.
Interpretation. “Nativeness” is not treated as one simple demographic label. It emerges from a hierarchy in which prosody organizes much of speaker space, while a separate acoustic-phonological dimension survives prosody control and can be selectively weakened.

Visual results and commentary

Layer-wise nativeness probe accuracy for wav2vec2 and Whisper
Figure 1. Layer-wise nativeness probe accuracy
What it shows: the diagnostic signal peaks in intermediate layers—layer 5 for wav2vec2 and layer 8 for Whisper—rather than simply increasing toward the output. This supports a layered account in which phonological speaker information is consolidated before the models' final, more task-specific stages.
wav2vec2 speakers projected onto the nativeness directionWhisper speakers projected onto the nativeness direction
Figure 2. Speakers projected onto the learned nativeness direction
What it shows: native and non-native speakers occupy non-overlapping regions in both models under leave-one-speaker-out evaluation. The separation is geometric and generalizes to held-out speakers; it is not caused by repeated utterances leaking between train and test sets.
Per-head attention entropy differences in wav2vec2Per-head attention entropy differences in Whisper
Figure 3. Attention-entropy differences across heads
What it shows: wav2vec2 exhibits a nearly uniform native-versus-non-native difference across attention heads, whereas Whisper is more mixed. That contrast fits their objectives: self-supervision preserves broad acoustic detail, while supervised ASR partially compresses speaker variation.
wav2vec2 entropy gap by sentenceWhisper entropy gap by sentence
Figure 4. Entropy gap across the ten analyzed sentences
What it shows: the processing difference is present across every sentence, not driven by a single prompt. Sentence tags make it possible to connect variation in the gap to the presence of emphatic, uvular, and pharyngeal segments.
wav2vec2 entropy gap across layers by phoneme categoryWhisper entropy gap across layers by phoneme category
Figure 5. Phoneme-category effects across layers
What it shows: emphatic- and uvular-containing sentences consistently alter the native/non-native attention gap, while the pharyngeal contrast is weak. The pharyngeal null result is interpreted cautiously because eight of ten prompts contain pharyngeals, leaving an underpowered comparison condition.
wav2vec2 nativeness and prosody layer specializationWhisper nativeness and prosody layer specialization
Figure 6. Prosody and nativeness peak at different depths
What it shows: prosody peaks earlier than nativeness—three layers earlier in wav2vec2 and one layer earlier in Whisper. Together with partial RSA and the prosody-only baseline, this argues that the nativeness axis is not merely slower speech, pitch, or duration under another name.
Nativeness and gender accuracy before and after causal ablation
Figure 7. Selective effect of causal ablation
What it shows: projecting out the nativeness direction causes a substantial nativeness-accuracy drop in both models while gender accuracy remains identical. The intervention therefore targets a specific representational direction rather than broadly damaging speaker information.

Limitations and next steps

  • The corpus covers one learner population, with non-native speakers primarily from Central and West African backgrounds; other L1 groups may produce different geometry.
  • Eighty speakers support the core analysis but limit finer dialect- and L1-specific comparisons.
  • Prosody controls do not identify every segmental cue contributing to the axis; targeted phoneme experiments are still needed.
  • The ablation study measures information removal, not downstream transcription quality. The next fairness test is whether weakening the axis improves recognition equity without harming ASR performance.

Overall conclusion: two models with opposing training objectives converge on a structured nativeness representation. Its selective causal role, persistence after prosody control, and interaction with Arabic-specific phonology make it a promising target for more precise fairness interventions.