Absorbed or Anchored? How Large Language Models Represent Embedded Words in Modern Standard Arabic–Egyptian Arabic Code-Switching

We study how linear probes read Modern Standard Arabic (MSA)–Egyptian Arabic code-switching in Linguistic Code-switching Evaluation (LinCE) tweets using Qwen3-8B-Base, ALLaM-7B-Instruct-preview, Gemma-2-9B and Fanar-1-9B. Probes trained on monolingual text classify 54–56% of MSA words in Egyptian contexts as Egyptian, but only 4–8% of Egyptian words in MSA contexts as MSA. Lexical and contextual controls reduce the gap, leaving 12–20 percentage points after regression and 19–28 after matching. Probes trained on balanced code-switched data recover the inserted word’s variety with 96–98% accuracy, so the asymmetry does not imply loss of variety information. Probe readings correlate with next-token variety preference. Replacing the outputs of five patching-selected attention heads with reference means changes both measures relative to layer-matched random heads, but reduces MSA-to-Egyptian misclassification by only 1–14 percentage points. Effects change direction with a different replacement mean and also occur in monolingual text. External news comments reproduce the directional readout gap (38–69 points), although the difference between lexically matched directional shifts remains unresolved. Register remains a possible contributor, and head selection and interventions use one tweet corpus. Probe-based analyses should therefore report switching directions separately and distinguish monolingual readouts from recoverable word variety.