Stance Labels Fail When They Matter Most: The Projection Problem in Stance Detection

arXiv cs.CL / 3/26/2026

💬 OpinionSignals & Early TrendsIdeas & Deep AnalysisModels & Research

Key Points

  • The paper argues that stance detection’s common three-way labeling scheme (Favor/Against/Neutral) fails to capture multi-dimensional attitudes toward complex targets.
  • It introduces the “projection problem,” where annotators compress multi-dimensional beliefs into a single label using different weighting schemes, causing disagreement that is not due to misunderstanding.
  • The study finds a conditional effect: when a text’s dimensions are consistent, stance labels agree relatively well regardless of annotator weighting, and three-way annotation performs adequately.
  • When a text’s dimensions conflict, overall label agreement collapses, even though agreement on individual dimensions remains strong, indicating the source of error is the projection to a single label.
  • A pilot analysis on SemEval-2016 Task 6 confirms a crossover pattern between label agreement and dimensional agreement, including large shifts for dimensions like Policy.

Abstract

Stance detection is nearly always formulated as classifying text into Favor, Against, or Neutral -- a convention inherited from debate analysis and applied without modification to social media since SemEval-2016. But attitudes toward complex targets are not unitary: a person can accept climate science while opposing carbon taxes, expressing support on one dimension and opposition on another. When annotators must compress such multi-dimensional attitudes into a single label, different annotators weight different dimensions -- producing disagreement that reflects not confusion but different compression choices. We call this the \textbf{projection problem}, and show that its cost is conditional: when a text's dimensions align, any weighting yields the same label and three-way annotation works well; when dimensions conflict, label agreement collapses while agreement on individual dimensions remains intact. A pilot study on SemEval-2016 Task 6 confirms this crossover: on dimension-consistent texts, label agreement (Krippendorff's \alpha = 0.307) exceeds dimensional agreement (\alpha = 0.082); on dimension-conflicting texts, the pattern reverses -- label \alpha drops to 0.085 while dimensional \alpha rises to 0.334, with Policy reaching 0.572. The projection problem is real -- but it activates precisely where it matters most.