AnimeScore: A Preference-Based Dataset and Framework for Evaluating Anime-Like Speech Style

arXiv cs.CL / 3/13/2026

💬 OpinionTools & Practical UsageModels & Research

共有:

Key Points

AnimeScore proposes a preference-based framework using pairwise ranking to evaluate anime-like speech, addressing the lack of an absolute MOS-style metric.
The study collects 15,000 pairwise judgments from 187 evaluators with free-form descriptions to capture nuanced perceptions beyond simple pitch cues.
Acoustic analysis reveals that anime-likeness arises from controlled resonance shaping, prosodic continuity, and deliberate articulation rather than high pitch alone.
Handcrafted features reach 69.3% AUC, while SSL-based ranking models achieve up to 90.8% AUC, indicating strong predictive performance and potential as a reward signal for generative speech optimization.
The dataset and framework provide a practical metric for evaluating anime-like speech and can guide improvements in anime-style voice generation.

Abstract

Evaluating 'anime-like' voices currently relies on costly subjective judgments, yet no standardized objective metric exists. A key challenge is that anime-likeness, unlike naturalness, lacks a shared absolute scale, making conventional Mean Opinion Score (MOS) protocols unreliable. To address this gap, we propose AnimeScore, a preference-based framework for automatic anime-likeness evaluation via pairwise ranking. We collect 15,000 pairwise judgments from 187 evaluators with free-form descriptions, and acoustic analysis reveals that perceived anime-likeness is driven by controlled resonance shaping, prosodic continuity, and deliberate articulation rather than simple heuristics such as high pitch. We show that handcrafted acoustic features reach a 69.3% AUC ceiling, while SSL-based ranking models achieve up to 90.8% AUC, providing a practical metric that can also serve as a reward signal for preference-based optimization of generative speech models.