Key Principle
Dialogue evaluation is a multi-dimensional problem with no single sufficient metric. Three stakeholders -- developers, users, and researchers -- drive divergent success criteria that can conflict (p. 91). The field has developed a progression of frameworks: PARADISE links objective measures to user satisfaction via regression; QoE maps the full causal chain from input factors to acceptability; Interaction Quality provides exchange-level granularity enabling real-time strategy adaptation. For open-domain systems, task success is undefined, forcing reliance on exchange-level proxies (SSA, ACUTE-EVAL) that miss dialogue-level coherence. BLEU and other MT-derived metrics systematically fail for dialogue because conversation has a one-to-many valid response space. Open-domain evaluation remains an unsolved problem.
Why This Matters
Without reliable evaluation, the field cannot distinguish genuine progress from metric gaming. Human judgements are expensive and unreliable -- "judgements can vary considerably across users and even within the same user on different occasions" (p. 92). Automated metrics are epistemically necessary but none yet captures what humans value in conversation. The credit assignment problem in modular systems means users who say the system "did not understand them" cannot distinguish ASR from NLU failure (p. 101). Developers who evaluate whole-system metrics alone risk fixing the wrong component. Lab evaluations systematically overestimate system quality because they remove environmental noise and cognitive pressure (pp. 92-93).
Good Examples
- PARADISE predicts user satisfaction via multiple linear regression: US = w1 x taskCompletion - w2 x dialogueLength, with weights learned from objective features (p. 116). This bridges objective metrics to subjective satisfaction and can serve as an RL reward function (p. 116).
- SSA (Sensibleness and Specificity Average) targets the core chatbot failure mode: generic safe responses. Specificity penalizes the degenerate strategy of replying "OK" to statements and "I don't know" to questions (p. 106). SSA correlates with perplexity, enabling automated proxy evaluation (p. 106).
- ACUTE-EVAL places two complete dialogues side by side for comparative evaluation. Humans are better at relative than absolute judgements, yielding "more fine-grained and more sensitive evaluation of multi-turn dialogues while reducing effort and costs" (p. 108).
- Interaction Quality evaluates at the exchange level during ongoing dialogue, not just at dialogue end. Expert IQ ratings correlate highly with user satisfaction, making expert annotation an economical proxy (p. 118). Predicted IQ can dynamically switch confirmation strategy mid-conversation (p. 120).
- WER is deceptive: "speech recognition errors may not always adversely affect the performance of the system as a whole as some of the errors might involve non-functional words" (p. 97). Teams that over-optimize aggregate WER waste effort on harmless errors.
Counterpoints
- BLEU fails for dialogue: An appropriate conversational response may share zero words with the input, and the one-to-many mapping between inputs and valid responses breaks the one-to-one assumption MT metrics require (p. 104). BLEU systematically penalizes creative or diverse responses.
- Automated NLG metrics "did not sufficiently reflect human ratings" (p. 100). Word-overlap, semantic similarity, and grammar metrics all fail to capture what humans judge as informative, natural, and high-quality.
- Automatic evaluation metrics "fail to take into account important aspects of multi-turn conversation such as the balance in the number of questions asked and answered" (p. 113).
- User simulators solve data quantity but introduce data quality concerns: "it was possible to distinguish simulated datasets from real datasets" (p. 95).
- Cross-conversation repetitiveness cannot be detected in single-session evaluation, creating a systematic bias toward bots that seem varied in one interaction but repeat across many (p. 109).
- Handcrafted NLG systems "outperformed Seq2Seq models in terms of measures of overall quality and complexity, length and diversity of outputs" (p. 100), suggesting neural is not universally superior.
Key Quotes
"One of the main goals of current approaches to evaluation is to have a procedure that is automatic and repeatable, and that correlates highly with human judgements" (p. 91)
"User-perceived quality is a subjective measure of the user's perceptions of their interactions with the system in relation to what they expect or desire from the interactions." (p. 116)
"The innovation in the IQ approach is that evaluation is carried out at the exchange level during the ongoing dialogue." (p. 118)
"Given an IQ score the dialogue strategy can be adapted dynamically during the ongoing interaction, thus contributing to subsequent interaction quality." (p. 120)
"There is no simple answer to this question as a number of different factors are involved" (p. 120)
Rules of Thumb
- Never use BLEU alone for dialogue evaluation. It penalizes the diverse, creative responses that make conversation engaging (p. 104).
- Read containment rate and abandonment rate jointly -- pushing containment too aggressively raises abandonment (pp. 96-97).
- Lab evaluations overestimate quality. Validate with in-the-wild data whenever possible (pp. 92-93).
- Use ACUTE-EVAL (side-by-side comparison) rather than independent Likert ratings when comparing two systems -- humans discriminate better relatively (p. 108).
- Exchange-level and dialogue-level metrics capture different things. Neither alone suffices; stack both (p. 106).
- Perplexity can serve as an automated proxy for SSA in neural models, but the correlation is imperfect (p. 106).
- The unified Venkatesh metric (engagement, coherence, domain coverage, depth, topical diversity) achieved r=0.66 correlation with user ratings on Alexa Prize data (p. 110).
- Crowdsourced evaluation is as reliable as lab evaluation for task performance and ranking, at lower cost (p. 95).
Related References
- neural-failure-modes.md -- the failure modes these evaluation methods attempt to measure
- hybrid-architectures.md -- why handcrafted components sometimes outperform neural on quality metrics
- pipeline-architecture.md -- the modular architecture that creates the credit assignment problem
- neural-dialogue-systems.md -- the systems being evaluated by these frameworks