Table of Contents
How to evaluate systems against human judgment on the presence of disagreement?
Evaluation measures
Evaluation measures
Alternative measures
Confusion matrix
Kappa statistic
K coefficient
K values
K values - example
K values
Computing K
Computing K
Computing K
Per-class agreement
Per-class agreement
Per-class agreement
Per-class agreement
Per-class agreement
Comparing system and coders
Conclusions
Conclusions
|