Task-Oriented Dialogues Evaluation

• Maximize task success

  • Tasks as Attribute Value Matrix (AVM) - frames to be filled during the dialogue.
  • Calculate the K-coefficient from a confusion matrix that summarizes how well an agent achieves the requirements of specific tasks: dialogue results vs scenario keys.
  • Task success is measured by the Kappa coefficient (how well the dialogue results agree with the scenario keys!)