Vol-2008
urn:nbn:de:0074-2008-1

Copyright © 2017 for the individual papers by the papers' authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors.

EVIA 2017
8th International Workshop on Evaluating Information Access

Proceedings of the 8th International Workshop on Evaluating Information Access
co-located with the 13th NTCIR Conference on the Evaluation of Information Access Technologies (NTCIR 2017)

Tokyo, Japan, December 5, 2017.

Edited by

Nicola Ferro *
Ian Soboroff **

* University of Padua, Department of Information Engineering, Via G. Gradenigo 6/B, 35131 Padova, Italy
*** National Institute of Standards and Technology, (NIST), 100 Bureau Drive, Gaithersburg, MD 20899, USA

Table of Contents

Preface i-ii
Nicola Ferro, Ian Soboroff

Evaluation Measures and Test Collections

Test Collections and Measures for Evaluating Customer-Helpdesk Dialogues 1-9
Zhaohao Zeng, Cheng Luo, Lifeng Shang, Hang Li, Tetsuya Sakai
An Interval-Like Scale Property for IR Evaluation Measures 10-15
Marco Ferrante, Nicola Ferro, Silvia Pontarollo
Evaluating Evaluation Measures with Worst-Case Confidence Interval Widths 16-19
Tetsuya Sakai

Assessment and Automatic Evaluation

Automatic Evaluation of World History Essay Using Chronological and Geographical Measures 20-23
Kotaro Sakamoto, Hideyuki Shibuki, Madoka Ishioroshi, Akira Fujita, Yoshinobu Kano, Teruko Mitamura, Tatsunori Mori, Noriko Kando
Towards Automatic Evaluation of Multi-Turn Dialogues: A Task Design that Leverages Inherently Subjective Annotations 24-30
Tetsuya Sakai
The Effect of Inter-Assessor Disagreement on IR System Evaluation: A Case Study with Lancers and Students 31-38
Tetsuya Sakai
Unanimity-Aware Gain for Highly Subjective Assessments 39-42
Tetsuya Sakai

2017-11-28: submitted by Nicola Ferro, metadata incl. bibliographic data published under Creative Commons CC0
2017-11-29: published on CEUR-WS.org |valid HTML5|