LLM4Eval 2024
Large Language Models for Evaluation in Information Retrieval 2024

Table of Contents

• Preface
  Summary: There were 26 papers submitted for peer-review to this workshop. Out of these, 7 papers were accepted for this volume as original research papers. Additionally, other 16 papers, 11 original contributions and 5 already published works, were accepted and presented during the workshop.

LLMJudge Challenge Overivew

• LLMJudge: LLMs for Relevance Judgments 1-3
  Hossein A. Rahmani, Emine Yilmaz, Nick Craswell, Bhaskar Mitra, Paul Thomas, Charles L. A. Clarke, Mohammad Aliannejadi, Clemencia Siro, Guglielmo Faggioli

Research Papers

• The Challenges of Evaluating LLM Applications: An Analysis of Automated, Human, and LLM-Based Approaches 4-18
  Bhashithe Abeysinghe, Ruhan Circi
• Exploring Large Language Models for Relevance Judgments in Tetun 19-30
  Gabriel de Jesus, Sérgio Nunes
• EXAM++: LLM-based Answerability Metrics for IR Evaluation 31-50
  Naghmeh Farzi, Laura Dietz
• A Novel Evaluation Framework for Image2Text Generation 51-65
  Jia-Hong Huang, Hongyi Zhu, Yixian Shen, Stevan Rudinac, Alessio M. Pacces, Evangelos Kanoulas
• Using LLMs to Investigate Correlations of Conversational Follow-up Queries with User Satisfaction 66-91
  Hyunwoo Kim, Yoonseo Choi, Taehyun Yang, Honggu Lee, Chaneon Park, Yongju Lee, Jin Young Kim, Juho Kim
• Evaluating RAG-Fusion with RAGElo: an Automated Elo-based Framework 92-112
  Zackary Rackauckas, Arthur Câmara, Jakub Zavrel
• Toward Automatic Relevance Judgment using Vision-Language Models for Image-Text Retrieval Evaluation 113-123
  Jheng-Hong Yang, Jimmy Lin