| Vol-3752 urn:nbn:de:0074-3752-3 Copyright © 2024 for the individual papers by the papers' authors. Copyright © 2024 for the volume as a collection by its editors. This volume and its papers are published under the Creative Commons License Attribution 4.0 International (CC BY 4.0).

# LLM4Eval 2024
Large Language Models for Evaluation in Information Retrieval 2024

### Proceedings of The First Workshop on Large Language Models for Evaluation in Information Retrieval (LLM4Eval 2024)
co-located with 10th International Conference on Online Publishing (SIGIR 2024)

### Washington D.C., USA, July 18, 2024.

Edited by

### Clemencia Siro *
Mohammad Aliannejadi *
Hossein A. Rahmani **
Nick Craswell ***
Charles L. A. Clarke ****
Guglielmo Faggioli *****
Bhaskar Mitra ***
Paul Thomas ***
Emine Yilmaz *,******

* University of Amsterdam, Faculty of Science, Informatics Institute, Amsterdam, The Netherlands
** University College London, Dept of Computer Science, London, UK
*** Microsoft
**** University of Waterloo, School of Computer Science, Waterloo, Ontario, Canada
***** University of Padua, Dept of Information Engineering, Padua, Italy
****** Amazon

## Table of Contents

- Preface
  Summary: There were 26 papers submitted for peer-review to this workshop. Out of these, 7 papers were accepted for this volume as original research papers. Additionally, other 16 papers, 11 original contributions and 5 already published works, were accepted and presented during the workshop.

### LLMJudge Challenge Overivew

- LLMJudge: LLMs for Relevance Judgments 1-3
  Hossein A. Rahmani, Emine Yilmaz, Nick Craswell, Bhaskar Mitra, Paul Thomas, Charles L. A. Clarke, Mohammad Aliannejadi, Clemencia Siro, Guglielmo Faggioli

### Research Papers

- The Challenges of Evaluating LLM Applications: An Analysis of Automated, Human, and LLM-Based Approaches 4-18
  Bhashithe Abeysinghe, Ruhan Circi
- Exploring Large Language Models for Relevance Judgments in Tetun 19-30
  Gabriel de Jesus, Sérgio Nunes
- EXAM++: LLM-based Answerability Metrics for IR Evaluation 31-50
  Naghmeh Farzi, Laura Dietz
- A Novel Evaluation Framework for Image2Text Generation 51-65
  Jia-Hong Huang, Hongyi Zhu, Yixian Shen, Stevan Rudinac, Alessio M. Pacces, Evangelos Kanoulas
- Using LLMs to Investigate Correlations of Conversational Follow-up Queries with User Satisfaction 66-91
  Hyunwoo Kim, Yoonseo Choi, Taehyun Yang, Honggu Lee, Chaneon Park, Yongju Lee, Jin Young Kim, Juho Kim
- Evaluating RAG-Fusion with RAGElo: an Automated Elo-based Framework 92-112
  Zackary Rackauckas, Arthur Câmara, Jakub Zavrel
- Toward Automatic Relevance Judgment using Vision-Language Models for Image-Text Retrieval Evaluation 113-123
  Jheng-Hong Yang, Jimmy Lin

2024-08-19: submitted by Guglielmo Faggioli, metadata incl. bibliographic data published under Creative Commons CC0
2024-09-02: published on CEUR Workshop Proceedings (CEUR-WS.org, ISSN 1613-0073) |valid HTML5|