| Vol-1609 urn:nbn:de:0074-1609-5 Copyright © 2016 for the individual papers by the papers' authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors.

# CLEF 2016
CLEF2016 Working Notes

### Working Notes of CLEF 2016 - Conference and Labs of the Evaluation forum

### Évora, Portugal, 5-8 September, 2016.

Edited by

### Krisztian Balog *
Linda Cappellato **
Nicola Ferro **
Craig Macdonald ***

* Department of Electrical Engineering and Computer Science, University of Stavenger, 4036 Stavanger, Norway
** Department of Information Engineering (DEI), University of Padua, Via Gradenigo 6/B, 35131, Padova, Italy
*** School of Computing Science, University of Glasgow, United Kingdom

## Table of Contents

- Preface
  Krisztian Balog, Linda Cappellato, Nicola Ferro, Craig Macdonald
- Author index

### CLEFeHealth

- Task 1 of the CLEF eHealth Evaluation Lab 2016: Handover Information Extraction 1-14
  Hanna Suominen, Liyuan Zhou, Lorraine Goeuriot, Liadh Kelly
- The IR Task at the CLEF eHealth Evaluation Lab 2016: User-centred Health Information Retrieval 15-27
  Guido Zuccon, Joao Palotti, Lorraine Goeuriot, Liadh Kelly, Mihai Lupu, Pavel Pecina, Henning Müller, Julie Budaher, Anthony Deacon
- Clinical Information Extraction at the CLEF eHealth Evaluation lab 2016 28-42
  Aurélie Névéol, Kevin Bretonnel Cohen, Cyril Grouin, Thierry Hamon, Thomas Lavergne, Liadh Kelly, Lorraine Goeuriot, Grégoire Rey, Aude Robert, Xavier Tannier, Pierre Zweigenbaum
- Comparison of Several Word embedding Sources for Medical Information Retrieval 43-46
  Julie Budaher, Mohannad Almasri, Lorraine Goeuriot
- SIBM at CLEF eHealth Evaluation Lab 2016: Extracting Concepts in French Medical Texts with ECMT and CIMIND 47-60
  Chloé Cabot, Lina F. Soualmia, Badisse Dahamna, Stéfan Darmoni
- ECSTRA-INSERM @ CLEF eHealth2016-task 2: ICD10 Code Extraction from Death Certificates 61-68
  Mohamed Dermouche, Vincent Looten, Rémi Flicoteaux, Sylvie Chevret, Julien Velcin, Namik Taright
- Wrappers for Feature Subset Selection in CRF-based Clinical Information Extraction 69-80
  Mike Ebersbach, Robert Herms, Christina Lohr, Maximilian Eibl
- LITL at CLEF eHealth2016: recognizing Entities in French Biomedical Documents 81-93
  Lydia-Mai Ho-Dac, Ludovic Tanguy, Céline Grauby, Nkauj Hnub Aurore Heu Mby, Justine Malosse, Laura Rivière, Amélie Veltz-Mauclair, Marine Wauquier
- BiTeM at CLEF eHealth Evaluation Lab 2016 Task 2: Multilingual Information Extraction 94-102
  Luc Mottin, Julien Gobeill, Anaïs Mottaz, Emilie Pasche, Arnaud Gaudinat, Patrick Ruch
- KISTI at CLEF eHealth 2016 Task 3: Ranking Medical Documents using Word Vectors 103-108
  Heung-Seon Oh, Yuchul Jung
- Distributional Semantics for Medical Information Extraction 109-122
  Lautaro Quiroz, Lydia Mennes, Mostafa Dehghani, Evangelos Kanoulas
- Task3 Patient-Centred Information Retrieval: Team CUNI 123-129
  Shadi Saleh, Pavel Pecina
- The Effectiveness of Query Expansion when searching for Health related Content: InfoLab at CLEF eHealth 2016 130-142
  Ricardo Silva, Carla Lopes
- Team GU-IRLAB at CLEF eHealth 2016: Task 3 143-146
  Luca Soldaini, Will Edman, Nazli Goharian
- ECNU at 2016 eHealth Task 1: Handover Information Extraction 147-156
  Yang Song, Yun He, Hongyu Liu, Qinmin Hu, Liang He, Yueyao Wang
- ECNU at 2016 eHealth Task 3: Patient-centred Information Retrieval 157-161
  Yang Song, Yun He, Hongyu Liu, Yueyao Wang, Qinmin Hu, Liang He
- Task 3: Patient-Centered Information Retrieval, IRTask 1: ad-hoc search - TEAM ub-botswana. 162-166
  Edwin Thuma, Nkwebi Peace Motlogelwa, Tebo Leburu-Dingalo
- KDEIR at CLEF eHealth 2016: Health Documents Re-ranking Based on Query Variations 167-170
  Md Zia Ullah, Masaki Aono
- Erasmus MC at CLEF eHealth 2016: Concept Recognition and Coding in French Texts 171-178
  Erik M. van Mulligen, Zubair Afzal, Saber A. Akhondi, Dang Vo, Jan A. Kors
- Semantic Tagging and Normalization of French Medical Entities 179-192
  Jorge Vivaldi, Horacio Rodriguez, Viviana Cotik
- WHUIRGroup at the CLEF 2016 eHealth Lab Task 3 193-197
  Ruixue Wang, Wei Lu, Ke Ren
- MayoNLPTeam at the 2016 CLEF eHealth Information Retrieval Task 1 198-204
  Yanshan Wang, Stephen Wu, Hongfang Liu
- LIMSI ICD10 coding Experiments on CépiDC Death Certificate Statements 205-218
  Pierre Zweigenbaum, Thomas Lavergne

### ImageCLEF

- Overview of the ImageCLEF 2016 Medical Task 219-232
  Alba García Seco De Herrera, Roger Schaer, Stefano Bromuri, Henning Müller
- Overview of the ImageCLEF 2016 Handwritten Scanned Document Retrieval Task 233-253
  Mauricio Villegas, Joan Puigcerver, Alejandro Toselli, Joan Andreu Sanchez, Enrique Vidal
- Overview of the ImageCLEF 2016 Scalable Concept Image Annotation Task 254-278
  Andrew Gilbert, Luca Piras, Josiah Wang, Fei Yan, Arnau Ramisa, Emmanuel Dellandrea, Robert Gaizauskas, Mauricio Villegas, Krystian Mikolajczyk
- DUTh at the ImageCLEF 2016 Image Annotation Task: Content Selection 279-287
  Georgios Barlas, Maria Ntonti, Avi Arampatzis
- Using Machine Learning Techniques, Textual and Visual Processing in Scalable Concept Image Annotation Challenge 288-298
  Alexandru Cristea, Adrian Iftene
- UAEMex at ImageCLEF 2016: Handwritten Retrieval 299-303
  Miguel Ángel García Calderón, René Arnulfo García Hernández, Yulia Ledeneva
- Traditional Feature Engineering and Deep Learning Approaches at Medical Classification Task of ImageCLEF 2016 304-317
  Sven Koitka, Christoph M. Friedrich
- Subfigure and Multi-Label Classification using a Fine-Tuned Convolutional Neural Network 318-321
  Ashnil Kumar, David Lyndon, Jinman Kim, David Feng
- Image Annotation and Two Paths to Text Illustration 322-333
  Herve Le Borgne, Etienne Gadeski, Ines Chami, Thi Quynh Nhi Tran, Youssef Tamaazousti, Alexandru Lucian Gînscã, Adrian Popescu
- UDEL CIS Working Notes in ImageCLEF 2016 334-346
  Pengyuan Li, Scott Sorensen, Abhishek Kolagunda, Xiangying Jiang, Xiaolong Wang, Chandra Kambhamettu, Hagit Shatkay
- MayoBMI at ImageCLEF 2016 Handwritten Document Retrieval Task 347-355
  Sijia Liu, Yanshan Wang, Saeed Mehrabi, Dingcheng Li, Hongfang Liu
- INAOE's participation at ImageCLEF 2016: Text Illustration Task 356-362
  Luis Pellegrin, Adrián Pastor López-Monroy, Hugo Jair Escalante, Manuel Montes-Y-Gómez
- MRIM-LIG at ImageCLEF 2016 Scalable Concept Image Annotation Task 363-370
  Maxime Portaz, Mateusz Budnik, Philippe Mulhem, Johann Poignant
- CNRS TELECOM ParisTech at ImageCLEF 2016 Scalable Concept Image Annotation Task: Overcoming the Scarcity of Training Data 371-385
  Hichem Sahbi
- NovaSearch at ImageCLEFmed 2016 Subfigure Classification Task 386-398
  David Semedo, João Magalhães
- CITlab ARGUS for Keyword Search in Historical Handwritten Documents - Description of CITlab's System for the ImageCLEF 2016 Handwritten Scanned Document Retrieval Task 399-412
  Tobias Strauß, Tobias Grüning, Gundram Leifert, Roger Labahn
- IPL at CLEF 2016 Medical Task 413-420
  Leonidas Valavanis, Spyridon Stathopoulos, Theodore Kalamboukis
- Joint Learning of CNN and LSTM for Image Captioning 421-427
  Yongqing Zhu, Xiangyang Li, Xue Li, Jian Sun, Xinhang Song, Shuqiang Jiang

### LifeCLEF

- Plant Identification in an Open-world (LifeCLEF 2016) 428-439
  Hervé Goëau, Pierre Bonnet, Alexis Joly
- LifeCLEF Bird Identification Task 2016: The arrival of Deep learning 440-449
  Hervé Goëau, Hervé Glotin, Willem-Pier Vellinga, Robert Planqué, Alexis Joly
- Floristic participation at LifeCLEF 2016 Plant Identification Task 450-458
  Julien Champ, Hervé Goëau, Alexis Joly
- Bluefield (KDE TUT) at LifeCLEF 2016 Plant Identification Task 459-468
  Siang Thye Hang, Atsushi Tatsuma, Masaki Aono
- Unsupervised Individual Whales Identification: Spot the Difference in the Ocean 469-480
  Alexis Joly, Jean-Christophe Lombardo, Julien Champ, Anjara Saloma
- SeaCLEF 2016: Object Proposal Classification for Fish Detection in Underwater Videos 481-489
  Jonas Jäger, Erik Rodner, Joachim Denzler, Viviane Wolff, Klaus Fricke-Neuderth
- Improving Bird Identification using Multiresolution Template Matching and Feature Selection during Training 490-501
  Mario Lasseck
- Plant Identification System based on a Convolutional Neural Network for the LifeClef 2016 Plant Classification Task 502-510
  Sue Han Lee, Yang Loong Chang, Chee Seng Chan, Paolo Remagnino
- Feature Learning via Mixtures of DCNNs for Fine-Grained Plant Classification 511-517
  Chris McCool, Zongyuan Ge, Peter Corke
- Open-set Plant Identification Using an Ensemble of Deep Convolutional Neural Networks 518-524
  Mostafa Mehdipour Ghazi, Berrin Yanikoglu, Erchan Aptoula
- Object Detection, Classification, Tracking and individual Recognition for Sea Images and Videos 525-533
  Dávid Papp, Dániel Lovas, Gábor Szücs
- Recognizing Bird Species in Audio Recordings using Deep Convolutional Neural Networks 534-543
  Karol Piczak
- Bag of MFCC-based Words for Bird Identification 544-546
  Julien Ricard, Hervé Glotin
- Audio Based Bird Species Identification using Deep Learning Techniques 547-559
  Elias Sprengel, Martin Jaggi, Yannic Kilcher, Thomas Hofmann
- Convolutional Neural Networks for Large-Scale Bird Song Classification in Noisy Environment 560-568
  Bálint Pál Tóth, Bálint Czeba
- Deep Learning and SVM Classification for Plant Recognition in Content-Based Large Scale Image Retrieval 569-578
  Bálint Pál Tóth, Márton Tóth, Dávid Papp, Gábor Szücs
- Very Deep Residual Networks with MaxOut for Plant Identification in the Wild 579-586
  Milan Šulc, Dmytro Mishkin, Jiří Matas

### LL4IR & NewsREEL: New Ideas

- Living Ranking: from Online to Real-time Information Retrieval Evaluation 587-588
  Lamjed Ben Jabeur, Laure Soulier, Paul Mousset, Lynda Tamine
- Research-Centres Centred Living Labs 589-590
  Liadh Kelly
- Ideas for a Standard LL4IR Extension - Living Labs from a System Operator's Perspective 591-592
  Philipp Schaer, Narges Tavakolpoursaleh

### News Recommendation Evaluation Lab (NEWSREEL)

- CLEF NewsREEL 2016: Comparing Multi-dimensional Offline and Online Evaluation of News Recommender Systems 593-605
  Benjamin Kille, Andreas Lommatzsch, Frank Hopfgartner, Martha Larson, Jonas Seiler, Davide Malagoli, Andras Sereny, Torben Brodt
- Development of a News Recommender System based on Apache Flink 606-617
  Alexandru Ciobanu, Andreas Lommatzsch
- CLEF NewsREEL 2016: Image based Recommendation 618-827
  Francesco Corsini, Martha Larson
- Real-time News Recommendations using Apache Spark 628-641
  Jaschar Domann, Jens Meiners, Lea Helmers, Andreas Lommatzsch
- Recommender Systems Evaluations : Offline, Online, Time and A/A Test 642-656
  Gebrekirstos Gebremeskel, Arjen P. de Vries
- Recommender Ensembles for News Articles based on Most-Popular Strategies 657-668
  Andreas Lommatzsch, Niels Johannes, Jens Meiners, Lea Helmers, Jaschar Domann
- Optimizing a Scalable News Recommender System 669-678
  Patrick Probst, Andreas Lommatzsch
- Clicks Pattern Analysis for Online News Recommendation Systems 679-690
  Jing Yuan, Andreas Lommatzsch, Benjamin Kille

### Digital Text Forensics (PAN)

- Clustering by Authorship Within and Across Documents 691-715
  Efstathios Stamatatos, Michael Tschuggnall, Ben Verhoeven, Walter Daelemans, Guenther Specht, Benno Stein, Martin Potthast
- Author Obfuscation: Attacking the State of the Art in Authorship Verification 716-749
  Martin Potthast, Matthias Hagen, Benno Stein
- Overview of the 4th Author Profiling Task at PAN 2016: Cross-Genre Evaluations 750-784
  Francisco Manuel Rangel Pardo, Paolo Rosso, Ben Verhoeven, Walter Daelemans, Martin Potthast, Benno Stein
- Age and Gender Identification using Stacking for Classification 785-790
  Madhulika Agrawal, Teresa Gonçalves
- Authorship Clustering using Multi-headed Recurrent Neural Networks 791-804
  Douglas Bagnall
- Author Profiling Using Support Vector Machines 805-814
  Rodwan Bakkar Deyab, José Duarte, Teresa Gonçalves
- Author Profiling using SVMs and Word Embedding Averages 815-823
  Roy Khristopher Bayot, Teresa Gonçalves
- CAPS: A Cross-genre Author Profiling System 824-835
  Ivan Bilan, Desislava Zhekova
- Author Profiling using Complementary Second Order Attributes and Stylometric Features 836-845
  Konstantinos Bougiatiotis, Anastasia Krithara
- GronUP: Groningen User Profiling 846-857
  Mart Busger Op Vollenbroek, Talvany Carlotto, Tim Kreutz, Maria Medvedeva, Chris Pool, Johannes Bjerva, Hessel Haagsma, Malvina Nissim
- Using Machine Learning Algorithms for Author Profiling In Social Media 858-863
  Daniel Dichiu, Irina Rancea
- Profile-based Approach for Age and Gender Identification 864-873
  Maria José Garciarena Ucelay, Maria Paula Villegas, Dario G. Funez, Leticia C. Cagnina, Marcelo L. Errecalde, Gabriela Ramírez-De-La-Rosa, Esau Villatoro-Tello
- PANcakes Team: A Composite System of Domain-Agnostic Features For Author Profiling 874-880
  Pepa Gencheva, Martin Boyanov, Elena Deneva, Preslav Nakov, Georgi Georgiev, Yasen Kiprov, Ivan Koychev
- Statistical Semantics in Context Space : Amrita_CEN@Author Profiling 881-889
  Barathi Ganesh H B, Anand Kumar M, Soman Kp
- Author Masking through Translation 890-894
  Yashwant Keswani, Harsh Trivedi, Parth Mehta, Prasenjit Majumder
- UniNE at CLEF 2016: Author Clustering 895-902
  Mirco Kocher
- UniNE at CLEF 2016: Author Profiling 903-911
  Mirco Kocher, Jacques Savoy
- Methods for Intrinsic Plagiarism Detection and Author Diarization 912-919
  Mikhail Kuznetsov, Anastasia Motrenko, Rita Kuznetsova, Vadim Strijov
- Evaluating Safety, Soundness and Sensibleness of Obfuscation Systems 920-928
  Matthias Liebeck, Pashutan Modaresi, Stefan Conrad
- Derivative Approach for Plagiarism Source Retrieval 929-931
  Rhulani Maluleka
- Multi Feature Space Combination for Authorship Clustering 932-938
  Muharram Mansoorizadeh, Mohammad Aminian, Taher Rahgooy, Mehdy Eskandari
- Author Obfuscation using WordNet and Language Models 939-946
  Muharram Mansoorizadeh, Taher Rahgooy, Mohammad Aminian, Mehdy Eskandari
- Adapting Cross-Genre Author Profiling to Language and Corpus 947-955
  Ilia Markov, Helena Gómez-Adorno, Grigori Sidorov, Alexander Gelbukh
- SU@PAN'2016: Author Obfuscation 956-969
  Tsvetomila Mihaylova, Georgi Karadjov, Yasen Kiprov, Georgi Georgiev, Ivan Koychev, Preslav Nakov
- Exploring the Effects of Cross-Genre Machine Learning for Author Profiling in PAN 2016 970-977
  Pashutan Modaresi, Matthias Liebeck, Stefan Conrad
- Profiling Microblog Authors using Concreteness and Sentiment - Know-Center at PAN 2016 Author Profiling 978-983
  Oliver Pimas, Andi Rexha, Mark Kröll, Roman Kern
- Exploring Word Embeddings and Character N-Grams for Author Clustering 984-991
  Yunita Sari, Mark Stevenson
- Cross-Genre Author Profile Prediction Using Stylometry-Based Approach 992-999
  Shaina Ashraf, Hafiz Rizwan Iqbal, Rao Muhammad Adeel Nawab
- Author Diarization Using Cluster-Distance Approach 1000-1007
  Abdul Sittar, Hafiz Rizwan Iqbal, Rao Muhammad Adeel Nawab
- A Big Increase in Known Unknowns: from Author Verification to Author Clustering 1008-1013
  Anna Vartapetiance, Lee Gillam
- Cross-Genre Age and Gender Identification in Social Media 1014-1017
  Anam Zahid, Aadarsh Sampath, Anindya Dey, Golnoosh Farnadi
- Experiments in Authorship-Link Ranking and Complete Author Clustering 1018-1023
  Valentin Zmiycharov, Dimitar Alexandrov, Hristo Georgiev, Yasen Kiprov, Georgi Georgiev, Ivan Koychev, Preslav Nakov

### Social Book Search (SBS)

- Overview of the SBS 2016 Interactive Track 1024-1038
  Maria Gaede, Mark Michael Hall, Hugo Huurdeman, Jaap Kamps, Marijn Koolen, Mette Skov, Toine Bogers, David Walsh
- Overview of the SBS 2016 Suggestion Track 1039-1052
  Marijn Koolen, Toine Bogers, Jaap Kamps
- Overview of the SBS 2016 Mining Track 1053-1063
  Toine Bogers, Iris Hendrickx, Marijn Koolen, Suzan Verberne
- Linking Task: Identifying Authors and Book Titles in Verbose Queries 1064-1071
  Anaïs Ollagnier, Sébastien Fournier, Patrice Bellot
- Verbose Query Reduction by Learning to Rank for Social Book Search Track 1072-1078
  Messaoud Chaa, Omar Nouali, Patrice Bellot
- SBS 2016 Track Mining: Classification with Linguistic Features for Book Search Requests Classification 1079-1088
  Mohamed Ettaleb, Chiraz Latiri, Brahim Douar, Patrice Bellot
- USTB at Social Book Search 2016 Suggestion Task: Active Books Set and Re-ranking 1089-1096
  Shao-Hui Feng, Bo-Wen Zhang, Xu-Cheng Yin, Zan-Xia Jin, Jian-Lin Jin, Jian-Wei Wu, Le-Le Zhang, Hao-Jie Pan, Fan Fang, Fang Zhou
- Hood or Hypertext: A Comparison of Offline and Online Book Search Sessions 1097-1105
  Maria Gaede, Vivien Petras
- Individual Differences and Task Behaviour 1106-1114
  Mark Michael Hall, Marijn Koolen
- SBS 2016 : Combining Query Expansion Result and Books Information Score for Book Recommendation 1115-1122
  Amal Htait, Sébastien Fournier, Patrice Bellot
- BM25 for Non-Textual Modalities in Social Book Search 1123-1129
  Melanie Imhof
- Social Book Search Track: ISM@CLEF'16 Suggestion Task 1130-1135
  Ritesh Kumar, Bhanodai Guggilla, Rajendra Pamula
- Word Embedding for Social Book Suggestion 1136-1144
  Nawal Ould Amer, Philippe Mulhem, Mathias Géry, Karam Abdulahhad
- OAUC at CLEF2016 SBS Lab: Using Appeal Elements to Improve Automatic Book Recommendation - Proof of Concept 1145-1154
  Michael Preminger, Gjertrud Fludal
- Query Expansion by Word Embedding in the Suggestion Track of CLEF 2016 Social Book Search Lab 1155-1165
  Shih-Hung Wu, Yi-Hsiang Hsieh, Liang-Pu Chen, Ping-Che Yang
- I Need More Time!: The Influence of Native Language on Search Behavior and Experience 1166-1182
  Pengyi Zhang, Chang Liu, Preben Hansen
- KNOW At The Social Book Search Lab 2016 Suggestion Track 1183-1189
  Hermann Ziak, Roman Kern
- KNOW At The Social Book Search Lab 2016 Mining Track 1190-1196
  Hermann Ziak, Andi Rexha, Roman Kern

### Cultural Microblog Contextualization (CMC)

- Cultural micro-blog Contextualization 2016 Workshop Overview: data and pilot tasks 1197-1200
  Liana Ermakova, Lorraine Goeuriot, Josiane Mothe, Philippe Mulhem Jian-Yun Nie Eric Sanjuan
- LIG at CLEF 2016 Cultural Microblog Contextualization: TimeLine illustration based on Microblogs 1201-1206
  Nayanika Dogra, Philippe Mulhem, Nawal Ould Amer, Lorraine Goeuriot
- Semantic Mapping: Towards Contextual and Trend Analysis of Behaviours and Practices 1207-1225
  Murtagh Fionn
- Building a Knowledge Base using Microblogs: the Case of Cultural MicroBlog Contextualization Collection 1226-1237
  Thi Bich Ngoc Hoang, Josiane Mothe
- Tweet Contextualization using Continuous Space Vectors: Automatic Summarization of Cultural Documents 1238-1245
  Elvys Linhares Pontes, Juan-Manuel Torres-Moreno, Stéphane Huet, Andréa Carneiro Linhares
- Tweet Data mining : the Cultural Microblog Contextualization Data Set 1246-1259
  Clémentine Scohy, Yassine Rkha Chaham, Sébastien Déjean, Josiane Mothe

2016-07-25: submitted by Nicola Ferro, metadata incl. bibliographic data published under Creative Commons CC0
2016-07-25: published on CEUR-WS.org |valid HTML5|