DIR Workshop

Concluding remarks of our last session of DIR 2021 1/2. Future of IR metrics? #DIR2021 @gkazai

Meta question: How to access a metric? What would be a good metric for a metric? #DIR2021 @gkazai

Offline eval looks easy? Looks simple, what could go wrong? Think again #DIR2021 @gkazai @arjenpdevries @maria_heuss

How are the offline eval pipelines used? Ranking experiments are the customers. Ranking experiment's success is judged by the eval pipeline #DIR2021 @gkazai @arjenpdevries @maria_heuss

A system's perspective on IR evaluation #DIR2021 @gkazai @arjenpdevries @maria_heuss

How does evaluation look like with a real search engine? An industry eval landscape from Bing #DIR2021 @gkazai @arjenpdevries @maria_heuss

How to setup an offline IR eval pipeline? A guideline #DIR2021 @gkazai @arjenpdevries @maria_heuss

Looking for references for Offline IR evaluation? Check out these books #DIR2021

Online/user-related metrics are harder to gather. Logs are noisy #DIR2021

Good overview of the Metric landscape, from a practical/industry perspective. How to define it offline v/s online #DIR2021 @gkazai @arjenpdevries @maria_heuss

Ex: of a 'Metric' and it's 'proxy'. User retention v/s Relevance #DIR2021 @gkazai @arjenpdevries @maria_heuss

How do you 'operationalize' a metric? Ans: Use a proxy. #DIR2021 @gkazai @arjenpdevries @maria_heuss

What is a 'metric'? #DIR2021 @gkazai @arjenpdevries @maria_heuss

How is an IR system evaluated @gkazai #DIR2021 @arjenpdevries

History of IR evaluation @gkazai #DIR2021

DIR 2021 1/2 keynote kickstart by @arjenpdevries & @maria_heuss #DIR2021 (1/n)

📢 Closing Week DIR 2021 1/2 - One day to go! 🚨 ⚡️One day to go for the closing session of DIR 2021 1/2. Don't miss it!! 🤝Join us to learn about: Industry perspective on IR evaluation by @gkazai from Microsoft. 🌎 Register here: meetup.com/SEA-Search-Eng… @maliannejadi @mdr

🚨📢 Last week - Keynote Announcement 🚨📢 For week 4 of DIR 2021 1/2, we have a very interesting keynote talk by Gabriella Kazai @gkazai from Microsoft. Topic of the talk: IR Evaluation - An Industry Perspective. 🌎Register here meetup.com/SEA-Search-Eng… @maliannejadi @mdr

This concludes week 3 of DIR 2021 1/2. This was the last week of lightning talks. ⚡️ Hope you all enjoyed it. Next week is the concluding week for DIR 2021, stay tuned for the latest updates. 😇

Want to try out the method and extend it? Link to code, and conclusions @HarrieOos #DIR2021