Multilingual characterization and extraction of narratives from online news: annotation guidelines

Abstract

This document provides the detailed annotation guidelines and annotation campaign design used for the creation of the datasets for the SemEval 2025 Task 10 on Multilingual Characterization and Extraction of Narratives from Online News. During this campaign, a total of 2419 documents were annotated by 35 individuals in 5 languages, covering over 96 fine-grained and 22 coarse-grained labels spanning 3 taxonomies. The annotation task is subdivided into three subtasks: Entity Framing, Narrative Classification, and Explanation of Narrative Classification. This document provides a detailed description of each task and the related taxonomies used, i.e., Named Entity Role taxonomy, and Fine-grained Narrative taxonomies covering two domains: Climate Change and Ukraine-Russia war. We also provide the annotation guidelines for all three tasks and the description of the overall annotation procedure management. The taxonomies are accompanied by examples for each label in all the 5 languages of the shared task. Additionally, full-fledged annotated examples for all tasks are provided. Furthermore, we describe Inception, the annotation platform used for the annotation process, and provide step-by-step instructions for the annotators on how to use it. Finally, based on the lessons learned we provide suggestions on how to run similar-in-nature large multilingual annotation campaigns.