Seminar: "Evaluating Pluralism in LLMs and Understanding Ambiguity in NLP Tasks"

Invited Speaker 1: Laura Majer (University of Zagreb)

Title: “Evaluating Pluralism in LLMs through Latent Perspectives”

Abstract: In NLP, linguistic diversity and disagreement are mainly acknowledged through two paradigms: perspectivism and pluralism. Perspectivism argues for preserving unaggregated labels throughout the NLP pipeline, where ‘perspective’ is usually represented as a set of static labels, and open-ended text analysis is rare. In parallel, pluralism refers to the analysis of LLM diversity, with prior work showing that LLMs exhibit notable homogeneity in generated text. This talk will showcase current work that extends the evaluation of pluralism in LLMs by estimating latent perspectives from open-ended text. By analysing the overlap between human and LLM-generated book reviews at multiple levels of abstraction, the goal is to identify the specific ways LLMs constrain diversity in their output and to propose potential remedies.

Bio: Laura is a third-year PhD student at the University of Zagreb, focusing on Natural Language Processing. She previously worked on various subjective tasks, a European fact-checking initiative, and synthetic data evaluation. Currently, she is primarily interested in perspectivism, pluralism, and annotation setups in subjective scenarios. In parallel, she is employed as a Research Engineer in industry.

Invited Speaker 2: Ieva Staliūnaitė (University of Cambridge)

Title: “Distinguishing Wanted and Unwanted Uncertainty: Ambiguity in NLP Tasks”

Abstract: Annotators sometimes disagree, and models are sometimes uncertain. This happens for different reasons, and is manifested differently across NLP tasks. This talk will discuss the different interactions between model probability distributions and human label variation. In automated fact-checking, underspecification is a powerful tool for the potential of misinformation. In Machine Translation, bias can lead to very confident predictions even in cases where gender is ambiguous. In LLM generations, uncertainty is canonically taken to be taken as a signal for incorrectness, however this varies depending on the calibration of the model and the ambiguity of the task. Across tasks, disagreement can be both valid and noisy, and separating the two is important.

Bio: Ieva is a fourth year PhD student at the University of Cambridge, working on Natural Language Processing. Her work focuses on fact-checking, ambiguity and uncertainty. She has a background in Linguistics from Utrecht University. She is currently working on a project on LLM correctness prediction and uncertainty quantification at the Alan Turing Institute. Ieva is open to job opportunities in academia and industry post-PhD!