Probing Relational Knowledge in Language Models via Word Analogies

Abstract

Understanding relational knowledge plays an integral part in natural language comprehension. When it comes to pre-trained language models (PLM), prior work has been focusing on probing relational knowledge this by filling the blanks in pre-defined prompts such as The capital of France is ---''. However, these probes may be affected by the co-occurrence of target relation words and entities (e.g. capital″, France″ and Paris″) in the pre-training corpus. In this work, we extend these probing methodologies leveraging analogical proportions as a proxy to probe relational knowledge in transformer-based PLMs without directly presenting the desired relation. In particular, we analysed the ability of PLMs to understand (1) the directionality of a given relation (e.g. Paris-France is not the same as France-Paris); (2) the ability to distinguish types on a given relation (both France and Japan are countries); and (3) the relation itself (Paris is the capital of France, but not Rome). Our results show how PLMs are extremely accurate at (1) and (2), but have clear room for improvement for (3). To better understand the reasons behind this behaviour and mistakes made by PLMs, we provide an extended quantitative analysis based on relevant factors such as frequency.

Type
Publication
Findings EMNLP 2022
Kiamehr Rezaee
Kiamehr Rezaee
PhD Student
Jose Camacho-Collados
Jose Camacho-Collados
Professor & UKRI Future Leaders Fellow