Séminaire ayant lieu dans le cadre du Doctorat en informatique cognitive, en collaboration avec le centre de recherche CRIA            

TITRE : Mechanistic Emergence of Grounding

Ziqiao MA

Jeudi le 17 septembre 2026 à 10h30

Local PK-5115 (Il est possible d'y assister en virtuel en vous inscrivant ici)              

RÉSUMÉ

What does it mean for a language model to actually ground a word in the world? Much of the current discussion treats grounding as an observed correspondence: a word like “horse” aligns with the right image region, so the model appears grounded. But correlation alone leaves a deeper question unanswered: how does this connection arise during learning, and what inside the model actually implements it? In this talk, I will approach grounding as a process rather than a property of a finished model. Starting from a minimal setting inspired by child language learning, we trace how models learn to connect linguistic symbols with corresponding information from the environment. Interestingly, models initially rely heavily on simple co-occurrence statistics, but later develop mechanisms that go beyond these surface correlations. By following information flow across training and intervening on individual attention heads, we find that grounding becomes concentrated in specialized aggregation mechanisms in the model's middle layers. I will then show how this picture extends from controlled experiments to vision-language models, and discuss what it suggests about language learning, multimodal model design, hallucination, and the broader symbol grounding debate.

BIOGRAPHIE

Ziqiao MA is a Member of Technical Staff at Thinking Machines Lab. He obtained his Ph.D. at the University of Michigan. His research stands at the intersection of language, interaction, and embodiment from a scalable and cognitive perspective, with the goal of grounding and aligning language agents to non-linguistic modalities and rich interactive contexts. He received an Outstanding Paper Award at ACL 2023, and an Amazon Alexa Prize Award.

RÉFÉRENCES

 Wu, S., Ma, Z., Luo, X., Huang, Y., Torres-Fonseca, J., Shi, F., & Chai, J. (2025). The Mechanistic Emergence of Symbol Grounding in Language Models. ICML. https://arxiv.org/abs/2510.13796 

Bick, A., Xing, E., & Gu, A. (2025). Understanding the Skill Gap in Recurrent Language Models: The Role of the Gather-and-Aggregate Mechanism. Proceedings of the 42nd International Conference on Machine Learning, PMLR 267, 4324–4344. https://arxiv.org/abs/2504.18574

Wang, L., Li, L., Dai, D., Chen, D., Zhou, H., Meng, F., Zhou, J., & Sun, X. (2023). Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learning. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 9840–9855. https://aclanthology.org/2023.emnlp-main.609/ 

Bisk, Y., Holtzman, A., Thomason, J., Andreas, J., Bengio, Y., Chai, J., Lapata, M., Lazaridou, A., May, J., Nisnevich, A., Pinto, N., & Turian, J. (2020). Experience Grounds Language. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, 8718–8735. https://aclanthology.org/2020.emnlp-main.703/

Bousselham, W., Petersen, F., Ferrari, V., & Kuehne, H. (2024). Grounding Everything: Emerging Localization Properties in Vision-Language Transformers. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3828–3837. https://arxiv.org/abs/2312.00878

Szot, A., Mazoure, B., Attia, O., Timofeev, A., Agrawal, H., Hjelm, D., Gan, Z., Kira, Z., & Toshev, A. (2025). From multimodal LLMs to generalist embodied agents: Methods and lessons. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 10644–10655. https://arxiv.org/abs/2412.08442 

Goulet, N., Massé, A. B., & Abdendi, M. (2025). Approaching the Source of Symbol Grounding with Confluent Reductions of Abstract Meaning Representation Directed Graphs. arXiv preprint arXiv:2508.11068.

Vincent‐Lamarre, P., Massé, A. B., Lopes, M., Lord, M., Marcotte, O., & Harnad, S. (2016). The latent structure of dictionaries. Topics in cognitive science, 8(3), 625-659.