Séminaire ayant lieu dans le cadre du Doctorat en informatique cognitive, en collaboration avec le centre de recherche CRIA              

TITRE : Perception tokens for visual reasoning

Ranjay KRISHNA

Jeudi le 19 novembre 2026 à 10h30

Local PK-5115 (Il est possible d'y assister en virtuel en vous inscrivant ici)                 

RÉSUMÉ

Today’s vision-language models can describe images fluently, yet they still struggle with the spatial and perceptual reasoning required for robotics. I will argue that visual reasoning—reasoning through points, sketches, trajectories, depth maps, and other grounded representations—will become more important than language reasoning for physical agents. It traces progress from benchmarks exposing fundamental perceptual failures to methods that teach models to reason directly in visual space. These ideas culminate in MolmoAct and MolmoAct2, open vision-language-action models that convert visual plans into robot actions. The resulting systems are interpretable, steerable, deployable zero-shot on affordable hardware, and competitive with proprietary models trained on substantially more data—advancing the goal of accessible, general-purpose robots for real-world environments.

BIOGRAPHIE

Ranjay KRISHNA is Assistant Professor in the Paul G. Allen School of Computer Science & Engineering at the University of Washington, where he co-directs the RAIVN lab. His research bridges computer vision, natural language processing, robotics, and human-computer interaction, focusing on teaching machines to perceive the visual world and interact with people by drawing on behavioral and social-science frameworks. He previously directed the multimodal and embodied AI team at the Allen Institute for AI.

RÉFÉRENCES 

Bigverdi, M., Luo, Z., Hsieh, C.-Y., Shen, E., Chen, D., Shapiro, L. G., & Krishna, R. (2025). Perception Tokens Enhance Visual Reasoning in Multimodal Language Models. Conference on Computer Vision and Pattern Recognition (CVPR).

Liu, B., Dong, Y., Wang, Y., Ma, Z., Tang, Y., Tang, L., Rao, Y., Ma, W.-C., & Krishna, R. (2025). Coarse Correspondences Boost Spatial-Temporal Reasoning in Multimodal Language Model. Conference on Computer Vision and Pattern Recognition (CVPR), 3783–3792.