Lost in Slots: Revisiting Object-Centric Representations in the Era of Foundation Models

Humans effortlessly perceive the world as a structured collection of objects — recognising that a red ball and a blue cube are distinct entities, each with their own attributes and relationships. This ability rests on a mechanism known as the binding problem: correctly associating distributed features like colour, shape, and location with the right objects. Even for humans, this is non-trivial, breaking down as scenes grow complex. Modern AI models exhibit far more pervasive failures — struggling with spatial reasoning, misattributing properties, and hallucinating objects. As vision-language models (VLMs) are deployed in settings where precise object-level understanding matters, solving binding has become a central challenge.

Object-centric representation learning (OCL) has emerged as a principled proposed solution. Slot-based OCL methods explicitly decompose a scene into structured object representations called ‘slots’, aiming to cleanly separate objects while retaining their attributes. Despite showing promise in compositional generalisation and visual reasoning, a critical question has gone unanswered: do slot-based representations actually provide a better substrate for binding than the dense patch features already learned by modern foundation models?

To answer this, a team of researchers—Priyam Dey, Aditya Sahdev, Omkar Mahesh Kashyap, and R Venkatesh Babu from the Vision & AI Lab, Indian Institute of Science—systematically evaluated slot-based and dense representations across the three core components of binding: scene segregation, object representation quality, and compositional reasoning, integrating both into pretrained foundation large language models (LLMs) via the large language vision assistant (LLaVA) instruction-tuning framework and evaluating on diverse visual question answering (VQA), grounding, and hallucination benchmarks.

The results are consistently unflattering for slot-based methods: dense representations outperform OC counterparts across all benchmarks, backbones, and LLM scales. Increasing slot count does not close the gap, and stronger LLMs amplify the advantage of dense features rather than helping OC representations catch up. Probing deeper, the team identifies three root causes: dense features already encode richer implicit scene grouping than explicit slot assignments; slot encoding discards fine-grained object property information that patch features preserve; and dense representations induce stronger spatial binding signals within the LLM than slot-encoded features do.

Yet, dense representations are not without fault. When task-irrelevant image regions are masked out before being passed to the VLM, performance consistently improves, revealing that even rich, expressive dense features carry visual noise that interferes with compositional reasoning. Digging deeper, a causal mediation analysis that directly intervenes on frozen patch token representations shows that this interference arises not just from the visual encoder, but also from within the LLM itself, which remains susceptible to cross-object distraction even when presented with high-quality visual tokens. Binding, it turns out, is a two-sided failure: neither the visual representation nor the language model alone is the culprit, motivating co-design across both components.

Together, these findings challenge the assumption that discrete slot-based abstraction is the right inductive bias for binding, and motivate future architectures that enforce selective, explicit binding while preserving the rich visual information that foundation models depend on.This research will be presented at NeurIPS 2026.

MENU