Thinking in Boxes: 3D Editing in Real Images

Generative models have become remarkably good at changing what is in a photo: swapping a colour, adding an object or restyling a scene. However, they are far less reliable at changing where things are. An instruction such as “turn the chair to the right” gives a model no precise sense of how far, in which direction or by what angle, and the same words could describe the object moving or the camera moving and are therefore ambiguous. Newer tools that let users draw 2D layout boxes help with placement in the frame, but a 2D box cannot indicate how an object should rotate or recede into depth. Methods that lift a photo into 3D using depth maps or meshes tend to break when an object undergoes large spatial transformations.

To address this, Pradhaan S Bhat, Naveen Chandra R, Rishubh Parihar, and Venkatesh Babu Radhakrishnan from the Indian Institute of Science (IISc), with Vaibhav Vavilala (Apple), D A Forsyth (University of Illinois Urbana-Champaign), and Anand Bhattad (Johns Hopkins University), developed ‘Thinking in Boxes’, a system in which a user edits a photograph by editing 3D boxes. The user places a box around each object of interest, with an off-the-shelf 3D detector proposing them, and then moves, rotates or resizes the boxes, or moves the virtual camera as per the intended edit. The original box and the edited box together specify exactly which transformation the user wants. In order to disambiguate 3D orientation, each face of a 3D box carries a fixed colour, so that the colours facing the camera tell the model which way an object is pointing. In addition, in order to distinguish object transformations from camera transformations, the boxes rest on a depth-aligned planar floor, which stays put when an object moves but shifts when the camera moves, removing the ambiguity between the two.

The box layouts are given, together with the photograph, to FLUX-Kontext, a large pretrained generative image editing model. The editor is adapted with small low-rank (LoRA) layers while its original weights stay frozen. The model is trained in two stages: first on about 110,000 synthetic scenes consisting of two objects from a pool of 10,000, in which objects are re-posed and the camera moves between two views, and then on 10,000 image pairs drawn from real-world videos.

Although trained mostly on synthetic scenes, the system edits everyday real photographs of offices, living rooms, streets, and outdoor landscapes. It preserves the lighting, background, and identity of the edited object, and fills in sides of the objects that were hidden in the original photo. Extensive experimental evaluation demonstrates that this method outperforms existing methods on precise geometric 3D edits, especially under large transformations. This is validated through a user study which demonstrates strong preference for the results of this method over existing works.

This work will be presented at the Conference on Neural Information Processing Systems (NeurIPS 2026).

Paper: arXiv 2606.20556

Project page: thinking-in-boxes.github.io

MENU