Felix Valdivieso

Gloria Fuertes.

Text is just one way to communicate with an AI system. But Physical AI should understand the world the way humans do; for example, through spatial, non-verbal interaction in real environments. When we communicate about the physical world, we naturally use spatial cues like pointing and framing areas of interest. In this experiment, we explore how gestures can become a spatial “prompt” that is interpreted by Newton, our Physical AI foundation model. That way, Newton can understand what the user is interested in. Using only a standard camera, our Head of Product Design, Lauren Bedal, directs Newton’s attention in real time by pointing at herself, framing objects in the scene, and moving in and out of focus. Newton applies reasoning only to the highlighted areas. The key idea: interaction itself is a modality for AI. Prompts don’t have to be static or textual; they can be spatial, continuous, and interactive. No special hardware required. Just Newton, a camera, and the way people already show things to each other.