CST-WM

Abstract

When the sensor moves with the agent, the agent's own action and what it observes stop being independent. Any predictor trained on that data faces a choice it is never told about: learn how the action moves the body and how the world answers, or learn the shortcut and read the answer straight off the command. Both fit the data. Only one is the relation you meant.

We call the second outcome causal hallucination. In embodied visual tracking it takes a precise and damaging form: the model writes the current action directly into the latent state that represents whether the target is still visible, instead of letting the action reach that state only through robot motion and the change in observation it produces. The rollouts stay plausible and the semantics go wrong — an agent planning on them cannot separate an action that keeps the target in view from one that loses it, and re-acquisition after occlusion is assigned no meaning at all.

CST-WM writes the relation among robot, target and environment into the transition itself. The latent state is decomposed into a target-evidence branch, a robot branch and an observation branch, and the factorization blocks any action-carrying representation from entering the target-evidence update while action remains available to robot motion and observation. The constraint is architectural rather than a penalty term, and it is verified rather than assumed: we measure the leakage Jacobian |∂H/∂a| and intervene by swapping actions at rollout time. Combined with rollout-based model-predictive control, one objective yields both stable following and temporary re-acquisition.

The idea

An agent that follows a person spends most of its time in the easy case. The moments that decide the episode are the other ones — the target occluded, drifting out of frame, or confusable with someone else — where the robot has to commit to a movement before reliable evidence comes back. The value of an action there is not how it moves the robot; it is what the action will do to the evidence.

That framing rules out both of the usual answers. A reactive tracker is myopic: once the target disappears it has nothing left to plan with. But a generic action-conditioned world model is not the answer either, because the very entanglement it is trained on offers it the shortcut. Action should reach target evidence by moving the body and changing what the body will see — not by being written into the target state directly. A predictor that takes the shortcut still produces plausible futures while conflating control with evidence, and no accuracy metric computed on those futures will tell you which model you have.

Method

Each egocentric frame yields a compact target-evidence token H — an open-vocabulary detection confidence plus the normalized target box area, i.e. whether the target is observable and how large it appears — together with a visual latent Z from a frozen VAE and a robot state x integrated from the action history. Crucially, H is a planning-oriented summary of observability and apparent scale, not a full external state estimate, so no privileged geometry is needed at test time.

The transition is a diffusion model whose three branches are denoised in a fixed order under strict masking: H is updated without reading the action; the robot branch absorbs the action and becomes the sole action carrier; the observation branch then fuses both. The computational graph contains no direct edge from the action to target evidence, and the zero Jacobian ∂Hℓ+1/∂aℓ = 0 serves as a diagnostic that the constraint really holds. Planning is model-predictive control with the Cross-Entropy Method, scoring rollouts directly on the predicted evidence tokens — so the planner never has to decode a full image at every horizon.

Is apparent scale a usable stand-in for distance?

Within the 1–3 m following range the protocol targets, the detector-derived evidence score rises monotonically with closeness and only saturates at very short range — enough to regulate following distance without ever estimating metric distance. Training it with an auxiliary in-range distance label (simulator-only, and never used at test time) closes most of the gap to a direct-distance oracle: 0.70 DRS versus the oracle's 0.72.

Demo

Real-world deployment on a physical robot, recorded across four environments that differ in lighting and layout, plus one clip isolating a target-loss and re-acquisition episode. Pick a segment:

Bright indoor. Office corridors under normal lighting. The robot holds a steady following distance down a straight corridor while the target walks away from it.

All four environment segments, uncut, in one file: full video (2:58).

Results

Evaluated on EVT-Bench and Habitat 3.0 from egocentric observations only, with scene-disjoint splits and three seeds. The adapted NWM baseline shares our observation interface, planning horizon, candidate budget, and action bounds — so the only remaining difference is the latent transition design.

Tracking

Recovery after temporary target loss

Does the structural constraint actually hold?

Three diagnostics say yes, and that it matters. The Jacobian of predicted target evidence with respect to the current action falls to 0.001 for CST-WM against 0.112 for the adapted NWM. Under an action-swap intervention the predicted evidence stays put (variance 0.006 vs. 0.084). And in a controlled check where the humanoid is held still while the robot moves, target-evidence stability reaches 0.98 while observation sensitivity stays at 0.84 — the model is selectively insensitive to action in the right branch, not numb to it everywhere.

Citation

@article{hu2026cstwm,
  title   = {CST-WM: A Causally Structured World Model for Embodied Visual Tracking},
  author  = {Hu, Junyi and Yuan, Shuaihang and Fang, Yi},
  journal = {arXiv preprint},
  year    = {2026}
}