Imagine watching a basketball clip. You ask which player took the shot, then follow up: “Where did he move next?” That tiny word, “he,” asks quite a lot of a model. It needs to carry a person, a place, and a moment from one answer into the next.
One more question
That first answer might only need one frame. Now let the player slip behind someone else, or switch to another camera. “He” still means the same person. The model has to carry that connection through the change, even when a fresh look at the image would leave it unsure.
This is the problem behind LMM-Track4D. The “4D” here means three-dimensional space evolving over time. The project asks how a large multimodal model can answer questions about a moving scene while maintaining a consistent account of the objects in it.
Two questions. Keep an eye on who “his” refers to.
The first question picks out player #6 at a particular moment.
The follow-up still refers to #6. What changes is the requested interval: 0.5–1.2 s. The target should carry over from the first answer.
Adapted from the two questions in Figure 1. The character stands for the player’s identity.
A path we can check
LMM-Track4D pairs ordinary answers with structured 3D trajectories when the question calls for them. A trajectory ties positions to time. Instead of leaving the whole answer inside a sentence, it gives us a path that can be compared with the scene.
That changes what we can ask of the model. “He moved across the court” is easy to read, but leaves plenty unspecified. A sequence of positions lets us examine which target was followed, which part of the clip the answer covers, and whether the movement stays coherent.
The task supports an entire short clip or a requested segment of a longer clip. That distinction matters in a conversation: a follow-up can narrow the time window without changing the person we are talking about.
Carrying the scene forward
Start with the camera. A player can move left in the image because they ran, because the camera moved, or both. Ray–Time Geometry Encoding (RTGE) adds geometry and time to the visual representation, giving the model more than image position to work with.
Then we need somewhere to keep track of the player. The persistent TRK state token carries dynamic information from one turn to the next. Think of a small working note beside the video: who we are following, and what has happened to them so far. In the model, this “note” is a learned representation.
When the question asks for a path, OSK-RA decodes structured 3D trajectories from this representation, conditioned on the query. We can now look at where the answer goes, as well as read what it says.
Testing continuity
Track4D-Bench gives us 526 clip-level dialogue samples, pairing questions and answers with trajectory supervision. That lets us check the words against the motion: a fluent answer still has a path to account for.
The evaluated baselines give us a comparison, and the reported results improve on them. But the questions are easier to picture than a table of scores: after a follow-up, are we still tracking the same person? Through an occlusion, does the path stay coherent? When the viewpoint changes, can we connect the two observations?
There are practical constraints, too. The formulation uses camera geometry and time information, and the paper notes that it still relies on structured supervision. The scale of the benchmark, the length of the temporal window, and the openness of the conversation remain limited. These details matter if you are wondering how directly the method transfers to a video from your phone.
Explore the project
The public repository currently contains the overview and visual demonstrations. Code, data, checkpoints, and reproduction materials are listed as coming soon. For now, the demo is the clearest way to see what this form of dialogue looks like; it is not yet a complete recipe for reproducing the experiments.
You can read the paper on arXiv or explore the project repository and demos. The repository is also where future code and data releases will be announced.