EgoPathBench

SHANGHAI JIAO TONG UNIVERSITY

EgoPathBench

From seeing a scene
to choosing a route.

Can a vision-language model turn what it sees into a complete, feasible, goal-reaching path?

Evaluating Zero-Shot Egocentric Waypoint Decision-Making in Vision-Language Models

Yang ZhaoZhuo ChenXubo Yang*

Shanghai Jiao Tong University   ·   *Corresponding author

Paper · arXiv Code Dataset Explore the tasks ↓
THE SPATIAL DECISION01 / FIRST-PERSON VIEW
Conceptual example from Figure 1: green and red waypoint routes in a room illustrate goal-reaching and failed route decisions.
THE GOAL

“Navigate to the coffee table
next to the sofa.”

Feasible route Failed routeConceptual example · Paper, Fig. 1
5spatial decision tasks
1,111benchmark questions
9foundation VLMs evaluated
31,852training questions + spatial CoT

01 / WHY THIS MATTERS

Spatial intelligence has to
hold up at every step.

A useful spatial decision brings several abilities together: recognize the goal, connect image locations to actions, account for the agent’s body, and keep the entire route feasible.

Existing spatial evaluations provide valuable measurements of objects, relations, directions, and distances. EgoPathBench examines how these abilities work together in an ordered, first-person waypoint decision.

By fixing the observation and action vocabulary, it isolates this decision from the mapping, memory, and control modules of a full navigation system.

01

A joint decision benchmark

Five tasks connect candidate feasibility, target grounding, and complete route planning through one visual interface.

02

Verifiable action consequences

Each waypoint is anchored in a 3D scene. Predictions are judged by their geometric feasibility and goal arrival.

03

A resource for learning

Training questions, geometry-grounded answers, reference routes, and spatial CoT support learning as well as evaluation.

02 / WHAT WE BUILT

One visual interface.
Five spatial demands.

Explore actual questions from the released benchmark. The model sees an image and a prompt, and returns visible waypoint IDs.

Point Traversability benchmark example with numbered visible candidate points.ACTUAL BENCHMARK OBSERVATION
POINT AGENT146 benchmark questions

Where can a point agent go?

Select every visible candidate that is traversable when the agent has no body width. This tests waypoint-to-scene correspondence and candidate feasibility.

MODEL PROMPT

The image shows numbered candidate points. Output a JSON array of display IDs that are walkable.

Expected output
An unordered set of traversable waypoint IDs.
What is evaluated
Candidate feasibility: balanced accuracy and F1.

The two traversability examples share a scene and view. The Point Path and Embodied Path examples also share a scene, view, and target.

01 — OBSERVE

Image + goal

First-person RGB and visible numbered waypoints are supplied to the model.

02 — DECIDE

Waypoint IDs

The prediction is a candidate set or an ordered route, expressed as a JSON array.

03 — VERIFY

Scene-grounded outcome

The evaluator checks agent-specific feasibility, every selected edge, and the endpoint.

A reference route certifies that a solution exists.

It is not the only answer a model may give. Any predicted route is evaluated against the same geometry and acceptable goal regions; path efficiency is measured after success. Scene geometry and ground-truth sidecars belong to the evaluator, not the model input.

How the dataset is constructed
Paper Figure 2: scenes, views, targets, waypoints, geometry-verified routes, and image-question and spatial CoT generation.

Scenes and views → visible waypoints and targets → point/embodied feasibility graphs → verified reference routes → questions and spatial CoT. Splits are formed by source group to keep related scans or regions together. The release contains 31,852 training, 1,345 validation, and 1,111 benchmark questions.

03 / WHAT WE FOUND

Complete routes expose
a substantial capability gap.

Nine foundation VLMs share the same benchmark and evaluator. These are results from the paper’s fixed evaluation, not a live ranking.

BEST ZERO-SHOT SCORE

28.3/ 100

Room to improve across five tasks

Gemini 3.1 Pro leads the foundation-model comparison on EgoPath Score.

EMBODIED PATH SUCCESS

2.9%

Body constraints change the problem

The top-ranked model reaches 35.9% on Point Path, but 2.9% on Embodied Path and 4.0% on Intent Path.

TRAINING-RESOURCE STUDY

3.9 → 38.9

Spatial supervision makes a difference

Qwen3.5-4B improves its EgoPath Score after fine-tuning with the released training resource.

LOOK INSIDE THE FAILURE

A legal first step
is only the beginning.

96.3–96.8% of outputs are evaluable routes. Yet legality often breaks later in the sequence, and selecting an acceptable endpoint remains difficult.

The central challenge is sustaining geometric and goal consistency across the entire decision.

Read the analysis · Figures 3–4 ↗

Pooled over nine VLMs; each rate uses all predictions for that task. These are parallel diagnostics, not sequential pass rates. Source: paper, Fig. 3.

Full foundation-model leaderboard
Native/default evaluation · all values on a 0–100 scale
ModelPoint
Trav. BA		Embodied
Trav. BA		Point
Path SR		Embodied
Path SR		Intent
Path SR		EgoPath
Score

BA: balanced accuracy. SR: success rate. EgoPath Score = 100 × [(2BAA1 − 1) + (2BAB1 − 1) + SRA2 + SRB2 + SRC] / 5, using task metrics in [0, 1]. Source: paper, Table 2.

A human reference on the same questions

Humans reach 54.2 EgoPath Score versus 28.6 for the strongest VLM on the matched human-evaluation subset. These subset scores are distinct from the full-benchmark leaderboard above.

04 / VALUE BEYOND EVALUATION

Train on spatial decisions.
Test beyond EgoPathBench.

The training resource includes geometry-grounded answers and spatial CoT: reasoning supervision that connects targets, candidate feasibility, and route structure.

QWEN3.5-4B · BEFORE / AFTER SFT
3.9Base
38.9+ EgoPathBench training

EgoPath Score on the full in-domain benchmark. Every metric reported in the paper’s training comparison improves.

31,852 direct-answer examples31,852 spatial CoT examples

Transfer to three independent benchmarks

Base After SFTAccuracy (%)

Four reported settings across three benchmarks. Gains are percentage-point differences, with each base/SFT pair evaluated on the same items. Source: selected paper, Table 3.

05 / EXPLORE & BUILD

A shared resource for
better spatial decisions.

THE PAPER

Read the full study

Task design, construction, nine-model evaluation, failure analysis, and training results.

arXiv:2609.16610 · Paper & supplementary materialOPEN-SOURCE CODE

Run the evaluator

Inference and evaluation tools, construction scripts, training utilities, and tests.

GitHub · MIT licenseDATA & TRAINING

Get the dataset

Download the complete train, validation, and benchmark splits, visual assets, evaluation metadata, and spatial CoT training examples.

Hugging Face · 33.39 GB · CC BY-NC-SA 4.0

Dataset use remains subject to upstream asset terms. Original scene meshes are not redistributed.

Questions or collaboration? Contact Yang Zhao ↗

Cite EgoPathBench

If you use the benchmark or training resource, please cite our paper.

@misc{zhao2026egopathbench,
  title={EgoPathBench: Evaluating Zero-Shot Egocentric Waypoint Decision-Making in Vision-Language Models},
  author={Yang Zhao and Zhuo Chen and Xubo Yang},
  year={2026},
  eprint={2609.16610},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2609.16610}
}