Embargoed draft. Do not share until the paper is published.
Seealpsee, Appenzell Alps25 June 2024 · fully autonomous
Robust Open-world Autonomous Mobility

Bridging vision-language reasoning and learned control for legged navigation in the wild

The VLM says what is in view and how risky it is. The robot turns that into where to walk, in alpine and urban terrain.

Scroll
1ETH Zurich  ·  2Apple  ·  *Corresponding author
Abstract

Autonomous navigation in diverse real-world environments remains one of the greatest challenges in robotics. It demands both robust low-level motion control to handle diverse terrain and high-level reasoning to interpret environmental cues. We present ROAM (Robust Open-world Autonomous Mobility), a system that enables legged robots to perform kilometer-scale navigation in complex alpine and urban settings. ROAM combines the world knowledge of a Vision-Language Model (VLM), pre-trained on internet-scale data, with robust locomotion and local planning policies learned in simulation via reinforcement learning (RL). The VLM provides common-sense reasoning and semantic interpretation of the scene, assessing risks, interpreting signs, and producing semantic labels with associated navigation costs. An open-vocabulary segmentation module converts these labels into a dense, real-time risk map, which conditions two pre-trained RL policies responsible for agile locomotion and local navigation: effectively coupling high-level VLM reasoning with low-level physical control. Real-world experiments showed the system’s ability to autonomously navigate a wide range of complex real-world environments. The robot completes kilometer-long hikes with minimal intervention, interprets signs to avoid danger, and evaluates its environment to identify safe areas on which to step. This work represents a step toward the reliable deployment of systems in unstructured environments. However, it also underscores the challenges that remain in VLM reliability and spatial reasoning, indicating that foundational models are a crucial component but not yet a complete solution for navigation in the wild.

How does it work?Scroll to play ↓
The VLM’s answer becomes a risk imageIt shows the robot where to walk. Drag any picture. go avoid
  1. 1 · The VLM names thingsand gives each a risk from 0 to 1: gravel_path 0 grass 0.5 cow 1
  2. 2 · Segmentation finds themin the camera image, pixel by pixel
  3. 3 · Each pixel gets its risk0 go 1 avoid
Risk image, Indoor hall: blue is go, red is avoidRobot front camera, Indoor hallIndoor hall
‹›
Risk image, Campus: blue is go, red is avoidRobot front camera, CampusCampus
‹›
Risk image, Forest road: blue is go, red is avoidRobot front camera, Forest roadForest road
‹›
Risk image, Test site: blue is go, red is avoidRobot front camera, Test siteTest site
‹›
Risk image, Seealpsee: blue is go, red is avoidRobot front camera, SeealpseeSeealpsee
‹›
Risk image, Alpine huts: blue is go, red is avoidRobot front camera, Alpine hutsAlpine huts
‹›
Risk image, Zermatt: blue is go, red is avoidRobot front camera, ZermattZermatt
‹›
Risk image, Mountain railway: blue is go, red is avoidRobot front camera, Mountain railwayMountain railway
‹›
Signs change the answer
Risk image, No sign: blue is go, red is avoidRobot front camera, No signNo sign
‹›
Risk image, Mine sign: blue is go, red is avoidRobot front camera, Mine signMine sign
‹›
Risk image, “Do not step”: blue is go, red is avoidRobot front camera, “Do not step”“Do not step”
‹›
Risk image, Danger sign: blue is go, red is avoidRobot front camera, Danger signDanger sign
‹›
01 The missing piece

It can already walk. The hard part is deciding where.

A person with a remote can drive this robot over almost any terrain. What the robot lacks is common sense: gravel is fine, a lake is not, and a sign can close a trail. A vision-language model (VLM) has it. ROAM (Robust Open-world Autonomous Mobility) is our system that hands that common sense to the robot's maps and legs.

What the VLM sees, and what the robot getsReal frames and logged answers. Pick a scene, then drag the line.
The VLM's answer

02 The pipeline

How ROAM works

The VLM never outputs coordinates or speeds. It only names what it sees and says how risky each thing is. Everything that has to be fast and precise happens downstream.

One second, in real timeThe line crosses once a second. Each tick is one update.
VLM
10 s for one answer
0.1 Hz
Segmentation
6 Hz
Elevation map
20 Hz
Navigation
10 Hz
Locomotion
50 Hz

Every picture is from the same moment of the Zermatt hike (15 August 2024), with the values logged on the robot.

Show the full pipeline figure (Fig. 1)
Diagram of the ROAM pipeline from camera images through the VLM, segmentation and elevation map to the RL policies
Fig. 1. The ROAM pipeline. High-level reasoning from the VLM reaches the motors through a dense semantic map and two RL policies.
03 Alpine expeditions

Kilometers, not meters

Two long hikes in the Swiss Alps test how far the system goes.

14–15 August 2024 · Valais

Zermatt

10.1km
distance
1220m
descent
~30%
steepest
25 June 2024 · Appenzell

Seealpsee

2.38km
loop
20m
up and down
🐄
on the trail
14–15 August 2024 · Valais

Zermatt

From glacier views down to the village. The route covered gravel, rock, grass, lakesides and train tracks, with slopes of about 30% at the steepest.

Play the video and the robot moves along the map; the other camera plays in the corner (click it to swap). Click the route, a dot or the profile to jump there. Right-drag (or two fingers) to tilt.

Drag across the profile to measure a section. Robot's GNSS track · elevation: swisstopo swissALTI3D · maps © swisstopo
25 June 2024 · Appenzell

Seealpsee

A loop around a mountain lake, sharing the trail with cows, hikers and boulders.

Play the video and the robot moves along the map; the other camera plays in the corner (click it to swap). Click the route, a dot or the profile to jump there. Right-drag (or two fingers) to tilt.

Drag across the profile to measure a section. Robot's GNSS track · elevation: swisstopo swissALTI3D · maps © swisstopo
3D Expedition · preview
Interactive · 3D expedition

The 3D Expedition

Fly along both hikes on the real terrain: the robot walks its GPS track, its own camera plays as you scroll, and every logged moment shows what it saw. Scroll, or press Play.

Enter the expedition →
25 June 2024

A day at Seealpsee

Behind the scenes: the setup, the 2.38 km lakeside route, cows on the trail, and the team explaining what the robot is doing. 5 min, with sound.

04 World knowledge in action

It reads the signs

Geometry can't tell an open path from a closed one. Text and symbols can, and ROAM reads them.

One place, different signs: the risk image changes with the sign
Risk image, No sign: blue is go, red is avoidRobot front camera, No signNo sign
‹›
Risk image, Mine sign: blue is go, red is avoidRobot front camera, Mine signMine sign
‹›
Risk image, “Do not step”: blue is go, red is avoidRobot front camera, “Do not step”“Do not step”
‹›
Risk image, Danger sign: blue is go, red is avoidRobot front camera, Danger signDanger sign
‹›
Experiment 1 · Fig. 4

Comparison of ROAM with baselines on a course with semantic and contextual challenges

ETH Hönggerberg, Zurich. Pick a trap, then pick who walks it.

    Show the paper figure (Fig. 4)
    Comparison of ROAM and baselines on a course with an offset global path, a mannequin and construction signs
    Experiment 2 · Fig. 5A(ii)

    Same terrain, different sign

    Arche test site. The robot's own camera, with the maps ROAM builds from the VLM's labels.

    camerarisksteppablepreferred
    goavoid

    What the VLM said
    risksteppablepreferred
    Show the paper figure (Fig. 5)
    Signs on pallets and cans change VLM risk labels and the robot's foot trajectories
    05 Locomotion

    Semantic information changed foot placement

    The locomotion policy reads a steppable layer next to the terrain height. It learns what that layer means in simulation; on the robot, the VLM fills it in.

    Training in simulation · RaiSim

    Steppable and non-steppable regions

    Non-steppable patches (red) come from Perlin noise. The policy sees them in its foot-height scan and is penalized for stepping on them.

    Experiment 3 · Fig. 5A(i)

    Exploding cans

    Arche test site. A sign on the wall reads "WARNING: Exploding Cans".

    Third-person view with the steppable map (top left) and elevation map (bottom left) burned in.

    What the VLM said
    risksteppablepreferred
    The sign turns harmless-looking cans into no-step zones. The robot places its feet between them.
    Extra experiment · not in the paper

    Banana peels on the ground

    The VLM labels the peels as risky and not steppable, so the robot places its feet around them. Without semantics, the baseline steps on them. ETH Zurich rooftop.

    06 Local navigation

    Finding its own way

    Given only a goal, where does the robot choose to walk? Geometry says "anything flat". ROAM prefers what a person would: paths, cleared ground, the side away from the tracks.

    Experiment 4 · Fig. 5B

    Same goal, different route

    Arche test site and a snowy city passage. Each run gives the local navigation policy a single goal location and nothing else: no waypoints and no global path. The policy picks the route on its own from the risk and preferred maps, and the geometry-only baseline gets the same goal.

    What the VLM said
    Indoor → urban → nature · one run

    From a hall to the fields, without changing anything

    ETH Hönggerberg. One continuous run starts inside a building, crosses the campus, heads out onto field and forest trails and farm roads, and ends in the next village. The system stays the same the whole way; only what the VLM sees changes.

    camerarisk
    risk: 0 go 1 avoid

    The robot's front camera (left) and the risk image ROAM builds from it (right), sped up 8×. Below: every answer the VLM gave on this run (one column each), the labels it used and the risk it gave them. Labels for floors and pillars give way to cobblestones and buildings, then to gravel, grass and trees. Click the chart to jump the video there.

    Experiment 5 · Fig. 5C

    The path is blocked. Now what?

    A cart and barrier tape block the trail. Watch what the robot did (4× speed), or jump to a step. Hönggerberg.

    Aerial view of the detour: obstacle at a, alternative route at b, grass shortcut at c, back to the global path at d
    07 One system, many places

    From mountain trails to stairwells

    We used the same system everywhere. No data was collected for any specific place.

    Fig. 2. Mountain trails (A–E), forest (F), grass (G), campus (H), roads (I), indoor spaces (J–L), smoke (M), train tracks (N) and signs (O).

    08 Supplementary movies

    Watch them in full

    The whole Seealpsee hike2.4 km around the lake, filmed from behind the robot.
    Reading construction signsLeaves the path when a sign says so, returns when told to, next to a geometry-only baseline.
    System componentsThe risk image, the navigation policy and the locomotion policy, each tested on its own.
    Indoor, urban and outdoorThe same system, no retuning, on stairs, in town and on trails.
    The whole Zermatt hike10.1 km from Gornergrat down to the cable car, filmed from behind the robot.
    09 Citation

    Cite this work

    @article{miki2026roam,
      title   = {Bridging vision-language reasoning and learned control for legged navigation in the wild},
      author  = {Miki, Takahiro and Schmid, Robin and Patel, Manthan and Tuna, Turcan and Richter, Julia and Nubert, Julian and Lee, Joonho and M{\"u}ller, Matthias and Hutter, Marco},
      journal = {Science Robotics},
      year    = {2026},
      doi     = {10.1126/scirobotics.aee1549}
    }