The Surprising Effectiveness of Few-Step Adaptation for Robotic Manipulation

Anonymous authors Under review, ICLR 2027

Pretrained robot policies with one expert demonstration per task.

FOSIL: a pretrained vision-language-action model is trained on 17 pretext behaviors, such as throwing an object, putting it in a basket, using a drawer and handing it over, then adapts from one demonstration. One gradient step enables environmental adaptation to new objects and spatial configurations; ten to thirty steps enable new compositions of familiar behaviors and skill adaptation such as scooping lentils.
Hover over a panel to play it. A pretext behavior also plays the held-out task built on it; hover over an object to switch the throwing rollout. Click a task for its full rollout.

Abstract

A pretrained vision-language-action (VLA) policy can learn a held-out manipulation task from one demonstration and one to ten gradient steps of its own training loss. We study this through Few-step One-Shot Imitation Learning (FOSIL): starting from a pretrained initialization such as π0.5 or MolmoAct2, we train on pretext behaviors collected on the target robot, then adapt the FOSIL model to each task with its architecture and objective unchanged. Tasks span environmental changes, new compositions of familiar skills, and skills absent from the pretext behaviors, on a two-arm manipulation platform and in LIBERO. With a π0.5 initialization, one gradient step raises average success on LIBERO-PRO task perturbations from 25% to 49%, compared with 21% and 25% for in-context variants of the same architecture. On the long-horizon compositions of LIBERO-10, ten steps raise success from 4.6% to 51.6%. We then examine what supports these few-step gains. At matched update magnitude, low-rank approximations preserve most of the adaptation gain with a robot-pretrained action expert and substantially less without robot pretraining. Fitting incidental motion details in the demonstration can also hinder adaptation. Refining the demonstrated actions with the FOSIL model, while preserving selected end-effector and gripper constraints, raises success on LIBERO-Object task perturbations from 52% to 76%. In our setting, in-context imitation benefits more from training examples that make the demonstration necessary than from increasing task count alone.

Pretext behaviors

Start from a pretrained VLA policy (π0.5 or MolmoAct2) and train it on pretext behaviors collected on the target robot. This gives the unadapted model.

For each held-out task, update all parameters on one demonstration for 1–30 gradient steps. Same architecture, same loss, no meta-training or rewards.

Six pretext behaviors: throwing an object into a basket; opening a drawer, placing an object inside and closing it; object handover; placing an object in a basket; opening a zipper, placing an object inside and closing it; whisking lentils. ▶ Play demonstration ▶ Play demonstration ▶ Play demonstration ▶ Play demonstration ▶ Play demonstration ▶ Play demonstration
Six of the 17 pretext behaviors (254 demonstrations total). Hover over a behavior to watch a demonstration; click to enlarge.

Adaptation settings. AdamW with fresh optimizer moments; 120 evenly spaced demonstration frames per step. For π0.5, learning rate 5 × 10−5 for one step and 10−5 for five or ten. Each simulation cell uses 50 episodes per task. In-context baselines see the same demonstration at inference instead of training on it.

Real-world adaptation

Each held-out task is shown beside the pretext behaviors closest to it. FOSIL learns the difference from one demonstration.

Environmental Adaptation

Environmental adaptation: robot throwing new objects into a basket, and bussing a table in a new spatial configuration.

Compositional Adaptation

Compositional adaptation: combining object handover with placing in a drawer, and removing from a drawer then placing in a zipper bag.

Skill Adaptation

Skill adaptation: a new wrist-rotation scooping move, and a new sweeping motion for tomato cleanup.
Success over 20 rollouts per task (60 for throwing), one demonstration each. Gradient steps: 1 for environmental, 10 for compositional, 30 for skill adaptation. Paper figure. Rollout videos are from π0.5 FOSIL: hover over a photo or a π0.5 FOSIL bar to preview one; click for the full comparison.

Environmental adaptation

A familiar skill with a new object, layout or goal.

1 demonstration · 1 gradient step

Throw a bar of soap into the basket

Pick up the Vaseline container soap and throw it into the basket.

1 Held-out task

Instruction: Pick up the soap and throw it into the basket.

2 Closest pretext behavior

Throw object in basket

Compositional adaptation

Familiar behaviors, chained in a sequence that was never demonstrated.

1 demonstration · 10 gradient steps

Hand the object over, then place it in the drawer

Handover then Place in drawer

1 Held-out task

Instruction: Pick up the object, hand it to the other arm, then place it inside the drawer.

2 Closest pretext behaviors

Handover
Place object in drawer

Skill adaptation

A motion that none of the pretext behaviors contains.

1 demonstration · 30 gradient steps

Scoop lentils with a spoon and pour them into a cup

1 Held-out task

Instruction: Pick up the spoon, scoop lentils from the bowl, and pour it into another cup.

2 Closest pretext behavior

Whisk lentils
The one demonstration

Simulation results

Why one demonstration is enough for a robot-pretrained policy, step by step on LIBERO-10.

The paper's heatmap of success rates for pi0.5, MolmoAct2, StarVLA-pi, StarVLA-OFT, and pi0.5-VL at 0, 1, 5, and 10 gradient steps.

LIBERO-10 · 10 long-horizon tasks

Success after 0, 1, 5, and 10 gradient steps on one demonstration

Each row is a FOSIL model trained on the same pretext behaviors from a different initialization. Each cell is the success rate after 0, 1, 5, or 10 gradient steps on one demonstration per task.

Robot pretraining

After one step, robot-pretrained models reach 29 to 33%; the rest stay under 15%

π0.5 and MolmoAct2 were pretrained on robot data; StarVLA-π, StarVLA-OFT, and π0.5-VL were not. The gap holds at five steps: 48% and 56% against at most 17%.

Same network

Re-initializing only π0.5's action expert halves its first-step gain

π0.5-VL is π0.5 with a freshly initialized action expert, trained on the same pretext behaviors. Its first step adds 13 points (2% to 15%) against 24 points for π0.5 (5% to 29%), and it needs all ten steps to reach 46%.

Why

One direction per weight matrix carries 87% of π0.5's improvement

Adaptation changes every weight matrix a little. We keep only the strongest direction of each change and test the policy again. For π0.5 that single direction keeps 87% of the improvement on LIBERO-10; for π0.5-VL it keeps 52%.

Our reading: robot pretraining has already built the direction a new task needs, so the demonstration adjusts it instead of constructing one from many smaller ones.

Update rank

π0.5-VL needs 16 to 64 directions to keep 90% of its improvement

Allowing k directions per matrix instead of one, π0.5 is near its full improvement from k = 1 on both suites (96% on LIBERO-Object). π0.5-VL climbs slowly, reaching about 90% at k = 16 on LIBERO-10 and k = 64 on LIBERO-Object.

Where

39% of the update's effect sits in the first six VLM layers

We give the unadapted π0.5 the update of a single block and measure how far it moves the policy toward the commanded object. The VLM carries 63% of the effect, and its layers 0 to 5 alone carry 39%. The action expert carries 10%, and no SigLIP layer more than 1.5%.

These are the layers where Swann et al. (2026) find salient, steerable features in π0.5.

Settings

Each LIBERO-10 cell uses 50 episodes per task. π0.5-VL keeps the architecture but re-initializes the action expert before training on the same pretext behaviors.

Rank truncation keeps the leading k singular directions of each hidden weight matrix's update and rescales it to the update's Frobenius norm; other parameters keep their full update. Rank is normalized to the dimensions of an attention head (d = 256), so k = 1 is not literally one direction in every differently sized matrix. Retained gain is (Ak − Aunadapted) / (Aadapted − Aunadapted). LIBERO-10 uses ten steps for both initializations at the same learning rate; LIBERO-Object uses one step for π0.5 and ten for π0.5-VL. Strictly, k counts directions per 256 dimensions, so a matrix wider than one attention head keeps proportionally more. The full update spreads its size over many directions (for π0.5 after one step on LIBERO-Object, an effective rank of about 35 per head); what concentrates in the top direction is its effect on success.

Layer shares are for LIBERO-Object task perturbation, one-step update, mean over 9 tasks, normalized to sum to one over all blocks; the action input and output projections count toward the action expert. Full simulation heatmap (PDF).

Analysis

Demonstration refinement

Fitting incidental details of a demonstration (wrist angle, speed) can hurt. Refinement keeps key end-effector waypoints and gripper commands, and lets the unadapted model fill in the rest of the motion.

For each demonstration observation, sample an action chunk from the unadapted model. Correct it minimally to match selected end-effector waypoints and the demonstrated gripper commands, spreading the correction smoothly over time. Adapt using these refined actions as supervision. Frames near the grasp are not refined.

LIBERO-Object task perturbation: 52% → 76%. No gain on Spatial swap (36% vs 35%). Spatial task-perturbation bars compare one step (demo) with eight steps (refined). Error bars as in the paper figure. Click a bar to watch its rollouts.
Refinement procedure and evaluation settings

The separate eight-task Spatial comparison matches both arms at eight AdamW steps, batch size 16 drawn from 128 frames, and learning rate 6.25 × 10−6. Success is 43% unadapted, 19% after adaptation on the recorded demonstration, and 68% after adaptation on the refinement. This subset and budget differ from the one-step suite results above.

Object tasks: correcting grasp and wrist motion

Instruction: Pick up the salad dressing and place it in the basket.

Adapted on demonstration
Adapted on refinement
The raw-demo policy repeats a wrist twist; the refined one completes the task.

Instruction: Pick up the ketchup and place it in the basket.

Adapted on demonstration
Adapted on refinement
The raw-demo policy closes its gripper too early; the refined one completes the task.
Spatial tasks: placing bowls on the plate

Instruction: Pick up the black bowl on top of the wooden cabinet and place it on the plate.

Adapted on demonstration
Adapted on refinement
The raw-demo policy times out after grasping; the refined one completes the task.

Instruction: Pick up the bowl next to the plate and place it on the plate.

Adapted on demonstration
Adapted on refinement
The raw-demo policy inherits the demo’s wrist orientation; the refined one completes the task.

Citation

@article{anonymous2026fosil,
  title   = {The Surprising Effectiveness of Few-Step Adaptation for Robotic Manipulation},
  author  = {Anonymous},
  journal = {Under review},
  year    = {2026}
}