GAP: Geometric Anchor Pre-training for Data-Efficient Visuomotor Learning of Manipulation Tasks

Abstract

Learning visuomotor policies from scarce expert demonstrations remains a core challenge in robotic manipulation. A primary hurdle lies in distilling high-dimensional RGB representations into control-relevant geometry without overfitting. While using frozen pretrained Vision Foundation Models (VFMs) improves data efficiency, it also shifts most task adaptation onto a small spatial pooling module, which can latch onto task-irrelevant shortcuts and lose geometric grounding when finetuned with few data samples.

We propose Geometric Anchor Pre-training (GAP), a simple, action-free warm-up stage that regularizes the spatial adapter before downstream imitation learning. GAP pre-trains the pooling layer on a lightweight simulated proxy task where object masks are available at no cost, encouraging the adapter to produce keypoints that lie on the object, cover its spatial extent, and remain sharp and repeatable over time. This yields stable geometric anchors that provide a reliable coordinate interface for few-shot policy learning, while keeping the VFM frozen.

We evaluate GAP on RoboMimic and ManiSkill under severe data scarcity (15–50 demonstrations) and domain shift. A simple adapter regularized with GAP consistently outperforms stronger attention-based poolers and end-to-end fine-tuning, achieving 62% success on RoboMimic Can with 15 demonstrations (+16% over AFA), 63% on the long-horizon high-precision Tool Hang task with 50 demonstrations (+13% over the best competitor), and 61% on ManiSkill StackCube with 30 demonstrations (+11% over full fine-tuning).

Contributions

We introduce GAP, an action-free pretraining strategy for spatial pooling layers that injects geometric priors via a cheap mask-supervised proxy task, preventing bottleneck collapse in few-shot visuomotor IL.

We demonstrate strong empirical gains on RoboMimic and ManiSkill in extremely low-data regimes (15–50 demonstrations) under domain and task shift.

We show that explicit geometric regularization is necessary — simply exposing the adapter to additional proxy data is insufficient and can cause negative transfer.

We demonstrate that GAP learns transferable geometric anchors that improve robustness across tasks, visual domain shifts, and simulation environments (pretrain on RoboMimic, transfer to ManiSkill).

Method

GAP addresses bottleneck collapse — a failure mode where the spatial pooling layer, trained with only a handful of demonstrations, latches onto background shortcuts instead of object-centric geometry. The fix is a short, action-free warm-up on a cheap proxy task (LiftCube in RoboMimic) using simulator-provided segmentation masks, with no expert action labels required.

Method overview. A frozen VFM backbone is followed by a lightweight spatial adapter with Spatial Softmax. GAP pre-trains this adapter on a proxy task using three geometric losses (centroid alignment, spread, diversity) before transferring to downstream few-shot imitation learning with a Diffusion Policy.

Centroid Alignment ($\mathcal{L}_{center}$)

$$\mathcal{L}_{center} = \| \bar{p}_t - c_t \|_2^2$$

Pulls the keypoint centroid onto the object mask centroid, grounding keypoints on task-relevant geometry rather than background distractors.

Geometric Spread ($\mathcal{L}_{spread}$)

$$\mathcal{L}_{spread} = \| \sigma_p - \sigma_{target} \|_2^2$$

Matches keypoint spatial variance to the object scale, preventing collapse to a single point and preserving orientation information.

Keypoint Diversity ($\mathcal{L}_{div}$)

$$\mathcal{L}_{div} = \frac{1}{K} \sum_{k=1}^K \left[ \max\left(0, \delta_{min} - \min_{j \neq k} \| p_{k,t} - p_{j,t} \|_2 \right) \right]^2$$

Enforces minimum separation between keypoints, encouraging coverage of distinct geometric extremities across the object surface.

Keypoints are partitioned per semantic entity in the scene, so each object receives its own subset of anchors. After GAP, the frozen encoder and regularized adapter transfer to downstream tasks — only the adapter and diffusion policy head are fine-tuned, with segmentation masks strictly discarded.

Results

We evaluate on four tasks of increasing difficulty — Can, Square Nut Assembly, and Tool Hang from RoboMimic, and StackCube from ManiSkill — with 15, 20, 30, and 50 expert demonstrations. All models use Diffusion Policies; GAP pre-trains once on LiftCube and transfers across all downstream tasks. Results are averaged over three seeds.

RoboMimic: Can

Method	15	20	30	50	Avg
E-E (Full FT)	0.55	0.76	0.88	0.95	0.79
R3M + SS	0.50	0.75	0.78	0.86	0.72
DINOv2 + SS	0.51	0.68	0.73	0.86	0.70
VC-1 + SS	0.49	0.64	0.82	0.89	0.71
AFA	0.46	0.74	0.78	0.93	0.73
GAP (Ours)	0.62	0.80	0.94	0.96	0.83

RoboMimic: Square

Method	15	20	30	50	Avg
E-E (Full FT)	0.15	0.19	0.29	0.38	0.25
R3M + SS	0.12	0.13	0.17	0.22	0.16
DINOv2 + SS	0.10	0.22	0.23	0.26	0.20
VC-1 + SS	0.07	0.24	0.23	0.32	0.22
AFA	0.15	0.19	0.32	0.43	0.27
GAP (Ours)	0.20	0.33	0.37	0.53	0.36

RoboMimic: Tool Hang

Method	15	20	30	50	Avg
E-E (Full FT)	0.06	0.20	0.13	0.33	0.18
R3M + SS	0.16	0.17	0.27	0.50	0.28
DINOv2 + SS	0.23	0.13	0.20	0.23	0.20
VC-1 + SS	0.20	0.17	0.20	0.43	0.25
AFA	0.20	0.23	0.30	0.45	0.30
GAP (Ours)	0.27	0.33	0.37	0.63	0.40

ManiSkill: StackCube

Method	15	20	30	50	Avg
E-E (Full FT)	0.22	0.23	0.50	0.66	0.40
R3M + SS	0.06	0.09	0.15	0.38	0.17
DINOv2 + SS	0.07	0.10	0.15	0.60	0.23
VC-1 + SS	0.04	0.08	0.28	0.63	0.26
AFA	0.09	0.14	0.25	0.44	0.23
GAP (Ours)	0.20	0.24	0.61	0.80	0.46

Multi-task evaluation. Success rates across four tasks and demo counts. ManiSkill results (gray) denote cross-simulator domain shift. GAP achieves state-of-the-art average performance on every task. Bold values indicate best per column.

GAP impact across different VFM backbones

GAP across backbones (Square, 30 demos). GAP consistently outperforms Global Avg. Pooling, unregularized Spatial Softmax, and AFA across R3M, DINOv2, and VC-1 backbones — confirming that geometric regularization is compatible with any frozen VFM.

Cite us

@article{buoso2026gap, title={GAP: Geometric Anchor Pre-training for Data-Efficient Visuomotor Learning of Manipulation Tasks}, author={Buoso, Davide and Protopapa, Andrea and Di Carlo, Stefano and Pistilli, Francesca and Averta, Giuseppe}, journal={arXiv preprint arXiv:2605.15836}, year={2026} }

GAP: Geometric Anchor Pre-training for Data-Efficient Visuomotor Learning of Manipulation Tasks

GAP introduces a lightweight, action-free warm-up stage that regularizes the spatial pooling layer before downstream imitation learning — preventing bottleneck collapse and enabling data-efficient visuomotor policies with as few as 15 demonstrations.

Abstract

Contributions

Method

Transferable Geometric Anchors

Sim-to-Real Transfer

Results

Cite us