Abstract
The sense of touch is central to manipulation, especially when vision is occluded or ambiguous. Although combining vision and touch improves manipulation, learning robust visuo-tactile policies requires substantial tactile data. Such data remains scarcer than visual data, because tactile sensors are fragile, specialized, and hard to standardize.
To address this, we present Feature-Extracted Latent Tactile (FELT), a learning-based framework that synthesizes per-finger pressure tactile images from RGB observations, reducing the need for tactile-equipped data collection. FELT uses a large frozen visual encoder and a lightweight query decoder to predict tactile signals in a single feed-forward pass. To respect the physical topology of dual-finger tactile sensors, FELT decodes the left and right tactile sensor panels through separate branches, capturing the asymmetric contact patterns during interactions such as wiping, insertion, and in-hand rotation. At inference time, FELT only requires RGB data, enabling it to retroactively augment vision-only data with tactile channels for visuo-tactile policy training and provide tactile signals online for robots without physical tactile sensors. Experiments on four contact-rich manipulation tasks demonstrate that FELT produces physically grounded tactile signals and improves downstream visuo-tactile policy success over vision-only baselines.
Method
Our tactile data generation framework. (a) Given fisheye RGB images, our model extracts visual features with a frozen DINOv2 ViT. (b) The tactile query decoder low pressure uses separate learnable, position-aware query embeddings for the left and right finger sensors. Each branch attends to the visual features via cross-attention and exchanges information via gated cross-panel connection. (c) A panel-aware convolutional readout head maps the decoded queries to tactile features, with learned side embeddings to distinguish the left and right sensors. The tactile features predict contact probability and pressure intensity for each tactile grid. (d) These generated tactile signals and features can augment real visuo-tactile demonstrations for downstream policy learning.
Tactile Prediction Results
We compare the predicted tactile images from FELT with ground-truth in our xArm test dataset.
Downstream Sensor-free Deployment
Policies trained with RGB and real tactile data, but deployed with FELT-generated tactile images in place of the physical sensor reading.