Original post · AI translation
看起来这是机器人领域一次出色的开源发布。
机器人技术面临的一个重大难题是：摄像头无法告诉机器人它捏得有多用力，也无法判断抓握是否正在打滑，但指尖传感器可以。
Robbyant 发布了 LingBot-Touch，这是一个开源模型（Apache 2.0），通过将指尖传感器读数转化为机器人控制软件可用的信号，赋予机器人触觉。
Robbyant 使用相同的触觉数据，为 LingBot-Touch 训练了 3 个规模相同的编码器（每个有 5.46M 个参数）。训练数据达到 4,000 小时时，在潜在空间预测被遮蔽部分的 JEPA 版本，在触觉→视频检索任务上的 top-5 准确率达到 49.3%。MAE 仍低于 9%。VAE 的结果为 3.1%，比使用 1,000 小时数据时还差。
我认为原因在于数据本身。手指只会按压少数几个位置，因此每一帧的大部分读数都接近 0，而那些非 0 读数又有噪声。重建这些读数意味着复制大量的 0 和无人能预测的噪声，而 JEPA 只预测表征，因此可以将两者都排除在外。论文指出，稀疏性和低信噪比是他们选择 JEPA 的原因。
稀疏性也影响了训练。如果从第 1 步起就遮蔽 80% 的触觉感测单元，许多片段中就没有可见的接触信息，梯度也会爆炸，因此他们在前 300 步将遮蔽比例从 40% 逐步提高到 80%。遵循 V-JEPA 2.1 的做法，目标来自 4 个深度的 EMA 编码器。
为单独考察多来源数据的影响，他们保持每种传感器自身的数据不变，只加入其他模态的数据。在目标传感器数据不变的情况下，多来源模型在 8 个传感器与任务组合中的 7 个上取得提升，表明来自不同类型传感器的数据能直接使每一种传感器受益。
Original
Looks like a brilliant open source release for robotics. A great problem of robotics is that a camera can’t tell a robot how hard it’s squeezing or whether a grip is slipping, but a fingertip sensor can. Robbyant released LingBot-Touch, an open-source model (Apache 2.0) that gives robots a sense of touch by turning fingertip sensor readings into signals their control software can use. Robbyant trained three same-size encoders (5.46M parameters each) on the same touch data for LingBot-Touch. At 4,000 hours, the JEPA version, which predicts the masked parts in latent space, got 49.3% top-5 on tactile→video retrieval. MAE stayed under 9%. The VAE got 3.1%, worse than it did at 1,000 hours. I think the reason is the data itself. A finger only presses a few spots, so most of every frame is close to zero, and the readings that aren't zero are noisy. Redrawing that means copying zeros and noise nobody can predict, and a JEPA only predicts the representation, so it can leave both out. The paper names the sparsity and the low signal-to-noise ratio as the reasons they went with JEPA. The sparsity shaped training too. With 80% of taxels masked from step one, a lot of clips had no contact left to see and the gradients blew up, so they ramp the mask from 40% to 80% over the first 300 steps. Targets come from an EMA encoder at four depths, following V-JEPA 2.1. To isolate the effect of multi-source data, they hold each sensor's own data fixed and add only the other modalities. With the target sensor's data unchanged, the multi-source model improves on seven of the eight sensor-task pairs, indicating that data from heterogeneous sensors directly benefits each individual sensor.