Can a World Model Learn to See by Touching?

Video world models have a known problem: they generate futures that look plausible but aren't physically correct. A cup slides through a table, a splash happens without contact, and the pixels barely complain. My working theory is that this happens because vision is an observational sense. Mass, friction, compliance and temperature are literally invisible. Touch is different. Contact is the only place where the dynamics of the world get measured directly, and touch only happens during action, which makes it interventional data rather than observational data.

So the hypothesis: train a JEPA-style world model on embodied sensation (contact pressure, thermal flux, proprioception) alongside vision, and the feeling channels should ground the visual pathway. The strong version, and the one worth testing: a model that learned by touching should judge physical properties better from vision alone than a model that only ever watched. A robot usually sees an object before it touches it. If touch experience transfers into the eyes, that's a result nobody has shown.

Nobody has the data for this at scale. Real tactile datasets top out around 10 hours because tactile sensors physically wear out. V-JEPA 2 trained on a million hours of video. That gap is the whole game, and simulation is the only way I can generate feeling at scale, so that's where this project starts.

The body: one fingertip

The agent is deliberately minimal. A 4x4 cm pad on a 3-axis gantry, like a fingertip glued to a 3D printer head. Its underside is an 8x8 array of small compliant collision geoms, each one a taxel that reports the normal force on it. A wrist camera looks at the workspace from the pad itself. Objects are procedurally sampled: six material classes (steel, aluminum, glass, plastic, wood, foam) with randomized temperature, friction, stiffness, density and size.

Temperature was the fun part, because no physics engine simulates thermal sensation. It turns out you don't need much. The felt temperature of a material is governed by its effusivity (the square root of conductivity times density times heat capacity). When skin touches an object, the interface temperature is the effusivity-weighted average of the two bulk temperatures. Steel has an effusivity around 13000, wood around 400, skin around 1300. So steel at 5 degrees pulls the interface way down toward 5, and wood at the same 5 degrees barely moves it. That's the actual physical reason metal feels colder than wood at identical temperature, and it's about 40 lines of code on top of the contact solver.

Steel and wood at an identical 5 degrees produce completely different traces, and glass at 60 degrees plateaus around 45 rather than climbing to 60, because blood perfusion keeps pulling the skin back toward baseline. These curves are the signal the model gets to learn from.

Here's the whole sensory loop in motion. Left is the scene, middle is the live pressure image, right is the live skin temperature. Pressing a cold steel sphere prints a round graded disc and a growing cold blob:

Sliding across a hot glass slab is a completely different signature: flat uniform pressure across the whole face, and the entire skin blooms hot:

Same skin, same physics, and the two interactions are instantly distinguishable from touch alone. That contrast is what the model is supposed to exploit.

Everything broke, and the gate is the fossil record

Nothing in this project worked on the first try. In order: a coordinate frame bug meant the pad never actually touched anything (all sensors read zero). Then flat contact produced pressure on exactly 4 corner pixels, because rigid body engines reduce flat-on-flat contact to a handful of points, so I rebuilt the skin as 64 individual soft taxels. Then a hot glass object registered a maximum skin temperature of 32 degrees, below the resting baseline, because the heat transfer constants were calibrated for the old contact regime. Then fixing that made skin reach 58 degrees on a 60 degree object, which is physically impossible (the interface temperature caps at 47.5), because I was recomputing the interface temperature from the evolving skin temperature instead of the bulk temperatures.

The response to each of these was the same: the bug becomes a permanent check. There's a validation script that has to print ALL PASS before any dataset gets generated, and it grew from 6 checks to 12 over the course of the project. Exact invariants (taxel forces must sum to the solver's contact force), ordinal relationships (steel must chill more than wood), absolute anchors (60 degree glass must drive skin into the 40 to 50 range), footprint shape (a sphere must print a round graded disc, not a square), and a camera check. Simulation is never accurate in the absolute sense. What you can enforce is that the relationships the model is supposed to learn are physically right.

The model

Small action-conditioned JEPA, about 17M parameters. Each 100ms frame becomes tokens: one for the camera image, one for the touch stack (pressure, skin temperature, heat flux), one for proprioception plus the motor command. The model reads 2 seconds of context plus the future motor commands (you always know what you're about to do, it's free supervision), and predicts the latents of the future sensations. Latents, not pixels, which is the JEPA part: no capacity wasted on rendering, and the EMA target encoder trick prevents collapse.

One design choice worth flagging: in the early experiments every encoder, including the vision one, is a small convnet trained from scratch. No pretrained backbone. Sparsh (Meta's touch representation work) got its best results distilling DINO-style objectives, and the natural upgrade for the vision pathway is a frozen DINOv2 encoder with only an adapter trained on top. That swap happens later in this story, and it both helped and backfired in an instructive way.

Evaluation is linear probes on frozen embeddings, split by episode (window-level splits leak object identity, which I caught right before it flattered my first numbers). The probes ask: does the latent encode material, temperature, friction, effusivity, density. None of these labels ever appear in training.

Experiment 1: null

2000 episodes, two variants (vision-only and vision plus touch), then probe both from vision-only input. The touch variant aced the probes when allowed to touch (temperature R2 of 0.80, effusivity 0.67), and gained nothing over the vision baseline when touch was masked. Two flaws explained it. First, the touch model had never seen masked touch during training, so masking was out of distribution and the embeddings degenerated. Second, I had colored objects by material class, so the vision baseline could read material off the paint job.

Fixes: modality dropout during training (touch hidden on 30% of samples, replaced by a learned absence token, while prediction targets stay full, which turns "judge by sight, predict the feel" into an explicit training signal), and colors randomized independently of material.

Experiment 2: null again, but the interesting kind

10,000 episodes, 17M parameters, both fixes in. The dropout worked mechanically (embeddings stay healthy under masking now). Still zero transfer. Vision-only probes of the touch model came out at chance.

The blue bars say the latents genuinely contain temperature, effusivity and material when touch is available. The orange bars say none of it survives when the model has to judge by sight.

The problem turned out to be the experiment design, not the model. By fully randomizing colors I didn't just remove the cheat, I removed every visual correlate of material. An orange cube at 5 degrees is now pixel-identical to an orange cube at 60 degrees. No amount of cross-modal grounding can recover information that isn't in the input. The correlation between how things look and how they feel is not a confound to be eliminated. It is the channel that touch supervision is supposed to exploit. In the real world steel looks metallic, and the entire point of touch teaching vision is that touch provides the labels for which appearance features predict which feel.

That reframes the next experiment cleanly: appearance should correlate with material the way it does in reality, noisily. Each material class gets a base look (steel grey, wood brown), each object varies around it with enough noise that color alone gets you 0.81 accuracy, informative but not a giveaway. Then material is inferable from vision in principle, both variants see identical cues, and the question becomes whether touch training makes a model use those cues better. It's a one-function change.

Experiment 3: touch taught vision

With class-anchored noisy colors, same data scale, same model, same training budget for both variants, probed from vision only on held-out episodes:

| probe | vision-only trained | touch-trained, judged by sight |
|---|---|---|
| material (chance 0.18) | 0.51 | 0.67 |
| effusivity R² | 0.16 | 0.66 |
| friction R² | 0.05 | 0.46 |
| temperature R² | -0.18 | -0.20 |

Both models saw the exact same images. The 16 point gap on material and the 4x gap on effusivity is purely what feeling during training taught the visual pathway. And the temperature row is the detail that makes me trust the rest: temperature has no visual correlate in this world (hot objects don't glow), so both models sit at zero. If the probes were leaking somehow, that row would be nonzero. The experiment ships with its own negative control.

To be clear about precedence: the qualitative idea that joint tactile-visual training helps vision-only inference goes back to at least Yuan et al. 2017 on fabric matching. What I haven't found anywhere is this exact controlled measurement (matched world models, frozen encoders, vision-only probing across data scales), or anyone using a thermal channel in representation learning at all.

Scaling it: touch is worth about 10x data

Same experiment at 10k, 30k and 100k episodes (100k is 37GB, generated in 3.5 hours on a laptop). Two findings. First, the touch-trained model at 10k episodes matches the vision-only model at 100k on material accuracy, so embodied contact buys roughly an order of magnitude of data efficiency. Second, on the feel-continuous properties the gap doesn't close: at 100k episodes the touch-trained model still holds effusivity R² 0.70 against 0.40, friction 0.51 against 0.30. Vision scaling alone was not converging to what touch teaches, at least not by 100k episodes.

Closing the loop: planning by imagined feel

The dreaming story from earlier isn't hypothetical anymore. With the frozen 100k checkpoint and no new training, a CEM planner samples action sequences, imagines the resulting touch latents, and scores them with a small head that reads predicted contact force. Task: press to a 3 newton target. The planner lands at 2.18 N mean error against 2.91 for random actions and 1.71 for an oracle PD controller that reads the true force. A world model trained only on babbling closes most of the gap to the oracle, planning entirely by predicted feel.

The cold shower: real photographs

The last step was the one I was most excited about. Swap the from-scratch vision encoder for frozen DINOv2 features with a small adapter, retrain on sim, and the sim probes jump to their best numbers yet (material 0.81). Then show it real photos it has never seen, with touch set to absent, and ask what things are made of. A small pilot on 8 curated photos looked great: both sponges read as foam, both steel pans read as steel, and the sponge's predicted effusivity landed on foam's actual physics value.

Then I ran the real exam, 400 photos from the Flickr Material Database, and the pilot fell apart. Overall accuracy: chance, at every data scale. Transfer exists exactly where the sim's appearance is honest (metal detection scales monotonically with data, well above chance) and fails where the sim lies (sim glass is an opaque colored box, so real glass collapses to confidently wrong). Worse, the control flipped: raw DINOv2 features with the same probe beat the full model on broad transfer, 0.24 to 0.17. Training on my narrow sim world was actively remapping DINOv2's general-purpose features toward the fake world, subtracting real-world signal except in the few places the sim was visually truthful.

So the sim-to-real claim I wanted isn't available yet, and the reason is specific: the visual world isn't faithful enough, and more episodes of the same unfaithful world buy nothing (10x data moved broad transfer by exactly zero). Diversity and honesty of appearance are the lever, not quantity.

Where this goes

Where things actually stand after all of this. Inside the simulation, the hypothesis held up under the controlled test: feeling teaches seeing, it's worth about 10x data, the advantage on feel properties persists at scale, and the resulting model can plan real contact behavior by imagining sensations. Crossing into the real world exposed the next wall, and it isn't the learning method, it's the fidelity of the world: transparent materials, real textures, honest appearance statistics. That's where the effort goes next, along with the in-sim program this was always building toward, which is spatial reasoning and eventually a body that does more than press and slide.

The sim engine remains the asset underneath all of it. Six hours of embodied experience per wall-clock hour per CPU core, with a thermal channel no public dataset or simulator has, and a validation gate that has caught every bug so far before it could poison a training run. Code, data and the full experiment log (including everything that broke, and the pilot result that reversed on a bigger test set) are in the repo.