WM-Craftnet: World Synesthesia Model for Generalizable and Robust Dexterous In-Hand Manipulation

Jie Yin · Zeyuan Zhao · Xiaojing Tan · Yang Liu · Chiyu Wang · Xinyang Gu

Sharpa Robotics · Conference on Robot Learning (CoRL) 2026

WM-Craftnet learns a World Synesthesia Model (WSM) for dexterous in-hand rotation. It fuses proprioception, noisy wrist depth, tactile contact, and action history into an action-conditioned recurrent state that captures geometry, contact, motion, and slip. That state is passed directly to an asymmetric actor–critic policy as a deployable representation—without imagined rollouts.

The model also reconstructs clean depth from noisy observations, which improves sim-to-real transfer to the human-sized, five-finger, 22-DoF Sharpa Wave hand.

1. Encode proprioception, wrist depth, tactile contact, and the previous action.
2. Predict action-conditioned dynamics with a Dreamer-style recurrent state-space model, supervised by clean depth and reward.
3. Control with the detached deterministic WSM state fed to the policy; reuse pretrained WSM weights for new object sets.

Duck · z-axis Stable finger contacts on a curved, asymmetric object during continuous z-axis rotation.	Flashlight · y-axis Target-axis rotation of an elongated body about its shorter axis under larger moment arms.	Stepped block · x-axis Rolling and fingertip-contact reallocation for gravity-sensitive x-axis rotation.
Bottle · thumb-guided re-centering 0–10 s: thumb draws the bottle toward the fingers, then y-axis rotation continues.	Coke Can · five-finger stabilization 0–5 s: all five fingers upright the can from a poor start, then z-axis rotation resumes.	Cross block · palm-to-finger transfer 0–7 s: fingers move the object from the palm onto the fingertips, then z-axis rotation continues.

More real-robot, multi-axis, depth-reconstruction, and unseen-object rollouts are on the project website.

Requirements

• Linux with an NVIDIA GPU and a compatible CUDA driver
• Python 3.8
• PyTorch 2.1.0 with CUDA 11.8
• NVIDIA Isaac Gym Preview 4

conda create -n wm-craftnet python=3.8
conda activate wm-craftnet

conda install pytorch=2.1.0 torchvision torchaudio pytorch-cuda=11.8 \
  -c pytorch -c nvidia
conda install -c fvcore -c iopath -c conda-forge fvcore iopath
conda install pytorch3d -c pytorch3d

pip install hydra-core gym numpy==1.22.2 tensorboardX tensorboard \
  wandb scipy imageio imageio-ffmpeg h5py trimesh rtree pillow ninja

Download NVIDIA Isaac Gym Preview 4, accept its license, then install:

tar -xzvf IsaacGym_Preview_4_Package.tar.gz
cd isaacgym/python
pip install -e . --no-deps

See install.md for additional notes.

Download the released z-axis checkpoint and run the test script:

hf download SharpaIT/WM-Craftnet wm_craftnet_set_z.pth --local-dir example_ckpt

CHECKPOINT=example_ckpt/wm_craftnet_set_z.pth \
bash scripts/test_wm_craftnet.sh

• Weights: wm_craftnet_set_z.pth on SharpaIT/WM-Craftnet
• Config: example_ckpt/wm_craftnet_set_z.yaml

The script opens the viewer by default. Pass extra Hydra arguments after the script name, or set HEADLESS=true when no display is available.

Each entry script selects the matching task config and accepts Hydra overrides:

# z-axis rotation on nine objects
bash scripts/train_wm_craftnet_z.sh task.env.objSet=set_z

# x-axis rotation on four objects
bash scripts/train_wm_craftnet_x.sh task.env.objSet=set_x4

# y-axis rotation on nine tool-like objects
bash scripts/train_wm_craftnet_y.sh task.env.objSet=set_y

Object sets

Object meshes, robot assets, and datasets may have terms outside this repository. See THIRD_PARTY_NOTICES.md before redistribution.

Memory settings

The paper setup uses num_envs=1024 and minibatch_size=4096 (WMCraftnetPPO.yaml). Scripts default to 256 envs and a 1024 minibatch so the WSM replay buffer fits on a 32 GiB host:

# default (memory-safe)
bash scripts/train_wm_craftnet_y.sh

# lower further if needed
NUM_ENVS=128 MINIBATCH_SIZE=512 bash scripts/train_wm_craftnet_y.sh

# paper settings on larger machines
NUM_ENVS=1024 MINIBATCH_SIZE=4096 bash scripts/train_wm_craftnet_y.sh

Edit the device switches at the top of each script for simulation, RL, and graphics GPUs. Outputs go to runs/.

Six optional heads are enabled by default—proprioception, clean depth, object pose, tactile contact, critic value, and BPS object shape—plus reward prediction. Configure them under task.env.cameraPolicy.worldModel in WMCraftnetRotation.yaml:

# legacy setup: proprioception + depth only (matches released checkpoint)
bash scripts/train_wm_craftnet_z.sh \
  task.env.cameraPolicy.worldModel.wm_pose_pred=False \
  task.env.cameraPolicy.worldModel.wm_tac_pred=False \
  task.env.cameraPolicy.worldModel.wm_value_pred=False \
  task.env.cameraPolicy.worldModel.wm_obj_pred=False

Checkpoint and head configs must match at train and test time. The released checkpoint uses proprioception + depth; evaluate with scripts/test_wm_craftnet.sh. Full-head runs need matching flags (see scripts/test_wm_craftnet_y.sh).

Hardware examples are under deploy/. Use Python 3.10 for deployment; simulation and training stay on Python 3.8.

Real Z-Axis Duck Rotation WM-Craftnet rotates a duck on the Sharpa Wave hand using wrist depth and tactile feedback.

Real Noisy Wrist Depth Raw depth from the wrist camera during the same rollout—noisy, incomplete, and hard to use directly for control.

Real Predicted Depth from WSM Latent WSM reconstructs a cleaner depth map that preserves hand–object geometry despite sensor noise.

On real hardware, the wrist depth stream is noisy and incomplete. The WSM latent reconstructs a cleaner depth representation that preserves hand–object geometry—supporting the same policy at deployment time. More rollouts are on the project website.

Download from the Sharpa download page or GitHub releases. On x86-64 Ubuntu, run from the repository root to install the public non-CUDA SDK v5.0.9 under deploy/sharpa_sdk/:

SDK_DEB=/tmp/sharpa-wave-sdk_5.0.9_amd64.deb
SDK_TMP=$(mktemp -d)

curl -fL \
  https://github.com/sharpa-robotics/sharpa-wave-sdk/releases/download/v5.0.9/sharpa-wave-sdk_5.0.9_amd64.deb \
  -o "$SDK_DEB"
echo "672c0b2e02f64c0db22760e8331f6e35e1fc9e7ef84363e81481e608a1278dfb  $SDK_DEB" \
  | sha256sum -c -
dpkg-deb -x "$SDK_DEB" "$SDK_TMP"

rm -rf deploy/sharpa_sdk/lib deploy/sharpa_sdk/python/sharpa
mkdir -p deploy/sharpa_sdk/python
cp -a "$SDK_TMP/opt/sharpa-wave-sdk/lib" deploy/sharpa_sdk/lib
cp -a "$SDK_TMP/opt/sharpa-wave-sdk/python/sharpa" deploy/sharpa_sdk/python/sharpa
cp "$SDK_TMP/opt/sharpa-wave-sdk/VERSION" deploy/sharpa_sdk/VERSION
cp "$SDK_TMP/opt/sharpa-wave-sdk/sdk-release-info.json" deploy/sharpa_sdk/sdk-release-info.json

rm -rf "$SDK_TMP" "$SDK_DEB"

The non-CUDA build receives 30 Hz tactile inference from the hand. The optional CUDA SDK build requires CUDA 13.x and TensorRT 10.x.

Download wm_craftnet_set_z.pth into example_ckpt/ with sibling example_ckpt/wm_craftnet_set_z.yaml. Set checkpoint, camera options, no_actuation, and max_steps in build_hardcoded_config() inside deploy/examples/wm_craftnet_infer.py:

conda activate inhand_deploy310
python deploy/examples/wm_craftnet_infer.py

The policy expects the same field of view and depth preprocessing as training—not just the same checkpoint.

Suggested workflow

1. Mount the physical depth camera to match cameraPolicy.sensor.pos / rot.
2. Set the RealSense stream; runtime resizes depth to 96 × 72 before applying depth_preprocess.crop.
3. If the real view differs from simulation, adjust crop margins—not the policy input size.
4. Start with no_actuation=True and verify the depth preview before enabling actuation.

WM-Craftnet builds on:

• Isaac Gym Environments (IsaacGymEnvs) — simulation and RL infrastructure
• World Model-based Perception (WMP) — world-model-conditioned policy learning
• Robot Synesthesia / in-hand rotation — dexterous manipulation task design
• DreamerV3 — recurrent state-space world model
• rl_games — PPO training backend

See NOTICE and THIRD_PARTY_NOTICES.md for license texts and attribution.

@inproceedings{yin2026wmcraftnet,
  title     = {WM-Craftnet: World Synesthesia Model for Generalizable and Robust Dexterous In-Hand Manipulation},
  author    = {Yin, Jie and Zhao, Zeyuan and Tan, Xiaojing and Liu, Yang and Wang, Chiyu and Gu, Xinyang},
  booktitle = {Conference on Robot Learning (CoRL)},
  year      = {2026}
}

Released under the Apache License, Version 2.0. See NOTICE and THIRD_PARTY_NOTICES.md for third-party attribution.