Grasp in Gaussians

Ayce Idil Aytekin, Xu Chen, Zhengyang Shen, Thabo Beeler, Helge Rhodin, Rishabh Dabral, Christian Theobalt

GraG reconstructs a hand and the object it manipulates from a single monocular video. The object is reconstructed once from a few keyframes with a video-adapted SAM3D, the hand is initialized with Dyn-HaMR, and both are then tracked with a compact Sum-of-Gaussians (SoG) objective. A 100-frame video takes about 16 minutes on one RTX 4090.

• Installation
• Quick start
• Pipeline
• Evaluation
• Citation

GraG uses Python 3.12, CUDA 12.1 and PyTorch 2.4.1 in a uv environment. It runs on one 24 GB GPU.

git clone https://github.com/aidilayce/GraG.git && cd GraG
uv venv --python 3.12 .venv
source .venv/bin/activate
uv sync --extra eval

Then build the CUDA extensions (PyTorch3D, gsplat, nvdiffrast, chumpy, kaolin) as described in INSTALL.md.

Download the following checkpoints as they are under their own licenses and put them in the written location:

SAM 3, Depth Anything 3 and GLM-4.1V are downloaded from Hugging Face on first use. Request access to facebook/sam3 and run huggingface-cli login. Per-frame contact flags are predicted with Gemini if GEMINI_API_KEY is set; otherwise contact is taken from hand/object mask proximity. Note that for our reported results in the paper, we used Gemini to get contact flags.

bash recon.sh path/to/video.mp4

The video is preprocessed into data/<video>/ and the reconstruction is written to outputs/<video>/<timestamp>/:

• refiner_final_state.pt: hand (MANO) and object trajectories and the canonical object mesh, in the camera frame.
• videos/<video>_final_mesh.mp4, videos/<video>_final_mesh_side.mp4: the reconstruction overlaid on the input and seen from the side.

GRAG_WORK_DIR and GRAG_LOG_ROOT change the data and output folders.

recon.sh runs two steps that can also be called separately.

Preprocessing, Stage 1 and Stage 2

python preprocessing/preprocess.py --video_path path/to/video.mp4

Every step is cached in the scene folder, so an interrupted run resumes where it stopped. --video_path also accepts a folder of frames called rgb/.

Stage 3: hand–object tracking

python optimization/opt.py --exp <video>

To render a saved result again:

python scripts/render_mesh_overlays.py --seq_name <video> \
    --checkpoint outputs/<video>/<run>/refiner_final_state.pt \
    --demo_root data --out_dir outputs/<video>/<run>/videos

We evaluate on the 18 HO3D v3 sequences used by HOLD and the 18 HOT3D clips listed in eval/hot3d/clips_table.csv, with HOLD's metrics. Ground truth is prepared once with HOLD (HO3D) and the HOT3D toolkit (HOT3D); see eval/README.md. The evaluator runs in HOLD's environment.

For one sequence, reconstruct and evaluate in one call:

export MANO_DIR=third_party/Dyn_HaMR/_DATA/mano
export HOLD_CODE_DIR=/path/to/hold/code          # HO3D ground truth
export HOT3D_GT_CACHE_DIR=/path/to/hot3d_gt      # HOT3D ground truth
export HOLD_PYTHON=/path/to/hold_env/bin/python
bash recon.sh path/to/hold_ABF12_ho3d/rgb ho3d --eval

For many sequences, list dataset,sequence,checkpoint per line in a runs file and call eval/run_eval.py (eval/README.md), then eval/summarize.py for the dataset means.

@inproceedings{aytekin2026grasp,
author = {Aytekin, Ayce Idil and Chen, Xu and Shen, Zhengyang and Beeler, Thabo and Rhodin, Helge and Dabral, Rishabh and Theobalt, Christian},
title = {Grasp in Gaussians: Fast Monocular Reconstruction of Dynamic Hand--Object Interactions},
year = {2026},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3829340.3842273},
doi = {10.1145/3829340.3842273},
booktitle = {Proceedings of the SIGGRAPH Asia 2026 Conference Papers},
numpages = {11},
location = {Kuala Lumpur, Malaysia},
series = {SA Conference Papers '26}
}

GraG is under MIT license. However, we build on SAM 3, SAM 3D Objects, MV-SAM3D, Depth Anything 3, GLM-4.1V-Thinking, Gemini, Dyn-HaMR with HaMeR, ViTPose, DROID-SLAM, HMP and Hand-BMC, MANO, PyTorch3D, gsplat, nvdiffrast and Kaolin. Evaluation follows HOLD on HO3D v3 and uses the HOT3D toolkit. Each component is under their own license.