Ming Dai¹, Sen Yang², Boqiang Duan², Wankou Yang¹, Jingdong Wang²

¹Southeast University; ²Baidu VIS

MomentSeg is a unified MLLM for pixel-level vision–language understanding, designed with a moment-centric sampling strategy to better capture fine-grained semantics in video. It flexibly supports a range of tasks, including referring and reasoning segmentation for images and videos, temporal sentence grounding, and image/video question answering.

• 2026.06.19 🔥 We release the training code, inference code, evaluation code, data package, and checkpoints for MomentSeg-3B and MomentSeg-7B.
• 2026.06.18 🎉 MomentSeg was accepted to ECCV 2026.
• 2025.10.12 🔥 We released the paper and video demo.

• Paper and Video Demo
• Model Checkpoints and Inference Instructions
• Training Code and Detailed Documentation
• Data Release

Create the environment and install the project dependencies:

conda create -n momentseg python=3.10 -y
conda activate momentseg

# Install the PyTorch build that matches your CUDA version first.
pip install -r requirements.txt

Prepare the base MLLM and mask decoder checkpoints under pretrained/:

pretrained/
├── Qwen2.5-VL-3B-Instruct/
├── Qwen2.5-VL-7B-Instruct/
└── sam2_hiera_large.pt

For example:

huggingface-cli download Qwen/Qwen2.5-VL-3B-Instruct \
  --local-dir pretrained/Qwen2.5-VL-3B-Instruct

huggingface-cli download Qwen/Qwen2.5-VL-7B-Instruct \
  --local-dir pretrained/Qwen2.5-VL-7B-Instruct

curl -L \
  https://dl.fbaipublicfiles.com/segment_anything_2/072824/sam2_hiera_large.pt \
  -o pretrained/sam2_hiera_large.pt

The released data package is available at MomentSeg Data. If you prepare the data manually, organize the original datasets and processed annotations under data/ as follows:

data/
├── video_datas/
│   ├── revos/
│   ├── mevis/
│   │   └── train/
│   ├── rvos/
│   ├── ref_sav_eval/
│   │   ├── videos/
│   │   ├── meta_expressions_valid.json
│   │   └── mask_dict.json
│   ├── chat_univi/
│   │   ├── Activity_Videos/
│   │   └── video_chat.json
│   ├── sam_v_full/
│   └── sam_v_final_custom.json
├── ref_seg/
│   ├── refcoco/
│   ├── refcoco+/
│   └── refcocog/
├── reason_seg/
├── glamm_data/
│   ├── images/
│   └── annotations/
├── llava_data/
│   ├── llava_images/
│   ├── LLaVA-Instruct-150K/
│   └── LLaVA-Pretrain/
└── VTG/
    └── NumPro_FT/
        ├── videos_1FPS/
        ├── train.caption_coco_format.json
        └── activitynet_captions_train.json

The main training configs read paths relative to ./data/. Please keep the folder names consistent with the layout above, or update the corresponding paths in:

projects/qwenvl_sam2/configs/momentseg-3B.py
projects/qwenvl_sam2/configs/momentseg-7B.py

MomentSeg provides training configs for both model scales:

projects/qwenvl_sam2/configs/momentseg-3B.py
projects/qwenvl_sam2/configs/momentseg-7B.py

Use the root training launcher:

bash train.sh

train.sh reads configs from CONFIG_LIST and uses NUM_GPUS and PORT for distributed launch. Edit CONFIG_LIST to select the model scale, or override the launch settings from the command line:

NUM_GPUS=8 PORT=29500 bash train.sh

You can also launch a config manually:

bash tools/dist.sh train projects/qwenvl_sam2/configs/momentseg-3B.py 8

After training, convert the checkpoint to Hugging Face format if needed:

python projects/qwenvl_sam2/hf/convert_to_hf_qwenv2.py \
  projects/qwenvl_sam2/configs/momentseg-3B.py \
  --pth-model work_dirs/momentseg-3B/iter_xxx.pth \
  --save-path work_dirs/momentseg-3B/hf_model

Run inference on an image, a video, or a folder of frames with demo/demo.py. The demo uses checkpoints/MomentSeg-3B by default and saves visualized results to output/demo unless --work-dir is specified:

CUDA_VISIBLE_DEVICES=0 python demo/demo.py \
  <IMAGE_OR_VIDEO_OR_FRAME_FOLDER> \
  --model_path <MODEL_DIR> \
  --work-dir output/demo \
  --text "xxx"

<MODEL_DIR> can be a released MomentSeg checkpoint directory or a locally converted Hugging Face model directory, such as:

checkpoints/MomentSeg-3B
checkpoints/MomentSeg-7B

For full evaluation, update MODEL_PATHS, NUM_GPUS, and MASTER_PORT in the provided script, then run:

bash test.sh

The main video segmentation evaluation entry point is:

torchrun --nproc_per_node=4 --master_port=29506 \
  projects/qwenvl_sam2/evaluation/ref_vos_eval.py \
  <MODEL_DIR> \
  --dataset REVOS \
  --launcher pytorch \
  --work_dir <OUTPUT_DIR> \
  --frame_num 8 \
  --inference_mode multi-frame \
  --video_max_frames 100

Metric scripts are provided under tools/eval/, including:

tools/eval/eval_revos.py
tools/eval/eval_mevis.py
tools/eval/eval_davis.py
tools/eval/eval_ref_sav.py
tools/eval/eval_reasonvos.py
tools/eval/eval_tvg.py

For example:

python tools/eval/eval_revos.py <OUTPUT_DIR>/REVOS.json \
  --save_json_name revos_valid.json

This project builds on the foundation of Sa2VA. We sincerely thank the Sa2VA team for releasing their codebase and model framework.

Please cite our paper if you find this project helpful.

@misc{momentseg,
      title={MomentSeg: Moment-Centric Sampling for Enhanced Video Pixel Understanding}, 
      author={Ming Dai and Sen Yang and Boqiang Duan and Wankou Yang and Jingdong Wang},
      year={2025},
      eprint={2510.09274},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2510.09274}, 
}