A Compact Benchmark Zoo for Fast and Reliable Model Evaluation

100+ ready-to-use compact versions of text, multimodal, and agent benchmarks.

Quick Start · Text Evaluation · Multimodal Evaluation · Agent Evaluation · Build Your Own · Citation

This repository is the official codebase for our EMNLP paper "ZipBench: Low-Cost Framework for Compressing Comprehensive Benchmarks of Large Language Models".

TL;DR. ZipBench provides 100+ ready-to-use compact versions of text, multimodal, and agent benchmarks, cutting evaluation cost by 60–80% while closely preserving full-benchmark scores and model rankings.

2026-08 - Our paper "ZipBench" has been accepted to the EMNLP'26 Main Conference! [Paper] 👈🎉Please read it！

The paper builds ZipBench Zoo, containing more than 100 compact benchmark versions across text, multimodal, and agent tasks. This repository provides ready-to-use evaluation for all three settings, together with the full benchmark-construction pipeline.

cd opencompass
python run.py zipbench/configs/eval_arc_challenge_small.py

The command evaluates the model configured inside the file on the small ARC-Challenge benchmark. See the text evaluation guide for installation and model configuration.

cd VLMEvalKit
python run.py \
  --data MMStar \
  --model Qwen2.5-VL-7B-Instruct \
  --subset small

Change small to tiny for a lower-cost run or to full for the complete benchmark. See the multimodal evaluation guide for setup and supported models.

Agent benchmarks use different official evaluation tools. ZipBench supplies the selected task IDs and weights, while each official tool still runs and scores the agent normally.

cd agent
pip install -e .
zipbench-agent list

Choose a benchmark, export its small or tiny task list, run the official evaluation tool, and pass its results back to zipbench-agent score. See the agent evaluation guide for complete commands.

cd zipbench
python compression/train_irt.py \
  --scenario LiveCodeBench \
  --train-pkl data/LiveCodeBench/LiveCodeBench_train.pkl \
  --output-dir runs/LiveCodeBench \
  --device cuda --seed 7 --run-id 1

Follow the ZipBench construction guide for a two-command reproduction and the full workflow for a new benchmark.

These values are averages across ZipBench Zoo. The actual size of each compact benchmark is selected separately according to its redundancy and evaluation fidelity.

Complete benchmark names and sample counts are listed in the three evaluation guides.

You do not need to run the construction pipeline to use the released small and tiny benchmarks. For readers who want to reproduce the method, ZipBench uses four steps:

1. Evaluate a few diverse models. The standard setup uses six anchor LLMs with different capability levels.
2. Expand the observed behavior. Additional response patterns are synthesized without evaluating more real models.
3. Learn which samples are informative. ZipBench learns a compact representation of how models behave on each sample.
4. Select and weight representative samples. Similar samples are grouped, and one weighted representative is retained from each group.

The implementation uses multidimensional 2PL item response theory and K-means clustering. The paper provides error and rank-consistency analyses for the resulting compact benchmarks.

ZipBench/
├── assets/          # Logo and repository assets
├── opencompass/     # Text evaluation on full, small, or tiny benchmarks
├── VLMEvalKit/      # Multimodal evaluation on full, small, or tiny benchmarks
├── agent/           # Agent evaluation through official benchmark tools
└── zipbench/        # Benchmark compression and reproducibility pipeline

The original upstream documentation is retained as opencompass/README_OpenCompass.md and VLMEvalKit/README_VLMEvalKit.md.

@misc{huang2026zipbenchlowcostframeworkcompressing,
      title={Zipbench: Low-Cost Framework for Compressing Comprehensive Benchmarks of Large Language Models}, 
      author={Zhongzhan Huang and Junxin Li and Guoming Ling and Yupei Lin and Shanshan Zhong and Hefeng Wu},
      year={2026},
      eprint={2609.12475},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2609.12475}, 
}

ZipBench builds on OpenCompass, VLMEvalKit, and py-irt. We thank their maintainers and the authors of all supported benchmarks.

This repository is released under the Apache 2.0 License. The bundled py-irt code retains its original MIT license.