UniComp: A Unified Evaluation of LLM Compression via Pruning, Quantization & Distillation

<\p> ---

conda create -n performance python=3.10
conda activate performance
pip install -e . -r ./setup/requirements_performance.txt

conda create -n light python=3.10
conda activate light
pip install -e . r ./setup/requirements_light.txt

sbatch run_performance.sh llama_8b_wanda_50 knowledge

bash sweep.sh

Local pruned/distilled models must be generated first — see compress/README.md

The general process works as follows: First, you serve the model you want to evaluate via vLLM and let it generate responses to all benchmark questions, which are saved in JSON. Then, LLM-as-judge (here GPT-4o-mini) is used to check the generated answers and score them.

conda create -n vllm python=3.10
conda activate vllm
pip install -r ./setup/requirements_vllm.txt

conda create -n trustllm python=3.10
conda activate trustllm
pip install -r ./setup/requirements_trust.txt

cd TrustLLM

1. Edit generate_all.py — set MODEL_PATH to your model path
2. Register the model you want to evaluate in trustllm_pkg/trustllm/config.py — add "/path/to/model": "model_name" to model_info/model_mapping and append "model_name" to the openai_model array. (Don't forget to add a "," after the model name, since its JSON - speaking from experience)
3. Serve the model on GPU:

conda activate vllm
vllm serve "/path/to/model" \
  --host 0.0.0.0 \
  --port 8000 \
  --dtype auto \
  --api-key localtoken \
  --served-model-name model_name

4. In config.py set openai_key="localtoken" and openai_api_base="http://localhost:8000/v1"
5. If you're on a cluster: SSH to the same compute node, otherwise just make sure you can access localhost:port_nr and run generation:

conda activate trustllm
python TrustLLM/generate_all.py

Responses are saved to UniComp/TrustLLM/generation_results/{model_name}/

Responses are evaluated by GPT-4 Turbo — an OpenAI API key is required.

1. Swap to OpenAI in config.py: set openai_key="YOUR_KEY" and openai_api_base="https://api.openai.com/v1"
2. Run evaluation:

conda activate trustllm
python evaluate.py --model_name "path/to/model"

Or via SLURM (uncomment the trustllm lines in run.sh):

sbatch run.sh

Results are printed to stdout.

sbatch run_performance.sh