DIVERSE: Disagreement-Inducing Vector Evolution For Rashomon Set Exploration [ICLR 2026]

Gilles Eerlings^1,2,3  ·  Brent Zoomers^1,2,4  ·  Jori Liesenborgs^1,2  ·  Gustavo Rovelo Ruiz^1,2  ·  Kris Luyten^1,2,3

¹ UHasselt Digital Future Lab  ·  ² Flanders Make  ·  ³ Flanders AI Research  ·  ⁴ FWO (Fonds Wetenschappelijk Onderzoek – Vlaanderen)

DIVERSE is a framework for systematically exploring the Rashomon set of neural networks, which is the collection of models that achieve similar accuracy to a reference model while differing in their predictive behavior.
The method augments pretrained networks with Feature-wise Linear Modulation (FiLM) layers and uses Covariance Matrix Adaptation Evolution Strategy (CMA-ES) to explore a latent modulation space, discovering diverse model variants without retraining or requiring gradient access.

• 🎛 FiLM-based modulation: Introduces lightweight FiLM layers into a frozen pretrained model, enabling controlled activation shifts via a latent vector.
• 🧬 CMA-ES optimization: Gradient-free evolutionary search over latent vectors, targeting disagreement while maintaining accuracy.
• 📊 Rigorous Rashomon protocol: Enforces Rashomon membership on a validation set and reports diversity only on a held-out test set.
• ⚡ Scalable exploration: Substantially reduces computational costs compared to retraining-based approaches.

System

• Tested on Ubuntu 22.04 LTS
• Requires a bash shell to run experiment scripts (.sh)
• Requires an NVIDIA GPU with CUDA support
  (tested on GeForce RTX 4090 with CUDA 12.4)
• Windows users may run the code inside WSL2 or a Linux container
  (not officially tested)

Software

• Anaconda
• Python 3.12.9

conda env create -f environment.yml

Activate the new environment: conda activate diverse

First, make the provided bash script executable:

chmod +x init.sh

Then run it:

./init.sh

This will:

1. Train the reference (pretrained) models for each dataset.
2. Generate initial latent vectors (Z seeds) used for CMA-ES exploration.

Before running, ensure the conda environment is activated:

conda activate diverse

This script performs an extensive hyperparameter sweep, which can take a long time and heavily use the GPU. Parallelism is controlled through subprocesses; to adjust the number of workers, edit utils/experiment_parameters.py

You have to repeat the following command for each epsilon and each model type (exception for the vision transformer):

python run_epsilon_CMA.py --model_type=<model_type> --epsilon=<epsilon>

Once you have run all CMA-ES runs, you can evaluate each epsilon, z dimension and dataset combination with the following:

python -m CMA.CMA_evaluation --model_type=<model_type> --epsilon=<epsilon> --z_dim=<z_dim>

Before running any baselines, ensure the conda environment is activated:

conda activate diverse

We provide an implementation of the dropout-based Rashomon exploration method described in Hsu et al. (ICLR 2024).

python -m baselines.dropout --model=<model_type> --epsilon=<epsilon> --search_budget=<search_budget>

Warning: Retraining is computationally expensive and may require significant time and GPU resources.

python -m baselines.retraining --model=<model> --start_seed=<start_seed> --search_budget=<search_budget>

After training, results can be evaluated on the test set. Outputs will be stored in a folder in the following path: baseline_evaluations/retraining/retraining_<model>/epsilon_<epsilon_value>/

python -m baselines.retraining_evaluator --model=<model> --epsilon=<epsilon> --search_budget=<search_budget>

To plot the results, you will first have to run each CMA search and evaluation for every dataset and epsilon, and have also run and evaluate all baselines for each dataset on the same epsilons.

Once the results are available, generate the plots with:

python -m utils.plotter

@inproceedings{
  eerlings2026diverse,
  title={{DIVERSE}: Disagreement-Inducing Vector Evolution for Rashomon Set Exploration},
  author={Gilles Eerlings and Brent Zoomers and Jori Liesenborgs and Gustavo Rovelo Ruiz and Kris Luyten},
  booktitle={The Fourteenth International Conference on Learning Representations},
  year={2026},
  url={https://openreview.net/forum?id=kQjSUHC84V}
}