LeMat-GenBench: A Unified Evaluation Framework for Crystal Generative Models

• Charge Neutrality: Ensures structures are charge-balanced using oxidation state analysis and bond valence calculations
• Minimum Interatomic Distance: Validates that atomic distances exceed minimum thresholds based on atomic radii
• Coordination Environment: Checks if coordination numbers match expected values for each element
• Physical Plausibility: Validates density, lattice parameters, crystallographic format, and symmetry

• Jensen-Shannon Distance (JSD): Measures similarity of categorical properties (space groups, crystal systems, elemental compositions) between generated and reference materials
• Maximum Mean Discrepancy (MMD): Measures similarity of continuous properties (volume, density) between generated and reference materials using kernel methods
• Fréchet Distance: Measures similarity of learned structural representations (embeddings) from MLIPs (ORB, MACE, UMA) between generated and reference materials

⚠️ Note: MMD calculations use a 15K sample from LeMat-Bulk dataset due to computational complexity.

• Element Diversity: Measures variety of chemical elements using Vendi scores and Shannon entropy
• Space Group Diversity: Measures variety of crystal symmetries present in generated structures
• Site Number Diversity: Measures variety in the number of atomic sites per structure
• Physical Size Diversity: Measures variety in physical properties (density, lattice parameters, packing factor)

• Novelty Ratio: Fraction of generated structures NOT present in LeMat-Bulk reference dataset
• BAWL Fingerprinting: Uses BAWL structure hashing to efficiently compare against ~5M known materials
• Structure Matcher: Alternative method using pymatgen StructureMatcher for structural comparison

• Uniqueness Ratio: Fraction of unique structures within the generated set (internal diversity)
• BAWL Fingerprinting: Uses BAWL structure hashing to identify duplicate structures efficiently
• Duplicate Detection: Counts and reports duplicate structures within the generated set

• Stability Ratio: Fraction of structures with energy above hull ≤ 0 eV/atom (thermodynamically stable)
• Metastability Ratio: Fraction of structures with energy above hull ≤ 0.1 eV/atom (metastable)
• Mean E_Above_Hull: Average energy above hull across multiple MLIPs (ORB, MACE, UMA)
• Formation Energy: Average formation energy across multiple MLIPs
• Relaxation Stability: RMSE between original and relaxed atomic positions
• Ensemble Statistics: Mean and standard deviation across MLIP predictions for uncertainty quantification

⚠️ Note: Energy above hull calculations may fail for charged species (e.g., Cs+, Br-) as phase diagrams expect neutral compounds.

• Production HHI: Measures supply risk based on concentration of element production sources
• Reserve HHI: Measures long-term supply risk based on concentration of element reserves

• SUN Rate: Fraction of structures that are simultaneously stable (e_above_hull ≤ 0), unique, and novel
• MetaSUN Rate: Fraction of structures that are simultaneously metastable (0 < e_above_hull ≤ 0.1), unique, and novel
• Combined Rate: Fraction of structures that are either stable or metastable, unique, and novel

uv run scripts/run_benchmarks.py --cifs /path/to/cif/directory --config comprehensive_multi_mlip_hull --name my_run

Create a text file with CIF paths:

# my_structures.txt
path/to/structure1.cif
path/to/structure2.cif
path/to/structure3.cif

Then run:

uv run scripts/run_benchmarks.py --cifs my_structures.txt --config comprehensive_multi_mlip_hull --name my_run

uv run scripts/run_benchmarks.py --csv my_structures.csv --config comprehensive_multi_mlip_hull --name my_csv_run

CSV Format Requirements:

• Must contain a column named structure, LeMatStructs, or cif_string
• The structure column should contain either:
  • JSON strings (pymatgen Structure dictionaries) - recommended
  • CIF strings (CIF format text)

Example CSV format:

material_id,structure,other_metadata
0,"{""@module"": ""pymatgen.core.structure"", ""@class"": ""Structure"", ""lattice"": {...}, ""sites"": [...]}",metadata1
1,"{""@module"": ""pymatgen.core.structure"", ""@class"": ""Structure"", ""lattice"": {...}, ""sites"": [...]}",metadata2

Note: You can only use one input method at a time (--cifs OR --csv, not both).

• MMD Reference Sample: Uses 15K samples from LeMat-Bulk for computational efficiency
• MLIP Models: Requires significant computational resources for stability benchmarks
• Memory Usage: Large structure sets may require substantial RAM
• Structure-Matcher: More accurate but computationally expensive than BAWL fingerprinting

• UMA Model: Requires HuggingFace access approval
• ORB Models: Automatically downloaded on first use
• MACE Models: Cached locally after first download

• Formation Energy: Works with charged species (Cs+, Br-, etc.)
• E_above_hull: May fail for charged species (expected behavior)

• Small Sets: Use --families to run only needed benchmarks
• Large Sets: Consider running benchmarks separately for memory efficiency
• Caching: Models are cached locally for faster subsequent runs
• SSH Optimization: Use run_benchmarks_ssh.py for high-core environments

1. UMA Access Denied:

# Ensure you're logged in
huggingface-cli login

# Check access status
huggingface-cli whoami

2. Memory Issues:

# Run fewer families at once
uv run scripts/run_benchmarks.py --cifs structures/ --config validity --families validity --name memory_test

3. Timeout Errors:

• Reduce structure count
• Use faster MLIP models (ORB instead of UMA)
• Increase timeout in configuration

4. Private Dataset Access Error:

# Error: 'Entalpic/LeMaterial-Above-Hull-dataset' doesn't exist on the Hub
# Solution: Download datasets locally (one-time setup)
uv run scripts/download_above_hull_datasets.py

5. Structure-Matcher Performance:

• Structure-matcher is more accurate but much slower than BAWL
• Consider using for smaller datasets or when accuracy is critical
• Use SSH-optimized script for large datasets

• Check the scripts documentation
• Review example configurations in src/config/
• Examine test files for usage patterns

• LeMat-Bulk Dataset: HuggingFace
  Siron, Martin, et al. "LeMat-Bulk: aggregating, and de-duplicating quantum chemistry materials databases." AI for Accelerated Materials Design-ICLR 2025.

• UMA Model: HuggingFace
  Wood, Brandon M., et al. "UMA: A Family of Universal Models for Atoms." arXiv preprint arXiv:2506.23971 (2025).

• ORB Models: GitHub
  Rhodes, Benjamin, et al. "Orb-v3: atomistic simulation at scale." arXiv preprint arXiv:2504.06231 (2025).

• MACE Models: GitHub
  Batatia, Ilyes, et al. "MACE: Higher order equivariant message passing neural networks for fast and accurate force fields." Advances in Neural Information Processing Systems 35 (2022): 11423-11436.

Distribution Metrics:

• Fréchet Distance: FCD Implementation
• Maximum Mean Discrepancy (MMD): Gretton et al. (2012)
  Gretton, Arthur, et al. "A kernel two-sample test." JMLR 13.1 (2012): 723-773.
• Jensen-Shannon Distance: Lin (1991)
  Lin, Jianhua. "Divergence measures based on the Shannon entropy." IEEE Transactions on Information Theory 37.1 (2002): 145-151.

Diversity Metrics:

• Vendi Score: Friedman & Dieng (2023)
  Friedman, Dan, and Adji Bousso Dieng. "The vendi score: A diversity evaluation metric for machine learning." arXiv preprint arXiv:2210.02410 (2022).

Supply Risk Metrics:

• Herfindahl-Hirschman Index (HHI): Mansouri Tehrani et al.
  Mansouri Tehrani, Aria, et al. "Balancing mechanical properties and sustainability in the search for superhard materials." Integrating Materials and Manufacturing Innovation 6.1 (2017): 1-8.