🗺️ StableHDMap

As a fundamental module in autonomous driving, online HD mapping has attracted increasing attention due to its cost-effectiveness and real-time capability. However, when a vehicle moves through highly dynamic environments, sensor displacement can cause drift in real-time map predictions. This temporal instability poses a fundamental challenge for downstream tasks. Existing online mapping methods mainly focus on single-frame accuracy, while temporal stability has not been systematically studied.

This work presents the first temporal stability benchmark for online vectorized HD mapping, including:

• A multi-dimensional stability framework: Presence consistency, Localization stability, and Shape stability
• A unified metric mAS (mean Average Stability)
• Large-scale experiments on 42 models and variants, showing that accuracy (mAP) and stability (mAS) are relatively independent performance dimensions
• An open benchmark and toolkit for easy reproduction and extension by the community

This toolkit supports both PKL and NPZ prediction formats. You can evaluate stability directly from saved predictions without rerunning model inference. It is already compatible with outputs from MapTR, PivotNet, BEVMapNet, StreamMapNet, and more.

• Features
• Installation
• Quick Start
• Stability Metrics
• Configurations and Data Formats
• Visualization
• Citation
• Acknowledgement and Contact

maptr_stability_eval/
├── src/maptr_stability_eval/     # Core code
│   ├── geometry/                 # Geometry (polyline and coordinate transform)
│   ├── stability/                # Stability metrics, alignment, and assigner
│   ├── data_parser/              # PKL/NPZ/NuScenes parsing
│   └── utils/                    # Config and utility functions
├── configs/                      # Model configs (maptr/pivotnet/bemapnet/streammapnet)
├── src/vis/                      # Visualization scripts
├── tools/                        # Evaluation and visualization shell scripts
├── main.py                       # Evaluation entry point
├── requirements.txt
└── README.md

• Python >= 3.7
• No GPU or deep learning framework required (evaluation and visualization only)
• Supports Linux / macOS / Windows

git clone <repository-url>
cd maptr_stability_eval
pip install -r requirements.txt
pip install -e .

Optional: install NuScenes support if needed:

pip install nuscenes-devkit

Core dependencies: numpy, scipy, shapely, tqdm, tabulate, matplotlib, seaborn, pandas (see requirements.txt).

python main.py \
    --data-format pkl \
    --prediction-file results.pkl \
    --config configs/maptr_trainval.py \
    --output-dir outputs

python main.py \
    --data-format npz \
    --prediction-file npz_folder/ \
    --config configs/pivotnet_trainval.py \
    --output-dir outputs

For more options (e.g., pred-rotate-deg, pred-swap-xy, pred-flip-x/y), run main.py --help.

Example output:

----------------------------------
MapTR Stability Index Results
----------------------------------
| class        | SI    | presence | localization | shape |
|--------------|-------|----------|--------------|-------|
| divider      | 0.8234| 0.9123   | 0.8456       | 0.7891|
| ped_crossing | 0.7891| 0.8765   | 0.8123       | 0.7456|
| boundary     | 0.8567| 0.9234   | 0.8678       | 0.8234|
| mean         | 0.8231| 0.9041   | 0.8419       | 0.7860|
----------------------------------

• Configurations: configs/ provides *_trainval.py / *_mini.py files by model and dataset (e.g., maptr_trainval.py, pivotnet_trainval.py), including field mapping, class mapping, and stability parameters.
• PKL: A single file where each list item is a sample dictionary, typically containing polylines, classes, scores, sample indices, and related fields (see each config's field_mapping).
• NPZ: One .npz file per token in a folder; expected fields include pts_3d, labels_3d, scores_3d, etc. (see config files).

For detailed field descriptions and examples, refer to comments in each configuration file. PKL/NPZ structures should match the corresponding field_mapping.

# Visualize stability of prediction results
python src/vis/vis_stability.py \
    --prediction-file results.pkl \
    --config configs/maptr_trainval.py \
    --output-dir vis_outputs \
    --data-format pkl

For GT and prediction visualization, see tools/vis_groundtruth.sh, tools/vis_prediction.sh, and scripts under src/vis/.

If this benchmark or toolkit helps your research, please cite:

@inproceedings{stablehdmap2026,
  title     = {Stability Under Scrutiny: Benchmarking Representation Paradigms for Online HD Mapping},
  author    = {Shan, Hao and Li, Ruikai and Jiang, Han and Fan, Yizhe and Yan, Ziyang and Li, Bohan and Hao, Xiaoshuai and Zhao, Hao and Cui, Zhiyong and Ren, Yilong and Yu, Haiyang},
  booktitle = {International Conference on Learning Representations (ICLR)},
  year      = {2026},
  url       = {https://openreview.net/forum?id=mxz5RqhCMe}
}

• Paper: OpenReview
• Project Page: https://stablehdmap.github.io/

• Thanks to OpenMMLab, MapTR, PivotNet, and related works for inspiration and contributions.
• Author: Hao Shan
• Email: [email protected]

Issues and pull requests are welcome.

MIT License

# Unit tests
pytest tests/

# NPZ loading and examples
python test_npz_loader.py
python example_usage.py
python demo_npz_usage.py