The MIT Datacenter Challenge provides high-performance computing (HPC) cluster data for research on workload characterization, resource utilization, anomaly detection, and system failure prediction.
The MIT Supercloud Dataset combines job scheduler records with job-specific measurements of compute, memory, storage, power, and temperature. It supports the development and evaluation of statistical and machine learning methods for the analysis of computing systems.
Version 1: Data and documentation | Version 2: Release status
Dataset description
The dataset includes high-level records from the Slurm Workload Manager and lower-level time series collected on the order of seconds for individual jobs. These complementary data sources allow joint analysis of scheduling decisions, resource consumption, and observed system behavior.
| Data category | Measurements and attributes |
|---|---|
| Scheduler records | Requested nodes and CPU, GPU, and memory resources; job run times; exit codes. |
| Resource utilization | CPU, GPU, and memory utilization; disk input/output. |
| Environmental measurements | Power consumption and temperature. |
Research topics
Scheduling characterization
Analysis of job submission rates and temporal patterns, evaluation of resource allocation policies, and prediction of job run times.
Workload characterization
Classification of traditional HPC workloads and AI training workloads using scheduler attributes and resource utilization measurements.
File system characterization
Characterization of file system activity through combined analysis of scheduler records and job-specific time series.
Anomaly detection and failure prediction
Development and evaluation of methods to detect anomalous behavior, investigate its causes, and predict job failures.
Publications
DCC Version 1
The dataset, collection methodology, and preliminary analyses are described in:
Samsi, Siddharth, Weiss, Matthew, Bestor, David, et al. “The MIT Supercloud Dataset.” 2021 IEEE High Performance Extreme Computing Conference (HPEC). IEEE, 2021.
arXiv:2108.02037 | IEEE Xplore
Please cite this publication when using or referencing the Version 1 dataset.
DCC Version 2
Piotr Luszczek, Daniel Burrill, William Bergeron, Vijay Gadepally, Matthew Hubbell, et al. “The MIT AI Systems Dataset: Environmental, Compute, and Workload Traces.” IEEE High Performance Extreme Computing Conference (HPEC), September 2026.
Data availability
Version 1 is available through the Amazon Open Data Registry. Download commands, storage requirements, and supporting documentation are provided on the Data page.
Version 2 is coming soon (October 2026). Release details and a DocuSign data use agreement will be provided in the Version 2 section.
Project context
The MIT Datacenter Challenge is part of the FastAI project within the MIT-USAF AI Accelerator, a joint program conducting fundamental AI research to support Department of the Air Force operations and broader societal needs.
FastAI studies the development of portable, high-performance AI applications, including programming languages, compiler technologies, comprehensive instrumentation, analytical productivity tools, and parallel algorithms.
Additional information: MIT-USAF AI Accelerator research.
Contact
Questions about the dataset and the Datacenter Challenge may be directed to [email protected].