🤗Dataset | 🖥️Code | 📄Paper

You can find our dataset on huggingface: 🤗ChartQAPro Dataset

ChartQAPro is supported in VLMEvalKit, the standard evaluation toolkit for Vision–Language Models.
We strongly recommend using VLMEvalKit for consistent and reproducible evaluation.
🔗 https://github.com/open-compass/VLMEvalKit

To evaluate your model on ChartQAPro using our repo codebase (NOT RECOMMENDED), follow the steps below:

Save your model's predictions in a .json file that contains a list of dictionaries.
Each dictionary should include the following keys (first three keys taken from the original huggingface dataset):

• "Answer": the ground truth answer
• "Question Type": the type of the question (e.g., Factoid, MCQ, etc.)
• "Year": useful for evaluating year-based answers
• "prediction": your model’s predicted answer

[
  {
    "Answer": ["2016"]
    "Question Type": "Factoid",
    "Year": ["YES"]
    "prediction": "2016"
  },
  ...
]

pip install anls pandas

python evaluate_predictions.py --predictions-file path/to/your/predictions.json

This will print your model’s performance across different question types and the overall score, following the official evaluation metrics used in the paper. 📊

If you have any questions about this work, please contact Ahmed Masry using the following email addresses: [email protected], [email protected], or [email protected].

If you use ChartQAPro in your research, please cite:

@inproceedings{masry-etal-2025-chartqapro,
    title = "{C}hart{QAP}ro: A More Diverse and Challenging Benchmark for Chart Question Answering",
    author = "Masry, Ahmed  and
      Islam, Mohammed Saidul  and
      Ahmed, Mahir  and
      Bajaj, Aayush  and
      Kabir, Firoz  and
      Kartha, Aaryaman  and
      Laskar, Md Tahmid Rahman  and
      Rahman, Mizanur  and
      Rahman, Shadikur  and
      Shahmohammadi, Mehrad  and
      Thakkar, Megh  and
      Parvez, Md Rizwan  and
      Hoque, Enamul  and
      Joty, Shafiq",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.findings-acl.978/",
    doi = "10.18653/v1/2025.findings-acl.978",
    pages = "19123--19151"
}