Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
81 changes: 81 additions & 0 deletions content/academia/chembench-maintainer-spotlight.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
---
name: Kevin Maik Jablonka
institution: Friedrich Schiller Universität Jena
department: Chemistry
projectName: ChemBench
projectRepo: https://github.com/lamalab-org/chembench
projectWebsite: https://chembench.lamalab.org/
maintainerProfiles:
- github: https://github.com/kjappelbaum
- orcid: https://orcid.org/0000-0003-4894-4660
badges: ["Academic Maintainer", "Research Software Engineer"]
description: "A comprehensive benchmarking framework with over 2,700 chemistry questions that evaluates how well AI language models understand chemistry, spanning basic concepts to complex reasoning tasks."
---

## What is this project, and what does it help people do?

ChemBench tests how well AI language models understand chemistry. We created over 2,700 chemistry questions spanning basic concepts to complex reasoning tasks. When we tested leading AI models against human experts, we found something surprising: the best models outperformed most humans on chemistry knowledge.

What makes ChemBench different? First, it can process chemical notation and molecular structures, not just text. Second, it includes practical features such as visual model testing and tracking when models refuse to answer. Third, unlike typical one-off evaluations, ChemBench includes governance and maintenance protocols.

Our results reveal clear patterns. AI models excel at textbook knowledge but struggle with novel reasoning. They perform well on certification-style questions but falter when asked to predict complex molecular behaviour. These gaps highlight where AI training still needs improvement.

The ChemBench framework has since grown beyond its original purpose. We now use its core architecture to test multimodal models in other scientific domains. By mapping both AI strengths and weaknesses in chemistry, we help researchers build better models and show educators where human expertise remains essential.

## What inspired you to start this project?

We needed ChemBench because it didn't exist. While training chemical language models, we couldn't find suitable tools to evaluate them. Existing benchmarks were not comprehensive enough to assess deep chemical understanding.

What started as filling a technical gap quickly became something more meaningful. Over 30 people joined the effort, including those writing questions, building evaluation tools, and establishing the human performance baseline. Experimental chemists collaborated closely with AI researchers throughout the process.

## How does this project connect to your academic work?

ChemBench emerged directly from my research team's work on chemical language models and helped establish a more systematic approach to benchmarking.

## Who contributes to the project?

Faculty, students, postdocs, and external contributors.

## How are students involved in the project?

Students contribute across all areas, including code, datasets, and testing.

## How is the project used in teaching or coursework?

The project contributes to PhD-level research and forms part of thesis work.

## What impact has this project had on your students?

Students develop strong software practices and gain visibility within the scientific community.

## What impact has the project had beyond the classroom or research?

The project has been adopted by researchers and AI labs, with a growing number of citations.

## What does it take to maintain the project?

Three PhD students currently serve as core maintainers.

## What have been the biggest challenges in maintaining the project?

Academic systems often undervalue software development, and securing long-term maintainers remains a challenge.

## How do you ensure the project remains sustainable over time?

We align research projects with the codebase and build partnerships to support ongoing development.

## How do you engage with your community?

Through documentation, onboarding resources, contribution guides, and community forums.

## Have you taken part in any open source programs or events?

Not yet, but we are planning to participate.

## What would you love to achieve by showcasing your project?

To attract new contributors and highlight the work of students involved in the project.

## Is there anything else you'd like to share about your project or open source journey?

GitHub tools such as Copilot and the Developer Pack have been invaluable.
Comment on lines +79 to +81
Loading