From 499d4f8e35273f8038c6ba230f6966f3150f1bc1 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Thu, 25 Jun 2026 09:25:17 +0000 Subject: [PATCH 1/3] Add ChemBench maintainer spotlight --- .../chembench-maintainer-spotlight.md | 113 ++++++++++++++++++ 1 file changed, 113 insertions(+) create mode 100644 content/academia/chembench-maintainer-spotlight.md diff --git a/content/academia/chembench-maintainer-spotlight.md b/content/academia/chembench-maintainer-spotlight.md new file mode 100644 index 00000000..f5a12218 --- /dev/null +++ b/content/academia/chembench-maintainer-spotlight.md @@ -0,0 +1,113 @@ +--- +name: Kevin Maik Jablonka +institution: Friedrich Schiller Universität Jena +department: Chemistry +projectName: ChemBench +projectRepo: https://github.com/lamalab-org/chembench +projectWebsite: https://chembench.lamalab.org/ +maintainerProfiles: + - github: https://github.com/kjappelbaum + - orcid: https://orcid.org/0000-0003-4894-4660 +badges: ["Academic Maintainer", "Research Software Engineer"] +description: "A comprehensive benchmarking framework with over 2,700 chemistry questions that evaluates how well AI language models understand chemistry, spanning basic concepts to complex reasoning tasks." +--- + +## What is this project, and what does it help people do? + +ChemBench tests how well AI language models understand chemistry. We created over 2,700 chemistry questions spanning basic concepts to complex reasoning tasks. When we tested leading AI models against human experts, we found something surprising: the best models outperformed most humans on chemistry knowledge. + +What makes ChemBench different? First, it can process chemical notation and molecular structures, not just text. Second, it includes practical features such as visual model testing and tracking when models refuse to answer. Third, unlike typical one-off evaluations, ChemBench includes governance and maintenance protocols. + +Our results reveal clear patterns. AI models excel at textbook knowledge but struggle with novel reasoning. They perform well on certification-style questions but falter when asked to predict complex molecular behaviour. These gaps highlight where AI training still needs improvement. + +The ChemBench framework has since grown beyond its original purpose. We now use its core architecture to test multimodal models in other scientific domains. By mapping both AI strengths and weaknesses in chemistry, we help researchers build better models and show educators where human expertise remains essential. + +## What inspired you to start this project? + +We needed ChemBench because it didn't exist. While training chemical language models, we couldn't find suitable tools to evaluate them. Existing benchmarks were not comprehensive enough to assess deep chemical understanding. + +What started as filling a technical gap quickly became something more meaningful. Over 30 people joined the effort, including those writing questions, building evaluation tools, and establishing the human performance baseline. Experimental chemists collaborated closely with AI researchers throughout the process. + +## How does this project connect to your academic work? + +ChemBench emerged directly from my research team's work on chemical language models and helped establish a more systematic approach to benchmarking. + +## Who contributes to the project? + +Faculty, students, postdocs, and external contributors. + +## How are students involved in the project? + +Students contribute across all areas, including code, datasets, and testing. + +## How is the project used in teaching or coursework? + +The project contributes to PhD-level research and forms part of thesis work. + +## What impact has this project had on your students? + +Students develop strong software practices and gain visibility within the scientific community. + +## What impact has the project had beyond the classroom or research? + +The project has been adopted by researchers and AI labs, with a growing number of citations. + +## What does it take to maintain the project? + +Three PhD students currently serve as core maintainers. + +## What have been the biggest challenges in maintaining the project? + +Academic systems often undervalue software development, and securing long-term maintainers remains a challenge. + +## How do you ensure the project remains sustainable over time? + +We align research projects with the codebase and build partnerships to support ongoing development. + +## How do you engage with your community? + +Through documentation, onboarding resources, contribution guides, and community forums. + +## Have you taken part in any open source programs or events? + +Not yet, but we are planning to. + +## What would you love to achieve by showcasing your project? + +To attract new contributors and highlight the work of students involved in the project. + +## Do you use AI tools in your day to day work on this project? If so, how? + +GitHub tools such as Copilot and the Developer Pack have been invaluable. + +## Do you implement AI into your classroom or coursework (if applicable)? If so, what does that look like in practice? + +n/a + +## Has AI changed how you maintain or manage your project? + +n/a + +## Have you experimented with AI driven or automated workflows in your project? What has that looked like? + +n/a + +## How do you see your contributors using AI when working on your project? + +n/a + +## What concerns or challenges, if any, do you have about the use of AI in your project or field? + +n/a + +## How has your approach to maintaining this project evolved over time? + +n/a + +## How do you see AI shaping the future of your project or field? + +n/a + +## Is there anything else you'd like to share? + +No additional comments. From 6f59d54181ff26c2b3aecac470e016d68872fc77 Mon Sep 17 00:00:00 2001 From: samus-aran Date: Thu, 25 Jun 2026 10:39:20 +0100 Subject: [PATCH 2/3] Update chembench-maintainer-spotlight.md --- .../chembench-maintainer-spotlight.md | 34 +------------------ 1 file changed, 1 insertion(+), 33 deletions(-) diff --git a/content/academia/chembench-maintainer-spotlight.md b/content/academia/chembench-maintainer-spotlight.md index f5a12218..1b7048b3 100644 --- a/content/academia/chembench-maintainer-spotlight.md +++ b/content/academia/chembench-maintainer-spotlight.md @@ -76,38 +76,6 @@ Not yet, but we are planning to. To attract new contributors and highlight the work of students involved in the project. -## Do you use AI tools in your day to day work on this project? If so, how? +## Is there anything else you'd like to share about your project or open source journey? GitHub tools such as Copilot and the Developer Pack have been invaluable. - -## Do you implement AI into your classroom or coursework (if applicable)? If so, what does that look like in practice? - -n/a - -## Has AI changed how you maintain or manage your project? - -n/a - -## Have you experimented with AI driven or automated workflows in your project? What has that looked like? - -n/a - -## How do you see your contributors using AI when working on your project? - -n/a - -## What concerns or challenges, if any, do you have about the use of AI in your project or field? - -n/a - -## How has your approach to maintaining this project evolved over time? - -n/a - -## How do you see AI shaping the future of your project or field? - -n/a - -## Is there anything else you'd like to share? - -No additional comments. From 0993f0ae4b116f552fac6b900741c30443087672 Mon Sep 17 00:00:00 2001 From: samus-aran Date: Thu, 25 Jun 2026 10:49:34 +0100 Subject: [PATCH 3/3] Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --- content/academia/chembench-maintainer-spotlight.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/academia/chembench-maintainer-spotlight.md b/content/academia/chembench-maintainer-spotlight.md index 1b7048b3..c02d1164 100644 --- a/content/academia/chembench-maintainer-spotlight.md +++ b/content/academia/chembench-maintainer-spotlight.md @@ -70,7 +70,7 @@ Through documentation, onboarding resources, contribution guides, and community ## Have you taken part in any open source programs or events? -Not yet, but we are planning to. +Not yet, but we are planning to participate. ## What would you love to achieve by showcasing your project?