idea: agent performance benchmark

[tfriedel]

It's important to know that your tool reduced token usage and doesn't degrade agent performance. One idea is to re-use benchmarks like this:
https://github.com/SWE-rebench/SWE-rebench-V2

I've been using it on a subset of 50 tasks for local benchmarks of Qwen models in Pi. It's relatively cheap or free in my case.

[Pringled]

Hey @tfriedel, this is really interesting, thanks for sharing. We've been thinking about expanding our benchmark with something like this, we'll have a look at this repo to see how feasible it is. Do you have any code you can share from your own benchmarks/experiments that we can have a look at as inspiration?

[tfriedel]

@Pringled
https://github.com/tfriedel/SWE-bench-fork

It's basically just a fork where I prompted claude to create a smaller sample and make it run with my local qwen models.