rcf/set-timeout! incompatible with clojure.test which doesn't guarantee test order

[didibus]

I have some tests where I need to set a specific timeout, say to assert that something does timeout. But it seems when we run tests through CI, I can't control what timeout set for which tests.

Normally I do:

(do
  (rcf/set-timeout! 200)
  (tests "That it times out"
    ...))

But like I said, I feel this doesn't generate a deftest that will execute with a 200ms timeout always.

[dustingetz]

What platform, JVM? Can you demonstrate it? Please confirm this is under clojure.test integration only ({:jvm-opts ["-Dhyperfiddle.rcf.generate-tests=true"]}), cannot reproduce without that flag?

[dustingetz]

Can you reproduce using the flag locally outside of CI? What CI provider, github actions?

[didibus]

I'll try and see if I can find a minimal way to reproduce.

It's on JDK 17, OpenSuse tumbleweed Linux.

It's not actually on CI, just running clojure -X:test locally with cognitect test runner.

I did set the flag above, as well as the omit stack trace flag.

I believe the issue is that clojure.test doesn't guarantee the test order without an explicit test-ns-hook or something like that.

So the set-timeouts! are not executed prior to each test as they are defined in the test file. So if you have many tests, each one needing a different timeout, you can't control which test is executed with what timeout when running the tests with a test runner for clojure.test.