NTRN-195 feat: relayer keyring tests - #31
Conversation
| env: jestEnvironment, | ||
| }; | ||
|
|
||
| const jestPromise = exec( |
There was a problem hiding this comment.
Just an idea, what if we just have interchain_kv_query test in package.json with postfixes equal to backends and defined RELAYER_VERSION for each run? (I hope someday we have all these tests running in CI).
There was a problem hiding this comment.
Of course, I considered this option during implementation, but it has several important (in my opinion) cons:
- Besides configuring the environment options, the test contains very feature-specific logic of watching in logs which backend is actually used by the relayer (
lookForKeyringInContainerLogsfunction). This logic should be implemented in the separate test scenario code, not in theinterchain_kv_queryor any wrapper around it. Without checking the actual keyring in logs there is a chance for test to become false-positive because of refactoring or something (configuration will break and all the test will pass because of usingtestkeyring for example). - It breaks the flow of running all the tests. In my current implementation running
yarn:testruns all the tests, including kering-related. In the proposed solution we will need some wrapper (shell script?) that will runyarn:test,RELAYER_VERSION=os yarn:interchain_kv_query, and so on. It gives a bigger chance for keyring tests to become lost or something. I tried to avoid adding additional layers not to increase the complexity. - As I mentioned in the PR description, the source of the issue is the mixed logic of setting up the environment and testing. Eventually we should separate those; so, some wrapper should set up the environment and run the tests on it. Your proposal is a step in that direction but it leads to some sort of inconsistency (some tests are run this way and some tests are run the other way). Choosing between inconsistent shitty code and consistent shitty code I chose the last.
- The keyring test doesn't need to depend exactly on
interchain_kv_querytest. It just needs any test that makes transactions to Neutron, sointerchain_kv_querymay be replaced with any other test, e.g. less time-consuming one.
There was a problem hiding this comment.
Please tell me if I got your idea wrong 🙃
There was a problem hiding this comment.
- I'm not sure that I really got the idea behind constant testing selection of keyring type. I mean once you have reached it working it can't be broken. It looks overcomplicated.
- Agree
- Agree
- So this way I guess a bit better option would be having some simple action in
relayer_keyring.test.tsitself bc calling one test from another looks not right
There was a problem hiding this comment.
1: Let's assume that logs aren't checked. Let's also assume somebody accidentally removed the line jestEnvironment['RELAYER_VERSION'] = relayerVersion; in the test's code or replaced image: "neutron-org/neutron-query-relayer:${RELAYER_VERSION}" to image: "neutron-org/neutron-query-relayer:base" in the Dockerfile or any other mistake of this kind. In these cases, the tests will falsely pass, and nobody notices (the only way to detect it is code review which is not 100% guaranteed).
4: Yeah, I also don't love it a lot, agree that it maybe should be replaced with something more simple… Still, the current design allows doing so in a very simple manner, while the proposed solution in your first message doesn't (if I got it right). The thing I'm afraid of is that it will contain a lot of copy-paste from the interchain_kv_query/interchain_tx_query tests. But yeah, it might be better than launching test from test.
There was a problem hiding this comment.
1: ok, got it. sounds reasonable 👍
4: So if there is much of c/p we can move some action parts into wrapper/helpers/classes/etc and reuse from the mentioned tests
There was a problem hiding this comment.
4: Actually, I thought about leaving it as technical debt and resolving it in a separate task/PR… It would touch the code that is really out-of-scope of this task which I really scared about (I struggle with the integration tests even while adding the new one, modifying them in the context of this task sounds really unattractive). But maybe merging this test into main in its current form is worse, I dunno.
There was a problem hiding this comment.
Got it. Agree. All-in-all the thing looks really impressive 👍
There was a problem hiding this comment.
Nah, I anyway reorganized the tests because it was a mess
# Conflicts: # setup/Makefile # src/helpers/env.ts # src/testcases/interchain_kv_query.test.ts
# Conflicts: # src/testcases/interchain_kv_query.test.ts
| import { getRemoteHeight } from '../helpers/wait'; | ||
|
|
||
| const lookForKeyringInContainerLogs = async ( | ||
| realayerImageName: string, |
There was a problem hiding this comment.
realayerImageName — misprint here and in several other places nearby
This PR contains the extension of the test framework with tests of the query-relayer configurations with different keyrings (pull request reflecting these changes in query-relayer: neutron-org/neutron-query-relayer#53).
The different configurations of query-relayer are managed by wrapper containers. The default image has a version "base" (see
.nevneardocker-compose.yml) and different configurations have appropriate version names. The version being provided to the test launching command forces docker-compose to take an appropriate version of the image.The main drawback is that we should be more careful with launching the query-relayer without specifying the version (the latest one might have some unpredictable configuration). To avoid it we can differentiate those images by their name and not by version.
The other drawback is that keyring tests don't work in a non-docker environment.
Notions: