Execution order of dynamically registered tests

[jandubois]

I just found out that dynamically registered tests have now been implemented (#349), and they are awesome! Thank you so much for implementing this!

I have one question though: could the dynamically registered tests be executed after the statically defined tests from the file? Right now they are executed first:

$ cat test.bats
@test 'First test' {
    true
}

second_test() {
    true
}

bats_test_function --description 'Second test' -- second_test

$ ./bats-core/bin/bats test.bats
test.bats
 ✓ Second test
 ✓ First test

2 tests, 0 failures

It is of course possible to force the order by dynamically registering all the tests in a file, but that feels a bit clumsy.

Maybe it is a lack of my imagination, but I don't see when I would ever want to run the static tests last. If I have a mix of static and dynamic tests, then the static tests may provide some initial setup that is being utilized by the dynamic test matrix later.

I know there is setup_file, but I may want to run the setup in @test functions because it is part of the functionality being tested, and may fail at various points too.

[martin-schulze-vireso]

Having tests depend in others is a bad idea. Features such as filtering, and shuffling assume that each test works in its own.

However, ist might still be a good idea to execute the tests in the same order as they we're declared.

[jandubois]

it might still be a good idea to execute the tests in the same order as they we're declared.

That seems to have been the case so far, and we have been relying on it for our integration tests.

I hope it will stay this way, that statically declared tests are executed in declaration order, and dynamically registered ones as well (I don't see any reason to interleave the two sets unless you explicitly request shuffling).

I would like to have dynamic tests executed last, but if that is either not desirable by you, or technically difficult, then there is always the workaround of registering everything dynamically.

---

This is mostly some background for what we are doing; not really relevant to the issue itself:

Having tests depend in others is a bad idea. Features such as filtering, and shuffling assume that each test works in its own.

I agree that this is a good idea for unit tests, but it doesn't work that well for integration tests of larger apps.

For example the test that made me file this issue runs tests that:

• Initialize and update some helm repositories
• Do a factory reset to a predictable starting configuration
• Start a virtual machine
• Start a container engine inside the VM
• Start a specific Kubernetes versions on top of the container runtime
• Deploy an app on the Kubernetes cluster
• Verify that the app works
• Tears down the app

This is not about testing the app that is being deployed in the cluster, but about testing the app that provisions the VM, container runtime, Kubernetes, etc.

There is no way to execute these steps in random order; each one depends on all the previous ones. You could put all the steps into a single @test block, but that doesn't give you any granularity in the test results.

I used dynamic test registration to run the test steps above in sequence once for each Kubernetes version specified as a parameter. Except the helm repo setup only needs to happen once, but was executed last until I used bats_test_function to register it first.

So yes, we don't use shuffling of tests, or running them in parallel because we know it won't work.

But otherwise BATS is working brilliantly for running these tests (we run the same tests on Linux, macOS, and Windows, just parameterized differently for each platform).