dynamic test registration

[martin-schulze-vireso]

I played a bit around with the framework to see if we could get parametrized tests working, so this is more of a proof of concept.
This needs #348 but I wanted to make those separate so we can play around here.

I see the following issues with this PR:

1. it adds another layer of execution to the test files, so we now run them n+2 times
2. with bats-exec-suite sourceing the test file, there is even more collision surface between tests and the framework, this becomes especially nasty when there are syntax errors in the test files

Closes #306

• I have reviewed the Contributor Guidelines.
• I have reviewed the Code of Conduct and agree to abide by it

[lucastheisen]

If i understand the code correctly, this basically allows you to define functions:

function assert_equals {
  if [[ "$1" != "$2" ]]; then
    echo "[$1] != [$2]"
  fi
}

Which can be run at any point later by invoking a another function:

bats_test_function --description "one is one" assert_equals "one" "one"
bats_test_function --description "two is two" assert_equals "two" "two"
bats_test_function --description "two is three" assert_equals "two" "three"

Then you would see a single set of results:

 ✓ one is one
 ✓ two is two
 ✗ two is three
   (from function `assert_equals' in file tests.bats, line ?)
     `?' failed

3 tests, 1 failure

This seems sufficiently dynamic for most use cases.

When you say

there is even more collision surface between tests and the framework

is that just to say that bats will be less helpful in pointing out where your syntax errors are? or that things might not actually work because your tests have vars that collide with bats internals?

[martin-schulze-vireso]

When you say

there is even more collision surface between tests and the framework

is that just to say that bats will be less helpful in pointing out where your syntax errors are? or that things might not actually work because your tests have vars that collide with bats internals?

The main problem is: when we hook code into bats, it might mess with internal variables. Additionally, if there is a syntax error in the sourced file, bats will go down with it, potentially swallowing the error (or giving unhelpful filenames due to the preprocessing). I am thinking about jailing this source into an extra bash invocation to avoid this.

[martin-schulze-vireso]

Any thoughts in this @brokenpip3 ?

[brokenpip3]

💪

[debarshiray]

I happened to notice something while reading the code:

[debarshiray]

There doesn't seem to be anything left that's using BATS_TEST_FILTER.

[debarshiray]

I tried to remove BATS_TEST_FILTER in #907

[debarshiray]

This broke test suites with multiple files loading common constants. I have tried to put together a possible fix with test cases to illustrate the problem in #904