@setup scenarios run even when Feature is skipped

[edwardsph]

I discovered this in 1.3.0RC2 minutes after 1.3.0 was released and have upgraded to confirm it still an issue.

I created a feature:

@test
Feature: sample karate test script
  Background:
    * url 'https://jsonplaceholder.typicode.com'

  @setup
  Scenario: setup
    * table tests
      | test    |
      | 'users' |

  Scenario Outline: get all <test>
    Given path '<test>'
    When method get
    Then status 200

    Examples:
      | karate.setup().tests |

If I run this without skipping any tags it reports:

features:     1 | skipped:    0 | efficiency: 0.05
scenarios:    2 | passed:     2 | failed: 0

Problem 1: Ideally a @setup scenario would not be included in the counts as it is misleading. It is only run for the benefit of the other scenario.

If I then run with .tags("~@test") it reports:

features:     1 | skipped:    0 | efficiency: 0.00
scenarios:    1 | passed:     1 | failed: 0

Problem 2: This is even more misleading since I wouldn't want any scenarios to count, nor should the feature be counted

If I convert it to a standard scenario with no @setup and run it with the tag skipped the results are:

features:     0 | skipped:    1 | efficiency: 0.00
scenarios:    0 | passed:     0 | failed: 0

This is what I would expect in the second case.

I think it makes sense to show the @setup scenario results in the report but it makes less sense to have it included in test results counts and to impact feature counts in the way it does.

[ptrthomas]

@edwardsph thanks for raising this, I was able to fix the .tags("~@test") case. but I'm still stuck trying to get the reports to not be counted. I'm exploring the option of adding a "fake" first step and treating the @setup like a call. will sleep on it

[ptrthomas]

@edwardsph just made a commit. would be great if you could build-locally and put it through some examples. I think table does not collapse in the HTML report, but other than that - please see if this is a reasonable solution.

we can collect tips on how the Karate HTML report can be improved, we hope to do a clean-up in the next version.

[ptrthomas]

1.3.1 released