suite: TestSubtestPanic is failing

[dolmen]

$ git log -1 --oneline --no-abbrev-commit          
331c520966e81450aef3b50956ea4a5db5e0ccdf (HEAD -> master, origin/master, origin/HEAD) Improve readability
$ go version
go version go1.21.3 darwin/amd64
$ go test -v -race -run TestSubtestPanic -failfast ./suite
=== RUN   TestSubtestPanic
=== RUN   TestSubtestPanic
=== RUN   TestSubtestPanic/TestSubtestPanic
=== RUN   TestSubtestPanic/TestSubtestPanic/subtest
    suite.go:87: test panicked: panic
        goroutine 35 [running]:
        runtime/debug.Stack()
        	/usr/local/Cellar/go/1.21.3/libexec/src/runtime/debug/stack.go:24 +0x67
        github.com/stretchr/testify/suite.failOnPanic(0xc000103040, {0x1646ec0, 0x176d450})
        	/Users/omengue/go/src/github.com/stretchr/testify/suite/suite.go:87 +0x3a
        github.com/stretchr/testify/suite.recoverAndFailOnPanic(0x913f1f8?)
        	/Users/omengue/go/src/github.com/stretchr/testify/suite/suite.go:82 +0x37
        panic({0x1646ec0?, 0x176d450?})
        	/usr/local/Cellar/go/1.21.3/libexec/src/runtime/panic.go:920 +0x270
        github.com/stretchr/testify/suite.(*subtestPanicSuite).TestSubtestPanic.func1()
        	/Users/omengue/go/src/github.com/stretchr/testify/suite/suite_test.go:673 +0x2b
        github.com/stretchr/testify/suite.(*Suite).Run.func1(0xc000103040)
        	/Users/omengue/go/src/github.com/stretchr/testify/suite/suite.go:113 +0x1fa
        testing.tRunner(0xc000103040, 0xc0001355e0)
        	/usr/local/Cellar/go/1.21.3/libexec/src/testing/testing.go:1595 +0x239
        created by testing.(*T).Run in goroutine 34
        	/usr/local/Cellar/go/1.21.3/libexec/src/testing/testing.go:1648 +0x82b
--- FAIL: TestSubtestPanic (0.01s)
    --- FAIL: TestSubtestPanic/TestSubtestPanic (0.00s)
        --- FAIL: TestSubtestPanic/TestSubtestPanic/subtest (0.00s)
--- PASS: TestSubtestPanic (0.01s)
PASS
ok  	github.com/stretchr/testify/suite	1.348s

[dolmen]

@linusbarth Could you fix this? I suppose it is related to #1471.

[linusbarth]

I'm not sure why your pull request fixes this. But I needed some time to even notice that not the test fails but rather this is just output from an internal test.

If I run the test (go test -v -race -run TestSubtestPanic -failfast ./suite) on the branch from your 1502-PR, I see no difference in output. Is this correct?

Also wouldn't it be better to somehow manipulate/mark/remove the output from the internal tests? (If that's even possible)

[dolmen]

@linusbarth #1502 is just a fix for an issue I found thanks to your mistake, but is NOT the fix for 1501. #1503 IS the fix.

[dolmen]

Well. I think that we need both #1502 and #1503. So I will just rebase #1503 on top of #1502.

[dolmen]

We have another test failure in suite:

$ go test -run TestSuiteWithStats -failfast -parallel 1 -count=1 ./suite
--- FAIL: WithStats (0.00s)
    --- FAIL: WithStats/TestPanic (0.00s)
        suite.go:87: test panicked: oops
            goroutine 21 [running]:
            runtime/debug.Stack()
            	/usr/local/Cellar/go/1.21.3/libexec/src/runtime/debug/stack.go:24 +0x5e
            github.com/stretchr/testify/suite.failOnPanic(0xc000134340, {0x13ef4a0, 0x14f7f10})
            	/Users/omengue/go/src/github.com/stretchr/testify/suite/suite.go:87 +0x39
            github.com/stretchr/testify/suite.Run.func1.1()
            	/Users/omengue/go/src/github.com/stretchr/testify/suite/suite.go:184 +0x24c
            panic({0x13ef4a0?, 0x14f7f10?})
            	/usr/local/Cellar/go/1.21.3/libexec/src/runtime/panic.go:914 +0x21f
            github.com/stretchr/testify/suite.(*suiteWithStats).TestPanic(0x0?)
            	/Users/omengue/go/src/github.com/stretchr/testify/suite/suite_test.go:549 +0x25
            reflect.Value.call({0xc0000f9e00?, 0xc00009b008?, 0x13?}, {0x146993b, 0x4}, {0xc00004be70, 0x1, 0x1?})
            	/usr/local/Cellar/go/1.21.3/libexec/src/reflect/value.go:596 +0xce7
            reflect.Value.Call({0xc0000f9e00?, 0xc00009b008?, 0x13d1b2f?}, {0xc000036670?, 0x11180cd?, 0x16d2098?})
            	/usr/local/Cellar/go/1.21.3/libexec/src/reflect/value.go:380 +0xb9
            github.com/stretchr/testify/suite.Run.func1(0xc000134340)
            	/Users/omengue/go/src/github.com/stretchr/testify/suite/suite.go:198 +0x467
            testing.tRunner(0xc000134340, 0xc00012e090)
            	/usr/local/Cellar/go/1.21.3/libexec/src/testing/testing.go:1595 +0xff
            created by testing.(*T).Run in goroutine 20
            	/usr/local/Cellar/go/1.21.3/libexec/src/testing/testing.go:1648 +0x3ad
--- FAIL: TestSuiteWithStats (0.00s)
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x131fd22]

goroutine 19 [running]:
testing.tRunner.func1.2({0x1409f20, 0x17a5c80})
	/usr/local/Cellar/go/1.21.3/libexec/src/testing/testing.go:1545 +0x238
testing.tRunner.func1()
	/usr/local/Cellar/go/1.21.3/libexec/src/testing/testing.go:1548 +0x397
panic({0x1409f20?, 0x17a5c80?})
	/usr/local/Cellar/go/1.21.3/libexec/src/runtime/panic.go:914 +0x21f
github.com/stretchr/testify/suite.TestSuiteWithStats(0x0?)
	/Users/omengue/go/src/github.com/stretchr/testify/suite/suite_test.go:571 +0x242
testing.tRunner(0xc0000831e0, 0x149d878)
	/usr/local/Cellar/go/1.21.3/libexec/src/testing/testing.go:1595 +0xff
created by testing.(*T).Run in goroutine 1
	/usr/local/Cellar/go/1.21.3/libexec/src/testing/testing.go:1648 +0x3ad
FAIL	github.com/stretchr/testify/suite	0.648s
FAIL

[dolmen]

#1503 didn't work. TestSubtestPanic wasn't the only test to have issues.

#1504 is (hopefully) the real fix: use consistent subtest-like names for subtests when running them via testing.InternalTest and testing.RunTests.