Regression from 4.11.x to 4.12 - 'java.lang.NoClassDefFoundError: android/util/Log$TerribleFailure' in test with no runner annotation

[mikehardy]

Description

We have code that does some configuration on ShadowLog and a test that runs without a specified test runner annotation, and while it worked fine in previous robolectric versions, it now fails with a java.lang.NoClassDefFoundError: android/util/Log$TerribleFailure on version 4.12

Perhaps there has been some inadvertent breaking change with how classpath's are handled/loaded if there is no test runner specified?

Steps to Reproduce

Do this in a test class setup() method:

        ShadowLog.stream = System.out
            // Filters for non-Timber sources. Prefer filtering in RobolectricDebugTree if possible
            // LifecycleMonitor: not needed as we already use registerActivityLifecycleCallbacks for logs
            .filter("^(?!(W/ShadowLegacyPath|D/LifecycleMonitor)).*$") // W/ShadowLegacyPath: android.graphics.Path#op() not supported yet.

Have a test that runs with no runner: https://github.com/ankidroid/Anki-Android/blob/a06e727f6094541b763e2dc18a300edb3e17fd26/AnkiDroid/src/test/java/com/ichi2/anki/RobolectricTestAnnotationTest.kt#L23-L26

import org.hamcrest.MatcherAssert.assertThat
import org.hamcrest.core.StringContains.containsString
import org.junit.Test
import kotlin.test.assertFailsWith

// explicitly missing @RunWith(AndroidJUnit4.class)
class RobolectricTestAnnotationTest : RobolectricTest() {
    @Test
    fun readableErrorIfNotAnnotated() {
        val exception = assertFailsWith<IllegalStateException> {
            @Suppress("UNUSED_VARIABLE")
            val unused = this.targetContext
        }
        assertThat(exception.message, containsString("RobolectricTestAnnotationTest"))
        assertThat(exception.message, containsString("@RunWith(AndroidJUnit4.class)"))
    }
}

Robolectric & Android Version

robolectric 4.12 and android API is default (which I assume means api34 will be used?)

Link to a public git repo demonstrating the problem:

Here's a hopefully useful deeper link to the dependabot PR that failed when it proposed this upgrade:
ankidroid/Anki-Android#16049

and the workflow showing how it failed (reproducible locally via ./gradlew jacocoUnitTestReport after pulling the PR (gh pr checkout 16049)
https://github.com/ankidroid/Anki-Android/actions/runs/8511327583/job/23310697425?pr=16049#step:8:234

[mikehardy]

All of our other tests pass (we've got hundreds! thank you) so I think this must have to do with the missing-on-purpose test runner annotation for the class triggering some sort of different behavior in the new release

[hoisie]

and a test that runs without a specified test runner annotation

I am a little confused by what this means. You have a test that invokes Robolectric APIs (e.g. ShadowLog APIs) but is not running in the context of a Robolectric test?

[hoisie]

Looks like this issue was caused by this commit:
e4d38f4

In retrospect the Android TerribleFailure is a @hide class, so we probably should't reference it directly from ShadowLog.

[hoisie]

Thankfully this specific issue can be fixed. However, just a word of caution: invoking Robolectric APIs outside of the context of a Robolectric ClassLoader is very unlikely to work :) Robolectric APIs often invoke underlying Android framework APIs, and those will not be available outside of Robolectric.

[hoisie]

https://github.com/robolectric/robolectric/releases/tag/robolectric-4.12.1 was just released which should fix this issue.

[mikehardy]

Fantastic! Thanks @hoisie - I hear your warning, and I'll add some documentation to that class in our repo to note that it works now but it is potentially fragile and any issues related to it may be interesting here (this one seems to have been interesting as it surfaced a @hide reference) but also may be uninteresting and the test could just be binned. Appreciated either way, robolectric is a big part of our test infra