Intermittent Stream closed exception after upgrading to 1.10.2
#1572
Assignees
Labels
Comments
bvolpato
added a commit
to bvolpato/DataflowTemplates
that referenced
this issue
Aug 3, 2023
…osed" Filed google/auto#1572 and rolling it back for now
|
I dug around in the code for a bit to see what might have changed between 1.8.2 and 1.10.2 that could be a cause for this, and what looked most likely to me is the change to parse files for It seems like that could be a problem here because All that said, I'm not sure why opening that stream seemingly must be succeeding in this stack trace and the exception is then occurring while reading from from it. Maybe something is closing the original jar stream that the static reference to |
alexeykukuku
pushed a commit
to alexeykukuku/DataflowTemplates
that referenced
this issue
Aug 3, 2023
…osed" Filed google/auto#1572 and rolling it back for now
|
Thanks @cgdecker, I think your theory is correct. From the stack traces, it looks like this is happening on Java 8. I think the issue is JDK-6947916, which was fixed in Java 9. We already have a workaround but as Colin notes that only applies when the original I put a lot of effort into making this workaround, including a ridiculously complicated regression test, then unknowingly undid it all with the change Colin mentions. While it's probably possible to come up with a new workaround, I think it would be a lot easier just to build with a more recent JDK. JDK 8 was released in 2014 and is more or less EOL now. Unless you can confirm that it is in fact a more recent JDK? (I'm assuming it's 8 because there are no module names in the stack traces.) |
|
(You can still build with |
|
Looking at the failed build link, it looks like Maven should be compiling with JDK 11 in this case: Specifically with: I wonder if changing the |
|
Thanks for looking into this! Right, it should be Java 11. I got locally too with After running It's hard to bisect because it takes a bit to run, but I am pretty confident it is not happening with a single thread (or the probability is minimal), as the loop ran for hours with Appreciate you guys looking -- but note that I understand this is a very specific use-case and this is not urgent. We reverted to 1.8.2 for the time being. |
|
I'm still puzzled at the absence of module names in those stack traces. Could you do this experiment in your environment? It's those I'm persisting here just because it still seems like a big coincidence that (1) the stack traces look like Java 8 and (2) the symptoms correspond exactly to a bug that was fixed in Java 9. |
|
Curiously, I just ran the |
|
For example, if the |
|
We also hit this problem quite often these day. Is there a plan to get this prioritized a bit? Thank you. |
Can you confirm whether you are using JDK 8 or a later JDK version? |
|
I can confirm we use Java 11 and compile with 11 as release target as well. We have Maven parallell build enabled. |
|
OK. If you have an exception stack-trace, can you show it? I'm curious if it contains module names, as I mentioned in an earlier comment. But I think it is probably possible to rework the code so that the existing workaround is applied more generally. |
|
Sure. There it is: |
|
Huh. That certainly does look like a Java 11 stacktrace. But JDK-6947916 is supposed to be fixed there. So perhaps there is another related issue? Anyway, I think this can be worked around and will look into that. |
copybara-service bot
pushed a commit
that referenced
this issue
Sep 7, 2023
We now always read template resources directly from the jar file containing them, rather than initially trying to use `getResourceAsStream`. That can trigger [JDK-6947916](https://bugs.openjdk.org/browse/JDK-6947916) and our existing fallback-to-workaround logic was ineffective with recent versions of EscapeVelocity. Even though the JDK bug was supposedly fixed in JDK 9 and later JDK 8 updates, people are still reporting issues with AutoValue that look exactly like it. Reading directly from the jar file should not be _too_ inefficient, and each read should only happen once per compilation, no matter how many `@AutoValue` classes there are in the compilation. Fixes #1572. RELNOTES=A workaround for a JDK bug with reading jar resources has been extended so it always applies, rather than just as a fallback. PiperOrigin-RevId: 563251940
copybara-service bot
pushed a commit
that referenced
this issue
Sep 7, 2023
We now always read template resources directly from the jar file containing them, rather than initially trying to use `getResourceAsStream`. That can trigger [JDK-6947916](https://bugs.openjdk.org/browse/JDK-6947916) and our existing fallback-to-workaround logic was ineffective with recent versions of EscapeVelocity. Even though the JDK bug was supposedly fixed in JDK 9 and later JDK 8 updates, people are still reporting issues with AutoValue that look exactly like it. Reading directly from the jar file should not be _too_ inefficient, and each read should only happen once per compilation, no matter how many `@AutoValue` classes there are in the compilation. Fixes #1572. RELNOTES=A workaround for a JDK bug with reading jar resources has been extended so it always applies, rather than just as a fallback. PiperOrigin-RevId: 563251940
copybara-service bot
pushed a commit
that referenced
this issue
Sep 8, 2023
We now always read template resources directly from the jar file containing them, rather than initially trying to use `getResourceAsStream`. That can trigger [JDK-6947916](https://bugs.openjdk.org/browse/JDK-6947916) and our existing fallback-to-workaround logic was ineffective with recent versions of EscapeVelocity. Even though the JDK bug was supposedly fixed in JDK 9 and later JDK 8 updates, people are still reporting issues with AutoValue that look exactly like it. Reading directly from the jar file should not be _too_ inefficient, and each read should only happen once per compilation, no matter how many `@AutoValue` classes there are in the compilation. Fixes #1572. RELNOTES=A workaround for a JDK bug with reading jar resources has been extended so it always applies, rather than just as a fallback. See #1572. PiperOrigin-RevId: 563251940
copybara-service bot
pushed a commit
that referenced
this issue
Sep 8, 2023
We now always read template resources directly from the jar file containing them, rather than initially trying to use `getResourceAsStream`. That can trigger [JDK-6947916](https://bugs.openjdk.org/browse/JDK-6947916) and our existing fallback-to-workaround logic was ineffective with recent versions of EscapeVelocity. Even though the JDK bug was supposedly fixed in JDK 9 and later JDK 8 updates, people are still reporting issues with AutoValue that look exactly like it. Reading directly from the jar file should not be _too_ inefficient, and each read should only happen once per compilation, no matter how many `@AutoValue` classes there are in the compilation. Fixes #1572. RELNOTES=A workaround for a JDK bug with reading jar resources has been extended so it always applies, rather than just as a fallback. See #1572. PiperOrigin-RevId: 563251940
copybara-service bot
pushed a commit
that referenced
this issue
Sep 8, 2023
We now always read template resources directly from the jar file containing them, rather than initially trying to use `getResourceAsStream`. That can trigger [JDK-6947916](https://bugs.openjdk.org/browse/JDK-6947916) and our existing fallback-to-workaround logic was ineffective with recent versions of EscapeVelocity. Even though the JDK bug was supposedly fixed in JDK 9 and later JDK 8 updates, people are still reporting issues with AutoValue that look exactly like it. Reading directly from the jar file should not be _too_ inefficient, and each read should only happen once per compilation, no matter how many `@AutoValue` classes there are in the compilation. Fixes #1572. RELNOTES=A workaround for a JDK bug with reading jar resources has been extended so it always applies, rather than just as a fallback. See #1572. PiperOrigin-RevId: 563251940
copybara-service bot
pushed a commit
that referenced
this issue
Sep 8, 2023
We now always read template resources directly from the jar file containing them, rather than initially trying to use `getResourceAsStream`. That can trigger [JDK-6947916](https://bugs.openjdk.org/browse/JDK-6947916) and our existing fallback-to-workaround logic was ineffective with recent versions of EscapeVelocity. Even though the JDK bug was supposedly fixed in JDK 9 and later JDK 8 updates, people are still reporting issues with AutoValue that look exactly like it. Reading directly from the jar file should not be _too_ inefficient, and each read should only happen once per compilation, no matter how many `@AutoValue` classes there are in the compilation. Fixes #1572. RELNOTES=A workaround for a JDK bug with reading jar resources has been extended so it always applies, rather than just as a fallback. See #1572. PiperOrigin-RevId: 563251940
|
I made it so that we no longer ever use |
|
@eamonnmcmanus Thank you so much getting this fixed! I'm looking forward to trying in the next release. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
and others
Hello!
We recently started observing an intermittent error after upgrading from 1.8.2 to 1.10.2 (GoogleCloudPlatform/DataflowTemplates#929).
I suspect it might be related to some racing/concurrency issues, as it happens sporadically and we are running Maven using multiple threads (
mvn -T8 ...). It is either very hard to get when running in a single thread (I tried a bunch of times and it didn't repro), or parallelism is actually the issue.Build https://github.com/GoogleCloudPlatform/DataflowTemplates/actions/runs/5734638367/job/15541176096?pr=934
Full stacktrace:
The text was updated successfully, but these errors were encountered: