[SPARK-48049][BUILD] Upgrade Scala to 2.13.14 #46288

Although we are waiting for Ammonite still, could you base this PR once more, @panbingkun ?

Add support for Scala 2.13.14 com-lihaoyi/Ammonite#1482

Sure, done.

panbingkun · 2024-05-02T09:03:22Z

https://repo1.maven.org/maven2/com/typesafe/genjavadoc/genjavadoc-plugin_2.13.14/0.19/

I have updated the version of genjavadoc in the file project/SparkBuild.scala.

dongjoon-hyun · 2024-05-02T14:19:32Z

Thanks!

dongjoon-hyun

Just a question. CI seems to complain the following. Is this related to this PR?

Error:  Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:3.4.1:enforce (enforce-versions) on project spark-tools_2.13: 
Error:  Rule 3: org.codehaus.mojo.extraenforcer.dependencies.EnforceBytecodeVersion failed with message:
Error:  Found Banned Dependency: org.jline:jline:jar:3.25.1

panbingkun · 2024-05-02T14:50:33Z

jline

I guess it may be related to the fact that the Ammonite supporting scala 2.13.14 has not been released.

panbingkun · 2024-05-02T14:52:00Z

connector/connect/client/jvm/pom.xml

@@ -73,7 +73,7 @@
    </dependency>
    <dependency>
      <groupId>com.lihaoyi</groupId>
-      <artifactId>ammonite_${scala.version}</artifactId>
+      <artifactId>ammonite_2.13.13</artifactId>


Only for test
After the Ammonite supports scala 2.13.14, I will restore it.

Yes, it's merged ten minutes ago. Looking forward to seeing the new release.

dongjoon-hyun · 2024-05-02T15:13:01Z

Although we already use ammonite_2.13.13 without this jline issue, let's wait and see~

I guess it may be related to the fact that the Ammonite supporting scala 2.13.14 has not been released.

panbingkun · 2024-05-03T13:41:11Z

@SethTisue
Just right, you are here too
scala/scala#10717
https://github.com/scala/scala/pull/10717/files#diff-87e6fe6a086cec9825e72357099f0ae6b84800ea025c5aee97b6cc7088c6cd4d

I found that in scala-2.13.14, the version of org.jline:jline was upgraded from 3.14.1 to 3.15.1 because:
scala/bug#12957

But I found that org.jline:jline:3.15.1 is compiled based on JDK 21.

So the maven plugin maven-enforcer-plugin helped us find this issue when Spark upgraded the scala version from 2.13.13 to 2.13.14.
Because our current Spark master branch supports JDK 17 and JDK 21.
Based on the above reasons, I would like to ask if it is also reasonable and feasible for us to exclude org.jline:jline:3.15.1 from org.scala-lang:scala-compiler:jar:2.13.14, and explicitly let it use org.jline:jline:3.14.1?

Thank you!

LuciferYang · 2024-05-03T14:26:14Z

https://github.com/jline/jline3/blob/c75301facc8716b59c1d57d3e3c5943358022560/pom.xml#L93-L95

https://github.com/jline/jline3/blob/c75301facc8716b59c1d57d3e3c5943358022560/pom.xml#L422-L434

The release version of jline 3.25.1 is still Java 8, this should not affect Spark's use of it.

LuciferYang · 2024-05-03T14:44:10Z

[INFO] Restricted to JDK 17 yet org.jline:jline:jar:3.25.1:compile contains org/jline/terminal/impl/ffm/CLibrary$termios.class targeted to 65.-257

Oh... It seems that CLibrary$termios.class is not compatible with Java 8.

In jline 3.25.1, CLibrary references some implementations that were moved to the java.lang.foreign package in Java 21, but in Java 17, they were still in the jdk.incubator.foreign package.

However, in jline 3.24.1, CLibrary also import classes in java.lang.foreign package, it's just that the final jline jar does not include the terminal-ffm module, so it won't trigger a compilation error from the maven-enforcer-plugin.

So it seems that using jline 3.25.1 should not pose any issues. It's unclear whether it's possible to workaround this by modifying the rules of the maven-enforcer-plugin. @panbingkun

LuciferYang · 2024-05-04T06:35:25Z

Do we have a corresponding issue in the Scala community or JLine community (or GitHub issue)?

IMO, a. We can ignore Scala 2.13.14 completely due to this issue. b. We can cut the transitive dependency as you mentioned as Option 1. However, it's Scala's dependency. So, the downstream projects or users (who are not aware of this) will complain again and again. c. We cannot choose Option 2 because the downstream projects or users will be affected and complain again.

For Apache Spark 4.0.0-preview, let's choose (a) because we didn't have enough time to play around with this new Scala version.

Until 4.0.0 release, we had better keep tracking the upstream community (Scala/JLine) activity about this topic. The best case is for them to fix it for all downstream (including us).

+1 for choice Option a if there are no critical issues that need to be fixed by Scala 2.13.14

one issue I can see is Fix ArrayBuilder regression in Scala 2.13.13 (OutOfMemoryError when adding empty arrays)

Let's create an issue in the Scala community? @panbingkun

panbingkun · 2024-05-06T01:11:49Z

Do we have a corresponding issue in the Scala community or JLine community (or GitHub issue)?

IMO, a. We can ignore Scala 2.13.14 completely due to this issue. b. We can cut the transitive dependency as you mentioned as Option 1. However, it's Scala's dependency. So, the downstream projects or users (who are not aware of this) will complain again and again. c. We cannot choose Option 2 because the downstream projects or users will be affected and complain again.

For Apache Spark 4.0.0-preview, let's choose (a) because we didn't have enough time to play around with this new Scala version.

Until 4.0.0 release, we had better keep tracking the upstream community (Scala/JLine) activity about this topic. The best case is for them to fix it for all downstream (including us).

@dongjoon-hyun @LuciferYang
Let's keep an eye on it. I have submitted this issue to the scala community
scala/bug#12994
Thank all!

panbingkun · 2024-05-06T08:48:25Z

@dongjoon-hyun @LuciferYang
Based on the detailed explanation of scala/bug#12994,

and double-check by debug jline on local env. The verification process is as follows:
1.Based on this pr upgrade (the version of scala is 2.13.14 , the version of jline is 3.25.1)
compile spark by commands:

./build/mvn -Pyarn -Pkubernetes -Pvolcano -Phive -Phive-thriftserver -Phadoop-cloud -Pspark-ganglia-lgpl -Pkinesis-asl clean package -DskipTests

2.When open spark-shell with the debug mode on local , and we will see the following run thread stack

export SPARK_SUBMIT_OPTS=-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=8888

3.The code logic of org.jline:jline:3.25.1 is as follows:
https://github.com/jline/jline3/blob/jline-parent-3.25.1/terminal/src/main/java/org/jline/terminal/TerminalBuilder.java#L654-L675

When loading org.jline.terminal.ffm, it will automatically skip because of the exception

java.lang.UnsupportedClassVersionError: org/jline/terminal/impl/ffm/FfmTerminalProvider has been compiled by a more recent version of the Java Runtime (class file version 65.65535), this version of the Java Runtime only recognizes class file versions up to 61.0

so we can completely ignore it in the maven plugin maven-enforcer-plugin#enforceBytecodeVersion, because it will automatically skip because of loading failure in the lower version of JDK (JDK < 21).

dongjoon-hyun · 2024-05-06T16:36:12Z

pom.xml

@@ -2981,6 +2981,9 @@
                    <maxJdkVersion>${java.version}</maxJdkVersion>
                    <ignoredScopes>test</ignoredScopes>
                    <ignoredScopes>provided</ignoredScopes>
+                    <ignoreClasses>
+                      <ignoreClass>org.jline.terminal.impl.ffm.*</ignoreClass>


Please add a comment with the pointer scala/bug#12994 .

dongjoon-hyun · 2024-05-06T16:36:39Z

Thank you, @panbingkun .

dongjoon-hyun · 2024-05-06T16:40:47Z

Is the PR ready?
ammonite 3.0.0-M1` is enough?
- https://ammonite.io/#3.0.0-M1

LuciferYang · 2024-05-07T05:51:46Z

I think we need wait a new Ammonite release version, as 3.0-M1 does not support Scala 2.13.14.

panbingkun · 2024-05-07T07:31:20Z

Is the PR ready?

ammonite 3.0.0-M1` is enough?

https://ammonite.io/#3.0.0-M1

I use this version [3.0.0-M1-19-a7973e17](https://mvnrepository.com/artifact/com.lihaoyi/ammonite_3.3.3/3.0.0-M1-19-a7973e17), which currently supports scala 2.13.14
https://mvnrepository.com/artifact/com.lihaoyi/ammonite

LuciferYang · 2024-05-07T11:42:43Z

Is the PR ready?

ammonite 3.0.0-M1` is enough?

https://ammonite.io/#3.0.0-M1

I use this version [3.0.0-M1-19-a7973e17](https://mvnrepository.com/artifact/com.lihaoyi/ammonite_3.3.3/3.0.0-M1-19-a7973e17), which currently supports scala 2.13.14 https://mvnrepository.com/artifact/com.lihaoyi/ammonite

This seems to be a snapshot or test version, not an official release？

panbingkun · 2024-05-07T11:49:08Z

I have asked the community to release an official version for us at his convenience
com-lihaoyi/Ammonite#1481 (comment)

dongjoon-hyun · 2024-05-07T15:53:28Z

Thank you for confirming. Ya, let's wait.

LuciferYang · 2024-05-14T09:13:02Z

https://github.com/com-lihaoyi/Ammonite/releases/tag/3.0.0-M2

3.0.0-M2 released ~ @panbingkun

panbingkun · 2024-05-14T09:22:15Z

https://github.com/com-lihaoyi/Ammonite/releases/tag/3.0.0-M2

3.0.0-M2 released ~ @panbingkun

Thanks~ ❤️
Updated.

LuciferYang · 2024-05-14T12:04:20Z

project/SparkBuild.scala

@@ -266,7 +266,7 @@ object SparkBuild extends PomBuild {
      .orElse(sys.props.get("java.home").map { p => new File(p).getParentFile().getAbsolutePath() })
      .map(file),
    publishMavenStyle := true,
-    unidocGenjavadocVersion := "0.18",
+    unidocGenjavadocVersion := "0.19",


Please upgrade GenJavadoc in a separate pr first. @panbingkun

Done
A separate pr for upgrade GenJavadoc: #46579

@LuciferYang
I think we can upgrade scala to 2.13.14 only after #46579 is merged.
1.https://mvnrepository.com/artifact/com.typesafe.genjavadoc/genjavadoc-plugin

2.https://github.com/panbingkun/spark/actions/runs/9079936342/job/24951167973

LuciferYang

+1, LGTM

dongjoon-hyun

+1, LGTM. Thank you, @panbingkun and @LuciferYang and all.
Merged to master for Apache Spark 4.0.0-preview2-rc2.

[SPARK-48049][BUILD] Upgrade Scala to 2.13.14

61446af

github-actions bot added BUILD DOCS labels Apr 29, 2024

dongjoon-hyun reviewed Apr 29, 2024

View reviewed changes

panbingkun changed the title ~~[WIP][SPARK-48049][BUILD] Upgrade Scala to 2.13.14~~ [SPARK-48049][BUILD] Upgrade Scala to 2.13.14 Apr 30, 2024

dongjoon-hyun reviewed May 1, 2024

View reviewed changes

Merge branch 'master' into SPARK-48049

4b21ea3

update genjavadoc to 0.19

22fcc87

only for test

7de0949

github-actions bot added the CONNECT label May 2, 2024

dongjoon-hyun reviewed May 2, 2024

View reviewed changes

panbingkun commented May 2, 2024

View reviewed changes

panbingkun added 3 commits May 3, 2024 06:24

Merge branch 'master' into SPARK-48049

dd13f5d

exclude jline 3.25.1

979a667

Merge branch 'master' into SPARK-48049

b3c7085

github-actions bot removed the CONNECT label May 3, 2024

panbingkun mentioned this pull request May 6, 2024

Scala 2.13.14 depends on jline-3.25.1, which contains a classfile that is not compatible with JDK17 (only JDK21 is supported) scala/bug#12994

Closed

panbingkun added 2 commits May 6, 2024 15:01

Merge branch 'master' into SPARK-48049

0067cf4

ignore org.jline.terminal.impl.ffm.*

3e5a24a

dongjoon-hyun reviewed May 6, 2024

View reviewed changes

Merge branch 'master' into SPARK-48049

a2fd3b4

Merge branch 'master' into SPARK-48049

7f37964

add some comment

0bd3a97

panbingkun added 2 commits May 14, 2024 17:19

Merge branch 'master' into SPARK-48049

b347491

update ammonite to 3.0.0-M2

431f7a8

panbingkun marked this pull request as ready for review May 14, 2024 09:24

LuciferYang reviewed May 14, 2024

View reviewed changes

panbingkun added 3 commits May 14, 2024 21:07

Merge branch 'master' into SPARK-48049

c8509f8

update

7e9fd24

Merge branch 'master' into SPARK-48049

626c0d4

LuciferYang approved these changes May 15, 2024

View reviewed changes

dongjoon-hyun approved these changes May 15, 2024

View reviewed changes

dongjoon-hyun closed this in 12820e1 May 15, 2024

[SPARK-48049][BUILD] Upgrade Scala to 2.13.14 #46288

[SPARK-48049][BUILD] Upgrade Scala to 2.13.14 #46288

Conversation

panbingkun commented Apr 29, 2024 • edited

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

panbingkun commented Apr 29, 2024

dongjoon-hyun left a comment

Choose a reason for hiding this comment

dongjoon-hyun commented Apr 29, 2024

panbingkun commented Apr 30, 2024

dongjoon-hyun left a comment

Choose a reason for hiding this comment

SethTisue commented May 1, 2024

dongjoon-hyun commented May 1, 2024

dongjoon-hyun commented May 1, 2024

panbingkun commented May 2, 2024

panbingkun commented May 2, 2024

dongjoon-hyun commented May 2, 2024

dongjoon-hyun left a comment

Choose a reason for hiding this comment

panbingkun commented May 2, 2024 • edited

panbingkun May 2, 2024

Choose a reason for hiding this comment

dongjoon-hyun May 2, 2024

Choose a reason for hiding this comment

dongjoon-hyun commented May 2, 2024

panbingkun commented May 3, 2024 • edited

LuciferYang commented May 3, 2024

LuciferYang commented May 3, 2024 • edited

LuciferYang commented May 4, 2024

panbingkun commented May 6, 2024

panbingkun commented May 6, 2024 • edited

dongjoon-hyun May 6, 2024

Choose a reason for hiding this comment

dongjoon-hyun commented May 6, 2024

dongjoon-hyun commented May 6, 2024

LuciferYang commented May 7, 2024

panbingkun commented May 7, 2024

LuciferYang commented May 7, 2024

panbingkun commented May 7, 2024 • edited

dongjoon-hyun commented May 7, 2024

LuciferYang commented May 14, 2024

panbingkun commented May 14, 2024

LuciferYang May 14, 2024

Choose a reason for hiding this comment

panbingkun May 14, 2024 • edited

Choose a reason for hiding this comment

panbingkun May 14, 2024 • edited

Choose a reason for hiding this comment

LuciferYang left a comment

Choose a reason for hiding this comment

dongjoon-hyun left a comment

Choose a reason for hiding this comment

panbingkun commented Apr 29, 2024 •

edited

panbingkun commented May 2, 2024 •

edited

panbingkun commented May 3, 2024 •

edited

LuciferYang commented May 3, 2024 •

edited

panbingkun commented May 6, 2024 •

edited

panbingkun commented May 7, 2024 •

edited

panbingkun May 14, 2024 •

edited

panbingkun May 14, 2024 •

edited