[SPARK-47234][BUILD] Upgrade Scala to 2.13.13 #45342

panbingkun · 2024-02-29T23:41:51Z

What changes were proposed in this pull request?

The pr aims to upgrade scala from 2.13.12 to 2.13.13.

Why are the changes needed?

The new version bring some bug fixes:
fix: SliceIterator to drop a max of Int.MaxValue scala/scala#10525
Fixes issue with blocking and BatchingExecutor scala/scala#10528
The release notes as follows: https://github.com/scala/scala/releases/tag/v2.13.13

Does this PR introduce any user-facing change?

Yes, The scala version is changed from 2.13.12 to 2.13.13.

How was this patch tested?

Pass GA.
After the master is upgraded to this version 2.13.13, we need to continue to observe.

Was this patch authored or co-authored using generative AI tooling?

No.

panbingkun · 2024-03-01T00:53:07Z

From GA testing, it seems that ammonite has not yet supported scala 2.13.13.
com.lihaoyi:ammonite_2.13.13:2.5.11

A PR has been submitted to the upstream ammonite community to address this issue:
com-lihaoyi/Ammonite#1430

dongjoon-hyun · 2024-03-01T04:53:59Z

Thank you, @panbingkun !

dongjoon-hyun

Actually, according to the release note, we need to upgrade SBT too.

Users of sbt must upgrade to sbt 1.9.9 to use the console task to access the 2.13.13 REPL (sbt/sbt#7502); error message is NoSuchMethodError: org.jline.utils.AttributedString.fromAnsi

I also saw your SBT 1.9.9 test PR and its failure.

[WIP][Only Test] Test sbt 1.9.9 #45345

In the worst case, we cannot upgrade to Scala 2.13.13.

panbingkun · 2024-03-01T05:53:23Z

Actually, according to the release note, we need to upgrade SBT too.

Users of sbt must upgrade to sbt 1.9.9 to use the console task to access the 2.13.13 REPL (sbt/sbt#7502); error message is NoSuchMethodError: org.jline.utils.AttributedString.fromAnsi

I also saw your SBT 1.9.9 test PR and its failure.

[WIP][Only Test] Test sbt 1.9.9 #45345

In the worst case, we cannot upgrade to Scala 2.13.13.

Yes, but unfortunately, from the test results of another PR, it seems that sbt 1.9.9 still has issues starting from version sbt 1.9.4. (When there is a corrupted jar in Maven's local repo, it will fail)

dongjoon-hyun · 2024-03-01T06:19:18Z

Yes, that's what I meant exactly. Scala 2.13.13 requires SBT 1.9.9 but SBT 1.9.9 is not working properly. In short, there is no working combination in Scala 2.13.13 and this is the worst case where we are currently standing. I don't know if SBT 1.9.10 might solve every issues, but it's also unlikely.

Yes, but unfortunately, from the test results of another PR, it seems that sbt 1.9.9 still has issues starting from version sbt 1.9.4. (When there is a corrupted jar in Maven's local repo, it will fail)

panbingkun · 2024-03-01T07:34:00Z

Yes, that's what I meant exactly. Scala 2.13.13 requires SBT 1.9.9 but SBT 1.9.9 is not working properly. In short, there is no working combination in Scala 2.13.13 and this is the worst case where we are currently standing. I don't know if SBT 1.9.10 might solve every issues, but it's also unlikely.

Yes, but unfortunately, from the test results of another PR, it seems that sbt 1.9.9 still has issues starting from version sbt 1.9.4. (When there is a corrupted jar in Maven's local repo, it will fail)

I have already submitted a bug report to SBT: sbt/sbt#7506

But it seems to be a problem with the coursier .

cc @LuciferYang

LuciferYang · 2024-03-01T08:19:42Z

From GA testing, it seems that ammonite has not yet supported scala 2.13.13. com.lihaoyi:ammonite_2.13.13:2.5.11

A PR has been submitted to the upstream ammonite community to address this issue: com-lihaoyi/Ammonite#1430

In fact, com-lihaoyi/Ammonite#1421 is already in the process of upgrading, there are probably some tests that have not passed yet

dongjoon-hyun · 2024-03-01T16:56:31Z

Thank you for updates for both.

dongjoon-hyun · 2024-03-14T21:07:37Z

It seems that 3.0.0-M1 is released last week.

Could you rebase the PR? I guess the only remaining issue is SBT?

panbingkun · 2024-03-15T00:51:08Z

It seems that 3.0.0-M1 is released last week.

Could you rebase the PR? I guess the only remaining issue is SBT?

Okay, let me try it. Thanks for reminding me. ❤️

dongjoon-hyun · 2024-03-15T02:35:26Z

pom.xml

@@ -226,7 +226,7 @@
    ./python/pyspark/sql/pandas/utils.py, and ./python/setup.py too.
    -->
    <arrow.version>15.0.0</arrow.version>
-    <ammonite.version>2.5.11</ammonite.version>
+    <ammonite.version>3.0.0-M1</ammonite.version>


Could you update the dependency list? The other tests seem to pass.

+++ b/dev/pr-deps/spark-deps-hadoop-3-hive-2.3 @@ -139,7 +139,7 @@ jettison/1.5.4//jettison-1.5.4.jar jetty-util-ajax/11.0.20//jetty-util-ajax-11.0.20.jar jetty-util/11.0.20//jetty-util-11.0.20.jar jline/2.14.6//jline-2.14.6.jar -jline/3.22.0//jline-3.22.0.jar +jline/3.24.1//jline-3.24.1.jar

BTW, it seems that we have two jline libraries at the same time.

Let me analyze it.

No problem, it has been checked before that their package names are different, so there will be no conflict

jline 2

jline 3

jline-2.14.6.jar

<groupId>jline</groupId> <artifactId>jline</artifactId> <version>2.14.6</version>

jline-3.22.0.jar

<groupId>org.jline</groupId> <artifactId>jline</artifactId> <version>3.22.0</version>

I'll see what depends on them. If they can be aligned, it would be great if only one is left.
Of course, that's another issue.

https://github.com/apache/hive/blob/36b32612ba0a20d3d305c77a813aca62d7129ed0/cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java#L41-L49

From the hive 2.3 version code, it seems that we cannot align here because its import of jline starts with jline....

Let me try in SPARK-47405 independently in the scope of upgrading spark-sql shell to use jline3 . If it's difficult, we can live with this as you confirmed that they are different packages now.

dongjoon-hyun · 2024-03-15T04:04:45Z

If CI passed, shall we land this to master first to investigate SBT issues further, @panbingkun and @LuciferYang ? It would be must faster to find any potential issues if this patch is in master branch because all people and Daily CIs will verify this.

panbingkun · 2024-03-15T05:25:33Z

If CI passed, shall we land this to master first to investigate SBT issues further, @panbingkun and @LuciferYang ? It would be must faster to find any potential issues if this patch is in master branch because all people and Daily CIs will verify this.

I agree with you!
Let me add to the description of this PR.

dongjoon-hyun

+1, LGTM. Thank you, @panbingkun and @LuciferYang .
Merged to master for Apache Spark 4.0.0.

### What changes were proposed in this pull request? The pr aims to upgrade scala from `2.13.12` to `2.13.13`. ### Why are the changes needed? - The new version bring some bug fixes: scala/scala#10525 scala/scala#10528 - The release notes as follows: https://github.com/scala/scala/releases/tag/v2.13.13 ### Does this PR introduce _any_ user-facing change? Yes, The `scala` version is changed from `2.13.12` to `2.13.13`. ### How was this patch tested? - Pass GA. - After the master is upgraded to this version `2.13.13`, we need to continue to observe. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#45342 from panbingkun/SPARK-47234. Authored-by: panbingkun <panbingkun@baidu.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

[SPARK-47234][BUILD] Upgrade Scala to 2.13.13

74f3f9f

github-actions bot added BUILD DOCS labels Feb 29, 2024

dongjoon-hyun reviewed Mar 1, 2024

View reviewed changes

panbingkun added 2 commits March 15, 2024 09:03

Merge branch 'master' into SPARK-47234

3a5f5e7

update ammonite to 3.0.0-M1

5df5f01

dongjoon-hyun reviewed Mar 15, 2024

View reviewed changes

fix spark-deps-hadoop-3-hive-2.3

c2f19e5

dongjoon-hyun changed the title ~~[WIP][SPARK-47234][BUILD] Upgrade Scala to 2.13.13~~ [SPARK-47234][BUILD] Upgrade Scala to 2.13.13 Mar 15, 2024

dongjoon-hyun marked this pull request as ready for review March 15, 2024 05:11

dongjoon-hyun approved these changes Mar 15, 2024

View reviewed changes

dongjoon-hyun closed this in 56cfc89 Mar 15, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-47234][BUILD] Upgrade Scala to 2.13.13 #45342

[SPARK-47234][BUILD] Upgrade Scala to 2.13.13 #45342

panbingkun commented Feb 29, 2024 •

edited

panbingkun commented Mar 1, 2024 •

edited

dongjoon-hyun commented Mar 1, 2024

dongjoon-hyun left a comment

panbingkun commented Mar 1, 2024

dongjoon-hyun commented Mar 1, 2024

panbingkun commented Mar 1, 2024 •

edited

LuciferYang commented Mar 1, 2024

dongjoon-hyun commented Mar 1, 2024

dongjoon-hyun commented Mar 14, 2024

panbingkun commented Mar 15, 2024 •

edited

dongjoon-hyun Mar 15, 2024

dongjoon-hyun Mar 15, 2024

panbingkun Mar 15, 2024

LuciferYang Mar 15, 2024

panbingkun Mar 15, 2024 •

edited

panbingkun Mar 15, 2024

panbingkun Mar 15, 2024

dongjoon-hyun Mar 15, 2024 •

edited

dongjoon-hyun commented Mar 15, 2024

panbingkun commented Mar 15, 2024

dongjoon-hyun left a comment

[SPARK-47234][BUILD] Upgrade Scala to 2.13.13 #45342

[SPARK-47234][BUILD] Upgrade Scala to 2.13.13 #45342

Conversation

panbingkun commented Feb 29, 2024 • edited

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

panbingkun commented Mar 1, 2024 • edited

dongjoon-hyun commented Mar 1, 2024

dongjoon-hyun left a comment

Choose a reason for hiding this comment

panbingkun commented Mar 1, 2024

dongjoon-hyun commented Mar 1, 2024

panbingkun commented Mar 1, 2024 • edited

LuciferYang commented Mar 1, 2024

dongjoon-hyun commented Mar 1, 2024

dongjoon-hyun commented Mar 14, 2024

panbingkun commented Mar 15, 2024 • edited

dongjoon-hyun Mar 15, 2024

Choose a reason for hiding this comment

dongjoon-hyun Mar 15, 2024

Choose a reason for hiding this comment

panbingkun Mar 15, 2024

Choose a reason for hiding this comment

LuciferYang Mar 15, 2024

Choose a reason for hiding this comment

panbingkun Mar 15, 2024 • edited

Choose a reason for hiding this comment

panbingkun Mar 15, 2024

Choose a reason for hiding this comment

panbingkun Mar 15, 2024

Choose a reason for hiding this comment

dongjoon-hyun Mar 15, 2024 • edited

Choose a reason for hiding this comment

dongjoon-hyun commented Mar 15, 2024

panbingkun commented Mar 15, 2024

dongjoon-hyun left a comment

Choose a reason for hiding this comment

panbingkun commented Feb 29, 2024 •

edited

panbingkun commented Mar 1, 2024 •

edited

panbingkun commented Mar 1, 2024 •

edited

panbingkun commented Mar 15, 2024 •

edited

panbingkun Mar 15, 2024 •

edited

dongjoon-hyun Mar 15, 2024 •

edited