[SPARK-42673][BUILD] Make `build/mvn` build Spark only with the verified maven version #40283

LuciferYang · 2023-03-05T08:23:49Z

What changes were proposed in this pull request?

build/mvn tends to use the new maven version to build Spark now, and GA starts to use 3.9.0 as the default maven version.

But there may be some uncertain factors when building Spark with unverified version.

For example, java-11-17 GA build task build with maven 3.9.0 has many error logs in master like follow:

Error: [ERROR] An error occurred attempting to read POM
org.codehaus.plexus.util.xml.pull.XmlPullParserException: UTF-8 BOM plus xml decl of ISO-8859-1 is incompatible (position: START_DOCUMENT seen <?xml version="1.0" encoding="ISO-8859-1"... @1:42) 
    at org.codehaus.plexus.util.xml.pull.MXParser.parseXmlDeclWithVersion (MXParser.java:3423)
    at org.codehaus.plexus.util.xml.pull.MXParser.parseXmlDecl (MXParser.java:3345)
    at org.codehaus.plexus.util.xml.pull.MXParser.parsePI (MXParser.java:3197)
    at org.codehaus.plexus.util.xml.pull.MXParser.parseProlog (MXParser.java:1828)
    at org.codehaus.plexus.util.xml.pull.MXParser.nextImpl (MXParser.java:1757)
    at org.codehaus.plexus.util.xml.pull.MXParser.next (MXParser.java:1375)
    at org.apache.maven.model.io.xpp3.MavenXpp3Reader.read (MavenXpp3Reader.java:3940)
    at org.apache.maven.model.io.xpp3.MavenXpp3Reader.read (MavenXpp3Reader.java:612)
    at org.apache.maven.model.io.xpp3.MavenXpp3Reader.read (MavenXpp3Reader.java:627)
    at org.cyclonedx.maven.BaseCycloneDxMojo.readPom (BaseCycloneDxMojo.java:759)
    at org.cyclonedx.maven.BaseCycloneDxMojo.readPom (BaseCycloneDxMojo.java:746)
    at org.cyclonedx.maven.BaseCycloneDxMojo.retrieveParentProject (BaseCycloneDxMojo.java:694)
    at org.cyclonedx.maven.BaseCycloneDxMojo.getClosestMetadata (BaseCycloneDxMojo.java:524)
    at org.cyclonedx.maven.BaseCycloneDxMojo.convert (BaseCycloneDxMojo.java:481)
    at org.cyclonedx.maven.CycloneDxMojo.execute (CycloneDxMojo.java:70)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:342)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:330)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:213)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:175)
    at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:76)
    at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:163)
    at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:160)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:260)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:172)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:100)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:821)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:270)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:77)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:568)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)

So this pr change the version check condition of build/mvn to make it build Spark only with the verified maven version.

Why are the changes needed?

Make build/mvn build Spark only with the verified maven version

Does this PR introduce any user-facing change?

No

How was this patch tested?

java-11-17 GA build task pass and no error message as above
Manual test:

Make the system use maven 3.9.0( > 3.8.7 ) by default:

run mvn -version

Apache Maven 3.9.0 (9b58d2bad23a66be161c4664ef21ce219c2c8584)
Maven home: /Users/${userName}/Tools/maven
Java version: 1.8.0_362, vendor: Azul Systems, Inc., runtime: /Users/${userName}/Tools/zulu8/zulu-8.jdk/Contents/Home/jre
Default locale: zh_CN, platform encoding: UTF-8
OS name: "mac os x", version: "13.2.1", arch: "aarch64", family: "mac"

and run build/mvn -version

Using `mvn` from path: /${basedir}/spark/build/apache-maven-3.8.7/bin/mvn
Using SPARK_LOCAL_IP=localhost
Apache Maven 3.8.7 (b89d5959fcde851dcb1c8946a785a163f14e1e29)
Maven home: /${basedir}/spark/build/apache-maven-3.8.7
Java version: 1.8.0_362, vendor: Azul Systems, Inc., runtime: /Users/${userName}/Tools/zulu8/zulu-8.jdk/Contents/Home/jre
Default locale: zh_CN, platform encoding: UTF-8
OS name: "mac os x", version: "13.2.1", arch: "aarch64", family: "mac"

We can see Spark use 3.8.7 in build directory when the system default maven > 3.8.7

Make the system use maven 3.8.7 by default:

run mvn -version

mvn -version
Apache Maven 3.8.7 (b89d5959fcde851dcb1c8946a785a163f14e1e29)
Maven home: /Users/${userName}/Tools/maven
Java version: 1.8.0_362, vendor: Azul Systems, Inc., runtime: /Users/${userName}/Tools/zulu8/zulu-8.jdk/Contents/Home/jre
Default locale: zh_CN, platform encoding: UTF-8
OS name: "mac os x", version: "13.2.1", arch: "aarch64", family: "mac"

and run build/mvn -version

Using `mvn` from path: /Users/${userName}/Tools/maven/bin/mvn
Using SPARK_LOCAL_IP=localhost
Apache Maven 3.8.7 (b89d5959fcde851dcb1c8946a785a163f14e1e29)
Maven home: /Users/${userName}/Tools/maven
Java version: 1.8.0_362, vendor: Azul Systems, Inc., runtime: /Users/${userName}/Tools/zulu8/zulu-8.jdk/Contents/Home/jre
Default locale: zh_CN, platform encoding: UTF-8
OS name: "mac os x", version: "13.2.1", arch: "aarch64", family: "mac"

We can see Spark use system default maven 3.8.7 when the system default maven is 3.8.7.

Make the system use maven 3.8.6( < 3.8.7 ) by default:

run mvn -version

mvn -version
Apache Maven 3.8.6 (84538c9988a25aec085021c365c560670ad80f63)
Maven home: /Users/${userName}/Tools/maven
Java version: 1.8.0_362, vendor: Azul Systems, Inc., runtime: /Users/${userName}/Tools/zulu8/zulu-8.jdk/Contents/Home/jre
Default locale: zh_CN, platform encoding: UTF-8
OS name: "mac os x", version: "13.2.1", arch: "aarch64", family: "mac"

and run build/mvn -version

Using `mvn` from path: /Users/${userName}/Tools/maven/bin/mvn
Using SPARK_LOCAL_IP=localhost
Apache Maven 3.8.7 (b89d5959fcde851dcb1c8946a785a163f14e1e29)
Maven home: /Users/${userName}/Tools/maven
Java version: 1.8.0_362, vendor: Azul Systems, Inc., runtime: /Users/${userName}/Tools/zulu8/zulu-8.jdk/Contents/Home/jre
Default locale: zh_CN, platform encoding: UTF-8
OS name: "mac os x", version: "13.2.1", arch: "aarch64", family: "mac"

We can see Spark use 3.8.7 in build directory when the system default maven < 3.8.7.

LuciferYang · 2023-03-05T08:45:04Z

for example:

LuciferYang · 2023-03-05T08:49:08Z

For check, I add a ./build/mvn -version before in java-11-17 GA task without this pr:

And it print as follows:

https://github.com/LuciferYang/spark/actions/runs/4328951282/jobs/7559134224

LuciferYang · 2023-03-05T09:53:16Z

no error message with this pr

LuciferYang · 2023-03-05T10:02:07Z

cc @dongjoon-hyun

pan3793 · 2023-03-06T05:22:50Z

build/mvn

@@ -119,7 +119,8 @@ install_mvn() {
  if [ "$MVN_BIN" ]; then
    local MVN_DETECTED_VERSION="$(mvn --version | head -n1 | awk '{print $3}')"
  fi
-  if [ $(version $MVN_DETECTED_VERSION) -lt $(version $MVN_VERSION) ]; then


why not use exact match here?

I am not sure when 3.9.x will be supported due to there is a pre-work need to complete, if the pre-work cannot be completed, the version after 3.9.0 also needs to be temporarily disabled

I mean change -lt to -ne, and always respect maven.version defined in pom.xml

LuciferYang · 2023-03-06T05:55:00Z

also cc @HyukjinKwon

hboutemy · 2023-03-06T07:50:56Z

there is a known issue in Maven 3.9.0 (related to plexus-utils XML stricter reading codehaus-plexus/plexus-utils#238 ) that is fixed in 3.9.1-SNAPSHOT: https://issues.apache.org/jira/browse/MNG-7697

3.9.1 will be released soon: can you eventually check with 3.9.1-SNAPSHOT if you're in a different case of this "too strict" XML parsing?

hboutemy · 2023-03-06T07:53:49Z

@cstamas do you know if the lax parsing covers that org.codehaus.plexus.util.xml.pull.XmlPullParserException: UTF-8 BOM plus xml decl of ISO-8859-1 is incompatible (position: START_DOCUMENT seen <?xml version="1.0" encoding="ISO-8859-1"... @1:42) ?

LuciferYang · 2023-03-06T07:59:52Z

https://issues.apache.org/jira/browse/MNG-7697

OK, let me test 3.9.1-SNAPSHOT later. @pan3793 Do you have any other issues besides those in GA task?

pan3793

LGTM

gnodet · 2023-03-06T08:16:27Z

@cstamas do you know if the lax parsing covers that org.codehaus.plexus.util.xml.pull.XmlPullParserException: UTF-8 BOM plus xml decl of ISO-8859-1 is incompatible (position: START_DOCUMENT seen <?xml version="1.0" encoding="ISO-8859-1"... @1:42) ?

It does not. This is a separate issue which is not actually a problem in maven or plexus-utils. See a similar fix for it in takari/polyglot-maven@14514b6
It has to be fixed in the org.cyclonedx.maven.BaseCycloneDxMojo.readPom method.

hboutemy · 2023-03-06T09:07:49Z

oh, I did not see that Spark was still using cyclonedx-maven-plugin old 2.7.3, thank you @gnodet: @LuciferYang @pan3793 you should upgrade to 2.7.5, which has completely changed the implementation and should not have the issue

LuciferYang · 2023-03-06T09:13:34Z

oh, I did not see that Spark was still using cyclonedx-maven-plugin old 2.7.3, thank you @gnodet: @LuciferYang @pan3793 you should upgrade to 2.7.5, which has completely changed the implementation and should not have the issue

Thanks for you suggestion, in #40065 I discussed with @dongjoon-hyun , I know 2.7.5 fixed these issues, but 2.7.5 has other issues like CycloneDX/cyclonedx-maven-plugin#284 and CycloneDX/cyclonedx-maven-plugin#289, so we hope wait until at least 2.7.6 to upgrade cyclonedx-maven-plugin

dongjoon-hyun

+1, LGTM. Thank you, @LuciferYang and all.
According to the context, we should stick to the fixed version of Maven and its plugin versions.

Merged to master/3.4/3.3/3.2.

…ied maven version ### What changes were proposed in this pull request? `build/mvn` tends to use the new maven version to build Spark now, and GA starts to use 3.9.0 as the default maven version. But there may be some uncertain factors when building Spark with unverified version. For example, `java-11-17` GA build task build with maven 3.9.0 has many error logs in master like follow: ``` Error: [ERROR] An error occurred attempting to read POM org.codehaus.plexus.util.xml.pull.XmlPullParserException: UTF-8 BOM plus xml decl of ISO-8859-1 is incompatible (position: START_DOCUMENT seen <?xml version="1.0" encoding="ISO-8859-1"... 1:42) at org.codehaus.plexus.util.xml.pull.MXParser.parseXmlDeclWithVersion (MXParser.java:3423) at org.codehaus.plexus.util.xml.pull.MXParser.parseXmlDecl (MXParser.java:3345) at org.codehaus.plexus.util.xml.pull.MXParser.parsePI (MXParser.java:3197) at org.codehaus.plexus.util.xml.pull.MXParser.parseProlog (MXParser.java:1828) at org.codehaus.plexus.util.xml.pull.MXParser.nextImpl (MXParser.java:1757) at org.codehaus.plexus.util.xml.pull.MXParser.next (MXParser.java:1375) at org.apache.maven.model.io.xpp3.MavenXpp3Reader.read (MavenXpp3Reader.java:3940) at org.apache.maven.model.io.xpp3.MavenXpp3Reader.read (MavenXpp3Reader.java:612) at org.apache.maven.model.io.xpp3.MavenXpp3Reader.read (MavenXpp3Reader.java:627) at org.cyclonedx.maven.BaseCycloneDxMojo.readPom (BaseCycloneDxMojo.java:759) at org.cyclonedx.maven.BaseCycloneDxMojo.readPom (BaseCycloneDxMojo.java:746) at org.cyclonedx.maven.BaseCycloneDxMojo.retrieveParentProject (BaseCycloneDxMojo.java:694) at org.cyclonedx.maven.BaseCycloneDxMojo.getClosestMetadata (BaseCycloneDxMojo.java:524) at org.cyclonedx.maven.BaseCycloneDxMojo.convert (BaseCycloneDxMojo.java:481) at org.cyclonedx.maven.CycloneDxMojo.execute (CycloneDxMojo.java:70) at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126) at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:342) at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:330) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:213) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:175) at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:76) at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:163) at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:160) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73) at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53) at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:260) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:172) at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:100) at org.apache.maven.cli.MavenCli.execute (MavenCli.java:821) at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:270) at org.apache.maven.cli.MavenCli.main (MavenCli.java:192) at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method) at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:77) at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke (Method.java:568) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282) at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406) at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347) ``` So this pr change the version check condition of `build/mvn` to make it build Spark only with the verified maven version. ### Why are the changes needed? Make `build/mvn` build Spark only with the verified maven version ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? - `java-11-17` GA build task pass and no error message as above - Manual test: 1. Make the system use maven 3.9.0( > 3.8.7 ) by default: run `mvn -version` ``` Apache Maven 3.9.0 (9b58d2bad23a66be161c4664ef21ce219c2c8584) Maven home: /Users/${userName}/Tools/maven Java version: 1.8.0_362, vendor: Azul Systems, Inc., runtime: /Users/${userName}/Tools/zulu8/zulu-8.jdk/Contents/Home/jre Default locale: zh_CN, platform encoding: UTF-8 OS name: "mac os x", version: "13.2.1", arch: "aarch64", family: "mac" ``` and run `build/mvn -version` ``` Using `mvn` from path: /${basedir}/spark/build/apache-maven-3.8.7/bin/mvn Using SPARK_LOCAL_IP=localhost Apache Maven 3.8.7 (b89d5959fcde851dcb1c8946a785a163f14e1e29) Maven home: /${basedir}/spark/build/apache-maven-3.8.7 Java version: 1.8.0_362, vendor: Azul Systems, Inc., runtime: /Users/${userName}/Tools/zulu8/zulu-8.jdk/Contents/Home/jre Default locale: zh_CN, platform encoding: UTF-8 OS name: "mac os x", version: "13.2.1", arch: "aarch64", family: "mac" ``` We can see Spark use 3.8.7 in build directory when the system default maven > 3.8.7 2. Make the system use maven 3.8.7 by default: run `mvn -version` ``` mvn -version Apache Maven 3.8.7 (b89d5959fcde851dcb1c8946a785a163f14e1e29) Maven home: /Users/${userName}/Tools/maven Java version: 1.8.0_362, vendor: Azul Systems, Inc., runtime: /Users/${userName}/Tools/zulu8/zulu-8.jdk/Contents/Home/jre Default locale: zh_CN, platform encoding: UTF-8 OS name: "mac os x", version: "13.2.1", arch: "aarch64", family: "mac" ``` and run `build/mvn -version` ``` Using `mvn` from path: /Users/${userName}/Tools/maven/bin/mvn Using SPARK_LOCAL_IP=localhost Apache Maven 3.8.7 (b89d5959fcde851dcb1c8946a785a163f14e1e29) Maven home: /Users/${userName}/Tools/maven Java version: 1.8.0_362, vendor: Azul Systems, Inc., runtime: /Users/${userName}/Tools/zulu8/zulu-8.jdk/Contents/Home/jre Default locale: zh_CN, platform encoding: UTF-8 OS name: "mac os x", version: "13.2.1", arch: "aarch64", family: "mac" ``` We can see Spark use system default maven 3.8.7 when the system default maven is 3.8.7. 3. Make the system use maven 3.8.6( < 3.8.7 ) by default: run `mvn -version` ``` mvn -version Apache Maven 3.8.6 (84538c9988a25aec085021c365c560670ad80f63) Maven home: /Users/${userName}/Tools/maven Java version: 1.8.0_362, vendor: Azul Systems, Inc., runtime: /Users/${userName}/Tools/zulu8/zulu-8.jdk/Contents/Home/jre Default locale: zh_CN, platform encoding: UTF-8 OS name: "mac os x", version: "13.2.1", arch: "aarch64", family: "mac" ``` and run `build/mvn -version` ``` Using `mvn` from path: /Users/${userName}/Tools/maven/bin/mvn Using SPARK_LOCAL_IP=localhost Apache Maven 3.8.7 (b89d5959fcde851dcb1c8946a785a163f14e1e29) Maven home: /Users/${userName}/Tools/maven Java version: 1.8.0_362, vendor: Azul Systems, Inc., runtime: /Users/${userName}/Tools/zulu8/zulu-8.jdk/Contents/Home/jre Default locale: zh_CN, platform encoding: UTF-8 OS name: "mac os x", version: "13.2.1", arch: "aarch64", family: "mac" ``` We can see Spark use 3.8.7 in build directory when the system default maven < 3.8.7. Closes #40283 from LuciferYang/ban-maven-3.9.x. Authored-by: yangjie01 <yangjie01@baidu.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit f70b8cf) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

LuciferYang · 2023-03-07T03:18:40Z

Thanks @dongjoon-hyun @pan3793 ~
Also thanks @gnodet @hboutemy

…ied maven version ### What changes were proposed in this pull request? `build/mvn` tends to use the new maven version to build Spark now, and GA starts to use 3.9.0 as the default maven version. But there may be some uncertain factors when building Spark with unverified version. For example, `java-11-17` GA build task build with maven 3.9.0 has many error logs in master like follow: ``` Error: [ERROR] An error occurred attempting to read POM org.codehaus.plexus.util.xml.pull.XmlPullParserException: UTF-8 BOM plus xml decl of ISO-8859-1 is incompatible (position: START_DOCUMENT seen <?xml version="1.0" encoding="ISO-8859-1"... 1:42) at org.codehaus.plexus.util.xml.pull.MXParser.parseXmlDeclWithVersion (MXParser.java:3423) at org.codehaus.plexus.util.xml.pull.MXParser.parseXmlDecl (MXParser.java:3345) at org.codehaus.plexus.util.xml.pull.MXParser.parsePI (MXParser.java:3197) at org.codehaus.plexus.util.xml.pull.MXParser.parseProlog (MXParser.java:1828) at org.codehaus.plexus.util.xml.pull.MXParser.nextImpl (MXParser.java:1757) at org.codehaus.plexus.util.xml.pull.MXParser.next (MXParser.java:1375) at org.apache.maven.model.io.xpp3.MavenXpp3Reader.read (MavenXpp3Reader.java:3940) at org.apache.maven.model.io.xpp3.MavenXpp3Reader.read (MavenXpp3Reader.java:612) at org.apache.maven.model.io.xpp3.MavenXpp3Reader.read (MavenXpp3Reader.java:627) at org.cyclonedx.maven.BaseCycloneDxMojo.readPom (BaseCycloneDxMojo.java:759) at org.cyclonedx.maven.BaseCycloneDxMojo.readPom (BaseCycloneDxMojo.java:746) at org.cyclonedx.maven.BaseCycloneDxMojo.retrieveParentProject (BaseCycloneDxMojo.java:694) at org.cyclonedx.maven.BaseCycloneDxMojo.getClosestMetadata (BaseCycloneDxMojo.java:524) at org.cyclonedx.maven.BaseCycloneDxMojo.convert (BaseCycloneDxMojo.java:481) at org.cyclonedx.maven.CycloneDxMojo.execute (CycloneDxMojo.java:70) at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126) at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:342) at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:330) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:213) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:175) at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:76) at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:163) at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:160) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73) at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53) at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:260) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:172) at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:100) at org.apache.maven.cli.MavenCli.execute (MavenCli.java:821) at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:270) at org.apache.maven.cli.MavenCli.main (MavenCli.java:192) at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method) at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:77) at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke (Method.java:568) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282) at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406) at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347) ``` So this pr change the version check condition of `build/mvn` to make it build Spark only with the verified maven version. ### Why are the changes needed? Make `build/mvn` build Spark only with the verified maven version ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? - `java-11-17` GA build task pass and no error message as above - Manual test: 1. Make the system use maven 3.9.0( > 3.8.7 ) by default: run `mvn -version` ``` Apache Maven 3.9.0 (9b58d2bad23a66be161c4664ef21ce219c2c8584) Maven home: /Users/${userName}/Tools/maven Java version: 1.8.0_362, vendor: Azul Systems, Inc., runtime: /Users/${userName}/Tools/zulu8/zulu-8.jdk/Contents/Home/jre Default locale: zh_CN, platform encoding: UTF-8 OS name: "mac os x", version: "13.2.1", arch: "aarch64", family: "mac" ``` and run `build/mvn -version` ``` Using `mvn` from path: /${basedir}/spark/build/apache-maven-3.8.7/bin/mvn Using SPARK_LOCAL_IP=localhost Apache Maven 3.8.7 (b89d5959fcde851dcb1c8946a785a163f14e1e29) Maven home: /${basedir}/spark/build/apache-maven-3.8.7 Java version: 1.8.0_362, vendor: Azul Systems, Inc., runtime: /Users/${userName}/Tools/zulu8/zulu-8.jdk/Contents/Home/jre Default locale: zh_CN, platform encoding: UTF-8 OS name: "mac os x", version: "13.2.1", arch: "aarch64", family: "mac" ``` We can see Spark use 3.8.7 in build directory when the system default maven > 3.8.7 2. Make the system use maven 3.8.7 by default: run `mvn -version` ``` mvn -version Apache Maven 3.8.7 (b89d5959fcde851dcb1c8946a785a163f14e1e29) Maven home: /Users/${userName}/Tools/maven Java version: 1.8.0_362, vendor: Azul Systems, Inc., runtime: /Users/${userName}/Tools/zulu8/zulu-8.jdk/Contents/Home/jre Default locale: zh_CN, platform encoding: UTF-8 OS name: "mac os x", version: "13.2.1", arch: "aarch64", family: "mac" ``` and run `build/mvn -version` ``` Using `mvn` from path: /Users/${userName}/Tools/maven/bin/mvn Using SPARK_LOCAL_IP=localhost Apache Maven 3.8.7 (b89d5959fcde851dcb1c8946a785a163f14e1e29) Maven home: /Users/${userName}/Tools/maven Java version: 1.8.0_362, vendor: Azul Systems, Inc., runtime: /Users/${userName}/Tools/zulu8/zulu-8.jdk/Contents/Home/jre Default locale: zh_CN, platform encoding: UTF-8 OS name: "mac os x", version: "13.2.1", arch: "aarch64", family: "mac" ``` We can see Spark use system default maven 3.8.7 when the system default maven is 3.8.7. 3. Make the system use maven 3.8.6( < 3.8.7 ) by default: run `mvn -version` ``` mvn -version Apache Maven 3.8.6 (84538c9988a25aec085021c365c560670ad80f63) Maven home: /Users/${userName}/Tools/maven Java version: 1.8.0_362, vendor: Azul Systems, Inc., runtime: /Users/${userName}/Tools/zulu8/zulu-8.jdk/Contents/Home/jre Default locale: zh_CN, platform encoding: UTF-8 OS name: "mac os x", version: "13.2.1", arch: "aarch64", family: "mac" ``` and run `build/mvn -version` ``` Using `mvn` from path: /Users/${userName}/Tools/maven/bin/mvn Using SPARK_LOCAL_IP=localhost Apache Maven 3.8.7 (b89d5959fcde851dcb1c8946a785a163f14e1e29) Maven home: /Users/${userName}/Tools/maven Java version: 1.8.0_362, vendor: Azul Systems, Inc., runtime: /Users/${userName}/Tools/zulu8/zulu-8.jdk/Contents/Home/jre Default locale: zh_CN, platform encoding: UTF-8 OS name: "mac os x", version: "13.2.1", arch: "aarch64", family: "mac" ``` We can see Spark use 3.8.7 in build directory when the system default maven < 3.8.7. Closes apache#40283 from LuciferYang/ban-maven-3.9.x. Authored-by: yangjie01 <yangjie01@baidu.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit f70b8cf) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

ban 3.9.x

f29de59

github-actions bot added the BUILD label Mar 5, 2023

LuciferYang changed the title ~~[SPARK-42673][BUILD] Ban 3.9.x for Spark Maven build~~ [SPARK-42673][BUILD] Ban Maven 3.9.x for Spark build Mar 5, 2023

add comments

c8a7bbb

pan3793 reviewed Mar 6, 2023

View reviewed changes

-lt -> -ne

08005ad

LuciferYang changed the title ~~[SPARK-42673][BUILD] Ban Maven 3.9.x for Spark build~~ [SPARK-42673][BUILD] Make build/mvn build Spark only with the verified maven version Mar 6, 2023

LuciferYang marked this pull request as draft March 6, 2023 05:38

LuciferYang marked this pull request as ready for review March 6, 2023 05:54

pan3793 approved these changes Mar 6, 2023

View reviewed changes

dongjoon-hyun approved these changes Mar 6, 2023

View reviewed changes

dongjoon-hyun closed this in f70b8cf Mar 6, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-42673][BUILD] Make `build/mvn` build Spark only with the verified maven version #40283

[SPARK-42673][BUILD] Make `build/mvn` build Spark only with the verified maven version #40283

LuciferYang commented Mar 5, 2023 •

edited

LuciferYang commented Mar 5, 2023

LuciferYang commented Mar 5, 2023 •

edited

LuciferYang commented Mar 5, 2023 •

edited

LuciferYang commented Mar 5, 2023

pan3793 Mar 6, 2023

LuciferYang Mar 6, 2023

pan3793 Mar 6, 2023 •

edited

LuciferYang Mar 6, 2023

LuciferYang Mar 6, 2023

LuciferYang commented Mar 6, 2023

hboutemy commented Mar 6, 2023

hboutemy commented Mar 6, 2023

LuciferYang commented Mar 6, 2023

pan3793 left a comment

gnodet commented Mar 6, 2023

hboutemy commented Mar 6, 2023

LuciferYang commented Mar 6, 2023 •

edited

dongjoon-hyun left a comment

LuciferYang commented Mar 7, 2023

[SPARK-42673][BUILD] Make build/mvn build Spark only with the verified maven version #40283

[SPARK-42673][BUILD] Make build/mvn build Spark only with the verified maven version #40283

Conversation

LuciferYang commented Mar 5, 2023 • edited

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

LuciferYang commented Mar 5, 2023

LuciferYang commented Mar 5, 2023 • edited

LuciferYang commented Mar 5, 2023 • edited

LuciferYang commented Mar 5, 2023

pan3793 Mar 6, 2023

Choose a reason for hiding this comment

LuciferYang Mar 6, 2023

Choose a reason for hiding this comment

pan3793 Mar 6, 2023 • edited

Choose a reason for hiding this comment

LuciferYang Mar 6, 2023

Choose a reason for hiding this comment

LuciferYang Mar 6, 2023

Choose a reason for hiding this comment

LuciferYang commented Mar 6, 2023

hboutemy commented Mar 6, 2023

hboutemy commented Mar 6, 2023

LuciferYang commented Mar 6, 2023

pan3793 left a comment

Choose a reason for hiding this comment

gnodet commented Mar 6, 2023

hboutemy commented Mar 6, 2023

LuciferYang commented Mar 6, 2023 • edited

dongjoon-hyun left a comment

Choose a reason for hiding this comment

LuciferYang commented Mar 7, 2023

[SPARK-42673][BUILD] Make `build/mvn` build Spark only with the verified maven version #40283

[SPARK-42673][BUILD] Make `build/mvn` build Spark only with the verified maven version #40283

LuciferYang commented Mar 5, 2023 •

edited

LuciferYang commented Mar 5, 2023 •

edited

LuciferYang commented Mar 5, 2023 •

edited

pan3793 Mar 6, 2023 •

edited

LuciferYang commented Mar 6, 2023 •

edited