console does not work with Scala 2.13.13, sbt 1.9.8

[xuwei-k]

steps

build.sbt

scalaVersion := "2.13.13"

run sbt console

problem

[info] Starting scala interpreter...
Welcome to Scala 2.13.13 (OpenJDK 64-Bit Server VM, Java 1.8.0_392).
Type in expressions for evaluation. Or try :help.


Failed to initialize compiler: NoSuchMethodError.
This is most often remedied by a full clean and recompile.
Otherwise, your classpath may continue bytecode compiled by
different and incompatible versions of scala.

java.lang.NoSuchMethodError: org.jline.utils.AttributedString.fromAnsi(Ljava/lang/String;Ljava/util/List;Ljava/lang/String;Ljava/lang/String;)Lorg/jline/utils/AttributedString;
	at org.jline.reader.impl.LineReaderImpl.fromAnsi(LineReaderImpl.java:4232)
	at org.jline.reader.impl.LineReaderImpl.expandPromptPattern(LineReaderImpl.java:4154)
	at org.jline.reader.impl.LineReaderImpl.setPrompt(LineReaderImpl.java:1253)
	at org.jline.reader.impl.LineReaderImpl.readLine(LineReaderImpl.java:628)
	at org.jline.reader.impl.LineReaderImpl.readLine(LineReaderImpl.java:512)
	at scala.tools.nsc.interpreter.jline.Reader.readOneLine(Reader.scala:44)
	at scala.tools.nsc.interpreter.shell.InteractiveReader.readLine(InteractiveReader.scala:38)
	at scala.tools.nsc.interpreter.shell.InteractiveReader.readLine$(InteractiveReader.scala:38)
	at scala.tools.nsc.interpreter.jline.Reader.readLine(Reader.scala:34)
	at scala.tools.nsc.interpreter.shell.ILoop.readOneLine(ILoop.scala:455)
	at scala.tools.nsc.interpreter.shell.ILoop.loop(ILoop.scala:460)
	at scala.tools.nsc.interpreter.shell.ILoop.run(ILoop.scala:993)
	at scala.tools.xsbt.ConsoleBridge.run(ConsoleBridge.scala:81)
	at sbt.internal.inc.AnalyzingCompiler.console(AnalyzingCompiler.scala:208)
	at sbt.Console.console0$1(Console.scala:65)
	at sbt.Console.$anonfun$apply$5(Console.scala:75)
	at sbt.Run$.executeSuccess(Run.scala:187)
	at sbt.Console.$anonfun$apply$4(Console.scala:75)
	at sbt.internal.util.Terminal.withRawInput(Terminal.scala:146)
	at sbt.internal.util.Terminal.withRawInput$(Terminal.scala:144)
	at sbt.internal.util.Terminal$ProxyTerminal$.withRawInput(Terminal.scala:424)
	at sbt.Console.$anonfun$apply$3(Console.scala:75)
	at sbt.internal.util.Terminal$TerminalImpl.withRawOutput(Terminal.scala:1028)
	at sbt.internal.util.Terminal$ProxyTerminal$.withRawOutput(Terminal.scala:463)
	at sbt.Console.apply(Console.scala:72)
	at sbt.Console.apply(Console.scala:50)
	at sbt.Console.apply(Console.scala:42)
	at sbt.Defaults$.$anonfun$consoleTask$1(Defaults.scala:2237)
	at sbt.Defaults$.$anonfun$consoleTask$1$adapted(Defaults.scala:2223)
	at scala.Function1.$anonfun$compose$1(Function1.scala:49)
	at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:63)
	at sbt.std.Transform$$anon$4.work(Transform.scala:69)
	at sbt.Execute.$anonfun$submit$2(Execute.scala:283)
	at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:24)
	at sbt.Execute.work(Execute.scala:292)
	at sbt.Execute.$anonfun$submit$1(Execute.scala:283)
	at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265)
	at sbt.CompletionService$$anon$2.call(CompletionService.scala:65)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)

expectation

notes

[SethTisue]

gah, this is bad. attn @eed3si9n @lrytz

over in scala/scala we merged the JLine 3.24.1 upgrade all the way back in October 2023, but of course it's common for nobody to notice a regression in nightlies, even over a period of months. and obviously the community build is no help with interactive stuff like this

Eugene, let's discuss, is this remediable in the sbt 1.9.x series, or do we have to try to get to 1.10.0 soon, or what?

[eed3si9n]

It works on my machine with sbt 1.10.0-M1:

sbt:foo> console
[info] Starting scala interpreter...
Welcome to Scala 2.13.13 (OpenJDK 64-Bit Server VM, Java 1.8.0_362).
Type in expressions for evaluation. Or try :help.

scala> 1 + 1
val res0: Int = 2

[eed3si9n]

Oh btw, thanks @xuwei-k for reporting this! 🙏🏼

[SethTisue]

It works on my machine with sbt 1.10.0-M1

Whew. Any chance of accelerating the progression to 1.10.0-RC1 and 1.10.0 final? Is that progression waiting on anything in particular?

[eed3si9n]

Or I guess we can just backport the JLine bump to 1.9.x series. At this point, the odds of it potentially breaking more things can be balanced with the fact that latest stable Scala 2.13.x can't console on sbt 1.9.x.

[eed3si9n]

Here's my PR - #7503

[SethTisue]

Or I guess we can just backport the JLine bump to 1.9.x series

Suits me!!

[lrytz]

community build is no help with interactive stuff like this

yeah, but it's still on us (scala/scala), we should have the most basic smoke test somewhere.

[eed3si9n]

good news is you can pipe a command from stdin:

$ echo "sys.exit()" | sbt console
[info] welcome to sbt 1.9.8 (Azul Systems, Inc. Java 1.8.0_352)
....
[info] Starting scala interpreter...
Welcome to Scala 2.13.13 (OpenJDK 64-Bit Server VM, Java 1.8.0_352).
Type in expressions for evaluation. Or try :help.

Failed to initialize compiler: NoSuchMethodError.
This is most often remedied by a full clean and recompile.
Otherwise, your classpath may continue bytecode compiled by
different and incompatible versions of scala.

java.lang.NoSuchMethodError: org.jline.utils.AttributedString.fromAnsi(Ljava/lang/String;Ljava/util/List;Ljava/lang/String;Ljava/lang/String;)Lorg/jline/utils/AttributedString;
	at org.jline.reader.impl.LineReaderImpl.fromAnsi(LineReaderImpl.java:4232)
	at org.jline.reader.impl.LineReaderImpl.expandPromptPattern(LineReaderImpl.java:4154)
	at org.jline.reader.impl.LineReaderImpl.setPrompt(LineReaderImpl.java:1253)
	at org.jline.reader.impl.LineReaderImpl.readLine(LineReaderImpl.java:628)
	at org.jline.reader.impl.LineReaderImpl.readLine(LineReaderImpl.java:512)
	at scala.tools.nsc.interpreter.jline.Reader.readOneLine(Reader.scala:44)
	at scala.tools.nsc.interpreter.shell.InteractiveReader.readLine(InteractiveReader.scala:38)
	at scala.tools.nsc.interpreter.shell.InteractiveReader.readLine$(InteractiveReader.scala:38)
	at scala.tools.nsc.interpreter.jline.Reader.readLine(Reader.scala:34)
	at scala.tools.nsc.interpreter.shell.ILoop.readOneLine(ILoop.scala:455)
	at scala.tools.nsc.interpreter.shell.ILoop.loop(ILoop.scala:460)
	at scala.tools.nsc.interpreter.shell.ILoop.run(ILoop.scala:993)
	at scala.tools.xsbt.ConsoleBridge.run(ConsoleBridge.scala:81)
	at sbt.internal.inc.AnalyzingCompiler.console(AnalyzingCompiler.scala:208)
	at sbt.Console.console0$1(Console.scala:65)
	at sbt.Console.$anonfun$apply$5(Console.scala:75)
	at sbt.Run$.executeSuccess(Run.scala:187)
	at sbt.Console.$anonfun$apply$4(Console.scala:75)
	at sbt.internal.util.Terminal.withRawInput(Terminal.scala:146)
	at sbt.internal.util.Terminal.withRawInput$(Terminal.scala:144)
	at sbt.internal.util.Terminal$ProxyTerminal$.withRawInput(Terminal.scala:424)
	at sbt.Console.$anonfun$apply$3(Console.scala:75)
	at sbt.internal.util.Terminal$DefaultTerminal.withRawOutput(Terminal.scala:1074)
	at sbt.internal.util.Terminal$ProxyTerminal$.withRawOutput(Terminal.scala:463)
	at sbt.Console.apply(Console.scala:72)
	at sbt.Console.apply(Console.scala:50)
	at sbt.Console.apply(Console.scala:42)
	at sbt.Defaults$.$anonfun$consoleTask$1(Defaults.scala:2237)
	at sbt.Defaults$.$anonfun$consoleTask$1$adapted(Defaults.scala:2223)
	at scala.Function1.$anonfun$compose$1(Function1.scala:49)
	at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:63)
	at sbt.std.Transform$$anon$4.work(Transform.scala:69)
	at sbt.Execute.$anonfun$submit$2(Execute.scala:283)
	at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:24)
	at sbt.Execute.work(Execute.scala:292)
	at sbt.Execute.$anonfun$submit$1(Execute.scala:283)
	at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265)
	at sbt.CompletionService$$anon$2.call(CompletionService.scala:65)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)

[success] Total time: 7 s, completed Feb 23, 2024 2:32:40 AM

bad news is that console task considers this to be a success and exits with 0:

$ echo $?
0

[xuwei-k]

bad news is that console task considers this to be a success and exits with 0:

• run task does not fail if throw LinkageError #7408 scripted test passes despite failing to run #6852 🤔

same result with sbt 1.5.7 😢

[dwijnand]

@eed3si9n do you know how the mechanics are working here? Why isn't the newer jline version being used as the runtime classpath? This can't be the scala version "mediator". It could just be selectively picking the scala jars to pull in, ignoring the other transitive dependencies metadata, like jline version. Or could it be some classloader or classloading layering issue. Do you know?

[eed3si9n]

Yea. We exclude JLine from the layered classloader unless useScalaReplJLine setting is true:

sbt/main/src/main/scala/sbt/Defaults.scala

Lines 669 to 672 in 48c2376

	val topLoader = if (!useScalaReplJLine.value) {
	// the JLineLoader contains the SbtInterfaceClassLoader
	classOf[org.jline.terminal.Terminal].getClassLoader
	} else classOf[Compilers].getClassLoader // the SbtInterfaceClassLoader

This is because JLine and JAnsi basically loads native libraries, and unlike other JVM libraries we haven't have multiple versions in one process space. My mental model of sbtn is like what I imagine the way telnet works, in a sense that the client program captures all keyboard inputs like VK_A or VK_UP etc, and there's a corresponding virtualized terminal emulator that sends the virtual key input VK_UP, and relays back that up-arrow does. Up-arrow is last history in both sbt shell and Scala REPL, for example so we run JLine terminal that's tracking the history etc. See 90dacc3.

There's further complication that happened for Scala 3, I think because they use compiler-interface, so #6199 changed the layered Classloader further, but the idea is the same.

Maybe a better solution would be to let sbtn fork its own java process.