Regression in scaladoc compilation when using Scala 3 definitions

[WojciechMazur]

Reproduction steps

Scala version: 2.13.13
Tested with Scala 3 versions for cross compiled project: 3.3.1 and 3.3.2

reproduction-sbt.zip
run with sbt "base/publishLocal; projectC/publishLocal"

1. Define ADT in Scala 3, create a jar out of it

package testlib

enum ADT:
  case SingletonCase
  case ClassCase(x: String)

2. Define Scala 2 project that uses Scala 3 JAR

package app

object Main {
  def main(args: Array[String]): Unit = {
    println(testlib.ADT.SingletonCase)
    println(testlib.ADT.ClassCase("foo"))
  }
}

3. Try to create documentation for Scala 2 project, (sbt Compile/doc)

Problem

Compilation fails with

error] stack trace is suppressed; run last projectC / Compile / doc for the full output
[error] (projectC / Compile / doc) scala.reflect.internal.Types$TypeError: Unsupported Scala 3 generic tuple type scala.Tuple in bounds of type MirroredElemTypes; found in  scala.deriving.Mirror.<refinement>.
[error] Total time: 0 s, completed Feb 22, 2024, 11:42:57 AM

[lrytz]

cc @bishabosha

[bishabosha]

writing here to confirm that it works in 2.13.12, so definitely something changed

[lrytz]

It's possible that the bug / behavior in the tasty reader is older but more symbols are forced in the new version of scaladoc.

[bishabosha]

I build scala 2.13 at scala/scala@ffc0297 (the commit before any of my changes) and it works, then at the classpath changes in scala/scala@e623261 it breaks, to confirm, I build at scala/scala@6b2c4ae (the commit before) and it works again.

I'm a bit surprised at this behavior, maybe there was some laziness that is now lost?

[lrytz]

Is it possible that before we didn't find all tasty files? Did this work for objects?

I guess I would add a few logs of when the classfile parser / tasty reader is invoked in both before and after.

[bishabosha]

Is it possible that before we didn't find all tasty files? Did this work for objects?

im not sure this is the reason, i.e. the tasty attribute was always in foo.class for object foo, never foo$.class, so in theory the same files should be forced.

I guess I would add a few logs of when the classfile parser / tasty reader is invoked in both before and after.

ill try that next

[bishabosha]

first thing I notice is that the symbol IDs are different

[bishabosha]

First noticed a bug in the filename logging, but not a big deal

however what this signals is that previously Mirror was being unpickled from the classfile, resolving the sibling tasty, but now it resolved directly from the tasty file, not sure if this is some sign

[bishabosha]

ok, for some reason, (found by adding a debug statement here) the doc tool is now trying to create doc pages for scala.deriving.Mirror, which then forces the fromTuple method which has a refinement to MirroredElemTypes, causing the crash

sbt --client last projectC/Compile/doc
[info] entering *experimental* thin client - BEEP WHIRR
[info] terminate the server with `shutdown`
> last projectC/Compile/doc
[info] Main Scala API documentation to /Users/jamie/Workspace/scaladoc-tasty-err/project-C/target/scala-2.13/api...

[debug] Returning already retrieved and compiled bridge: /Users/jamie/.ivy2/local/org.scala-lang/scala2-sbt-bridge/2.13.14-bin-SNAPSHOT/jars/scala2-sbt-bridge.jar.
[debug] Calling Scaladoc with arguments:
[debug] 	-d
[debug] 	/Users/jamie/Workspace/scaladoc-tasty-err/project-C/target/scala-2.13/api
[debug] 	-Ytasty-reader
[debug] 	-bootclasspath
[debug] 	/Users/jamie/.ivy2/local/org.scala-lang/scala-library/2.13.14-bin-SNAPSHOT/jars/scala-library.jar
[debug] 	-classpath
[debug] 	/Users/jamie/.ivy2/local/org.foo/base_3/0.1.0-SNAPSHOT/jars/base_3.jar:/Users/jamie/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-library_3/3.3.1/scala3-library_3-3.3.1.jar
[debug] 	/Users/jamie/Workspace/scaladoc-tasty-err/project-C/src/main/scala/Main.scala
[info] Creating doc template for object Main
[info] Creating doc template for object Enum
[info] Creating doc template for trait Enum
[info] Creating doc template for object Mirror
[info] Creating doc template for class SingletonProxy
[info] Creating doc template for trait Singleton
[info] Creating doc template for trait Sum
[info] Creating doc template for trait Product
[info] Creating doc template for trait Mirror
[info] Creating doc template for object SourceFile
[info] Creating doc template for class SourceFile
[info] Creating doc template for object Child
[info] Creating doc template for class Child
[info] Creating doc template for class Tuple$package
[info] Creating doc template for class ADT
[error] stack trace is suppressed; run 'last projectC / Compile / doc' for the full output
[error] Total time: 0 s, completed Feb 23, 2024, 4:18:49 PM
[error] (projectC / Compile / doc) scala.reflect.internal.Types$TypeError: Unsupported Scala 3 generic tuple type scala.Tuple in bounds of type MirroredElemTypes; found in  scala.deriving.Mirror.<refinement>.

and on that old working commit scala/scala@6b2c4ae we get instead

sbt --client last projectC/Compile/doc
[info] entering *experimental* thin client - BEEP WHIRR
[info] terminate the server with `shutdown`
> last projectC/Compile/doc
[info] Main Scala API documentation to /Users/jamie/Workspace/scaladoc-tasty-err/project-C/target/scala-2.13/api...

[debug] Calling Scaladoc with arguments:
[debug] 	-d
[debug] 	/Users/jamie/Workspace/scaladoc-tasty-err/project-C/target/scala-2.13/api
[debug] 	-Ytasty-reader
[debug] 	-bootclasspath
[debug] 	/Users/jamie/.ivy2/local/org.scala-lang/scala-library/2.13.13-bin-SNAPSHOT/jars/scala-library.jar
[debug] 	-classpath
[debug] 	/Users/jamie/.ivy2/local/org.foo/base_3/0.1.0-SNAPSHOT/jars/base_3.jar:/Users/jamie/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-library_3/3.3.1/scala3-library_3-3.3.1.jar
[debug] 	/Users/jamie/Workspace/scaladoc-tasty-err/project-C/src/main/scala/Main.scala
[info] Creating doc template for object Main
[info] Creating doc template for object SourceFile
[info] Creating doc template for class SourceFile
[info] Creating doc template for object Child
[info] Creating doc template for class Child
[info] Creating doc template for class ADT
[info] Main Scala API documentation successful.
[success] Total time: 0 s, completed Feb 23, 2024, 5:05:55 PM

Actually this looks wrong even in the "working" case. if I depend on the 2.13 compiled version of base then there should only be a doc page created for Main, there is no reason why ADT, Child etc are also getting made in scala/scala@6b2c4ae

[bishabosha]

Finally I have solved it.

Those were not being ignored because the sourceFile on the symbol was != null so was assumed to be a source file in the project. I had not added the case to ignore the associatedFile if it ends in ".tasty".