Repository navigation
Force scalaVersion when using scala scalafmt extension #1273
Description
Related to #579
Currently I am having an issue in upgrading Kafka's scala's spotless plugin to the latest version of scalafmt (3.5.8), you can see the PR here apache/kafka#12475 (comment). While the spotless scala plugin works fine when running with the default Scala version that is 2.13.8, when happen to build Kafka with a different scala version by using the -PscalaVersion flag it causes spotless to fail, i.e.
./gradlew :spotlessScalaCheckRuns fine without any problems, but if you do
./gradlew -PscalaVersion=2.12 :spotlessScalaCheckThen you get the following stack trace
> Configure project :
Starting build with version 3.4.0-SNAPSHOT (commit id de2ccf1f) using Gradle 7.5.1, Java 18 and Scala 2.12.15
Build properties: maxParallelForks=10, maxScalacThreads=8, maxTestRetries=0
> Task :spotlessScala FAILED
Step 'scalafmt' found problem in 'streams/streams-scala/src/main/scala/org/apache/kafka/streams/scala/FunctionsCompatConversions.scala':
null
java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:119)
at com.diffplug.spotless.scala.ScalaFmtStep$State.createFormat(ScalaFmtStep.java:98)
at com.diffplug.spotless.FormatterStepImpl$Standard.format(FormatterStepImpl.java:80)
at com.diffplug.spotless.FormatterStep$Strict.format(FormatterStep.java:88)
at com.diffplug.spotless.Formatter.compute(Formatter.java:230)
at com.diffplug.spotless.PaddedCell.calculateDirtyState(PaddedCell.java:203)
at com.diffplug.spotless.PaddedCell.calculateDirtyState(PaddedCell.java:190)
at com.diffplug.gradle.spotless.SpotlessTaskImpl.processInputFile(SpotlessTaskImpl.java:102)
at com.diffplug.gradle.spotless.SpotlessTaskImpl.performAction(SpotlessTaskImpl.java:88)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:125)
at org.gradle.api.internal.project.taskfactory.IncrementalInputsTaskAction.doExecute(IncrementalInputsTaskAction.java:32)
at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:51)
at org.gradle.api.internal.project.taskfactory.AbstractIncrementalTaskAction.execute(AbstractIncrementalTaskAction.java:25)
at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:29)
at org.gradle.api.internal.tasks.execution.TaskExecution$3.run(TaskExecution.java:236)
at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:29)
at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:26)
at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
at org.gradle.internal.operations.DefaultBuildOperationRunner.run(DefaultBuildOperationRunner.java:47)
at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:68)
at org.gradle.api.internal.tasks.execution.TaskExecution.executeAction(TaskExecution.java:221)
at org.gradle.api.internal.tasks.execution.TaskExecution.executeActions(TaskExecution.java:204)
at org.gradle.api.internal.tasks.execution.TaskExecution.executeWithPreviousOutputFiles(TaskExecution.java:187)
at org.gradle.api.internal.tasks.execution.TaskExecution.execute(TaskExecution.java:165)
at org.gradle.internal.execution.steps.ExecuteStep.executeInternal(ExecuteStep.java:89)
at org.gradle.internal.execution.steps.ExecuteStep.access$000(ExecuteStep.java:40)
at org.gradle.internal.execution.steps.ExecuteStep$1.call(ExecuteStep.java:53)
at org.gradle.internal.execution.steps.ExecuteStep$1.call(ExecuteStep.java:50)
at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:204)
at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:199)
at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
at org.gradle.internal.operations.DefaultBuildOperationRunner.call(DefaultBuildOperationRunner.java:53)
at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:73)
at org.gradle.internal.execution.steps.ExecuteStep.execute(ExecuteStep.java:50)
at org.gradle.internal.execution.steps.ExecuteStep.execute(ExecuteStep.java:40)
at org.gradle.internal.execution.steps.RemovePreviousOutputsStep.execute(RemovePreviousOutputsStep.java:68)
at org.gradle.internal.execution.steps.RemovePreviousOutputsStep.execute(RemovePreviousOutputsStep.java:38)
at org.gradle.internal.execution.steps.CancelExecutionStep.execute(CancelExecutionStep.java:41)
at org.gradle.internal.execution.steps.TimeoutStep.executeWithoutTimeout(TimeoutStep.java:74)
at org.gradle.internal.execution.steps.TimeoutStep.execute(TimeoutStep.java:55)
at org.gradle.internal.execution.steps.CreateOutputsStep.execute(CreateOutputsStep.java:51)
at org.gradle.internal.execution.steps.CreateOutputsStep.execute(CreateOutputsStep.java:29)
at org.gradle.internal.execution.steps.CaptureStateAfterExecutionStep.executeDelegateBroadcastingChanges(CaptureStateAfterExecutionStep.java:124)
at org.gradle.internal.execution.steps.CaptureStateAfterExecutionStep.execute(CaptureStateAfterExecutionStep.java:80)
at org.gradle.internal.execution.steps.CaptureStateAfterExecutionStep.execute(CaptureStateAfterExecutionStep.java:58)
at org.gradle.internal.execution.steps.ResolveInputChangesStep.execute(ResolveInputChangesStep.java:48)
at org.gradle.internal.execution.steps.ResolveInputChangesStep.execute(ResolveInputChangesStep.java:36)
at org.gradle.internal.execution.steps.BuildCacheStep.executeWithoutCache(BuildCacheStep.java:181)
at org.gradle.internal.execution.steps.BuildCacheStep.lambda$execute$1(BuildCacheStep.java:71)
at org.gradle.internal.Either$Right.fold(Either.java:175)
at org.gradle.internal.execution.caching.CachingState.fold(CachingState.java:59)
at org.gradle.internal.execution.steps.BuildCacheStep.execute(BuildCacheStep.java:69)
at org.gradle.internal.execution.steps.BuildCacheStep.execute(BuildCacheStep.java:47)
at org.gradle.internal.execution.steps.StoreExecutionStateStep.execute(StoreExecutionStateStep.java:36)
at org.gradle.internal.execution.steps.StoreExecutionStateStep.execute(StoreExecutionStateStep.java:25)
at org.gradle.internal.execution.steps.RecordOutputsStep.execute(RecordOutputsStep.java:36)
at org.gradle.internal.execution.steps.RecordOutputsStep.execute(RecordOutputsStep.java:22)
at org.gradle.internal.execution.steps.SkipUpToDateStep.executeBecause(SkipUpToDateStep.java:110)
at org.gradle.internal.execution.steps.SkipUpToDateStep.lambda$execute$2(SkipUpToDateStep.java:56)
at org.gradle.internal.execution.steps.SkipUpToDateStep.execute(SkipUpToDateStep.java:56)
at org.gradle.internal.execution.steps.SkipUpToDateStep.execute(SkipUpToDateStep.java:38)
at org.gradle.internal.execution.steps.ResolveChangesStep.execute(ResolveChangesStep.java:73)
at org.gradle.internal.execution.steps.ResolveChangesStep.execute(ResolveChangesStep.java:44)
at org.gradle.internal.execution.steps.legacy.MarkSnapshottingInputsFinishedStep.execute(MarkSnapshottingInputsFinishedStep.java:37)
at org.gradle.internal.execution.steps.legacy.MarkSnapshottingInputsFinishedStep.execute(MarkSnapshottingInputsFinishedStep.java:27)
at org.gradle.internal.execution.steps.ResolveCachingStateStep.execute(ResolveCachingStateStep.java:89)
at org.gradle.internal.execution.steps.ResolveCachingStateStep.execute(ResolveCachingStateStep.java:50)
at org.gradle.internal.execution.steps.ValidateStep.execute(ValidateStep.java:114)
at org.gradle.internal.execution.steps.ValidateStep.execute(ValidateStep.java:57)
at org.gradle.internal.execution.steps.CaptureStateBeforeExecutionStep.execute(CaptureStateBeforeExecutionStep.java:76)
at org.gradle.internal.execution.steps.CaptureStateBeforeExecutionStep.execute(CaptureStateBeforeExecutionStep.java:50)
at org.gradle.internal.execution.steps.SkipEmptyWorkStep.executeWithNoEmptySources(SkipEmptyWorkStep.java:254)
at org.gradle.internal.execution.steps.SkipEmptyWorkStep.execute(SkipEmptyWorkStep.java:91)
at org.gradle.internal.execution.steps.SkipEmptyWorkStep.execute(SkipEmptyWorkStep.java:56)
at org.gradle.internal.execution.steps.RemoveUntrackedExecutionStateStep.execute(RemoveUntrackedExecutionStateStep.java:32)
at org.gradle.internal.execution.steps.RemoveUntrackedExecutionStateStep.execute(RemoveUntrackedExecutionStateStep.java:21)
at org.gradle.internal.execution.steps.legacy.MarkSnapshottingInputsStartedStep.execute(MarkSnapshottingInputsStartedStep.java:38)
at org.gradle.internal.execution.steps.LoadPreviousExecutionStateStep.execute(LoadPreviousExecutionStateStep.java:43)
at org.gradle.internal.execution.steps.LoadPreviousExecutionStateStep.execute(LoadPreviousExecutionStateStep.java:31)
at org.gradle.internal.execution.steps.AssignWorkspaceStep.lambda$execute$0(AssignWorkspaceStep.java:40)
at org.gradle.api.internal.tasks.execution.TaskExecution$4.withWorkspace(TaskExecution.java:281)
at org.gradle.internal.execution.steps.AssignWorkspaceStep.execute(AssignWorkspaceStep.java:40)
at org.gradle.internal.execution.steps.AssignWorkspaceStep.execute(AssignWorkspaceStep.java:30)
at org.gradle.internal.execution.steps.IdentityCacheStep.execute(IdentityCacheStep.java:37)
at org.gradle.internal.execution.steps.IdentityCacheStep.execute(IdentityCacheStep.java:27)
at org.gradle.internal.execution.steps.IdentifyStep.execute(IdentifyStep.java:44)
at org.gradle.internal.execution.steps.IdentifyStep.execute(IdentifyStep.java:33)
at org.gradle.internal.execution.impl.DefaultExecutionEngine$1.execute(DefaultExecutionEngine.java:76)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeIfValid(ExecuteActionsTaskExecuter.java:139)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:128)
at org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:77)
at org.gradle.api.internal.tasks.execution.FinalizePropertiesTaskExecuter.execute(FinalizePropertiesTaskExecuter.java:46)
at org.gradle.api.internal.tasks.execution.ResolveTaskExecutionModeExecuter.execute(ResolveTaskExecutionModeExecuter.java:51)
at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:57)
at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:56)
at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:36)
at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.executeTask(EventFiringTaskExecuter.java:77)
at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:55)
at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:52)
at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:204)
at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:199)
at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
at org.gradle.internal.operations.DefaultBuildOperationRunner.call(DefaultBuildOperationRunner.java:53)
at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:73)
at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:52)
at org.gradle.execution.plan.LocalTaskNodeExecutor.execute(LocalTaskNodeExecutor.java:69)
at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:327)
at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:314)
at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:307)
at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:293)
at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.execute(DefaultPlanExecutor.java:420)
at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.run(DefaultPlanExecutor.java:342)
at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
Caused by: java.lang.NoClassDefFoundError: scala/collection/IterableOnce
at org.scalafmt.Scalafmt$.format$default$2(Scalafmt.scala:171)
at org.scalafmt.Scalafmt.format$default$2(Scalafmt.scala)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
... 123 more
Caused by: java.lang.ClassNotFoundException: scala.collection.IterableOnce
at com.diffplug.spotless.FeatureClassLoader.findClass(FeatureClassLoader.java:81)
... 126 more
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':spotlessScala'.
> java.lang.reflect.InvocationTargetException
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
* Get more help at https://help.gradle.org
The reason behind the stacktrace is the fact that the latest version of Scalafmt (3.5.8) is compiled against Scala 2.13.x but the usage of -PscalaVersion=2.12 happens to also override the Scala version used by Scalafmt. Note that the reason why we specify the version of
Scala using -PscalaVersion=2.12 is because of building and then releasing Kafka against multiple Scala versions.
Ultimately the point is that whatever version Scalafmt happens to run is irrelevant to what Scala version you are using to build your project and in this specific case we would ideally want to force a scalaVersion that scalafmt with regardless of the -PscalaVersion=2.12 flag, i.e. something like
spotless {
scala {
version = "2.13.8" // Force Scala version to be 2.13.8 regardless of user specific scala flag
target 'streams/**/*.scala'
scalafmt("$versions.scalafmt").configFile('checkstyle/.scalafmt.conf')
licenseHeaderFile 'checkstyle/java.header', 'package'
}
}Relevant details:
Starting build with version 3.4.0-SNAPSHOT (commit id de2ccf1f) using Gradle 7.5.1, Java 18 and Scala 2.12.15
Build properties: maxParallelForks=10, maxScalacThreads=8, maxTestRetries=0
Spotless version: 6.9.0
Operating system: MacOS Montery 12.5 M1
Spotless Configuration
spotless {
scala {
target 'streams/**/*.scala'
scalafmt("$versions.scalafmt").configFile('checkstyle/.scalafmt.conf')
licenseHeaderFile 'checkstyle/java.header', 'package'
}
}
Activity
nedtwigg commentedon Aug 10, 2022
I would expect that the scala version wouldn't depend on
-PscalaVersion, and here is why:spotless/lib/src/main/java/com/diffplug/spotless/scala/ScalaFmtStep.java
Line 83 in ce98b68
The scalaFmt classpath is resolved in an isolated way, as shown above. The
provisionerfield is provided as belowspotless/plugin-gradle/src/main/java/com/diffplug/gradle/spotless/GradleProvisioner.java
Lines 111 to 136 in ce98b68
I don't know how Gradle handles Scala, but I suppose that the scala plugin might modify the dependency resolution process somehow. Happy to take a PR that fixes this, but it seems like this should probably be pretty easy to workaround by disabling the spotless tasks on non-friendly Scala versions.
mdedetrich commentedon Aug 10, 2022
Scalafmt is a scala project so it requires scala to be on the classpath to run so yes I do believe the gradle Scala plugin is actually providing that version of Scala on the classpath which is then picked up by spotless.
Indeed I think as a workaround this should be fine and since it appears that scalafmt is using an isolated class it should be possible to provide an alternate Scala version, I will have a look at this tomorrow.
mdedetrich commentedon Aug 11, 2022
I created an issue at gradle (see gradle/gradle#21502) to also see if they have any ideas but I suspect to solve this issue we will need to use an isolated classloader due to the fact that we are going to have different scala versions in the same JVM instance.
There might also be an argument that in general any of the spotless integrations should run in an isolated classloader since most of the formatters/linters are cli "like" tools that work on context free grammers (i.e. they just parse source files)
mdedetrich commentedon Aug 12, 2022
@nedtwigg So I did some research on this issue (see gradle/gradle#21502 (comment)) and I better understand the problem/solutions.
So the essentially problem is that spotless-scala hardcodes the Scala version (see https://github.com/diffplug/spotless/blob/0fd20bb80c6c426d20e0a3157c3c2b89317032da/lib/src/main/java/com/diffplug/spotless/scala/ScalaFmtStep.java) depending on which scalafmt version you configure. Due to spotless not knowing anything about gradle-scala plugin, this causes spotless-scala to pick Scala 2.13.x even if you configure the Scala version via gradle-scala to be 2.12.x and configuring gradle-scala to Scala 2.12 will override the Scala version globally in gradle causing gradle to load the scala lib/runtime for that version.
This leaves us with 2 solutions
wdyt? For me I have a preference of detecting if gradle-scala is being used and just picking the scala version from that. Also willing to do a PR however I am not that familiar with gradle so if there is some quick reference/sample code and/or documentation on how to dynamically detect plugins that would be great.
nedtwigg commentedon Aug 21, 2022
A few things - for one thing, if we wanted to remove the reflection, that is possible
Also we are using an isolated classloader. We grab the jar files for ScalaFmt here
spotless/lib/src/main/java/com/diffplug/spotless/scala/ScalaFmtStep.java
Line 83 in b535dce
and then use that isolated classloader here
spotless/lib/src/main/java/com/diffplug/spotless/scala/ScalaFmtStep.java
Line 88 in b535dce
You can do that like so:
spotless/plugin-gradle/src/main/java/com/diffplug/gradle/spotless/ScalaExtension.java
Lines 73 to 76 in b535dce
mdedetrich commentedon Aug 22, 2022
@nedtwigg I just made a PR that converts the scalafmt integration to use compile time source sets (see #1283). I still need to test this with Kafka, but if the issue due to reflection this should be enough to solve it. If not I will do a future PR which will try to detect a currently configured Scala version using the classpath.
I want to test if this PR solves my problem by publishing
spotlesslocally using./gradlew publishToMavenLocalhowever I am getting problems with signing, i.e.Do you have any ideas how to temporarily disable signing just for local publish?
nedtwigg commentedon Aug 22, 2022
Add
-x signPluginMavenPublicationto the gradle command line.mdedetrich commentedon Aug 22, 2022
Thanks for the tip, I ended up just disabling the
signingplugin which allowed me to publish locally however I am getting problems resolving amavenLocal()repository due to conflicts between with the defaultgradlePluginPortal()(a clash due to the same groupId with artifactory + snapshots is causing artifactory to return a 409).In any case I have already implemented what I described at #1283 (comment), I just haven't committed the changes yet.
nedtwigg commentedon Aug 23, 2022
Thanks to @mdedetrich for the new
majorScalaVersionparameter, published inplugin-gradle 6.10.0andplugin-maven 2.25.0.mdedetrich commentedon Aug 23, 2022
Thanks, I can confirm that the new release has also solved the underlying problem I had wrt Scala binary version.