How to setup spotless for a git pre commit hook that only check changed files

I'm using spotless with an android kotlin project now. And I have spotless set up like this

apply plugin: 'com.diffplug.gradle.spotless'
spotless {
    kotlin {
        target "**/*.kt"
        ktlint(versions.ktlint)
    }
}

What I want to do is set up a git pre commit hook that apply gradlew spotlessCheck before the commit but only on the commited files. It's something like this eslint-pre-commit-check
I have search around the repo but can't figure out how to achieve this. Is there any way to do this? Thanks.

The spotless gradle plugin has fast up-to-date checking built-in, so performance should be very good even without limiting to changed files. If your objective is to enforce the check only on changed files, that is a good idea, but it is not supported at this time. EDIT: now supported via ratchetFrom.

(bad advice from me deleted here)

Understood. Following is my workaround now. Hope it helps someone else who want to achieve this too.

#!/bin/sh
# From gist at https://gist.github.com/chadmaughan/5889802

echo '[git hook] executing gradle spotlessCheck before commit'

# stash any unstaged changes
git stash -q --keep-index

# run the spotlessCheck with the gradle wrapper
./gradlew spotlessCheck --daemon

# store the last exit code in a variable
RESULT=$?

# unstash the unstashed changes
git stash pop -q

# return the './gradlew spotlessCheck' exit code
exit $RESULT

Our latest versions (4.2.1 as of writing) have a feature called ratchetFrom. It's not exactly the same as this, but very close, and should perform much better. I'm closing this issue, but I might be wrong. If ratchetFrom does not make this (excellent!) workaround obsolete for you, please let me know!

ratchetFrom is very useful, but it does not, in fact make this obsolete, since it does not differentiate between changes staged for the commit and changes not staged for the commit. In order for a pre-commit hook to be useful, it should preferably only consider staged changes. Otherwise running spotlessApply in a hook makes it possible to make commits which do not in fact contain the formatting fixes.

(Also, if there is a way to make spotlessApply return a non-zero value if it does in fact change any files, that would be extremely useful as well. Now I'm running spotlessCheck first to find out if the hook should fail or not, then running spotlessApply to fix the problems automatically before failing. This probably does warrant opening a separate issue.)

I've got a slanted view here, since I use a git client that ignores the staging concept. I know that some users get a lot of power out of the staging area, and I'd love for Spotless to be helpful to them.

Perhaps what you'd really like is spotlessApply --staged, which would look only at staged files, and apply the result to the staging area, ignoring the checked-out files in the WC. Making that work would automatically work for spotlessCheck as well.

If spotlessApply --staged existed, would you care about the return value anymore? You could use spotlessCheck --staged if you wanted errors, or spotlessApply --staged if you just wanted it to just always be right.

--staged sounds like a good solution. I'm not sure whether it is possible for a pre-commit hook to actually stage the changes it makes, though, so it might still be useful to have an option to make spotlessApply to return a non-zero value if it changes anything.

(Also, since it is possible to have both staged and non-staged changes in the same file, --staged should be smart enough to only look at staged changes, not only staged files.)

I'm not sure whether it is possible for a pre-commit hook to actually stage the changes

Looks like it can.

(Also, since it is possible to have both staged and non-staged changes in the same file, --staged should be smart enough to only look at staged changes, not only staged files.)

The staging area is not set of changes. For each file, git has three binary blobs. The working copy, the head, and the index (aka staging area). Each of those is a binary blob, and the "changes" are backed-out by diffing the blobs, not the other way around.

In the simple way to use git, a file is either unstaged (index == head) or staged (index == working copy). In the "power" way, where only some "changes" are staged, all it means is that (index != head != working copy).

spotlessApply --staged would apply to the index blob, so it would do what I think you want.

I created a pre-commit hook that formats the code and adds the formatted files that were initially in the staging area back to the staging area in order to commit the actual formatted code. It leaves the other files as is.

you can install it by running this in your repository root:

curl -o .git/hooks/pre-commit https://gist.githubusercontent.com/toefel18/23c8927e28b7e7cf600cb5cdd4d980e1/raw/163337f2ae596d4b3a55936c2652b1e8227db5b7/pre-commit && chmod +x ./.git/hooks/pre-commit

#!/bin/bash
set -e

filesToAddAfterFormatting=()
containsJavaOrKotlin=0

# Collect all files currently in staging area, and check if there are any java or kotlin files
for entry in $(git status --porcelain | sed -r 's/[ \t]+/-/g')
do
  # entry can be for example:
  # MM-src/main/java/net/project/MyController.java
  # -M-src/main/java/net/project/AnotherController.java

 if [[ $entry == M* ]] ; then
    filesToAddAfterFormatting+=(${entry:2}) # strips the prefix
 fi

 if [[ $entry == *.java ]] || [[ $entry == *.kt ]] ; then
    containsJavaOrKotlin=1
 fi
done;


# If any java or kotlin files are found, run spotlessApply
if [ "$containsJavaOrKotlin" == "1" ] ; then
  echo "Kotlin and/or Java found in staging, running:  ./gradlew -PdisableSpotlessCheck spotlessApply"
  ./gradlew -PdisableSpotlessCheck spotlessApply
else
  echo "Not running spotlessApply"
fi

# Add the files that were in the staging area
for fileToAdd in $filesToAddAfterFormatting
do
  echo "re-adding $fileToAdd after formatting"
  git add "$fileToAdd"
done;

@toefel18

Thanks for sharing @toefel18.

@nedtwigg

@nedtwigg you are welcome :D, is there an option to make spotlessApply format one single file?

is there an option to make spotlessApply format one single file?

Yes, the IDE hook.

I created a pre-commit hook that formats the code and adds the formatted files that were initially in the staging area back to the staging area in order to commit the actual formatted code. It leaves the other files as is.

you can install it by running this in your repository root:
curl -o .git/hooks/pre-commit https://gist.githubusercontent.com/toefel18/23c8927e28b7e7cf600cb5cdd4d980e1/raw/163337f2ae596d4b3a55936c2652b1e8227db5b7/pre-commit && chmod +x ./.git/hooks/pre-commit

nice one - but that one seem to fail when using git commit --amend:

git commit --amend
Kotlin and/or Java found in staging, running:  ./gradlew -PdisableSpotlessCheck spotlessApply

Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.7/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 827ms
6 actionable tasks: 6 up-to-date
re-adding src/main/java/dev/jbang/U-il.java after formatting
fatal: pathspec 'src/main/java/dev/jbang/U-il.java' did not match any files

this is failing on OSX since sed here does not support \t for tab but instead replaces t's.

I fixed it by replacing the \t with an actual tab character.

Not sure if it is helpful nowadays for someone but I created a pre-commit hook that runs spotless maven plugin goals in maven projects.

Hey, thanks for getting back :)

By a large number of files, I say for instance I have like 16k java files in my project that come under the target.

If I understood correctly the isClean method is called for every file (at least what I found out when I was testing with my project and I was not using Gradle cache at that time).

So maybe calling the method 16k times in my case makes this slower.

I will check and reply here if I find something more :)

maybe calling the method 16k times in my case makes this slower.

For the very first run ever, it will call the method 16k times, and that will indeed be very slow. But after that, Gradle will only call the method for files which have changed on disk, and it will use its daemon filesystem monitor to prune that number to single digits. If you are using the gradle build cache, then even on a fresh clone it should not need to call the method 16k times because it can use the cached value.

This is a great thread, nice to see this has been discussed in such depth already. Is there currently a specific recommendation for folks who want to run spotlessApply as a pre-commit hook? (Wish github issues had a conclusion or summary section, so you don't have to try to infer it from all the comments!)

So from how I understand this thread, simply adding an option to pass multiple files to spotless instead of just a single file would already be a great improvement? As that allows for custom git hooks that don't have to start a new spotless process for each file. And a --staged option would help, but not be strictly necessary.

Edit: It seems there was a DspotlessFiles option that's been deprecated due to being slow and error prone as it was using regexes. I wonder if something like -DspotlessFileList ['/path/to/file1','/path/to/file2'] that's simply a list of file paths would be reasonable (where the argument is a JSON array since at least Maven does not have a good way to pass multiple values)

I've been using lint-staged to run spotless on staged files. It's been working beautifully, but lint-staged is an npm package. It only makes sense to use lint-staged if your project is using node.

adding an option to pass multiple files to spotless instead of just a single file would already be a great improvement

Sorta...

For the long-tail of users who are not happy with our current integration with git workflows, I think I don't understand how you see the problem. The way I see the problem is that spotless is designed to enforce a formatting invariant, always. That formatting invariant is either

all files must be this way (standard mode)
all files that have changed since origin/main must be this way (ratchetFrom)
all files in my staging area must be this way (spotlessApply --staged #623)
the file in my IDE must be this way (IDE_HOOK.md)

I can imagine that you might have some arbitrary list of files, so maybe bringing back the spotlessFiles to Gradle Spotless would be good. I'm not completely against it if it's implemented cleanly and documented well. In practice, it nukes configuration cache and made a bunch of other stuff hard, at least last time. Maybe there's a better way to do it. But mostly I don't understand why you would have such a unique list of files to format in the first place.

tl;dr If --staged can be implemented, don't bother with an option to pass individual files (in my opinion).

Thanks very much for the detailed explanation. I guess I should have been a little clearer on my thought process and what our use case is.

Our project currently doesn't use any automatic, but we'd like to add it. I'm currently checking out various options what we can use and how we can integrate it with git hooks. This plugin looks pretty promisingin that it isn't tied to any specific formatter and provides a standardized way to integrate with different formatters.

Our goal is pretty standard: ensure all files are formatted and that people don't commit unformatted code. In that sense, the invariant enforced by standard mode would be all we need. But since it's a large project with many files -- definitely on the order of magnitude of 16k files, for which somebody in this thread mentioned performance issues when the plugin needs to check all files.

So yes, the proposed --staged option (hadn't seen there was a different issue already) should be enough for integrating nicely with git pre-commit hooks. I only see giving an arbitrary list of files as a workaround if the --staged feature cannot be implemented -- it would make it possible to find the staged files within the git hook via a bash command and pass on those files to this plugin.

A while ago I had experimented a little bit with git-code-format-maven-plugin on another smaller project. It does exactly the same as the proposed --staged option. It even sets up git hooks automatically, which is pretty convenient, but not strictly necessary as their are other plugins for setting up git hooks.

We probably won't be using that formatting plugin as it only formats Java files, but it illustrates a nice approach to the problem. Formatting staged files is also what prettier suggests, using other such as lint-staged and calling the prettier command with a list of staged files.

To summarize, I'll need to experiment a little bit more with this plugin, and maybe current caching strategy is already fast enough so that no optimizations are needed. Being able to format only staged files would be a nice option to have if or when the hook turns out to need too much time (and some developers have slower machines than I do).

maybe current caching strategy is already fast enough so that no optimizations are needed

With the simplest "spotlessApply 16k files" option, you have Gradle using build cache so that it's only ever slow once, and you also have Gradle file-monitor based up-to-date checking, so it's extra super fast on a given machine after the first run.

Setting up git hooks automatically is a great idea, I added that to #623. Implementing --staged is definitely possible, but it's not on my personal shortlist. PR's welcome! Thanks to Gradle up-to-date mechanisms, I doubt that --staged will actually be faster. It's conceivable it could even be slower, because it might thrash against the caching / up-to-date mechansims. IMO --staged is a good idea because of the behavior spec, not the performance.

you have Gradle using build cache so that it's only ever slow once, and you also have Gradle file-monitor based up-to-date checking, so it's extra super fast on a given machine after the first run.

Well, we are using Maven, not Gradle, but we'll see how performance turns out to be. I'm glad your open for PRs and I might take a look at it depending on how it turns out.

Hi, finally we understood how the spotlessFiles are working :) and we put this in pre-commit hook
It will check list of git staged files
as we are using it for Java project, we filtered out the non-java files
after that, we converted the absolute path to only the filename
then adding .* at the beginning as it must be a pattern
then we have add , between the files to pass the comma-separated file name pattern

./mvnw spotless:check -DspotlessFiles=$(git diff --staged --name-only | grep .java | xargs -I {} basename {} | sed 's/^/.*/' | paste -sd ',' -)

merged main into swerve-cleanup

The solution described below should work for most situations. The pre-commit script described below eliminates the need to use spotlessFiles by instead taking advantage of ratchetFrom and git stash.

Maven

#!/bin/sh
echo '[git hook] executing spotless:apply to format code before commit'

# Stash any unstaged changes
git stash -q --keep-index

# Run spotless:apply
# We pass in ratchetFrom here to ensure that we only format the files that have changed since the last commit
if command -v mvn > /dev/null 2>&1 # Check if mvn command exists to support GitHub Desktop on Windows
then
    mvn spotless:apply -DratchetFrom=HEAD # Requires Maven to be installed
else
  ./mvnw spotless:apply -DratchetFrom=HEAD # Otherwise call maven wrapper for Mac-OS / Unix / Git for Windows
fi

# Store the last exit code in a variable
RESULT=$?

# Stage formatting changes
git add -u

# Un-stash the stashed changes
git stash pop -q

# Return the 'spotless:apply' exit code
exit $RESULT

Handles stashing unstaged changes
Only performs spotless on files that have changed since HEAD
Re-adds the formatted files back to git staging
Restores stashed files

Gradle

The Gradle equivalent would be:

#!/bin/sh
echo '[git hook] executing spotlessApply to format code before commit'

# Stash any unstaged changes
git stash -q --keep-index

# Run spotlessApply
# We pass in ratchetFrom here to ensure that we only format the files that have changed since the last commit
if command -v gradle > /dev/null 2>&1 # Check if gradle command exists to support GitHub Desktop on Windows
then
  gradle spotlessApply -PratchetFrom=HEAD # Requires Gradle to be installed
else
  ./gradlew spotlessApply -PratchetFrom=HEAD # Otherwise call gradle wrapper for Mac-OS / Unix / Git for Windows
fi

# Store the last exit code in a variable
RESULT=$?

# Stage formatting changes
git add -u

# Un-stash the stashed changes
git stash pop -q

# Return the 'spotlessApply' exit code
exit $RESULT

But the above Gradle solution assumes a few things:

Your grade is configured to install the above script. For me, I have this script in a directory called .hooks and created a custom Gradle task:

tasks.register('installLocalGitHook', Copy) {
    from new File(rootProject.rootDir, '.hooks/pre-commit')
    into { new File(rootProject.rootDir, '.git/hooks') }
    filePermissions {
        user {
            read true
            write true
            execute true
        }
        group {
            read true
            write true
            execute true
        }
        other {
            read true
            write false
            execute true
        }
    }
}

Your spotless gradle configuration is modified like to support the new -p property we are passing in

spotless {
    String ref = project.properties["ratchetFrom"]
    if (ref != null) {
        ref = ref.trim()
        if (ref.length() > 0) {
            ratchetFrom ref
        }
    }
    ...
}

@mrlonis

@mrlonis this can fail if there are staged and unstaged changes in the same file and applying formatting causes conflicts with unstaged section.

A variant of @bhaskarmelkani script that calls spotlessApply once (with ratchetFrom("HEAD")) worked best for me.

I didn't bother doing this for Gradle, but for Maven I think I can resolve the concerns of conflicts with unstaged changes with the following pre-commit and post-commit:

pre-commit

#!/bin/sh
handleMergeCommit() {
  if [ -f .git/MERGE_HEAD ]
  then
    echo "[git pre-commit hook] - Merge in progress. Exiting pre-commit hook."
    exit 0
  fi
}

handleEmptyCommitAtStart() {
  git diff --cached --quiet
  GIT_DIFF=$?
  if [ "$GIT_DIFF" -eq 0 ]; then
    echo "[git pre-commit hook] - No changes to commit! This is likely a rebase or merge commit. Exiting pre-commit hook."
    exit 0
  fi
}

stageUnStagedChangesInStagedFiles() {
  STAGED_FILES=$(git diff --name-only --cached --diff-filter=ad)
  if [ -n "$STAGED_FILES" ]; then
    echo "$STAGED_FILES" | xargs git add
  fi
}

spotlessApply() {
  # We pass in ratchetFrom here to ensure that we only format the files that have changed since the last commit
  if command -v mvn >/dev/null 2>&1; then    # Check if mvn command exists to support GitHub Desktop on Windows
    mvn spotless:apply -DratchetFrom=HEAD -q # Requires Maven to be installed
  else
    ./mvnw spotless:apply -DratchetFrom=HEAD -q # Otherwise call maven wrapper for Mac-OS / Unix / Git for Windows
  fi
}

handleMergeConflicts() {
  conflictedFiles="$(git diff --name-only --diff-filter=U)"
  if [ -n "$conflictedFiles" ]; then
    for conflictedFile in $conflictedFiles; do
      echo "[git pre-commit hook] - Resolving conflict for $conflictedFile"
      git checkout --theirs "$(pwd)/$conflictedFile"
      git restore --staged "$(pwd)/$conflictedFile"

      if command -v mvn >/dev/null 2>&1; then
        mvn spotless:apply -DspotlessFiles=".*$conflictedFile" -q
      else
        ./mvnw spotless:apply -DspotlessFiles=".*$conflictedFile" -q
      fi

      git add "$conflictedFile"
    done
  fi
}

handleEmptyCommitAtEnd() {
  git diff --cached --quiet
  GIT_DIFF=$?
  if [ "$GIT_DIFF" -eq 0 ]; then
    echo "[git pre-commit hook] - No changes to commit! Aborting commit!"
    # We end up with no changes to commit here due to the stashing and un-stashing of changes
    # where we stash a bad formatting change, apply the formatting, and then un-stash the bad formatting change.
    #
    # Example:
    #   - We accidentally indented 1 line in a file by an extra space. This causes us to stash the extra space, apply the
    #     formatting which will remove the extra space, and then un-stash the extra space back into the file that the
    #     formatter just removed. By re-running spotless:apply, we remove the extra space again and remove the file from
    #     the staging area.
    #
    # This results in no changes to commit, so we exit with 1 to prevent committing an empty commit.
    # We run spotless:apply again here to ensure that the files are formatted correctly and to remove the file from the staging area.
    spotlessApply
    exit 1
  fi
}

handleMergeCommit
handleEmptyCommitAtStart
stageUnStagedChangesInStagedFiles
git stash clear
git stash -q --keep-index
echo "[git pre-commit hook] - Running spotless:apply"
spotlessApply
SPOTLESS_APPLY_RESULT=$?
git add -u
git stash pop -q
handleMergeConflicts
handleEmptyCommitAtEnd
exit $SPOTLESS_APPLY_RESULT

post-commit

#!/bin/sh
# We pass in ratchetFrom here to ensure that we only format the files that have changed since the last commit
if command -v mvn >/dev/null 2>&1; then # Check if mvn command exists to support GitHub Desktop on Windows
  mvn spotless:apply -DratchetFrom=HEAD -q # Requires Maven to be installed
else
  ./mvnw spotless:apply -DratchetFrom=HEAD -q # Otherwise call maven wrapper for Mac-OS / Unix / Git for Windows
fi

This seems to handle most scenarios I throw at it

Super necro, but for gradle I've come up with this:

build.gradle:

spotless {
    String ref = project.properties["ratchetFrom"]
    if (ref != null) {
        ref = ref.trim()
        if (ref.length() > 0) {
            ratchetFrom ref
        }
    }

    java {
        target "**/*.java"
        googleJavaFormat("1.18.0").aosp()
        importOrder "", "javax", "java", "\\#"
    }

    groovyGradle {
        target "**/*.gradle"
        greclipse().configFile("tools/spotless-gradle.properties")
    }

    format 'styling', {
        target '**/*.md', '**/*.yml', '**/*.yaml', '**/*.json'
        prettier()
    }
}

pre-commit__spotless-apply.sh:

#!/usr/bin/env bash
set -euo pipefail

if [ "${PRE_COMMIT:-0}" -ne 1 ]; then
  echo "This script should only be run by pre-commit! Exiting." >&2
  exit 1
fi

REQUIREMENTS=("git" "java")
for i in "${REQUIREMENTS[@]}"; do
  if ! command -v "${i}" &> /dev/null; then
    echo "'${i}' is required to run this script. Please install it." >&2
    exit 1
  fi
done

RATCHET_REF="${PRE_COMMIT_FROM_REF:-HEAD^}"
echo "Running spotlessApply on files changed since ${RATCHET_REF}"

# shellcheck source=../gradlew
./gradlew --no-daemon --console=plain :spotlessApply -PratchetFrom="${RATCHET_REF}"

.pre-commit-config.yaml:

repos:
  - repo: local
    hooks:
      - id: gradle-spotless-apply
        name: Run Gradle Spotless Apply
        language: script
        entry: ./scripts/pre-commit__spotless-apply.sh
        types_or:
          - java
          - groovy
          - yaml
          - markdown
          - json
        pass_filenames: false
        require_serial: true

In normal use (ie. as a git hook) it will ratchetFrom HEAD^. If you run pre-commit with --from-ref=$somesha, it'll ratchetFrom $somesha. This means it works nicely when run in CI (from merge target, to HEAD).

Obviously this is pretty tightly coupled with pre-commit, so YMMV with the needed context stuff.

Navigation Menu

Repository navigation

How to setup spotless for a git pre commit hook that only check changed files #178

6 remaining items

Maven

Gradle

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

How to setup spotless for a git pre commit hook that only check changed files #178

Description

Activity

nedtwigg commented on Dec 14, 2017

Zkffkah commented on Dec 14, 2017

nedtwigg commented on Jun 5, 2020

ehuisman commented on Jun 24, 2020

nedtwigg commented on Jun 24, 2020

ehuisman commented on Jun 25, 2020

nedtwigg commented on Jun 25, 2020

toefel18 commented on Nov 28, 2020

nedtwigg commented on Nov 29, 2020

toefel18 commented on Nov 30, 2020

nedtwigg commented on Dec 1, 2020

maxandersen commented on Dec 28, 2020

ejba commented on Jan 23, 2021

6 remaining items

bhaskarmelkani commented on Aug 20, 2021

nedtwigg commented on Aug 20, 2021

husam-e commented on Oct 27, 2022

blutorange commented on Nov 14, 2022

jonsmithers commented on Nov 14, 2022

nedtwigg commented on Nov 14, 2022

blutorange commented on Nov 16, 2022

nedtwigg commented on Nov 16, 2022

blutorange commented on Nov 16, 2022

mnafshin commented on Aug 21, 2023

mrlonis commented on Jul 5, 2024

Maven

Gradle

lucas-mpc-brant commented on Jul 9, 2024

mrlonis commented on Jul 10, 2024

whi-tw commented on Aug 2, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions