Calculate filesystem node's hash lazily.

[candid82]

The main motivation behind this change is to speed up status operation. Currently it's very slow, especially for repositories with lots of ignored files (e.g. node.js repository with node_modules directory). One of the reasons for this slowness is the fact that traversing filesystem involves calculating file hashes for all files, even if those hashes are not needed in the end because the files are in gitignore.

On my machine, for a random repository with sizable (ignored) node_modules directory this changes bring the execution time for Worktree.Status from ~3.5s to ~1.4s. This is still very slow, but a significant improvement.

A better fix (instead of or in addition to this one) would be to avoid traversing ignored files in the first place. However, such change seem to be more intrusive and will require much deeper understanding of the codebase.

[pjbgf]

@candid82 thanks for your PR. Do you mind providing a benchstat of the performance improvements please?

Also, re-word the commit message to align with the contribution guidelines.

A better fix (instead of or in addition to this one) would be to avoid traversing ignored files in the first place. However, such change seem to be more intrusive and will require much deeper understanding of the codebase.

Happy for this to be a follow-up PR.

[candid82]

@pjbgf Below please find benchstat results for one of my local repositories with sizable node_modules directory. Obviously, the results depend on the size of git repo and the number of ignored files.

> benchstat ~/old.txt  ~/new.txt
goos: darwin
goarch: amd64
pkg: github.com/go-git/go-git/v5
cpu: Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
          │ /Users/candid/old.txt │       /Users/candid/new.txt        │
          │        sec/op         │   sec/op    vs base                │
Status-16              3.875 ± 1%   1.392 ± 3%  -64.07% (p=0.000 n=10)

The benchmark code:

func BenchmarkStatus(b *testing.B) {
	repo, _ := PlainOpen("<path to repo>")
	worktree, _ := repo.Worktree()
	for i := 0; i < b.N; i++ {
		s, _ := worktree.Status()
		if len(s) > 0 {
			b.Errorf("error!")
		}
	}
}

I also adjusted the commit message.

[pjbgf]

Thank you for providing the benchstat and rewording the commit.

Doing another review I noticed that this change could introduce an issue by which once the Hash() is called and the hash is calculated, changes to the underlying fs will not trigger a new hash recalculation. By looking for the places where this is used, I see no red flags that this would be a problem. Please add some comments in the Hash() func to that effect.

Otherwise, LGTM.

[codablock]

@candid82 @pjbgf Any updates on this PR? If there is lack of time on @candid82's side, I'm also happy to provide the requested comments/docs after this PR is merged.

Merging this PR would alleviate a lot of pain in regard to #181. On my side, it would help me avoid a temporary workaround that would exec-out into git status, so I'm very happy to instead help here :)

[candid82]

@codablock I moved on to other things and currently don't have the time to properly evaluate the concern brought up by @pjbgf in the previous comment. Please feel free to pick this work up.

[codablock]

@pjbgf As requested in Slack, I have tested this PR locally (by replacing github.com/go-git/go-git/v5 with my local version inside go.mod) and can confirm that Status() performance gets improved a lot. I used the notorious node_modules to test things out and Status() runtime has been reduced to below a second from previously 5-10 seconds.

If you merge this PR, I will follow up with a PR that adds the docs/comments.

[pjbgf]

@codablock thanks for the testing. @candid82 thank you for your original contribution.

[codablock]

Here is the PR with the requested comment update: #992