Add support to follow `.gitignore` (or custom `.treefmtignore`) file for ignoring files or directories?

[aldoborrero]

Rationale

Sometimes, you want to exclude directories that don't belong to any specific language but rather because you don't want treefmt to traverse those. As far as I'm aware, you need to exclude those from one specific formatter.

Note: There's global support for ignoring directories.

We can add support to follow the .gitignore options or if .treefmtignore is present, add those directories to the global config.

We can achieve a similar behaviour by leveraging something like gitignore crate.

[zimbatm]

At the moment it looks for .ignore files during the tree traversal if I'm not mistaken, per the ignore crate. I think it can be extended to also load .gitignore files?

I think the big question is: are there cases where you want to treefmt a file that is not in git? If yes, then it would be circumventing that use case.

[zimbatm]

Related to #154 and #182

[adam-gaia]

I think treefmt has unintentionally lost the ability to respect ignore files like it used to.

I use treefmt via treefmt-nix as a pre-commit hook.
A while back, I added support for a formatter, typos, to treefmt-nix.

I almost never invoke treefmt (or typos) directly, but instead as pre-commit hooks. This means, that treefmt (and formatters invoked by treefmt such as typos) only operates on changed files that are git staged for committing.

To my surprise (and horror), I invoked treefmt manually today (not via pre-commit) and typos tried to spellcheck every binary file in my rust target/ directory 😆.

It took me a while to figure out what was going on because the typos formatter printed the contents of every binary file to my terminal as it did its "spellchecking", which meant it was pretty much impossible to scroll back in the history to find some useful info on what had happened.

I run many different formatters under treefmt. Eventually I figured out it was typos causing the issue.

Typos will automatically ignore binary and gitignored, unless it is explicitly passed those files as arguments, which is what treefmt is designed to do.

I wrote the treefmt-nix module for typos and set includes = [ "*" ];, thinking treefnt would respect .gitignore before invoking typos.
Due to wanting every file to be spellchecked, there isn't a good way to update the treefmt-nix module for typos. No file extension filter will catch every binary file. I believe filtering of binary/hidden/ignored files should happen upstream, by treefmt itself.

[brianmcgee]

Is this repo public? If not, can you provide a simple reproducer?

We should be respecting .gitignore when using the git walker.

There was one edge case introduced when we allowed traversing untracked files recently, where it wouldn't respect .gitignore. That was only on main and was fixed before https://github.com/numtide/treefmt/releases/tag/v2.2.0 was released last week.

treefmt-nix will be updated with 2.2.0 as the default later today when I look at some breaks introduced when updating nixpkgs: numtide/treefmt-nix#332

[adam-gaia]

Thanks for the quick reply!!

I have been using typos v2.2.0 by setting treefmt.nix.nixpkgs.follows = nixpkgs unstable.

Here is a simple reproduction:
https://github.com/adam-gaia/treefmt-typos-ignore-bug

The reproduction repo's flake's default devshell provides my pre-comit/treefmt/typos setup, while also adding treefmt + typos to the path.

There is a justfile that I think is self explaniary, but I'm still happy to provide more into!

$ just setup                                                                                                                                                                                                                                 
Creating file with intentional typos that is gitignore'd: foo.txt

$ just check                                                                                                                                                                                                                                typos --version
typos-cli 1.30.3
treefmt --version
treefmt v2.2.0
Will not format .gitignore'd files:
typos

Should not format .gitignore'd files, but will:
treefmt -vv
DEBU walk | git: processing file: /home/agaia/tmp/treefmt-typos-bug/.direnv/bin/nix-direnv-reload
DEBU walk | git: processing file: /home/agaia/tmp/treefmt-typos-bug/.direnv/flake-inputs/5hvi4h69xrzdl0d16yi25mhks1wd5ys2-source
DEBU walk | git: processing file: /home/agaia/tmp/treefmt-typos-bug/.direnv/flake-inputs/8bh3jgq1riy3jxm07vy4xxzvk9xd74pc-source
DEBU walk | git: processing file: /home/agaia/tmp/treefmt-typos-bug/.direnv/flake-inputs/8kpx53qi52yhjai1vdw8zpa95iqa61bv-source
DEBU walk | git: processing file: /home/agaia/tmp/treefmt-typos-bug/.direnv/flake-inputs/g5v3sgqy6a0fsmas7mnapc196flrplix-source
DEBU walk | git: processing file: /home/agaia/tmp/treefmt-typos-bug/.direnv/flake-inputs/p0kdydbkd1kd89zmxgscswi4y4dzwhfj-source
DEBU walk | git: processing file: /home/agaia/tmp/treefmt-typos-bug/.direnv/flake-inputs/r2l9q8mznwbgn34668v8n9c8biap1g84-source
DEBU walk | git: processing file: /home/agaia/tmp/treefmt-typos-bug/.direnv/flake-inputs/sw0baymdl83qlspwj8654c232ph4nfyb-source
DEBU walk | git: processing file: /home/agaia/tmp/treefmt-typos-bug/.direnv/flake-inputs/yj1wxm9hh8610iyzqnz75kvs6xl8j3my-source
DEBU walk | git: processing file: /home/agaia/tmp/treefmt-typos-bug/.direnv/flake-profile-a5d5b61aa8a61b7d9d765e1daf971a9a578f1cfa
DEBU walk | git: processing file: /home/agaia/tmp/treefmt-typos-bug/.direnv/flake-profile-a5d5b61aa8a61b7d9d765e1daf971a9a578f1cfa.rc
DEBU walk | git: processing file: /home/agaia/tmp/treefmt-typos-bug/.pre-commit-config.yaml
DEBU walk | git: processing file: /home/agaia/tmp/treefmt-typos-bug/foo.txt
DEBU walk | git: processing file: /home/agaia/tmp/treefmt-typos-bug/.envrc
DEBU walk | git: processing file: /home/agaia/tmp/treefmt-typos-bug/.gitignore
DEBU walk | git: processing file: /home/agaia/tmp/treefmt-typos-bug/README.md
DEBU walk | git: processing file: /home/agaia/tmp/treefmt-typos-bug/devshell.nix
DEBU walk | git: processing file: /home/agaia/tmp/treefmt-typos-bug/flake.lock
DEBU walk | git: processing file: /home/agaia/tmp/treefmt-typos-bug/flake.nix
DEBU walk | git: processing file: /home/agaia/tmp/treefmt-typos-bug/formatter.nix
DEBU walk | git: processing file: /home/agaia/tmp/treefmt-typos-bug/justfile
DEBU walk | git: processing file: /home/agaia/tmp/treefmt-typos-bug/treefmt.nix
DEBU walk | git: processing file: /home/agaia/tmp/treefmt-typos-bug/typos.toml
DEBU walk | cache: read 13 files from delegate
DEBU read 13 files
DEBU formatter | typos: match: /home/agaia/tmp/treefmt-typos-bug/.direnv/bin/nix-direnv-reload
DEBU formatters signature for key typos: f422f60c84a5d5544e3ae0dd9aa3cd62
DEBU formatter | typos: match: /home/agaia/tmp/treefmt-typos-bug/.direnv/flake-profile-a5d5b61aa8a61b7d9d765e1daf971a9a578f1cfa.rc
DEBU formatter | typos: match: /home/agaia/tmp/treefmt-typos-bug/foo.txt
DEBU formatter | typos: match: /home/agaia/tmp/treefmt-typos-bug/.envrc
DEBU path matched global excludes: .gitignore
DEBU formatter | typos: match: /home/agaia/tmp/treefmt-typos-bug/README.md
DEBU formatter | typos: match: /home/agaia/tmp/treefmt-typos-bug/devshell.nix
DEBU path matched global excludes: flake.lock
DEBU formatter | typos: match: /home/agaia/tmp/treefmt-typos-bug/flake.nix
DEBU formatter | typos: match: /home/agaia/tmp/treefmt-typos-bug/formatter.nix
DEBU formatter | typos: match: /home/agaia/tmp/treefmt-typos-bug/justfile
DEBU formatter | typos: match: /home/agaia/tmp/treefmt-typos-bug/treefmt.nix
DEBU formatter | typos: match: /home/agaia/tmp/treefmt-typos-bug/typos.toml
DEBU formatter | typos: executing: /nix/store/0xjm5ikxz9rqyrvjff9laj9bqgifc321-typos-1.30.3/bin/typos --write-changes .direnv/bin/nix-direnv-reload .direnv/flake-profile-a5d5b61aa8a61b7d9d765e1daf971a9a578f1cfa.rc foo.txt .envrc README.md devshell.nix flake.nix formatter.nix justfile treefmt.nix typos.toml
ERRO formatter | typos: failed to apply with options '[--write-changes]': exit status 2

error: `wether` should be `weather`, `whether`
  --> foo.txt:1:111
  |
1 | The quick brownn fox jumpped over the lazyy dogg, not noticing the rainn that had beggun to fall. Despite the wether, it continued it's jurney with great determanation.
  |                                                                                                               ^^^^^^
  |
error: `wether` should be `weather`, `whether`
  --> justfile:9:122
  |
9 |     @echo "The quick brownn fox jumpped over the lazyy dogg, not noticing the rainn that had beggun to fall. Despite the wether, it continued it's jurney with great determanation." > {{typofile}}
  |                                                                                                                          ^^^^^^
  |

traversed 13 files
emitted 11 files for processing
formatted 0 files (0 changed) in 30ms
DEBU no more files to read
Error: failed to finalise formatting: formatting failures detected
error: Recipe `check` failed on line 21 with exit code 1

---

Update: I still think treefmt should respect ignored files, but I just found a workaround, specifically for typos, while creating the reproduction.

Typos now provides a --force-exclude flag which forces it to respect ignored files, even when explicitly passed those files as arguments. I'll update the treefmt-nix typos module to use this. I think this will help in cases where typo's config excludes something that is otherwise not gitignore'd, and thus would get picked up by treefmt regardless.

That will fix typos via treefmt linting the 'justfile' in my example above, even though I've set it to be ignored in my typos.toml in the reproduction repo.

[veselyn]

Is this repo public? If not, can you provide a simple reproducer?

We should be respecting .gitignore when using the git walker.

There was one edge case introduced when we allowed traversing untracked files recently, where it wouldn't respect .gitignore. That was only on main and was fixed before https://github.com/numtide/treefmt/releases/tag/v2.2.0 was released last week.

treefmt-nix will be updated with 2.2.0 as the default later today when I look at some breaks introduced when updating nixpkgs: numtide/treefmt-nix#332

Hi @brianmcgee,

I'm hitting a similar problem as @adam-gaia.

Here's a repository that reproduces it: https://github.com/veselyn/numtide-treefmt-issues-198

Reproducible with treefmt v2.2.0. I couldn't reproduce it with treefmt 2.1.1 in another project of mine.

[brianmcgee]

Thanks both of you for reporting this. You've caught a bad release.

I checked the tag for 2.2.0 and it's pointing to this commit 0773976. This was my fault, I had a stale local checkout when I cut the release.

cca79a8 was added after this commit which repects gitignore correctly after added the untracked files changes.

I'll cut a new release now.

[brianmcgee]

Here's the new release https://github.com/numtide/treefmt/releases/tag/v2.2.1.

Should propagate through Hydra in the next couple days.

[adam-gaia]

@brianmcgee No worries, it happens. Thanks for the quick response!