Was not expecting re-add to add an external file

[Lockszmith-GH]

This is an approximation of my output chezmoi (v2.60.1)

> cz status
 M .local/bin/tv
MM some-file-i-changed

> cz status --exclude externals
MM some-file-i-changed

> cz re-add --exclude externals

> cz status
chezmoi: .local/bin/tv: inconsistent state (~/.local/share/chezmoi/_home.macos/private_dot_local/bin/executable_tv, https://github.com/alexpasmantier/television/releases/download/0.10.10/tv-0.10.10-macos-arm64.tar.gz defined in ~/.local/share/chezmoi/_home.macos/.chezmoiexternal.yaml.tmpl)

> OFFLINE=1 cz forget ~/.local/bin/tv

> cz status
 M .local/bin/tv

> cz apply

> cz status

As you can see, when I did a re-add, it added the ~/.local/bin/tv file, breaking chezmoi until I removed it from the source.

Was not expecting that.
Does this match your expectation here?

[Lockszmith-GH]

I've updated the output above - the issue is the same even when using --exclude externals. At this point, this seems like a bug.

[twpayne]

Yes, this behavior is suspicious.

I tried to reproduce what you saw in #4366, but everything is working as-expected for me.

[Lockszmith-GH]

I'll try and reproduce it in a controlled way

[Lockszmith-GH]

I was able to reproduce it.

At the bottom you can find a complete Dockerfile, compose.yml a test-script.sh, and the invocation command I used to reproduce the issue.

tl;dr:

I tried testing, just like you, with a file - and indeed, the issue did not reproduced.
However, when the destination is in an archive-file, the re-add misidentified the new-file.

Invocation and full output

Click to expand/collapse the complete reproduction scenario

Once all files below are in a folder, invoke the test with:

> UID=$UID GID=$GID docker compose build && docker compose run --rm -it bash
[+] Building 0.2s (9/9) FINISHED                                                                    docker:default
 => [bash internal] load build definition from Dockerfile                                                     0.0s
 => => transferring dockerfile: 395B                                                                          0.0s
 => [bash internal] load metadata for docker.io/library/alpine:latest                                         0.1s
 => [bash internal] load .dockerignore                                                                        0.0s
 => => transferring context: 2B                                                                               0.0s
 => [bash 1/4] FROM docker.io/library/alpine:latest@sha256:a8560b36e8b8210634f77d9f7f9efd7ffa463e380b75e2e74  0.0s
 => CACHED [bash 2/4] RUN printf "%s=%s\n" "PUID" "2001" "PGID" "1001"                                        0.0s
 => CACHED [bash 3/4] RUN apk add --no-cache bash git shadow chezmoi file curl     && groupadd -g 1001 group  0.0s
 => CACHED [bash 4/4] WORKDIR /home/user                                                                      0.0s
 => [bash] exporting to image                                                                                 0.0s
 => => exporting layers                                                                                       0.0s
 => => writing image sha256:5998fa7b4413f2f5be77b94677ec239e242010284a813ca99c551e2c06f17241                  0.0s
 => => naming to docker.io/library/test-chezmoi-bash                                                          0.0s
 => [bash] resolving provenance for metadata file                                                             0.0s
21: managed-v1 created
27: managed-v1 added
30: external-v1 created
50: external-v1 ready managed-v1 added
53: As expected, file does not exist locally yet.
57: Setup complete.
test-ext-file
66: Changes applied internally and externally. Testing...
chezmoi: test-ext-from-archive-file: inconsistent state (/home/user/.local/share/chezmoi/private_executable_test-ext-from-archive-file, file:///mnt/test/test-ext-file.tgz defined in /home/user/.local/share/chezmoi/.chezmoiexternal.yaml)

The test-script.sh

#! /usr/bin/env bash

set -e

# (re-)create simple executable script
mkex() {
    [ -e "${1}" ] && rm "${1}"
    install -m 700 <(echo "echo '${@:2}'") "$1"
}

# (re-)create simple executable script (with mkex) and a tgz archive containing it
mkex-ext() {
    mkex "${@}"
    (cd "${1%/*}"; tar cvzf "${1##*/}".tgz "${1##*/}")
}

chezmoi init
EXPECTED="managed-v1"
mkex ~/test-managed-file "${EXPECTED}"
TEST="$(~/test-managed-file)"
[ "$TEST" == "${EXPECTED}" ] && echo "$LINENO: managed-v1 created" >&2

chezmoi add ~/test-managed-file
EXPECTED="test-managed-file"
TEST="$(chezmoi status)" && [ "$TEST" == "" ] || (chezmoi --debug status; exit 2)
TEST="$(chezmoi managed)"
[ "$TEST" == "${EXPECTED}" ] && echo "$LINENO: managed-v1 added" >&2

TEST="$(mkex-ext /mnt/test/test-ext-file "external-v1")"
[ "$TEST" == "test-ext-file" ] && echo "$LINENO: external-v1 created" >&2

CZDIR="$(chezmoi source-path)"
CZEXCFG="$CZDIR/.chezmoiexternal.yaml"
cat > "${CZEXCFG}" <<CZEXCFG
test-ext-file:
  type: file
  executable: yes
  url: file:///mnt/test/test-ext-file

test-ext-from-archive-file:
  type: archive-file
  executable: yes
  path: test-ext-file
  url: file:///mnt/test/test-ext-file.tgz
CZEXCFG

EXPECTED=" A test-ext-file
 A test-ext-from-archive-file"
TEST="$(chezmoi status)" || chezmoi --debug status
[ "$TEST" == "${EXPECTED}" ] && echo "$LINENO: external-v1 ready managed-v1 added" >&2

TEST="$(~/test-ext-file 2> /dev/null)" || true
[ "$TEST" == "" ] && echo "$LINENO: As expected, file does not exist locally yet." >&2

chezmoi apply
TEST="$(~/test-ext-file)" || true
[ "$TEST" == "external-v1" ] && echo "$LINENO: Setup complete." >&2

mkex ~/test-managed-file managed-v2
mkex-ext /mnt/test/test-ext-file external-v2

EXPECTED=" M test-ext-file
 M test-ext-from-archive-file
MM test-managed-file"
TEST="$(chezmoi status)"
[ "$TEST" == "${EXPECTED}" ] && echo "$LINENO: Changes applied internally and externally. Testing..." >&2

chezmoi re-add
EXPECTED="chezmoi/.chezmoiexternal.yaml
chezmoi/private_executable_test-managed-file"
# The following shows that the file sourced in the archive was added
TEST="$(cd "${CZDIR}/.."; find chezmoi -xdev -type f -not -path 'chezmoi/.git/*')"
[ "$TEST" == "${EXPECTED}" ] && echo "$LINENO: Changes applied internally and externally. Testing..." >&2

# if above breaks, the following creates the inconsistent state message
chezmoi status

# The Error:
# chezmoi: test-ext-from-archive-file: inconsistent state (/home/user/.local/share/chezmoi/private_executable_test-ext-from-archive-file, file:///mnt/test/test-ext-file.tgz defined in /home/user/.local/share/chezmoi/.chezmoiexternal.yaml)

Dockerfile

FROM alpine

ARG PUID=2001
ARG PGID=1001

RUN printf "%s=%s\n" "PUID" "$PUID" "PGID" "$PGID"

RUN apk add --no-cache bash git shadow chezmoi file curl \
    && groupadd -g $PGID group \
    && useradd -u $PUID -g $PGID -m user \
    && install -o $PUID -g $PGID -d /mnt/test

VOLUME /mnt/test
VOLUME /home/user

USER user

WORKDIR /home/user

CMD ["bash"]

compose.yml

services:
  bash:
    build:
      context: .
      args:
        PUID: "${UID:-1000}"
        PGID: "${GID:-1000}"
    command: './test-script.sh'
    volumes:
    - ./test-script.sh:/home/user/test-script.sh:ro

[Lockszmith-GH]

@twpayne were you able to reproduce?

[twpayne]

Not yet. I need to unpick the reproduction in the Dockerfile and convert it to a testscript that I can run and debug.