some application image builds failing with APKO-provided Kaniko

[crispysipper]

Actual behavior
Environment: using APKO kaniko as a capability within a CI/CD utility image to power our pipeline infrastructure
The CI/CD utility image is built upon wolfi-base:latest
CI environment is exclusively Gitlab CI with Kubernetes executors and unprivileged containers

During the build phase, what appears to be filesystem and/or layering errors occur such as this:

INFO[0000] Resolved base name node:18 to build          
INFO[0000] Retrieving image manifest node:18            
INFO[0000] Retrieving image node:18 from registry index.docker.io 
INFO[0000] Retrieving image manifest nginx:alpine       
INFO[0000] Retrieving image nginx:alpine from registry index.docker.io 
INFO[0001] Built cross stage deps: map[0:[/app/build]]  
INFO[0001] Retrieving image manifest node:18            
INFO[0001] Returning cached image manifest              
INFO[0001] Executing 0 build triggers                   
INFO[0001] Building stage 'node:18' [idx: '0', base-idx: '-1'] 
INFO[0001] Unpacking rootfs as cmd COPY package*.json ./ requires it. 
error building image: error building stage: failed to get filesystem from image: mkdir /usr/lib/terminfo: file exists

Common sense workarounds have been attempted, like deleting/recreating that filepath but the result is always the same.

Of note - using our previous CI/CD Utility image, built on public.ecr.aws/docker/library/bash and kaniko added by copying over the public Kaniko binaries from gcr.io/kaniko-project/executor:debug via Dockerfile does not result in these errors.

Expected behavior
Successful application image build

To Reproduce
Steps to reproduce the behavior:

1. Obtain wolfi-base image loaded with APKO kaniko
2. Attempt to build application image using dockerfile, example:

# Stage 1: Build the React app
FROM node:18 AS build

WORKDIR /app

COPY package*.json ./
RUN npm install

COPY . .

RUN npm run build

# Stage 2: Serve the build with nginx
FROM nginx:alpine

COPY --from=build /app/build /usr/share/nginx/html

EXPOSE 80

CMD ["nginx", "-g", "daemon off;"]

package.json:

{
  "name": "my-react-app",
  "version": "1.0.0",
  "private": true,
  "dependencies": {
    "react": "^18.2.0",
    "react-dom": "^18.2.0",
    "react-scripts": "5.0.1"
  },
  "scripts": {
    "start": "react-scripts start",
    "build": "react-scripts build",
    "test": "react-scripts test",
    "eject": "react-scripts eject"
  },
  "browserslist": {
    "production": [
      ">0.2%",
      "not dead",
      "not op_mini all"
    ],
    "development": [
      "last 1 chrome version",
      "last 1 firefox version",
      "last 1 safari version"
    ]
  }
}

• Kaniko Image (fully qualified with digest)
  See below dockerfile for our CI/CD Utility Image build:

# Stage: hauler
FROM docker.io/hauler/hauler-debug:1.2.4 AS hauler
# Stage: packer
FROM docker.io/hashicorp/packer:latest AS packer
FROM cgr.dev/chainguard/wolfi-base:latest
# Metadata arguments
ARG SOURCE_APK_REPO='https://apk.cgr.dev/chainguard'
ARG NCTL_VERSION='4.3.0'
ARG DEPENDENCY_CHECK_VERSION='12.1.0'
ARG SONAR_VERSION='7.1.0.4889'
# Upgrade system and install packages
# Install packages w/ kaniko
RUN apk update && \
    apk upgrade --no-cache && \
    apk add --no-cache \
        python3 openjdk-17 nodejs npm go bash make maven perl ruby py3-pip \
        podman buildah buildkit skopeo docker-credential-ecr-login crane \
        fuse-overlayfs kubectl aws-cli cosign opentofu flux harbor-cli argo-cd \
        kaniko kaniko-warmer kaniko-compat kaniko-warmer-compat \
        sonar-scanner-cli grype syft unzip jq curl yq glab bubblewrap ncurses openssl

# Environment setup
ENV JAVA_HOME=/usr/lib/jvm/java-17-openjdk \
    GOBIN=/usr/local/bin \
    PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$JAVA_HOME/bin"

# install hauler
RUN curl -sfL https://get.hauler.dev | bash

# install harborcli
COPY --from=registry.goharbor.io/harbor-cli/harbor-cli:latest /harbor /usr/local/bin/harbor

# Set environment
ENV JAVA_HOME=/usr/lib/jvm/java-17-openjdk \
GOBIN=/usr/local/bin \
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$JAVA_HOME/bin"
# Install tools in a single RUN for caching efficiency
RUN set -eux; \
    # Download and extract policies
    curl -sSL -H "PRIVATE-TOKEN:<redacted>" \
        "https://gitlab.com/api/v4/projects/<project-id>/repository/archive.tar.gz" -o policies.tar.gz && \
    mkdir -p policies && tar xf policies.tar.gz -C policies --strip-components 1 && \
    # install argo-cli
    # curl -sSL -o argocd-linux-amd64 https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64 && install -m 755 argocd-linux-amd64 /usr/local/bin/argocd && /bin/cp -a /usr/local/bin/argocd /usr/bin/argocd && \
    # Install Nirmata CLI
    curl -sSL -o nctl.zip "https://nirmata-downloads.s3.us-east-2.amazonaws.com/nctl/nctl_${NCTL_VERSION}/nctl_${NCTL_VERSION}_linux_386.zip" && \
    unzip -o nctl.zip && install -m 755 nctl /usr/bin/nctl && \
    # # install sonar-scanner-cli
    # curl -sSOL https://binaries.sonarsource.com/Distribution/sonar-scanner-cli/sonar-scanner-cli-${SONAR_VERSION}.zip && unzip sonar-*.zip && /bin/cp -a /sonar-scanner-${SONAR_VERSION}/bin/sonar-scanner /usr/bin && /bin/cp -a /sonar-scanner-${SONAR_VERSION}/bin/sonar-scanner-debug /usr/bin && \
    # ln -s /sonar-scanner-${SONAR_VERSION}/lib/sonar-scanner-cli-${SONAR_VERSION}.jar /usr/lib/sonar-scanner-cli-${SONAR_VERSION}.jar && \
    # # install dependency-check and download NVD
    # curl -sSL -o dc.zip "https://github.com/jeremylong/DependencyCheck/releases/download/v${DEPENDENCY_CHECK_VERSION}/dependency-check-${DEPENDENCY_CHECK_VERSION}-release.zip" && \
    # unzip -q dc.zip -d /opt && ln -s /opt/dependency-check-${DEPENDENCY_CHECK_VERSION} /opt/dependency-check && \
    # /opt/dependency-check/bin/dependency-check.sh --updateonly --nvdApiDelay 16000 --nvdApiKey e12bd712-54c8-4662-87d0-e237ec130d6b && \
    # Install chainctl
    curl -sSL -o chainctl "https://dl.enforce.dev/chainctl/latest/chainctl_$(uname -s | tr '[:upper:]' '[:lower:]')_$(uname -m | sed 's/aarch64/arm64/')" && \
    install -m 755 chainctl /usr/local/bin/chainctl && \
    # Install apko & melange
    go install chainguard.dev/apko@latest && \
    go install chainguard.dev/melange@latest && \
    # Cleanup
    rm -rf *.zip *.gz nctl chainctl policies.tar.gz

# Copy tools from builder stages
COPY --from=hauler /usr/local/bin/hauler /usr/local/bin/hauler
COPY --from=packer /bin/packer /usr/local/bin/packer




 **Triage Notes for the Maintainers**
 <!-- 🎉🎉🎉 Thank you for an opening an issue !!! 🎉🎉🎉
We are doing our best to get to this. Please help us by helping us prioritize your issue by filling the section below -->


 | **Description** | **Yes/No** |
 |----------------|---------------|
 | Please check if this a new feature you are proposing        | <ul><li>- [ ] </li></ul>|
 | Please check if the build works in docker but not in kaniko | <ul><li>- [Yes] </li></ul>|
 | Please check if this error is seen when you use `--cache` flag | <ul><li>- [ ] </li></ul>|
 | Please check if your dockerfile is a multistage dockerfile | <ul><li>- [ ] </li></ul>|

[crispysipper]

using executor with the --single-snapshot flag was also no help

[crispysipper]

same behavior with a django backend build:

INFO[0000] Retrieving image manifest python:3.11-slim   
INFO[0000] Retrieving image python:3.11-slim from registry index.docker.io 
INFO[0000] Built cross stage deps: map[]                
INFO[0000] Retrieving image manifest python:3.11-slim   
INFO[0000] Returning cached image manifest              
INFO[0000] Executing 0 build triggers                   
INFO[0000] Building stage 'python:3.11-slim' [idx: '0', base-idx: '-1'] 
INFO[0000] Unpacking rootfs as cmd RUN apt-get update && apt-get install -y     build-essential     libpq-dev     && rm -rf /var/lib/apt/lists/* requires it. 
error building image: error building stage: failed to get filesystem from image: mkdir /usr/lib/terminfo: file exists

[crispysipper]

updating as I find things and for your records:

adding this flag to the executor command appears to circumvent the issue:

--ignore-path=/usr/lib/terminfo

still worth noting though, that we don't experience this in previous iterations of public kaniko binaries.

[crispysipper]

also

--ignore-path=/lib/terminfo

[mzihlmann]

Just to put it up-front. Kaniko is not meant to be run in any other image than the kaniko images themselves. The builder is not the just binary, but the entire container constitutes the builder, that's one of the trade-offs that is made to make image building without containerization possible. From the Readme:

• Running kaniko in any Docker image other than the official kaniko image is not
  supported due to implementation details.
  - This includes copying the kaniko executables from the official image into
  another image (e.g. a Jenkins CI agent).
  - In particular, it cannot use chroot or bind-mount because its container must
  not require privilege, so it unpacks directly into its own container root
  and may overwrite anything already there.

The issue you're seeing arises if your custom builder image (or the environment you're running in for that matter) and the image you're trying to build both contain the same file, /usr/lib/terminfo in your case. This is one of the occasions where the magic trick that kaniko is trying to pull off falls apart and the under-the-hood workings become obvious and mundane. Kaniko is not actually building up a container like other tools do, instead it just copies files around as instructed by the Dockerfile and snapshots itself. If however, the files it tries to create pre-exist, then the whole process breaks down. To avoid these kinds of problems whilst providing a shell, kaniko-debug ships with busybox instead of a common base images like ubuntu or debian for example, as busybox can be very well-contained.

I can't see the contents of wolfi-base:latest, but the public.ecr.aws/docker/library/bash doesn't contain /usr/lib/terminfo. I think the root cause is not caused by the new kaniko binary, but the new base image. My suggestion is to follow the suggestions of the readme and not copy kaniko into other images. If other tools are required, I would copy them into the kaniko image, but make sure that they are installed in as few and as awkward places as possible, to reduce the likelihood of them ever clashing with a payload image.

Edit: Ok, I see you're referring to cgr.dev/chainguard/wolfi-base:latest which doesn't contain the file either. hmmmm? On a sidenote, I think distributing kaniko via apk is actually a bad idea UX-wise, as it encourages creating custom kaniko images, which will then fail to build alpine images.

[crispysipper]

I mean since the project has been abandoned by Google, Chainguard has taken up the cause and here we are. I am tracking that Kaniko isn't "officially" supported when added to custom images by lifting and shifting the binaries or in any other manner other than leveraging the "official" images.

"Kaniko is not actually building up a container like other tools do." Indeed.

But yes, I am utilizing Chainguard's APK Kaniko offering, thus why I am submitting the issue as recommended by Chainguard support themselves. They are picking up the pieces of a poorly maintained and abandoned project, but one that certainly still has a great deal of use case in this industry.

[thejan2009]

I've had some success with inverting the builder building approach - adding statically linked executables into kaniko/executor:debug image.