Flower SuperLink in graminized container

[YKDZ]

Hi everyone!
In short, I’m trying to run flower-superlink and serverapp (which is launched as a subprocess by flower-superlink) inside a single image built with gsc. However, when starting federated learning, I keep running into this error:

superlink  | [P1:T1:python] error: failed sending PAL handles as part of checkpoint: Invalid argument (EINVAL)
superlink  | [P1:T1:python] error: process creation failed
superlink  | error: libos_init() failed in receive_checkpoint_and_restore: Permission denied (EACCES)
superlink  | WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
superlink  | I0000 00:00:1763912589.633508       1 fork_posix.cc:71] Other threads are currently calling into gRPC, skipping fork() handlers

Does this indicate that some libraries might be using system calls that are not supported by Gramine?

Here is my manifest template:

fs.mounts = [
  { path = "/tmp", type = "tmpfs" },
  { path = "/opt/flower/apps", type = "tmpfs" },
  { path = "/opt/flower/storage", type = "tmpfs" },
]

loader.env.SSL_CERTFILE = { passthrough = true }
loader.env.SSL_KEYFILE = { passthrough = true }
loader.env.SSL_CA_CERTFILE = { passthrough = true }
loader.env.FLOWER_TLS_MODE = { passthrough = true }
loader.env.SERVERAPPIO_API_ADDRESS = { passthrough = true }
loader.env.FLEET_API_ADDRESS = { passthrough = true }
loader.env.CONTROL_API_ADDRESS = { passthrough = true }
loader.env.FLOWER_DATABASE_URL = { passthrough = true }
loader.env.FLOWER_STORAGE_DIR = { passthrough = true }
loader.env.FLOWER_SUPERLINK_EXTRA_ARGS = { passthrough = true }

sys.enable_sigterm_injection = true
sys.insecure__allow_eventfd = true
sgx.enclave_size = "1024M"
sgx.max_threads = 128
sgx.remote_attestation = "dcap"

and my Dockerfile:

ARG SUPERLINK_IMAGE=flwr/superlink:latest

FROM ${SUPERLINK_IMAGE} AS superlink

USER root

ENV FLWR_HOME=/opt/flower \
    PATH="/opt/flower/bin:/python/venv/bin:$PATH" \
    UV_SYSTEM_PYTHON=1 \
    PYTHONDONTWRITEBYTECODE=1

COPY --from=ghcr.io/astral-sh/uv:latest /uv /bin/uv
COPY --from=ghcr.io/astral-sh/uv:latest /uvx /bin/uvx

WORKDIR /opt/flower/apps
COPY pyproject.toml ./
COPY flower_tutorial ./flower_tutorial

RUN set -eux; \
    sed -i '/flwr\[simulation\]/d' pyproject.toml; \
    sed -i '/"docker>=/d' pyproject.toml; \
    sed -i '/"jinja2"/d' pyproject.toml; \
    sed -i '/"tomli"/d' pyproject.toml; \
    sed -i '/"tomli-w"/d' pyproject.toml; \
    sed -i '/"pyyaml"/d' pyproject.toml; \
    HTTPS_PROXY=http://192.168.2.23:20171 \
    HTTP_PROXY=http://192.168.2.23:20171 \
    uv pip install --system --no-cache-dir .; \
    rm -rf /root/.cache/uv;

# Patch shutil
RUN find /python/venv -name install.py -path "*/flwr/cli/*" -exec sed -i \
    -e 's/shutil.copy2/shutil.copyfile/g' \
    -e 's/dirs_exist_ok=True)/dirs_exist_ok=True, copy_function=shutil.copyfile)/g' \
    -e '/import shutil/a shutil.copystat = lambda *args, **kwargs: None' \
    {} +

COPY certificates/ /etc/flower/certs/
RUN chmod -R 644 /etc/flower/certs/ && \
    chmod 755 /etc/flower/certs/

COPY docker/start_superlink.sh /opt/flower/bin/start-superlink.sh
RUN chmod +x /opt/flower/bin/start-superlink.sh

EXPOSE 9091 9092 9093

ENV SSL_CERTFILE=/etc/flower/certs/server.crt \
    SSL_KEYFILE=/etc/flower/certs/server.key \
    SSL_CA_CERTFILE=/etc/flower/certs/ca.crt \
    FLOWER_TLS_MODE=ssl \
    SERVERAPPIO_API_ADDRESS=0.0.0.0:9091 \
    FLEET_API_ADDRESS=0.0.0.0:9092 \
    CONTROL_API_ADDRESS=0.0.0.0:9093

ENTRYPOINT ["/opt/flower/bin/start-superlink.sh"]

and my entrypoint script:

#!/usr/bin/env bash
set -euo pipefail

CERT_PATH=${SSL_CERTFILE:-/run/ratls/server.crt}
KEY_PATH=${SSL_KEYFILE:-/run/ratls/server.key}
CA_PATH=${SSL_CA_CERTFILE:-}
TLS_MODE=${FLOWER_TLS_MODE:-ratls}
SERVERAPPIO=${SERVERAPPIO_API_ADDRESS:-0.0.0.0:9091}
FLEET_API_ADDRESS=${FLEET_API_ADDRESS:-0.0.0.0:9092}
CONTROL_API_ADDRESS=${CONTROL_API_ADDRESS:-0.0.0.0:9093}
STORAGE_DIR=${FLOWER_STORAGE_DIR:-/opt/flower/storage}
DATABASE_URL=${FLOWER_DATABASE_URL:-:flwr-in-memory:}

mkdir -p "$(dirname "${CERT_PATH}")" "$(dirname "${KEY_PATH}")" "${STORAGE_DIR}"

echo "DEBUG: TLS_MODE=${TLS_MODE}"
echo "DEBUG: CERT_PATH=${CERT_PATH}"
echo "DEBUG: KEY_PATH=${KEY_PATH}"
echo "DEBUG: CA_PATH=${CA_PATH}"

if [[ -f "${CERT_PATH}" ]]; then
    echo "DEBUG: CERT_PATH exists"
else
    echo "DEBUG: CERT_PATH does not exist"
fi

if [[ -f "${KEY_PATH}" ]]; then
    echo "DEBUG: KEY_PATH exists"
else
    echo "DEBUG: KEY_PATH does not exist"
fi

if [[ -n "${CA_PATH}" ]]; then
    if [[ -f "${CA_PATH}" ]]; then
        echo "DEBUG: CA_PATH exists"
    else
        echo "DEBUG: CA_PATH does not exist"
    fi
fi

if [[ "${TLS_MODE}" == "insecure" ]]; then
  TLS_ARGS=(--insecure)
else
  if [[ -z "${CA_PATH}" && "${TLS_MODE}" == "ratls" ]]; then
    CA_PATH="${CERT_PATH}"
  fi

  TLS_ARGS=(--ssl-certfile "${CERT_PATH}" --ssl-keyfile "${KEY_PATH}")
  if [[ -n "${CA_PATH}" ]]; then
    TLS_ARGS+=(--ssl-ca-certfile "${CA_PATH}")
  fi
fi

EXTRA_ARGS=()
if [[ -n "${FLOWER_SUPERLINK_EXTRA_ARGS:-}" ]]; then
  # shellcheck disable=SC2206
  EXTRA_ARGS=(${FLOWER_SUPERLINK_EXTRA_ARGS})
fi

exec /python/venv/bin/flower-superlink \
  "${TLS_ARGS[@]}" \
  --serverappio-api-address "${SERVERAPPIO}" \
  --fleet-api-address "${FLEET_API_ADDRESS}" \
  --control-api-address "${CONTROL_API_ADDRESS}" \
  --storage-dir "${STORAGE_DIR}" \
  --database "${DATABASE_URL}" \
  "${EXTRA_ARGS[@]}" \
  "$@"

I’d like to know how to fix this issue. Do you have any suggestions? Or is there a better approach to run both SuperLink and ServerApp inside an Intel SGX TEE?

[monavij]

Fork is very expensive and slow in Gramine and looks like its just not two processes but you are forking several child processes in your solution. You maybe running out of EPC and hence fork failing. How much EPC do you have on your system? See if you can reduce the EPC size and/or number of threads. You can try EDMM that may help as it will not hog EPC at start time. Otherwise think through which processes do really need to run inside SGX and not try to run the whole system inside SGX.

[YKDZ]

Hello, thank you for your response. To get Flower federated learning working, I followed your suggestion and separated the serverapp (superexec) process from superlink, placing superlink outside the Gramine TEE. This indeed helped me move forward, but I’ve started encountering recurring process-creation errors such as:

[P7:T45:python] error: process creation failed

Does this, as you mentioned, indicate that I may be running out of EPC memory? (My system has 8 GB of EPC.)

Do you have any recommendations regarding this? For example, would increasing my EPC size likely resolve the issue? (Changing the Flower superexec system to avoid forking child processes would require extensive modifications, which I’d prefer to avoid.)

Additionally, if I only care about getting the project running for now and not about security, is there any configuration compromise that could help bypass this limitation?

Edit: I also tried enabling EDMM using the method you suggested, but that caused Gramine to fail to run the startup script — it just stayed stuck during the startup process for more than twenty minutes. I believe this is a separate issue.

[monavij]

Few things I can suggest are 1) Try increasing the EPC size on your system, if available. 2) Try reducing your enclave size to be as small as possible and also reduce the number of threads required to be minimal. 3) Try running your workload under Gramine-direct to identify any non-SGX issues. Also can you share how many processes are your forking? I am not sure why EDMM is failing.