Skip to content

Commit 284770b

Browse files
r-casulaPHEMS bot
andauthored
Develop (#37)
* Version bump to 0.6.0 * Idp image (#22) * idp-image: Moved the idp initializer logic to its own docker image, to avoid installing at runtime * idp-image: Added dedicated image for the idp initializer. Added imagepullpolicy * idp-image: Hadolint fixes for pip-related errors * Azure bugfixes (#23) * azure-bugfixes: Incresed ttl removal time for jobs to 30s. Added service account name to jobs. Added extra check for aks clusters, as sometimes jobs get deleted but the pod is still there. * azure-bugfixes: Version bump. Changelog update * azure-bugfixes: Big refactor on crd representation. Added env, inputs, outputs as crd fields to give more flexibility. Removed dataset as a required field. * azure-bugfixes: Try catch to autorestart the watcher * azure-bugfixes: Moved bunch of logic for github push to the controller helper docker image * azure-bugfixes: Changelog update. Hadolint fixes * azure-bugfixes: Fixed missing mocks on file open for ci tests * azure-bugfixes: removed the docker copy of scripts * azure-bugfixes: Added a while true in the main * azure-bugfixes: Adjusted few paths mounted * azure-bugfixes: changed the delete policy on idp-initializer * azure-bugfixes: Addressed an issue with the controller creating too many jobs at the moment of updating the annotations * azure-bugfixes: Renamed user sync script to something more intuitive * azure-bugfixes: small refactor on create_retry_job * develop: Added summary to pipeline * develop: Removed vuln scan from pipeline, we will rely on github * develop: Moved an init job as post install as the delete policy was not working * develop: Added post-upgrade * develop: Added a timeout in case a pod doesn't exist, leaving the controller hanging * develop: Updated tests and changelog * develop: Updated references for alpine to be more dynamic as we are not using 3.19 as only tag * develop: Bumped version * develop: Updated CHANGELOG.md * Version bump to 0.7.2 * not-alpine: Moved away from python alpine, added support for db-virtu… (#26) * not-alpine: Moved away from python alpine, added support for db-virtualization * not-alpine: hadolint fixes * not-alpine: renamed task body field * not-alpine: renamed fields in the crd class and schema * not-alpine: Added Changelogs * HEAD: develop: Added before-hook-creation for init job. Copy secret is now conditional on keycloak info, as when used as subchart, the main chart will take care of that * develop: version bump * develop: Updated changelog * branch-name: Changed the branch name where the results will be pushed to (#29) * 0.7.3 (#28) * Version bump to 0.6.0 * Idp image (#22) * idp-image: Moved the idp initializer logic to its own docker image, to avoid installing at runtime * idp-image: Added dedicated image for the idp initializer. Added imagepullpolicy * idp-image: Hadolint fixes for pip-related errors * Azure bugfixes (#23) * azure-bugfixes: Incresed ttl removal time for jobs to 30s. Added service account name to jobs. Added extra check for aks clusters, as sometimes jobs get deleted but the pod is still there. * azure-bugfixes: Version bump. Changelog update * azure-bugfixes: Big refactor on crd representation. Added env, inputs, outputs as crd fields to give more flexibility. Removed dataset as a required field. * azure-bugfixes: Try catch to autorestart the watcher * azure-bugfixes: Moved bunch of logic for github push to the controller helper docker image * azure-bugfixes: Changelog update. Hadolint fixes * azure-bugfixes: Fixed missing mocks on file open for ci tests * azure-bugfixes: removed the docker copy of scripts * azure-bugfixes: Added a while true in the main * azure-bugfixes: Adjusted few paths mounted * azure-bugfixes: changed the delete policy on idp-initializer * azure-bugfixes: Addressed an issue with the controller creating too many jobs at the moment of updating the annotations * azure-bugfixes: Renamed user sync script to something more intuitive * azure-bugfixes: small refactor on create_retry_job * develop: Added summary to pipeline * develop: Removed vuln scan from pipeline, we will rely on github * develop: Moved an init job as post install as the delete policy was not working * develop: Added post-upgrade * develop: Added a timeout in case a pod doesn't exist, leaving the controller hanging * develop: Updated tests and changelog * develop: Updated references for alpine to be more dynamic as we are not using 3.19 as only tag * develop: Bumped version * develop: Updated CHANGELOG.md * Version bump to 0.7.2 * not-alpine: Moved away from python alpine, added support for db-virtu… (#26) * not-alpine: Moved away from python alpine, added support for db-virtualization * not-alpine: hadolint fixes * not-alpine: renamed task body field * not-alpine: renamed fields in the crd class and schema * not-alpine: Added Changelogs * HEAD: develop: Added before-hook-creation for init job. Copy secret is now conditional on keycloak info, as when used as subchart, the main chart will take care of that * develop: version bump * develop: Updated changelog --------- Co-authored-by: PHEMS bot <[email protected]> * main: negated the kc-secrets condition * branch-name: Changed the branch name where the results will be pushed to * branch-name: version bump * branch-name: added user info to the other create_helper_job call * branch-name: Updated changelog --------- Co-authored-by: PHEMS bot <[email protected]> * develop: v1.0.0 * Version bump to 1.1.0 * 144-aws-storage: Added support for AWS EFS (#34) * 144-aws-storage: Added support for AWS EFS * 144-aws-storage: Added default storageclass definition * 144-aws-storage: Added dynamic pv and pvc naming so capacity can be changed at upgrade time * 144-aws-storage: Added default value for storage capacity on template * 144-aws-storage: wrong condition on copy-secrets * 144-aws-storage: added post-install hooks on copy template * 144-aws-storage: Reverted Changes * develop: Vulnerabilities fix * develop: makefile change * 34 base image upgrade (#33) * 34-base-image-upgrade: Migrated to python 3.13 * 34-base-image-upgrade: Fixed the path for the python venv * 34-base-image-upgrade: hadolint fixes on test.dockerfile * 34-base-image-upgrade: Moved to uv from pip on the helper image * 34-base-image-upgrade: more hadolint fixes * 34-base-image-upgrade: updated uv.lock * Version bump to 1.2.0 * Review with third (#18) * chart-integration: Added cases where common alues are picked up from global rather than root values * 81-third-party: Added new way to push results. Restructured the CRD to have a clearer separation and initial validation at cluster level. * 81-third-party: Added more unittests * 81-third-party: hadolint fixes * 81-third-party: Missing key * 81-third-party: Assumed the result credentials are in a secret already. Adjusted tests. Pylint fixes * 81-third-party: removed unnecessary verify. There is a custom wrapper for it * 81-third-party: Refactored some mocking to be more narrowed down * 81-third-party: Added a field for tasks to indicate we are triggering it from the fntc * 81-third-party: Added pre-install hook as the idp initializer needs it when installed as subchart * 81-third-party: Added hooks to configmap * 81-third-party: added hook to secret copy * 81-third-party: Added weight to cm * 81-third-party: Moved away from pre-hooks, using weights. pre is causing a lot of sync errors as a subchart * 81-third-party: Added a global value to check if the chart is used as subchart, so that we won't override regcred contents * 81-third-party: Renamed regcreds file * 81-third-party: Moved the check in the main if * 81-third-party: Added generalization for namespaces * 81-third-party: Adjusted global templates * 81-third-party: wrong if * chart-integration: Removed outdated makefile option. Bumped version * review-with-third: Added check for review. To make optional * review-with-third: Added feature flag * 81-third-party: removed organization field from the CRD as is not used anymore. * 81-third-party: Changelog * review-with-third: Fixed tests * review-with-third: pylint fix for unused imports * review-with-third: Changed the result extension to zip * review-with-third: Fixed python path bin --------- Co-authored-by: Riccardo Casula <[email protected]> --------- Co-authored-by: PHEMS bot <[email protected]> Co-authored-by: Riccardo Casula <[email protected]>
1 parent 8fb3157 commit 284770b

File tree

17 files changed

+869
-1063
lines changed

17 files changed

+869
-1063
lines changed

CHANGELOG.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,7 @@
11
# Releases Changelog
22

3+
# 1.2.0
4+
- Added a dynamic configuration to `global.taskReview` in the values files. This is only effective when deployed with the federated node. It will hold the result release until approved from the api, which in turns should set the task's CRD annotation `approved` to `"true"` and allowing result delivery. Defaults to `false`.
35
# 1.1.0
46
- Added support for AWS EFS persistent volume through the csi driver `efs.csi.aws.com`
57
To configure it, set in the values file:

Dockerfile

Lines changed: 8 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,28 +1,25 @@
1-
FROM python:3.12.2-slim
1+
FROM python:3.13.5-slim
22

33
COPY controller /app/controller
4-
COPY Pipfile* /app
4+
COPY pyproject.toml /app
55

66
WORKDIR /app
77

88
ENV PYTHONUNBUFFERED=1
99
ENV PYTHONIOENCODING=UTF-8
1010

11-
# hadolint detects pipenv as another invocation of pip
12-
# hadolint ignore=DL3013,DL3008
11+
SHELL ["/bin/bash", "-o", "pipefail", "-c"]
12+
# hadolint ignore=DL3008
1313
RUN apt-get update \
1414
&& apt-get install --no-install-recommends -y \
1515
gcc gh jq curl openssl tar \
16-
&& pip install --no-cache-dir --upgrade pip \
17-
&& python3 -m pip install --no-cache-dir pipenv \
18-
&& pipenv lock \
19-
&& pipenv install --system --deploy --categories packages \
20-
&& pip uninstall -y pipenv \
16+
&& curl -LsSf https://astral.sh/uv/install.sh | sh \
17+
&& /root/.local/bin/uv sync \
2118
&& apt-get clean \
2219
&& rm -rf /var/lib/apt/lists/*
2320

2421
# hadolint ignore=DL3013,DL3008
25-
RUN curl -sSL -O https://packages.microsoft.com/config/debian/12/packages-microsoft-prod.deb \
22+
RUN curl -sSL -O https://packages.microsoft.com/config/debian/"$(grep VERSION_ID /etc/os-release | cut -d '"' -f 2)"/packages-microsoft-prod.deb \
2623
&& dpkg -i packages-microsoft-prod.deb \
2724
&& apt-get update \
2825
&& apt-get install --no-install-recommends -y azcopy \
@@ -33,4 +30,5 @@ RUN curl -sSL -O https://packages.microsoft.com/config/debian/12/packages-micros
3330
&& rm -rf /var/lib/apt/lists/*
3431
3532
ENV PYTHONPATH=/app/controller
33+
ENV PATH="/app/.venv/bin:$PATH"
3634
CMD ["python3", "-m", "controller"]

Makefile

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
SHELL=/bin/bash
2-
IMAGE ?= ghcr.io/aridhia-open-source/fn_task_controller:0.7.0
2+
IMAGE ?= ghcr.io/aridhia-open-source/fn_task_controller:1.2.0
33
TESTS_IMAGE ?= ghcr.io/aridhia-open-source/fn_task_controller_tests
44
TEST_CONTAINER ?= fn-controller-tests
55

@@ -9,6 +9,9 @@ build_docker:
99
build_test_container:
1010
docker build . -f test.Dockerfile -t ${TESTS_IMAGE}
1111

12+
build_helper:
13+
docker build build/helper -t ghcr.io/aridhia-open-source/fn_task_controller_helper:1.0.0
14+
1215
run_test_container: cleanup_test_container
1316
docker run --name ${TEST_CONTAINER} ${TESTS_IMAGE}
1417
docker cp ${TEST_CONTAINER}:/app/artifacts/coverage.xml artifacts/coverage.xml

Pipfile

Lines changed: 0 additions & 23 deletions
This file was deleted.

Pipfile.lock

Lines changed: 0 additions & 1002 deletions
This file was deleted.

build/helper/Dockerfile

Lines changed: 7 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
FROM alpine/k8s:1.29.4
1+
FROM alpine/k8s:1.33.2
22

33
ARG USERNAME=fednode
44
ARG USER_UID=1001
@@ -9,18 +9,15 @@ RUN addgroup -S "${USERNAME}" --gid "${USER_GID}" \
99

1010
WORKDIR /apps
1111
# Can't find venv-related files
12+
SHELL ["/bin/ash", "-eo", "pipefail", "-c"]
1213
# hadolint ignore=SC1091
13-
RUN apk add --no-cache \
14-
'jq=~1.7' \
15-
'curl=~8.5' \
16-
'openssl=~3.1' \
17-
'github-cli=~2.39' \
18-
&& python3 -m venv venv \
19-
&& . venv/bin/activate \
20-
&& pip install --no-cache-dir "requests==2.32.3" \
14+
RUN curl -LsSf https://astral.sh/uv/install.sh | sh \
15+
&& /root/.local/bin/uv init \
16+
&& /root/.local/bin/uv add requests \
2117
&& chown "${USER_UID}:${USER_GID}" /apps
2218

2319
COPY . /apps/
2420
USER ${USER_UID}
2521

26-
ENTRYPOINT [ "/apps/venv/bin/python3", "idp-init.py" ]
22+
ENV PATH="/apps/.venv/bin:$PATH"
23+
ENTRYPOINT [ "python3", "idp-init.py" ]

controller/__main__.py

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
"""
22
Entrypoint for the FNTC
33
"""
4+
45
from .controller import start
56

67
print("Starting the controller")

controller/helpers/pod_watcher.py

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,6 @@
66

77
import base64
88
import logging
9-
import json
109
import re
1110
import subprocess
1211
from kubernetes.watch import Watch

controller/models/crd.py

Lines changed: 17 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
import json
22
from math import exp
3+
import os
34
import re
45

56
from exceptions import CRDException
@@ -45,7 +46,22 @@ def can_trigger_task(self) -> bool:
4546
return self.annotations.get(f"{self.domain}/user") and not self.annotations.get(f"{self.domain}/done")
4647

4748
def can_deliver_results(self) -> bool:
48-
return self.annotations.get(f"{self.domain}/done") and not self.annotations.get(f"{self.domain}/results")
49+
"""
50+
Overcomplicated flow control, but there are few requirements to
51+
fetch results:
52+
- done HAS to be there, which means task pod is done
53+
- results HAS NOT to be there, meaning results have not been fetched and delivered yet
54+
55+
TASK_REVIEW and approved annotation should make the whole check fail when:
56+
TASK_REVIEW is set and approved is not "true". So we check for this
57+
case, and negate it.
58+
"""
59+
return self.annotations.get(f"{self.domain}/done") and \
60+
not self.annotations.get(f"{self.domain}/results") and \
61+
not (
62+
os.getenv("TASK_REVIEW") is not None and \
63+
self.annotations.get(f"{self.domain}/approved", "false").lower() != "true"
64+
)
4965

5066
def should_skip(self) -> bool:
5167
return bool(self.is_delete or self.annotations.get(f"{self.domain}/results"))

controller/tests/conftest.py

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -142,6 +142,30 @@ def mock_crd_api_basic_done(mock_crd_task_done):
142142
}}
143143
return deepcopy(mock_crd_task_done)
144144

145+
@pytest.fixture
146+
def mock_crd_azcopy_done(mock_crd_task_done):
147+
mock_crd_task_done["object"]["spec"]["results"] = {"other": {
148+
"url": "https://fancyresultsplace.com/api/storage",
149+
"auth_type": "AzCopy"
150+
}}
151+
return deepcopy(mock_crd_task_done)
152+
153+
@pytest.fixture
154+
def mock_crd_api_done(mock_crd_task_done):
155+
mock_crd_task_done["object"]["spec"]["results"] = {"other": {
156+
"url": "https://fancyresultsplace.com/api/storage",
157+
"auth_type": "Bearer"
158+
}}
159+
return deepcopy(mock_crd_task_done)
160+
161+
@pytest.fixture
162+
def mock_crd_api_basic_done(mock_crd_task_done):
163+
mock_crd_task_done["object"]["spec"]["results"] = {"other": {
164+
"url": "https://fancyresultsplace.com/api/storage",
165+
"auth_type": "Basic"
166+
}}
167+
return deepcopy(mock_crd_task_done)
168+
145169
@pytest.fixture(autouse=True)
146170
def k8s_config(mocker):
147171
mocker.patch('kubernetes.config.load_kube_config', return_value=Mock())
@@ -286,3 +310,7 @@ def delivery_open(request, mocker):
286310
if getattr(request, "param", None):
287311
file_contents = request.param
288312
return mocker.patch("models.crd.open", mock_open(read_data=json.dumps(file_contents)))
313+
314+
@pytest.fixture
315+
def review_env(monkeypatch):
316+
monkeypatch.setenv("TASK_REVIEW", "enabled")

0 commit comments

Comments
 (0)