Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch to Kubernetes + Kueue service backend #723

Open
wants to merge 219 commits into
base: main
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
219 commits
Select commit Hold shift + click to select a range
d48ee86
OPS: Increase minor version number
cortadocodes Jan 9, 2025
a74f752
DEP: Temporarily remove `numpy` and `pandas` dev dependencies
cortadocodes Jan 9, 2025
d40192b
DEP: Drop support for python 3.8 and 3.9
cortadocodes Jan 9, 2025
7140f76
DEP: Add `numpy` and `pandas` back to dev dependencies
cortadocodes Jan 9, 2025
52eddd0
DEP: Update lock file
cortadocodes Jan 9, 2025
a4c1450
TST: Update test for python3.13
cortadocodes Jan 9, 2025
cbe45ad
FEA: Add `octue question ask` CLI command
cortadocodes Jan 13, 2025
96b145e
ENH: Deserialise `octue question ask` inputs from JSON
cortadocodes Jan 13, 2025
665fb41
FEA: Add ability to ask remote question from CLI
cortadocodes Jan 13, 2025
eb0066a
FIX: Fix incorrect type
cortadocodes Jan 13, 2025
aa2fd4b
REF: Split `octue question ask` into `ask` and `ask-local` commands
cortadocodes Jan 13, 2025
9693950
ENH: Add `async` option to `octue question ask`
cortadocodes Jan 13, 2025
37e20c1
ENH: Deprecate `octue run` command
cortadocodes Jan 13, 2025
823d6f5
FIX: Return question UUID for async `octue question ask`
cortadocodes Jan 13, 2025
8a4bfb6
REF: Rename `answer_question` to `answer_pub_sub_question`
cortadocodes Jan 13, 2025
3d4c525
FIX: Fix event and attributes generation
cortadocodes Jan 13, 2025
bd07a3d
FIX: Add missing kwarg
cortadocodes Jan 13, 2025
f5e21f3
ENH: Support parsing already-extracted questions
cortadocodes Jan 13, 2025
40c3dfa
ENH: Remove unnecessary log message
cortadocodes Jan 13, 2025
ef684b7
FEA: Add `octue question events raw` command
cortadocodes Jan 13, 2025
386f508
FEA: Add `octue question events replay` CLI command
cortadocodes Jan 13, 2025
1fce3ba
REF: Move `ask*` subcommands under `octue question ask` group
cortadocodes Jan 13, 2025
d3faaf7
FIX: Fix argument name and backend format in CLI command
cortadocodes Jan 13, 2025
f1c6dd9
FIX: Fix `--asynchronous` option in command
cortadocodes Jan 13, 2025
390be58
FIX: Fix setting attributes for question event
cortadocodes Jan 13, 2025
ecc4ef9
ENH: Improve description of commands
cortadocodes Jan 13, 2025
69cada1
REF: Rename `octue question events raw` to `octue question events get`
cortadocodes Jan 13, 2025
bbcf14b
ENH: Suppress google crc32c warning
cortadocodes Jan 13, 2025
4c4e559
FIX: Return data from `octue question ask remote` properly
cortadocodes Jan 13, 2025
da628a5
FIX: Await publishing of `result` event
cortadocodes Jan 13, 2025
f491210
FIX: Return early for async question in `octue question ask remote`
cortadocodes Jan 14, 2025
444093f
ENH: Improve description of `octue question ask local`
cortadocodes Jan 14, 2025
e5fc542
STY: Add missing ruff isort pre-commit check
cortadocodes Jan 23, 2025
50a62e0
OPS: Correct version number
cortadocodes Jan 23, 2025
0d7c8c2
ENH: Move `get-diagnostics` command to `octue question diagnostics`
cortadocodes Jan 23, 2025
41996c2
ENH: Allow passing of attributes to `octue question ask local`
cortadocodes Jan 27, 2025
b50b207
TST: Update mock paths
cortadocodes Jan 27, 2025
fa86c28
FIX: Avoid assuming question is a dict
cortadocodes Jan 27, 2025
cb867c4
ENH: Allow setting of Octue services topic name via envvar
cortadocodes Jan 27, 2025
23fa101
TST: Fix test for different python versions
cortadocodes Jan 27, 2025
6d157aa
FIX: Fix typo in `OCTUE_SERVICES_TOPIC_NAME` envvar
cortadocodes Jan 27, 2025
4cc32b2
FIX: Remove non-existent service revision check for now
cortadocodes Jan 27, 2025
dd47a92
FIX: Add missing `elif`
cortadocodes Jan 27, 2025
680ec63
FIX: Convert attributes to correct types in `make_attributes`
cortadocodes Jan 28, 2025
f0bbd98
FIX: Extract attributes correctly from passed-in format
cortadocodes Jan 28, 2025
a94fb35
ENH: Log when question event extraction is complete
cortadocodes Jan 28, 2025
0bd2f68
FIX: Explicitly pass credentials to publisher/subscriber clients
cortadocodes Feb 3, 2025
89c5362
FIX: Refresh credentials before returning
cortadocodes Feb 3, 2025
9081704
REV: Revert "FIX: Refresh credentials before returning"
cortadocodes Feb 3, 2025
584d398
REV: Revert "FIX: Explicitly pass credentials to publisher/subscriber…
cortadocodes Feb 4, 2025
521666b
ENH: Log question acknowledgement to pub/sub
cortadocodes Feb 4, 2025
2ea1cc3
ENH: Allow use of service registries with `octue question ask remote`
cortadocodes Feb 4, 2025
f6b9c94
FIX: Make service config optional for `octue question ask remote`
cortadocodes Feb 4, 2025
28c72e8
FEA: Allow requesting compute resource amounts per question
cortadocodes Feb 5, 2025
2ad4d00
REF: Remove Cloud Run flask app
cortadocodes Feb 5, 2025
6b83793
REF: Move dockerfiles into their own directory
cortadocodes Feb 5, 2025
a3afdda
REF: Move `answer_pub_sub_question` module and rename
cortadocodes Feb 5, 2025
a909e23
TST: Move cloud run deployment test
cortadocodes Feb 5, 2025
9dd7ea5
REF: Remove push subscription creation logic
cortadocodes Feb 5, 2025
c8ba8bd
ENH: Remove cloud run runtime warning
cortadocodes Feb 5, 2025
9076d5b
ENH: Add `GOOGLE_KUEUE` to compute providers and remove old ones
cortadocodes Feb 5, 2025
e5769d7
REF: Remove unused global variable
cortadocodes Feb 5, 2025
b5b50e6
ENH: Remove option to parse events from Cloud Run
cortadocodes Feb 5, 2025
1c09aa7
REF: Remove other mentions of cloud run
cortadocodes Feb 5, 2025
3626b6a
OPS: Remove terraform config
cortadocodes Feb 5, 2025
8a17a14
REF: Remove unused dockerfile and workflow
cortadocodes Feb 5, 2025
e9fd3d0
REF: Split pub/sub event/attributes extraction into two functions
cortadocodes Feb 5, 2025
1f8ee61
REF: Move attributes extraction into `events` subpackage
cortadocodes Feb 5, 2025
c564ae3
CHO: Deprecate `octue get-diagnostics` CLI command
cortadocodes Feb 5, 2025
31fc7bb
ENH: Add log message explaining waiting for question acceptance
cortadocodes Feb 5, 2025
9eca502
FIX: Only add resource request attributes if non-None
cortadocodes Feb 5, 2025
a4e7e0e
REF: Rename function
cortadocodes Feb 5, 2025
f4c1c2b
FEA: Add `cancel` method to `Service` and `Child`
cortadocodes Feb 5, 2025
a514d4c
WIP: Update event schema locally
cortadocodes Feb 5, 2025
cf91180
ENH: Simplify question cancellation
cortadocodes Feb 6, 2025
ac1ba02
FEA: Add `octue question cancel` CLI command
cortadocodes Feb 6, 2025
a108f0e
ENH: Avoid trying to cancel question if it's already finished
cortadocodes Feb 6, 2025
0912281
FIX: Avoid passing invalid ID to child in `octue question cancel`
cortadocodes Feb 6, 2025
2c330fa
FIX: Add missing key
cortadocodes Feb 6, 2025
7b96f43
FIX: Remove restriction on attributes for `PARENT` sender type
cortadocodes Feb 6, 2025
2d3647a
FIX: Fix log message
cortadocodes Feb 6, 2025
cdc0c0c
FIX: Fix string interpolations in log messages
cortadocodes Feb 6, 2025
9a86bce
FIX: Wait for event emission future by default
cortadocodes Feb 6, 2025
74f11d8
REF: Factor out attributes into `EventAttributes` class
cortadocodes Feb 11, 2025
0a49780
REF: Factor out getting local `octue` version
cortadocodes Feb 11, 2025
a42c781
FIX: Fix incorrect path change in dockerfiles
cortadocodes Feb 11, 2025
27c712d
REF: Move CRC32C warning catch below imports
cortadocodes Feb 11, 2025
02c0e8e
TST: Fix test
cortadocodes Feb 11, 2025
c3bafa4
DOC: Document resource parameters in `ask` methods
cortadocodes Feb 11, 2025
38b38c4
DOC: Document `cancel` methods
cortadocodes Feb 11, 2025
973c11c
ENH: Log warning instead of raising if question already finished
cortadocodes Feb 11, 2025
da00d8c
DOC: Fix docstrings
cortadocodes Feb 11, 2025
97e9bfa
REF: Rename function
cortadocodes Feb 11, 2025
29ac6fb
DOC: Update docstrings in `answer_pub_sub_question`
cortadocodes Feb 11, 2025
8bfee3f
REF: Rename `answer_pub_sub_question` to `answer_question`
cortadocodes Feb 11, 2025
5578fdf
ENH: Remove outdated `octue run` CLI command
cortadocodes Feb 11, 2025
5a7a010
ENH: Return result from `Service.answer` and `octue question ask local`
cortadocodes Feb 11, 2025
6e89bfc
REF: Allow `create_sruid` to create default SRUIDs
cortadocodes Feb 11, 2025
de2b272
REF: Simplify `octue question ask local`
cortadocodes Feb 11, 2025
d3c9444
DOC: Improve documentation of `octue question ask remote`
cortadocodes Feb 11, 2025
3a3160a
REF: Tidy up `octue question events get`
cortadocodes Feb 11, 2025
9baf194
REF: Tidy up `octue question events replay`
cortadocodes Feb 11, 2025
a1e364b
ENH: Avoid getting non-result events from event store if unnecessary
cortadocodes Feb 11, 2025
ea32f31
ENH: Warn if no result is returned by `octue question events replay`
cortadocodes Feb 11, 2025
9013444
DOC: Improve docstring
cortadocodes Feb 11, 2025
8b6c8d7
FIX: Pass service/app config correctly in `octue question ask local`
cortadocodes Feb 12, 2025
ffbcc55
ENH: Make service/app config mandatory in `answer_question`
cortadocodes Feb 12, 2025
d2f4b4f
DOC: Remove outdated docstring param
cortadocodes Feb 12, 2025
0f923be
FIX: Pass CLI JSON output to stdout as valid JSON
cortadocodes Feb 12, 2025
adf2b2e
TST: Test `octue question ask remote` command
cortadocodes Feb 12, 2025
7c1f9a6
REF: Simplify making an originator question
cortadocodes Feb 12, 2025
6019f36
TST: Test `octue question ask local` command
cortadocodes Feb 12, 2025
6c8c659
FIX: Use encoder when dumping to JSON in CLI
cortadocodes Feb 12, 2025
4109684
FIX: Send output manifests to `stdout` in serialised form
cortadocodes Feb 12, 2025
6bccdd9
TST: Test output manifests are passed to `stdout` properly from CLI
cortadocodes Feb 12, 2025
235c295
TST: Factor out repeated fixture
cortadocodes Feb 12, 2025
8ed4728
ENH: Log warning if no events found in `octue question events get`
cortadocodes Feb 12, 2025
d403ed4
TST: Test `octue question events get` CLI command
cortadocodes Feb 12, 2025
2337d09
ENH: Log warning if no events found in `octue question events replay`
cortadocodes Feb 12, 2025
cf5838a
REF: Move no events/result warnings into underlying functions
cortadocodes Feb 12, 2025
4dc099b
ENH: Remove redundant `--only-handle-result` option from `replay` com…
cortadocodes Feb 12, 2025
2d26da3
ENH: Return no result from `replay` command if no events/result
cortadocodes Feb 12, 2025
808d987
TST: Test `octue question events replay` CLI command
cortadocodes Feb 12, 2025
a83a734
TST: Test `octue get-diagnostics` is deprecated
cortadocodes Feb 12, 2025
3bb11f5
ENH: Remove cloud run specific commands in dockerfiles
cortadocodes Feb 12, 2025
4fa4886
ENH: Delete unsupported python3.9 dockerfile
cortadocodes Feb 12, 2025
7c71b58
MRG: Merge pull request #709 from octue/add-kueue-service-backend
cortadocodes Feb 17, 2025
f0b689f
WIP: Temporarily hard-code WIF details into CI workflow
cortadocodes Feb 17, 2025
23a7575
MRG: Merge branch 'rc-kueue' into factor-out-event-attributes
cortadocodes Feb 17, 2025
f72470d
FIX: Avoid deserialising input manifest in `octue question ask local`
cortadocodes Feb 17, 2025
cd02dd1
FIX: Deserialise input manifest from JSON in `octue question ask local`
cortadocodes Feb 17, 2025
1394334
FIX: Remove `GOOGLE_KUEUE` from log timestamp exclusion
cortadocodes Feb 25, 2025
5013eae
CHO: Ignore GCP credentials files
cortadocodes Feb 26, 2025
313bf5c
TST: Replace cloud run deployment test with kueue equivalent
cortadocodes Feb 26, 2025
4ea98a0
OPS: Move `pydocstyle` config inside `pyproject.toml`
cortadocodes Feb 26, 2025
7ec3943
OPS: Use `pydocstyle` within `ruff` in pre-commit checks
cortadocodes Feb 26, 2025
fd6289d
DOC: Update docstring
cortadocodes Feb 26, 2025
305a09c
MRG: Merge branch 'rc-kueue' into factor-out-event-attributes
cortadocodes Feb 26, 2025
16233a4
ENH: Set better defaults in `EventAttributes`
cortadocodes Feb 26, 2025
22b7e98
FIX: Avoid calling `to_dict` method on dictionary
cortadocodes Feb 26, 2025
5a501ae
TST: Fix tests
cortadocodes Feb 26, 2025
32fec93
FIX: Fix question attributes defaults
cortadocodes Feb 26, 2025
59dd51e
REF: Simplify attribute defaults
cortadocodes Feb 26, 2025
cc66592
FIX: Remove question attributes from `make_response_attributes`
cortadocodes Feb 26, 2025
9e6eb79
DOC: Add missing param to docstring
cortadocodes Feb 26, 2025
bc9f198
FIX: Remove default question attributes from `EventAttributes`
cortadocodes Feb 26, 2025
7dd7040
FIX: Only remove attributes if they're present
cortadocodes Feb 26, 2025
31ced10
FIX: Fix `save_diagnostics` default in `make_question_event`
cortadocodes Feb 26, 2025
eda2ec3
DOC: Add `attributes` param to docstrings
cortadocodes Feb 26, 2025
7adfe38
REF: Add default args and document `make_question_event`
cortadocodes Feb 26, 2025
2b57073
REF: Factor out emitting result event
cortadocodes Feb 26, 2025
5d6285f
DOC: Fix docstring
cortadocodes Feb 26, 2025
8188e7c
REF: Expect `datetime` attr as `datetime` object in `EventAttributes`
cortadocodes Feb 27, 2025
bd33a93
REF: Rename `EventAttributes` methods and add docstrings
cortadocodes Feb 27, 2025
b40bfb0
TST: Test `EventAttributes` class
cortadocodes Feb 27, 2025
3cbfa9a
REF: Make sender/recipient swap clearer
cortadocodes Feb 27, 2025
16e4e24
ENH: Remove need to (de)serialise `None` attribute values
cortadocodes Feb 27, 2025
dd3ce1d
REF: Use `EventAttributes` in `MockService`
cortadocodes Feb 27, 2025
571a4b6
DOC: Improve docstring
cortadocodes Feb 27, 2025
45c37ea
REV: Revert "WIP: Temporarily hard-code WIF details into CI workflow"
cortadocodes Feb 27, 2025
a35c2bf
WIP: Comment out question cancellation methods for now
cortadocodes Feb 27, 2025
4df4719
MRG: Merge pull request #721 from octue/factor-out-event-attributes
cortadocodes Feb 27, 2025
1d35b50
TST: Use new project for example service
cortadocodes Feb 27, 2025
3ea2289
TST: Wait 30s longer for deployment test question
cortadocodes Feb 27, 2025
b2b00fd
ENH: Increase max heartbeat interval by 1 minute
cortadocodes Mar 3, 2025
fb3f9a7
TST: Move deployment test to example repo
cortadocodes Mar 3, 2025
2411eda
OPS: Add terraform config for testing
cortadocodes Mar 3, 2025
ac802cc
ENH: Only allow boolean for `forward_logs` attribute in schema
cortadocodes Mar 3, 2025
5fff971
OPS: Use updated terraform module input
cortadocodes Mar 3, 2025
db88914
ENH: Raise error if service registry request fails
cortadocodes Mar 4, 2025
5b5f756
FIX: Only declare service revision found if response code is 200
cortadocodes Mar 4, 2025
6e6432c
TST: Test response to service registry request failure
cortadocodes Mar 4, 2025
643320b
ENH: Make log message more specific
cortadocodes Mar 4, 2025
6213f0c
ENH: Make requests to service registries authenticated
cortadocodes Mar 4, 2025
9fdb7cc
OPS: Use version `0.1.1` of `terraform-octue-twined-core`
cortadocodes Mar 5, 2025
bdb1ccc
ENH: Disallow additional properties in service comms schema
cortadocodes Mar 5, 2025
022a9c8
CHO: Update version compatibility data
cortadocodes Mar 5, 2025
28aa7b2
DOC: Remove outdated explanation about service compatibility
cortadocodes Mar 5, 2025
e103bec
REF: Remove as-yet-unused `cancellation` event from schema
cortadocodes Mar 5, 2025
f2cbfcb
ENH: Add new resource attributes to schema
cortadocodes Mar 5, 2025
9146a60
ENH: Use version `0.15.0` of service communication schema
cortadocodes Mar 6, 2025
b6969d8
TST: Patch ID token request for service registry requests
cortadocodes Mar 6, 2025
fa97144
FIX: Ensure event UUID and datetime is unique for each event
cortadocodes Mar 6, 2025
d2f9756
DOC: Replace "Octue service" with "Twined service" in docs
cortadocodes Mar 6, 2025
31d805c
FIX: Remove extra `'` from log message
cortadocodes Mar 6, 2025
0b3670f
REF: Simplify allowing service configuration to not exist
cortadocodes Mar 6, 2025
479ff2e
ENH: Set default table ID in `get_events`
cortadocodes Mar 6, 2025
57f64e6
ENH: Remove requirement for service config for `events get` command
cortadocodes Mar 6, 2025
e7ea523
ENH: Remove requirement for service config for `events replay` command
cortadocodes Mar 6, 2025
2330239
DOC: Update "asking questions" documentation
cortadocodes Mar 6, 2025
3bf7e54
DOC: Update "creating services" documentation
cortadocodes Mar 6, 2025
87d91e3
DOC: Update "updating services" documentation
cortadocodes Mar 6, 2025
b20fe07
DOC: Update docstring
cortadocodes Mar 6, 2025
0508e48
DOC: Update "running services locally" documentation
cortadocodes Mar 6, 2025
ad56feb
DOC: Fix "an Twined" typo
cortadocodes Mar 6, 2025
e0c85ce
DOC: Rewrite "deploying services" documentation
cortadocodes Mar 10, 2025
08f2a3a
DOC: Fix workflow name in docs
cortadocodes Mar 10, 2025
4db5555
DOC: Update "troubleshooting services" docs
cortadocodes Mar 10, 2025
9007fde
DOC: Update "authentication" documentation
cortadocodes Mar 10, 2025
01502ba
DOC: Remove boilerplate bibliography
cortadocodes Mar 10, 2025
10ae13c
DOC: Improve docstring
cortadocodes Mar 10, 2025
cdf1b1a
DOC: Fix list in docs
cortadocodes Mar 10, 2025
8561091
DOC: Update example service links
cortadocodes Mar 10, 2025
2f45dea
DOC: Improve "deploying services" documentation
cortadocodes Mar 10, 2025
a32d272
DOC: Split deployment docs into service and network deployment
cortadocodes Mar 13, 2025
26fe592
DOC: Update scope of existing services mentioned in docs
cortadocodes Mar 13, 2025
71b6a07
REF: Move service registry code into `octue.cloud.registry` module
cortadocodes Mar 13, 2025
398129d
REF: Rename `EventAttributes.refresh`
cortadocodes Mar 13, 2025
c802c41
REF: Simplify `EventAttributes.make_opposite_attributes`
cortadocodes Mar 13, 2025
10f0448
ENH: Remove overkill service exception handling
cortadocodes Mar 13, 2025
be41e9d
REF: Split `EventAttributes` into question and response attributes
cortadocodes Mar 13, 2025
9ecc28d
REF: Factor out question attributes deserialisation
cortadocodes Mar 13, 2025
d744a08
DOC: Add missing docstring
cortadocodes Mar 13, 2025
0b2f636
ENH: Remove unnecessary setting of missing attributes before validation
cortadocodes Mar 13, 2025
6317b3a
REF: Move attribute extraction into `EventAttributes` class
cortadocodes Mar 13, 2025
ca09b78
REF: Remove extraneous event validation arguments
cortadocodes Mar 13, 2025
da6faa5
REF: Factor out default maximum heartbeat interval
cortadocodes Mar 13, 2025
a0a6317
REF: Rename `utils` module to `question`
cortadocodes Mar 13, 2025
85b27bd
FIX: Ensure only `google_crc32c` warning is caught
cortadocodes Mar 13, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
FIX: Fix incorrect type
  • Loading branch information
cortadocodes committed Jan 13, 2025
commit eb0066aa31bf9a0c9bdcf1f872b53598f75e0c1c
2 changes: 1 addition & 1 deletion octue/cli.py
Original file line number Diff line number Diff line change
@@ -97,7 +97,7 @@ def question():
@click.option(
"-p",
"--project-name",
type="str",
type=str,
default=None,
help="If asking a remote question, the name of the Google Cloud project the service is deployed in. If not "
"provided, the project name is detected from the local Google application credentials if present.",