Skip to content

feat: add support for task execution through redis queue workers #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 34 commits into from
Jul 6, 2025
Merged

feat: add support for task execution through redis queue workers #2

merged 34 commits into from
Jul 6, 2025

Conversation

danellecline
Copy link
Contributor

@danellecline danellecline commented Jun 26, 2025 •
edited
Loading

This is a significant enhancement that adds:

  • Redis queue workers
  • Saves results to JSON output to a data mount
  • Multiple GPU execution for speed-up
  • Better logging
  • Tests for most endpoints
  • Health check on FastAPI-vss service

@danellecline
refactoring to add rq based execution
0a8fd3a
@danellecline
working rq worker and added precommit
bd497a4
@danellecline
added health check and pass filenames to queue
ab3f428
@danellecline
added environment settings for LOG_PATH, LOG_LEVEL, and GIT_VERSION f…
8d21900 
…or dev
@danellecline
testproject config.yml
d5e9a8a
@danellecline
added database load to compose and some refactoring of test images
bb772b6
@danellecline
added github workflow to api test to run before release and working t…
49aa10b 
…est_main.http
@danellecline
add absolute paths to example.env for api test
d69d247
@danellecline
trying no quotes
a8832c7
@danellecline
added build and relative path to example.env
97540cc
@danellecline
revert to .env with substituted paths for best compatibility
6227526
@danellecline
clean compose.yml
de91d6d
@danellecline
switch redis to non-default port 6380
9018c43
@danellecline
removed white space and not needed clean
b931942
@danellecline
longer sleep for compose stack to build
27567b2
@danellecline
remove API ready
f9ec735
@danellecline
switch to httpie action
23f3965
@danellecline
improved indentation
030c5e8
@danellecline
found port issue and revert to httpyac which is working locally
3505996
@danellecline
revert api check
9a1e4ce
@danellecline
replace server name - action works with act
578fab5
@danellecline
check api
0332201
@danellecline
check api
70cb317
@danellecline
separate dev compose for api-test and added hourly predictor store
f36dcfc
@danellecline
refactor LOG_PATH
2f79844
@danellecline
set userid to runner
8de83b0
@danellecline
hourly and second timestampped output; prediction aligned with scores…
a0d50a2 
…, and put logs in volumes for api test
@danellecline
rq worker user
6df86b9
@danellecline
fixed health check and reduct ttl to 10 seconds
ad49740
@danellecline
refactor to SimpleWorker for single model load and simplification of …
a774ef6 
…compose
@danellecline danellecline merged commit d6cc7ba into main Jul 6, 2025
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@danellecline