Skip to content

Latest commit

 

History

History
79 lines (77 loc) · 3.25 KB

notes.org

File metadata and controls

79 lines (77 loc) · 3.25 KB

Notes

Major Tasks

Shared State (to run like a server with REST?)

tui interface

consider to use https://crates.io/crates/config instead of current configuration?

figure if it can get rid of returning yaml_value in favour of some generic configuration type

fix RoundRobin queue

i think it’s done

Redis backend should return RedisError, now custom ones.

mongodb queue

postgres queue

benchmarks criterion

Tasks

Need some kind of statistics

Figure out if it’s possible to implement similar api for capp-rs:

“` let app_state = AppState { sqlite: Arc::new(sqlite), }; let router = axum::Router::new() .route(“/authors”, post(create_author)) .layer(trace_layer) .with_state(app_state); “`

need workflows

  • [ ] coverage
  • [X] tests/clippy/check/fmt
  • [ ] integration tests (require only redis now)

healthcheck

  • [X] internet
  • [ ] proxy with httpbin
  • [ ] optional redis
  • [ ] optional mongodb
  • [ ] optional postgres

Refactor worker. Add basic monitoring capabilities.

  • [ ] adjust async tests to [tokio::test] and drop initialization of async runtime in code
  • [ ] expose http endpoint with json collected basic stats from workers

Refactor processor/task_runner/computation.

  • [ ] shared context for Computation. way to transform &Worker to SharedContext
  • [ ] add storage error types context error types to TaskProcessorError

Refactor storage (task deport)

Executor

  • [X] better task handling in executor, queue, started, finished and TaskStatus enum

Task Deport

  • [ ] add queued time, make started time optional (should be something when execution is started)
  • [ ] drop task exceed time for execution
  • [ ] separate thread to clean up staled tasks (or find the way how to not make them stalled)
  • [ ] add more tests

Important!!!!

  • [-] TaskProcessor::process should not panic? (ugh it’s complicated to apply catch_unwind to async function)
  • [ ] Use tokio per core executor to make tasks Sync maybe? if possible
  • [X] i think need to remove unnecessary workspaces
  • [ ] fix tests sometimes not drop database keys

Workers manager

Need something to manager workers. I think worker should return anyhow:Result<ComputationResult> and if Computation failed manager run worker again. Also there can be thread somewhere which will run another worker instead of failed.

Workers communications

Worker should have bi-directional channels to communicate with nodes and channel to accept termination signal. add Anyhow::Error for worker run Manager should collect stats about workers

  • [X] worker sleep timeout - make it std::time::Duration

Computation function

  • [X] computation function should return anyhow::Error
  • [ ] computation could be paused and resumed

InMemory storage backend

  • [ ] Probably i need to Box list and hashmap since they could grow large and quickly overflow the stack

Consider to use Tower as abstraction for function

Renaming?

  • [X] task_deport -> storage
  • [X] task_executor -> manager

prelude rs

tower integration

I think need to update Status states with channels.

redis storage backend with bloom filter

redis storage backend with priorities

Notes

httpbin could be launched as container “docker run -p 80:80 kennethreitz/httpbin”