WIP: Service rewrite#99

Draft

1ntEgr8 wants to merge 129 commits intoerdos-project:mainfrom

1ntEgr8:service-rewrite

Commits on Oct 1, 2024

scale runtime based on max number of tasks
1ntEgr8
committed

Commits on Oct 2, 2024

fix bug in runtime calc
1ntEgr8
committed

Commits on Nov 4, 2024

Commits on Nov 7, 2024

Commits on Nov 8, 2024

Commits on Nov 12, 2024

WIP: service changes to handle first tpch taskgraph
Dhruv Garg
committed

Commits on Nov 13, 2024

Fix the tick() function to dequeue all events upto n, pass runtime_unit in tpch_loader
Dhruv Garg
committed

Commits on Nov 20, 2024

Commits on Nov 21, 2024

enqueue scheduler start event in register task graph
1ntEgr8
committed
bug fixes
1ntEgr8
committed
add a lock, haven't checked all places
1ntEgr8
committed
Add tests for notify task completion
Dhruv Garg
committed
add cancelled field to proto

1ntEgr8
authored and
Dhruv Garg
committed
populate cancelled field in rpc response

1ntEgr8
authored and
Dhruv Garg
committed
Updates to spark erdos service documentation
Dhruv Garg
committed
implement register driver
1ntEgr8
committed
quick fixes in regsister and de-register driver
Dhruv Garg
committed
Updates to spark erdos documentation
Dhruv Garg
committed
register driver bug fix
1ntEgr8
committed
Add override_worker_cpu_count flag
Dhruv Garg
committed

Commits on Nov 22, 2024

Commits on Nov 23, 2024

Separate out internal state mgmt for driver and application
Dhruv Garg
committed
re-add impl for override_worker_cpu_count
Dhruv Garg
committed
fix: correct enqueue of task_finished and sched_start in notifyTaskCompletion
Dhruv Garg
committed
enable task cancellations to be sent back to backend
Dhruv Garg
committed
update test script to include cancel_task scenario (needs to be sped up)
Dhruv Garg
committed

Commits on Nov 25, 2024

allow service to use different schedulers based on args
Dhruv Garg
committed
format service
1ntEgr8
committed
add enforce deadlines flag
1ntEgr8
committed

Commits on Nov 28, 2024

update docs about sched time discretization
Dhruv Garg
committed
added comments about setting deadlines in generate_task_graph
Dhruv Garg
committed
updates to test script (verified across edf, fifo, dsched)
Dhruv Garg
committed
update documentation for the service
Dhruv Garg
committed

Commits on Nov 29, 2024

Commits on Dec 2, 2024

add launch script
1ntEgr8
committed

Commits on Dec 3, 2024

add shutdown rpc method
1ntEgr8
committed
add service experiment runner
1ntEgr8
committed
sleep for a bit before launching queries
1ntEgr8
committed
remove extraneous sleep
1ntEgr8
committed
set start_time of task to its placement time
Dhruv Garg
committed
add testcase to verify new taskgraph termination approach
Dhruv Garg
committed
nit documentation in erdos-spark setup
Dhruv Garg
committed
more explainable msgs from GetPlacements
Dhruv Garg
committed
reorder event queue priority to process scheduler events before task placement
Dhruv Garg
committed
improvements to experiment runner
1ntEgr8
committed
fix profile path in tpch loader
1ntEgr8
committed
add spark-master-ip flag
1ntEgr8
committed

Commits on Dec 14, 2024

Commits on Dec 15, 2024

Commits on Jan 11, 2025

remove extraneous print
1ntEgr8
committed

Commits on Jan 21, 2025

Commits on Jan 29, 2025

TetriSchedScheduler: Reconsider tasks that couldn't be placed
rohanbafna
committed

Commits on Feb 12, 2025

tpch partitioning based on space-time analysis
Dhruv Garg
committed

Commits on Feb 20, 2025

bucketize space-time for tpch-partitioning, cleaned up code
Dhruv Garg
committed
tpch partitions for 100g, 250g and diff max executors
Dhruv Garg
committed