Skip to content

Commit 9cf701a

Browse files
author
Dhruv Garg
committed
update documentation for the service
1 parent 8fd7f2a commit 9cf701a

File tree

1 file changed

+20
-2
lines changed

1 file changed

+20
-2
lines changed

rpc/spark_erdos_setup.md

Lines changed: 20 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -151,9 +151,23 @@ pip install -r rpc/requirements.txt
151151
python -m grpc_tools.protoc -I./rpc/protos --python_out=. --grpc_python_out=. ./rpc/protos/rpc/erdos_scheduler.proto
152152
```
153153

154-
### Run the service using
154+
### Run the service
155155
```bash
156-
python -m rpc.service
156+
python -m rpc.service --enforce_deadlines --scheduler_runtime=0
157+
```
158+
159+
The above command uses the default argument values from the `service.py` and `main.py`. The default scheduler is `EDF`. Other options available for the
160+
service are `FIFO` and `TetriSched`. The DSched scheduler is a specific instantiation of the `TetriSched` scheduler. The other schedulers can be run
161+
as follows:
162+
163+
#### To instantiate FIFO scheduler for the service:
164+
```bash
165+
python -m rpc.service --scheduler=FIFO --enforce_deadlines --scheduler_runtime=0
166+
```
167+
168+
#### To instantiate DSched scheduler for the service:
169+
```bash
170+
python -m rpc.service --scheduler=TetriSched --enforce_deadlines --scheduler_runtime=0 --release_taskgraphs --opt_passes=CRITICAL_PATH_PASS --opt_passes=CAPACITY_CONSTRAINT_PURGE_PASS --oppasses=DYNAMIC_DISCRETIZATION_PASS --retract_schedules --scheduler_max_occupancy_threshold=0.999 --finer_discretization_at_prev_solution --scheduler_selective_rescheduling --scheduler_reconsideration_period=0.6 --scheduler_time_discretization=1 --scheduler_max_time_discretization=5 --finer_discretization_window=5 --scheduler_log_to_file
157171
```
158172

159173
### Run local tests for the erdos-spark service
@@ -168,6 +182,10 @@ pytest tests/test_service.py
168182
```bash
169183
python -m rpc.service
170184
```
185+
Refer to the above section to instantiate different schedulers for the service.
186+
187+
> NOTE: Since we emulate a 20-node spark cluster on a single system, an additional flag `--override_worker_cpu_count` needs to be passed in the
188+
> service launch command.
171189
172190
### Start all components of the spark cluster
173191
Run the following commands from the root directory of the `spark-mirror` repository.

0 commit comments

Comments
 (0)