Merge branch 'kebe-temp'

Author: kebe7jun

packages/2024-08-21-kubecon-hk/slides.md

@@ -1233,6 +1233,20 @@ glowSeed: 230
 
 </div>
 
+<!--
+Let’s dive into the core features of kcover.
+
+[click] Firstly, there is the “Firewatch of Workloads.” This feature will continuously monitor the status of training task workloads, such as PyTorchJobs, and promptly detect any error messages.
+
+[click] Next, “Enhanced Observability” will be implemented by utilizing various means to determine the status of jobs, such as observing logs and real-time system calls, thus enhancing the observability of training jobs.
+
+[click] Through “Periodic Inspection,” we will regularly test the status of jobs, the environment, or infrastructure to ensure that the resources committed to training jobs meet the required conditions, ensuring smooth training progress.
+
+[click] With “Cascading Shutdown,” when a fault occurs that prevents the training task from continuing, the entire task will be restarted through Cascading Shutdown. This prevents the training framework from waiting due to a non-working part, thus avoiding the waste of valuable hardware resources.
+
+[click] Finally, “Intelli-Migration” will intelligently assess the health status of nodes to determine whether they can continue running jobs, ensuring maximized resource utilization while safeguarding training efficiency.
+-->
+
 ---
 class: py-10
 ---
@@ -1290,6 +1304,14 @@ class: py-10
 
 </div>
 
+<!--
+The architecture of Kcover consists of two parts: Collector and Controller (also known as Recovery Manager).
+
+The Collector runs as a Daemonset on each Node, responsible for gathering information. This includes executing the dcgmi command, analyzing the logs and events of each Pod, and invoking some system calls, such as checking the status of PCIE devices, to determine the operational status of tasks. It reports any exceptional events back to the APIServer.
+
+The Controller monitors these events from the APIServer and makes further assessments of the data collected by the Collector to determine whether a Job needs to be restarted. If a restart is required, it will execute the restart of the entire Job and may mark the node as unschedulable.
+-->
+
 ---
 class: py-10
 glow: right
@@ -1317,6 +1339,13 @@ glowSeed: 230
 
 </v-clicks>
 
+<!--
+[click] Once a training task is labeled, kcover will continuously analyze this information. 
+This includes [click] node status, [click] container logs (such as CUDA, NCCL, or OOM errors, [click] as well as specific exit codes). 
+[click] If a problem is detected, 
+[click] we will record the event through the Collector [click] and may initiate a Cascading Shutdown to restart the task, allowing it to resume training from the last known state. [click] Additionally, through ongoing diagnostic tools, we will analyze network status, GPU hardware status, PCIE status, and kernel status to ensure that the system always operates at optimal conditions.
+-->
+
 ---
 class: py-10
 ---
@@ -1327,7 +1356,7 @@ class: py-10
 
 #### Install
 
-```shell
+```shell {|3}
 helm repo add baizeai https://baizeai.github.io/charts
 helm repo update baizeai
 helm -n kcover-system --create-namespace install kcover baizeai/kcover
@@ -1363,6 +1392,13 @@ metadata:
 
 </div>
 
+<!--
+To start using kcover, you can initially install kcover onto your system with a few simple helm commands. 
+[click] You only need to execute the helm install command to install kcover on your cluster.
+[click] Subsequently, when submitting training tasks, such as a PyTorchJob, you only need to set a label for the job. 
+[click] This allows kcover to continuously monitor the job, ensuring that it can be quickly recovered after a failure without the need for manual intervention.
+-->
+
 ---
 class: py-10
 ---
@@ -1453,6 +1489,11 @@ class: py-10
   </div>
 </div>
 
+<!--
+The above discusses some of the current features and technical details of kcover. 
+[click] This project is now open source, and you can find it at here. We warmly welcome everyone to help us; suggestions or feedback are greatly appreciated.
+-->
+
 ---
 class: py-10
 ---