new-cloud-skeleton

ℹ️ NOTE: This guide will change as SkyPilot develops. Please check back often to make sure you have the most up-to-date version. The read me in this doc has been derived from the Google Doc

Skeleton repo for what's needed to add a new cloud.

Detailed instructions upcoming! Contact the dev team on SkyPilot Slack to get access.

Introduction to SkyPilot

SkyPilot is an intercloud broker -- a framework for running workloads on any cloud. Here are some useful links to learn more:

1. Introductory Blogpost [Start here if you are new]
2. Documentation
3. The Sky Above the Clouds
4. GitHub

How does SkyPilot work?

Here's a simplified overview of SkyPilot's architecture.

TODO: diagram in google doc

In this diagram, the user has two clouds enabled (AWS and GCP). This is what happens when a user launches a job with sky launch:

1. The optimizer reads AWS Catalog and GCP Catalog and runs an algorithm to decide which cloud to run the job on. (Let's suppose the optimizer chooses AWS.) This information is then sent to the provisioner+executor.
   • A catalog is a list of instance types and their prices.
2. The provisioner+executor executes ray commands to launch a cluster on AWS.
   • AWS Node Provider is the interface between ray and AWS, translating ray function calls to AWS API calls.
3. Once the cluster is launched, the provisioner+executor ssh’s into the cluster to execute some AWS Setup commands. This is used to download some important packages on the cluster.
4. The provisioner+executor submits the job to the cluster and the cluster runs the job.

When all is done, the user can run sky down and provisioner+executor will tear down the cluster by executing more ray commands.

Getting Started

Now let's say you have a new cloud, called FluffyCloud, that you want SkyPilot to support. What do you need to do?

You need to:

1. Write a NodeProvider for FluffyCloud. This is the most important part.
2. Add the FluffyCloud catalog to SkyPilot and write functions that read this catalog.
3. Write FluffyCloud setup code.
4. Add FluffyCloud credential check to verify locally stored credentials. This is needed for a user to enable FluffyCloud.

For reference, here is an actual merged PR for adding a new cloud to help you estimate what is required:

• Lambda Cloud

By completing the following steps, you will be able to run SkyPilot on FluffyCloud.

• Step 0
• Step 1
• Step 2
• Step 3
• Step 4
• Step 5