Skip to content

MaciolekMichal/azure-databricks-dev-platform

Repository files navigation

Environment Setup

🐧1. Setup WSL Follow this guide to install Linux on Windows with WSL

🐋2. Install Docker Desktop Follow this guide to install Docker Desktop on your Windows machine

💻 3. Setup VSCode Follow this guide to install VS Code on your Windows machine Make sure to install Dev Containers extension

🐱 4. Clone this repository How to clone this Github repository?

After cloning the repository, make sure Docker Desktop is running and open the project in VS Code. VS Code will automatically detect the .devcontainer/ configuration and prompt you to Reopen in Container. Click it to build and start the development environment. Alternatively, use Ctrl + Shift + P → Dev Containers: Reopen in Container.

🔑 5. Create an Azure subscription To continue, you need an active Azure subscription. If you don’t have one yet, you can create a free account here.

Recommendation: When creating the account, consider choosing the Free Tier with $200 in credits (valid for 30 days). This is more than enough to complete the exercises..

🧱 6. Create a first databricks workspace Manually create a sandbox Databricks workspace (via Azure Portal or Azure CLI), then copy the generated Databricks account ID into the `.env` file.
☁️ 7. Authenticate to Azure Authenticate your local environment with Azure (via Azure CLI) to enable local interaction with Azure resources:
make authenticate-azure
🧱 8. Authenticate to Databricks Authenticate your local environment with Databricks on account level (via Databricks CLI) to enable local interaction with Databricks resources:
make authenticate-databricks-on-account-level
🐱 9. Authenticate to GitHub To enable Terraform to interact with GitHub (for example, to create environments, environment variables, and secrets), a fine-grained GitHub Personal Access Token (PAT) is required. Creating a fine-grained personal access token

Once created, export the token as an environment variable on your local machine named GITHUB_PAT.

The token is then automatically forwarded into the dev container via .devcontainer/devcontainer.json:

"remoteEnv": {
  "GITHUB_PAT": "${localEnv:GITHUB_PAT}"
}

This makes the token available to Terraform inside the container, enabling authenticated GitHub operations while keeping secrets out of source control.


How to interact with the platform?

🌱 1. Setup environment variables

All non-sensitive environment variables should be defined in the .env file.

All sensitive environment variables (e.g. tokens, secrets) should be set as environment variables on your local machine.
These variables are then passed into the dev container via .devcontainer/devcontainer.json, for example:

"remoteEnv": {
  "GITHUB_PAT": "${localEnv:GITHUB_PAT}"
}

This ensures that sensitive values are not committed to the repository while still being available inside the dev container.

🗄️ 2. Create Terraform backends

Create the Terraform remote state backends for each environment.

First, create the global backend:

source infra/backends/create_backend.sh global

Then repeat the same step for the remaining environments:

  • dev
  • prod

This ensures that each environment has its own isolated Terraform state.

🏗️ 3. Run terraform code

For every environment, starting with global run:

make init-terraform  TERRAFORM_DIR=<dir> ENV=<env>
make plan-terraform  TERRAFORM_DIR=<dir> ENV=<env>
make apply-terraform TERRAFORM_DIR=<dir> ENV=<env>

Environment-specific variables:

Environment TERRAFORM_DIR ENV
Global global_env global
Dev stage_env dev
Prod stage_env prod

🤝 Contributing

Contributions are welcome.
Feel free to open an issue or submit a pull request to improve the code, or documentation.

About

No description, website, or topics provided.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published