Kubespot (Azure)

Compliance Oriented Kubernetes Setup for AWS, Google Cloud and Microsoft Azure.

Kubespot is an open source terraform module that attempts to create a complete compliance-oriented Kubernetes setup on AWS, Google Cloud and Azure. These add additional security such as additional system logs, file system monitoring, hard disk encryption and access control. Further, we setup the managed Redis and SQL on each of the Cloud providers with limited access to the Kubernetes cluster so things are further locked down. All of this should lead to setting up a HIPAA / PCI / SOC2 being made straightforward and repeatable.

This covers how we setup your infrastructure on AWS, Google Cloud and Azure. These are the three Cloud Providers that we currently support to run Kubernetes. Further, we use the managed service provided by each of the Cloud Providers. This document covers everything related to how infrastructure is setup within each Cloud, how we create an isolated environment for Compliance and the commonalities between them.

Tools & Setup

brew install kubectl kubernetes-helm google-cloud-sdk terraform

Keys

How to get key for cluster creation (client id and secret)

1. Sign in to Azure portal
2. Navigate to the Azure Active Directory
3. Select "App registrations"
4. If there is application already use existing one or create new one as follows
5. Click on the "New registration" button to create a new application registration
6. select the appropriate supported account type (e.g., "Accounts in this organizational directory only")
7. Click on the "Register" button to create the application.
8. After application is created, Under "Certificates & secrets," click on the "New client secret" button to create a new client secret.
9. Copy the client id and client secret and pass it to cluster creation opszero module

Deployment

terraform init
terraform plan
terraform apply -auto-approve

Teardown

terraform destroy -auto-approve

Providers

Name	Version
azurerm	~> 4.30.0

Inputs

Name	Description	Type	Default	Required
acr_id	azure container resource id to provide access for aks	string	""	no
ad_group_ids	ActiveDirectory Groups that have access to this cluster	list	[]	no
ad_user_ids	ActiveDirectory users that have access to the kubernetes admin group and attached to the cluster	list	[]	no
address_spaces	List of address spaces for the VNet	list(string)	[ "null" ]	no
aks_sku_tier	aks sku tier. Possible values are Free ou Paid	string	"Free"	no
allocation_method	Allocation method for the public IP	string	"Static"	no
azurerm_resource_group_enabled	Enable creation of a new DDoS protection plan	bool	true	no
cidr	The address space that is used the virtual network	string	"10.0.0.0"	no
cluster_name	n/a	string	"aks"	no
cmk_enabled	Flag to control resource creation related to cmk encryption.	bool	false	no
create_nat_gateway	Flag to create NAT Gateway	bool	false	no
default_node_pool	Default node pool configuration	object({ name = string vm_size = string os_disk_type = string os_disk_size_gb = number auto_scaling_enabled = bool node_public_ip_enabled = bool count = number min_count = number max_count = number max_pods = number type = string })	n/a	yes
default_tags	Base tags applied to all resources	map(string)	{ "ManagedBy": "OpsZero", "Repositories": "https://github.com/opszero/terraform-azurerm-kubespot" }	no
delegation	Delegation of subnet resources	map(list(object({ name = string service_delegation = list(object({ name = string actions = list(string) })) })))	{}	no
dns_servers	List of DNS servers	list(string)	[]	no
edge_zone	Specifies the Edge Zone within the Azure Region where this Managed Kubernetes Cluster should exist. Changing this forces a new resource to be created.	string	null	no
enable	Enable creation of a new DDoS protection plan	bool	true	no
enable_azure_policy	Enable Azure Policy Addon.	bool	true	no
enable_ddos_pp	Enable creation of a new DDoS protection plan	bool	false	no
enable_http_application_routing	Enable HTTP Application Routing Addon (forces recreation).	bool	false	no
enable_route_table	Flag to enable Route Table	bool	false	no
enabled	Enable resource group creation and related resources.	bool	true	no
environment_name	Name of the environment to create resources	string	""	no
existing_ddos_pp	Use an existing DDoS protection plan ID	string	null	no
image_cleaner_enabled	(Optional) Specifies whether Image Cleaner is enabled.	bool	false	no
image_cleaner_interval_hours	(Optional) Specifies the interval in hours when images should be cleaned up. Defaults to 48.	number	48	no
key_vault_id	Specifies the URL to a Key Vault Key (either from a Key Vault Key, or the Key URL for the Key Vault Secret	string	""	no
kubernetes_version	Version of Kubernetes to deploy	string	"1.32.2"	no
linux_profile	Username and ssh key for accessing AKS Linux nodes with ssh.	object({ username = string, ssh_key = string })	null	no
local_account_disabled	Whether local account should be disable or not	bool	false	no
location	The Azure region where the resource group will be created.	string	"East US"	no
lock_level	The level of the lock. Can be 'CanNotDelete' or 'ReadOnly'.	string	"CanNotDelete"	no
mariadb_sql_enabled	Specify whether the mariadb is enabled	bool	true	no
mariadb_sql_version	Specify the version of MariaDB to use. Possible values are 10.2 and 10.3	string	"10.2"	no
nat_gateway_idle_timeout	Timeout in minutes for idle NAT Gateway	number	4	no
network_plugin	Network plugin to use for networking.	string	"azure"	no
network_policy	(Optional) Sets up network policy to be used with Azure CNI. Network policy allows us to control the traffic flow between pods. Currently supported values are calico and azure. Changing this forces a new resource to be created.	string	null	no
node_resource_group	Name of the resource group in which to put AKS nodes. If null default to MC_	string	null	no
nodes_desired_capacity	The number of Amazon EC2 instances that should be running in the group	number	1	no
nodes_pools	List of additional node pools	list(object({ name = string vm_size = string os_type = string os_disk_type = string os_disk_size_gb = number auto_scaling_enabled = bool node_count = number min_count = number max_count = number max_pods = number node_public_ip_enabled = bool mode = string orchestrator_version = string node_taints = list(string) host_group_id = string # capacity_reservation_group_id = string # workload_runtime = string # zones = list(string) }))	[]	no
notes	Optional notes about the lock.	string	"Resource group lock to prevent accidental deletion"	no
outbound_type	The outbound (egress) routing method which should be used for this Kubernetes Cluster. Possible values are loadBalancer and userDefinedRouting.	string	"loadBalancer"	no
postgres_sql_enabled	Specify whether postgres sql is enabled	bool	false	no
postgres_sql_version	Specify the version of PostgreSQL to use. Valid values are 9.5, 9.6, 10, 10.0, and 11	string	"11"	no
prefix	(Optional) The prefix for the resources created in the specified Azure Resource Group. Omitting this variable requires both var.cluster_log_analytics_workspace_name and var.cluster_name have been set. Only one of var.prefix,var.dns_prefix_private_cluster can be specified.	string	""	no
private_cluster_enabled	Configure AKS as a Private Cluster : https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/kubernetes_cluster#private_cluster_enabled	bool	false	no
private_dns_zone_id	Id of the private DNS Zone when <private_dns_zone_type> is custom	string	null	no
private_dns_zone_type	n/a	string	null	no
redis_capacity	The size of the Redis cache to deploy	number	1	no
redis_enabled	Specify whether the redis cluster is enabled	bool	false	no
redis_family	The SKU family/pricing group to use. Valid values are C (for Basic/Standard SKU family) and P (for Premium)	string	"C"	no
redis_shard_count	Only available when using the Premium SKU The number of Shards to create on the Redis Cluster	number	0	no
redis_sku_name	The SKU of Redis to use. Possible values are Basic, Standard and Premium	string	"Standard"	no
region	The Azure Region where the Resource Group should exist.	string	"Central US"	no
registry_enabled	Specify whether the container registry is enabled	bool	true	no
registry_name	n/a	string	"acrprodtyj"	no
resource_group_name	Use an existing DDoS protection plan ID	string	""	no
resource_lock_enabled	Flag to enable resource lock on the resource group.	bool	false	no
role_based_access_control_enabled	Whether role based acces control should be enabled or not	bool	true	no
route_table_name	Name of the route table	string	null	no
routes	Routes to be added to the route table	list(object({ name = string address_prefix = string next_hop_type = string next_hop_in_ip_address = string }))	[]	no
service_cidr	CIDR used by kubernetes services (kubectl get svc).	string	"10.0.0.0/16"	no
service_endpoint_policy_ids	Service Endpoint Policy IDs for subnet	list(string)	null	no
service_endpoints	Service Endpoints for subnet	list(string)	[]	no
sku	SKU for the Public IP	string	"Standard"	no
sku_name	SKU for NAT Gateway	string	"Standard"	no
specific_name_subnet	Specific subnet names if needed	bool	false	no
specific_subnet_names	n/a	list(string)	[]	no
sql_master_password	The Password associated with the administrator_login for the PostgreSQL/MariaDB Server	string	""	no
sql_master_username	The Administrator login for the PostgreSQL/MariabDB Server	string	"prod"	no
sql_sku_name	Specify the SKU Name for this PostgreSQL Server	string	"GP_Gen5_2"	no
sql_storage_in_mb	Max storage allowed for a MariaDB server	number	10240	no
subnet_names	Subnet names	list(string)	[]	no
subnet_prefixes	Subnet prefixes for address allocation	list(string)	[]	no
tags	Additional tags that override or extend default_tags	map(string)	{}	no
timeouts	Timeout values for resource group operations.	object({ create = optional(string, "30m") read = optional(string, "5m") update = optional(string, "30m") delete = optional(string, "30m") })	{}	no
zones	Zones for NAT Gateway	list(string)	[]	no

Resources

Name	Type
azurerm_container_registry.acr	resource
azurerm_disk_encryption_set.main	resource
azurerm_key_vault_access_policy.key_vault	resource
azurerm_key_vault_access_policy.kubelet_identity	resource
azurerm_key_vault_access_policy.main	resource
azurerm_key_vault_key.example	resource
azurerm_kubernetes_cluster.aks	resource
azurerm_kubernetes_cluster_node_pool.node_pools	resource
azurerm_management_lock.default	resource
azurerm_nat_gateway.natgw	resource
azurerm_nat_gateway_public_ip_association.pip_assoc	resource
azurerm_network_ddos_protection_plan.main	resource
azurerm_postgresql_database.qa	resource
azurerm_postgresql_server.default	resource
azurerm_postgresql_virtual_network_rule.default	resource
azurerm_public_ip.pip	resource
azurerm_redis_cache.default	resource
azurerm_resource_group.default	resource
azurerm_role_assignment.aks_acr_access_object_id	resource
azurerm_role_assignment.aks_acr_access_principal_id	resource
azurerm_role_assignment.aks_system_identity	resource
azurerm_role_assignment.aks_uai_private_dns_zone_contributor	resource
azurerm_role_assignment.aks_uai_vnet_network_contributor	resource
azurerm_role_assignment.aks_user_assigned	resource
azurerm_role_assignment.azurerm_disk_encryption_set_key_vault_access	resource
azurerm_route_table.rt	resource
azurerm_subnet.subnet	resource
azurerm_subnet_nat_gateway_association.subnet_assoc	resource
azurerm_subnet_route_table_association.main	resource
azurerm_user_assigned_identity.aks_user_assigned_identity	resource
azurerm_virtual_network.default	resource
azurerm_client_config.current	data source
azurerm_subscription.current	data source

Outputs

Name	Description
aks_cluster_name	The name of the AKS cluster
aks_fqdn	FQDN of the AKS cluster
aks_kube_config	Raw kubeconfig for the AKS cluster
nat_gateway_id	The ID of the NAT Gateway
redis_id	ID of the Redis cache
registry_name	Name of the Azure Container Registry
resource_group_location	The location of the created resource group
resource_group_name	The name of the created resource group
subnet_id	n/a
subnet_ids	The IDs of the created subnets
vnet_id	The ID of the created Virtual Network

🚀 Built by opsZero!

Since 2016 opsZero has been providing Kubernetes expertise to companies of all sizes on any Cloud. With a focus on AI and Compliance we can say we seen it all whether SOC2, HIPAA, PCI-DSS, ITAR, FedRAMP, CMMC we have you and your customers covered.

We provide support to organizations in the following ways:

• Modernize or Migrate to Kubernetes
• Cloud Infrastructure with Kubernetes on AWS, Azure, Google Cloud, or Bare Metal
• Building AI and Data Pipelines on Kubernetes
• Optimizing Existing Kubernetes Workloads

We do this with a high-touch support model where you:

• Get access to us on Slack, Microsoft Teams or Email
• Get 24/7 coverage of your infrastructure
• Get an accelerated migration to Kubernetes

Please schedule a call if you need support.