Skip to content

Add azure_storage as backup agent #134085

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 72 commits into from
Feb 24, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
72 commits
Select commit Hold shift + click to select a range
6690660
Bump pylamarzocco to 1.4.3
zweckj Dec 24, 2024
e79e7e4
Add azure_storage integration
zweckj Dec 27, 2024
6a2aa89
Merge branch 'dev' into azure-storage/init
zweckj Dec 27, 2024
abb028e
Add exc translations
zweckj Dec 27, 2024
a973895
Add as MSFT integration
zweckj Dec 27, 2024
9db9200
adapt get backup
zweckj Dec 27, 2024
f903ec8
Merge branch 'dev' into azure-storage/init
zweckj Dec 27, 2024
a60fffa
extra error handle
zweckj Dec 27, 2024
e93c1df
extra error handle
zweckj Dec 27, 2024
35a8679
comments
zweckj Dec 27, 2024
813d504
remove const, test cleanuo
zweckj Dec 28, 2024
84c85c7
further tweak testes
zweckj Dec 28, 2024
315d213
rename test
zweckj Dec 28, 2024
65e1538
set diagnostics exempt
zweckj Dec 29, 2024
f2bda6e
move return out of try
zweckj Jan 7, 2025
89d3770
Create issue when container does not exist
zweckj Jan 12, 2025
e3cc22f
quality scale
zweckj Jan 12, 2025
58a0c33
use decorator for error handling
zweckj Jan 12, 2025
ee2f0bd
rename vars
zweckj Jan 12, 2025
29a6f54
cleanup
zweckj Jan 12, 2025
9160e1d
Add tests
zweckj Jan 12, 2025
93aa56d
Add tests
zweckj Jan 12, 2025
046fe4d
Merge branch 'dev' into azure-storage/init
zweckj Jan 12, 2025
90fa8bd
fix test
zweckj Jan 13, 2025
9b8dc54
do without deppcopy
zweckj Jan 13, 2025
589e08f
use deepcopy, assert upload
zweckj Jan 13, 2025
e137e98
backpu not found test
zweckj Jan 13, 2025
64e2205
drop snapshots
zweckj Jan 13, 2025
ffee804
use as_frontend_json
zweckj Jan 14, 2025
29e5f31
Remove issue raising, remove translations
zweckj Jan 16, 2025
4dcbe48
increase timeout
zweckj Jan 17, 2025
488ac42
Revert "increase timeout"
zweckj Jan 17, 2025
6c73bac
Merge branch 'dev' into azure-storage/init
zweckj Jan 18, 2025
12dd2c0
Increase aiohttp timeout
zweckj Jan 19, 2025
27292c2
rename metadata version
zweckj Jan 23, 2025
4ec57f0
some test changes
zweckj Jan 24, 2025
b719531
parenthesis
zweckj Jan 24, 2025
2258ea9
Merge branch 'dev' into azure-storage/init
zweckj Jan 24, 2025
439be62
bdraco optimizations
zweckj Jan 27, 2025
b7f6f99
bdraco optimizations
zweckj Jan 27, 2025
8fcdf48
bdraco optimizations
zweckj Jan 27, 2025
3fc9e1f
Merge branch 'dev' into azure-storage/init
zweckj Jan 28, 2025
b7e38da
Add unique id
zweckj Jan 28, 2025
635e344
remove slugify
zweckj Jan 28, 2025
ffbb19f
rename default folder in CF
zweckj Jan 28, 2025
56bce81
Merge branch 'dev' into azure-storage/init
zweckj Jan 29, 2025
8f6b5a7
fix tests due to new flag
zweckj Jan 29, 2025
b7a35e5
Merge branch 'dev' into azure-storage/init
zweckj Feb 1, 2025
f7e1caf
migrate to suggested name
zweckj Feb 1, 2025
c6944ef
Merge branch 'dev' into azure-storage/init
zweckj Feb 1, 2025
f8d090d
Consider backup metadata as blob
zweckj Feb 7, 2025
4225cc1
Merge branch 'dev' into azure-storage/init
zweckj Feb 7, 2025
640350d
fix tests
zweckj Feb 7, 2025
8ab9a9c
stale docstring
zweckj Feb 7, 2025
56b640c
Add check for metadata version
zweckj Feb 7, 2025
4dd4dc6
Merge branch 'dev' into azure-storage/init
zweckj Feb 11, 2025
dfaf20e
use const for metadata version
zweckj Feb 11, 2025
968d29c
import from private file
zweckj Feb 12, 2025
7f20403
Improve coverage
zweckj Feb 12, 2025
c396d52
Merge branch 'dev' into azure-storage/init
zweckj Feb 17, 2025
cdc759d
switch to new unload
zweckj Feb 17, 2025
c8fa87c
Further simplify
zweckj Feb 17, 2025
adfc15e
Merge branch 'dev' into azure-storage/init
zweckj Feb 21, 2025
4f9084b
Use new state change
zweckj Feb 21, 2025
ca11355
Update tests/components/azure_storage/test_backup.py
zweckj Feb 24, 2025
fab2c6c
Update homeassistant/components/azure_storage/backup.py
zweckj Feb 24, 2025
174e108
Change log level from error to debug
zweckj Feb 24, 2025
ff6266e
Improve error message in backup operation
zweckj Feb 24, 2025
bc37bbd
Fix formatting and typo in backup.py
zweckj Feb 24, 2025
6631b32
fix test
zweckj Feb 24, 2025
5e1a8b9
break long string
zweckj Feb 24, 2025
52c6456
Break line
MartinHjelmare Feb 24, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .strict-typing
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,7 @@ homeassistant.components.auth.*
homeassistant.components.automation.*
homeassistant.components.awair.*
homeassistant.components.axis.*
homeassistant.components.azure_storage.*
homeassistant.components.backup.*
homeassistant.components.baf.*
homeassistant.components.bang_olufsen.*
Expand Down
2 changes: 2 additions & 0 deletions CODEOWNERS

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions homeassistant/brands/microsoft.json
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
"azure_devops",
"azure_event_hub",
"azure_service_bus",
"azure_storage",
"microsoft_face_detect",
"microsoft_face_identify",
"microsoft_face",
Expand Down
82 changes: 82 additions & 0 deletions homeassistant/components/azure_storage/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
"""The Azure Storage integration."""

from aiohttp import ClientTimeout
from azure.core.exceptions import (
ClientAuthenticationError,
HttpResponseError,
ResourceNotFoundError,
)
from azure.core.pipeline.transport._aiohttp import (
AioHttpTransport,
) # need to import from private file, as it is not properly imported in the init
from azure.storage.blob.aio import ContainerClient

from homeassistant.config_entries import ConfigEntry
from homeassistant.core import HomeAssistant
from homeassistant.exceptions import ConfigEntryError, ConfigEntryNotReady
from homeassistant.helpers.aiohttp_client import async_create_clientsession

from .const import (
CONF_ACCOUNT_NAME,
CONF_CONTAINER_NAME,
CONF_STORAGE_ACCOUNT_KEY,
DATA_BACKUP_AGENT_LISTENERS,
DOMAIN,
)

type AzureStorageConfigEntry = ConfigEntry[ContainerClient]


async def async_setup_entry(
hass: HomeAssistant, entry: AzureStorageConfigEntry
) -> bool:
"""Set up Azure Storage integration."""
# set increase aiohttp timeout for long running operations (up/download)
session = async_create_clientsession(
hass, timeout=ClientTimeout(connect=10, total=12 * 60 * 60)
)
container_client = ContainerClient(
account_url=f"https://{entry.data[CONF_ACCOUNT_NAME]}.blob.core.windows.net/",
container_name=entry.data[CONF_CONTAINER_NAME],
credential=entry.data[CONF_STORAGE_ACCOUNT_KEY],
transport=AioHttpTransport(session=session),
)

try:
if not await container_client.exists():
await container_client.create_container()
except ResourceNotFoundError as err:
raise ConfigEntryError(
translation_domain=DOMAIN,
translation_key="account_not_found",
translation_placeholders={CONF_ACCOUNT_NAME: entry.data[CONF_ACCOUNT_NAME]},
) from err
except ClientAuthenticationError as err:
raise ConfigEntryError(
translation_domain=DOMAIN,
translation_key="invalid_auth",
translation_placeholders={CONF_ACCOUNT_NAME: entry.data[CONF_ACCOUNT_NAME]},
) from err
except HttpResponseError as err:
raise ConfigEntryNotReady(
translation_domain=DOMAIN,
translation_key="cannot_connect",
translation_placeholders={CONF_ACCOUNT_NAME: entry.data[CONF_ACCOUNT_NAME]},
) from err

entry.runtime_data = container_client

def _async_notify_backup_listeners() -> None:
for listener in hass.data.get(DATA_BACKUP_AGENT_LISTENERS, []):
listener()

entry.async_on_unload(entry.async_on_state_change(_async_notify_backup_listeners))

return True


async def async_unload_entry(
hass: HomeAssistant, entry: AzureStorageConfigEntry
) -> bool:
"""Unload an Azure Storage config entry."""
return True
182 changes: 182 additions & 0 deletions homeassistant/components/azure_storage/backup.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,182 @@
"""Support for Azure Storage backup."""

from __future__ import annotations

from collections.abc import AsyncIterator, Callable, Coroutine
from functools import wraps
import json
import logging
from typing import Any, Concatenate

from azure.core.exceptions import HttpResponseError
from azure.storage.blob import BlobProperties

from homeassistant.components.backup import (
AgentBackup,
BackupAgent,
BackupAgentError,
BackupNotFound,
suggested_filename,
)
from homeassistant.core import HomeAssistant, callback

from . import AzureStorageConfigEntry
from .const import DATA_BACKUP_AGENT_LISTENERS, DOMAIN

_LOGGER = logging.getLogger(__name__)
METADATA_VERSION = "1"


async def async_get_backup_agents(
hass: HomeAssistant,
) -> list[BackupAgent]:
"""Return a list of backup agents."""
entries: list[AzureStorageConfigEntry] = hass.config_entries.async_loaded_entries(
DOMAIN
)
return [AzureStorageBackupAgent(hass, entry) for entry in entries]


@callback
def async_register_backup_agents_listener(
hass: HomeAssistant,
*,
listener: Callable[[], None],
**kwargs: Any,
) -> Callable[[], None]:
"""Register a listener to be called when agents are added or removed."""
hass.data.setdefault(DATA_BACKUP_AGENT_LISTENERS, []).append(listener)

@callback
def remove_listener() -> None:
"""Remove the listener."""
hass.data[DATA_BACKUP_AGENT_LISTENERS].remove(listener)
if not hass.data[DATA_BACKUP_AGENT_LISTENERS]:
hass.data.pop(DATA_BACKUP_AGENT_LISTENERS)

return remove_listener


def handle_backup_errors[_R, **P](
func: Callable[Concatenate[AzureStorageBackupAgent, P], Coroutine[Any, Any, _R]],
) -> Callable[Concatenate[AzureStorageBackupAgent, P], Coroutine[Any, Any, _R]]:
"""Handle backup errors."""

@wraps(func)
async def wrapper(
self: AzureStorageBackupAgent, *args: P.args, **kwargs: P.kwargs
) -> _R:
try:
return await func(self, *args, **kwargs)
except HttpResponseError as err:
_LOGGER.debug(
"Error during backup in %s: Status %s, message %s",
func.__name__,
err.status_code,
err.message,
exc_info=True,
)
raise BackupAgentError(
f"Error during backup operation in {func.__name__}:"
f" Status {err.status_code}, message: {err.message}"
) from err

return wrapper


class AzureStorageBackupAgent(BackupAgent):
"""Azure storage backup agent."""

domain = DOMAIN

def __init__(self, hass: HomeAssistant, entry: AzureStorageConfigEntry) -> None:
"""Initialize the Azure storage backup agent."""
super().__init__()
self._client = entry.runtime_data
self.name = entry.title
self.unique_id = entry.entry_id

@handle_backup_errors
async def async_download_backup(
self,
backup_id: str,
**kwargs: Any,
) -> AsyncIterator[bytes]:
"""Download a backup file."""
blob = await self._find_blob_by_backup_id(backup_id)
if blob is None:
raise BackupNotFound(f"Backup {backup_id} not found")
download_stream = await self._client.download_blob(blob.name)
return download_stream.chunks()

@handle_backup_errors
async def async_upload_backup(
self,
*,
open_stream: Callable[[], Coroutine[Any, Any, AsyncIterator[bytes]]],
backup: AgentBackup,
**kwargs: Any,
) -> None:
"""Upload a backup."""

metadata = {
"metadata_version": METADATA_VERSION,
"backup_id": backup.backup_id,
"backup_metadata": json.dumps(backup.as_dict()),
}

await self._client.upload_blob(
name=suggested_filename(backup),
metadata=metadata,
data=await open_stream(),
length=backup.size,
)

@handle_backup_errors
async def async_delete_backup(
self,
backup_id: str,
**kwargs: Any,
) -> None:
"""Delete a backup file."""
blob = await self._find_blob_by_backup_id(backup_id)
if blob is None:
return
await self._client.delete_blob(blob.name)

@handle_backup_errors
async def async_list_backups(self, **kwargs: Any) -> list[AgentBackup]:
"""List backups."""
backups: list[AgentBackup] = []
async for blob in self._client.list_blobs(include="metadata"):
metadata = blob.metadata

if metadata.get("metadata_version") == METADATA_VERSION:
backups.append(
AgentBackup.from_dict(json.loads(metadata["backup_metadata"]))
)

return backups

@handle_backup_errors
async def async_get_backup(
self,
backup_id: str,
**kwargs: Any,
) -> AgentBackup | None:
"""Return a backup."""
blob = await self._find_blob_by_backup_id(backup_id)
if blob is None:
return None

return AgentBackup.from_dict(json.loads(blob.metadata["backup_metadata"]))

async def _find_blob_by_backup_id(self, backup_id: str) -> BlobProperties | None:
"""Find a blob by backup id."""
async for blob in self._client.list_blobs(include="metadata"):
if (
backup_id == blob.metadata.get("backup_id", "")
and blob.metadata.get("metadata_version") == METADATA_VERSION
):
return blob
return None
72 changes: 72 additions & 0 deletions homeassistant/components/azure_storage/config_flow.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
"""Config flow for Azure Storage integration."""

import logging
from typing import Any

from azure.core.exceptions import ClientAuthenticationError, ResourceNotFoundError
from azure.core.pipeline.transport._aiohttp import (
AioHttpTransport,
) # need to import from private file, as it is not properly imported in the init
from azure.storage.blob.aio import ContainerClient
import voluptuous as vol

from homeassistant.config_entries import ConfigFlow, ConfigFlowResult
from homeassistant.helpers.aiohttp_client import async_get_clientsession

from .const import (
CONF_ACCOUNT_NAME,
CONF_CONTAINER_NAME,
CONF_STORAGE_ACCOUNT_KEY,
DOMAIN,
)

_LOGGER = logging.getLogger(__name__)


class AzureStorageConfigFlow(ConfigFlow, domain=DOMAIN):
"""Handle a config flow for azure storage."""

async def async_step_user(
self, user_input: dict[str, Any] | None = None
) -> ConfigFlowResult:
"""User step for Azure Storage."""

errors: dict[str, str] = {}

if user_input is not None:
self._async_abort_entries_match(
{CONF_ACCOUNT_NAME: user_input[CONF_ACCOUNT_NAME]}
)
container_client = ContainerClient(
account_url=f"https://{user_input[CONF_ACCOUNT_NAME]}.blob.core.windows.net/",
container_name=user_input[CONF_CONTAINER_NAME],
credential=user_input[CONF_STORAGE_ACCOUNT_KEY],
transport=AioHttpTransport(session=async_get_clientsession(self.hass)),
)
try:
await container_client.exists()
except ResourceNotFoundError:
errors["base"] = "cannot_connect"
except ClientAuthenticationError:
errors[CONF_STORAGE_ACCOUNT_KEY] = "invalid_auth"
except Exception:
_LOGGER.exception("Unknown exception occurred")
errors["base"] = "unknown"
if not errors:
return self.async_create_entry(
title=f"{user_input[CONF_ACCOUNT_NAME]}/{user_input[CONF_CONTAINER_NAME]}",
data=user_input,
)

return self.async_show_form(
data_schema=vol.Schema(
{
vol.Required(CONF_ACCOUNT_NAME): str,
vol.Required(
CONF_CONTAINER_NAME, default="home-assistant-backups"
): str,
vol.Required(CONF_STORAGE_ACCOUNT_KEY): str,
}
),
errors=errors,
)
16 changes: 16 additions & 0 deletions homeassistant/components/azure_storage/const.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
"""Constants for the Azure Storage integration."""

from collections.abc import Callable
from typing import Final

from homeassistant.util.hass_dict import HassKey

DOMAIN: Final = "azure_storage"

CONF_STORAGE_ACCOUNT_KEY: Final = "storage_account_key"
CONF_ACCOUNT_NAME: Final = "account_name"
CONF_CONTAINER_NAME: Final = "container_name"

DATA_BACKUP_AGENT_LISTENERS: HassKey[list[Callable[[], None]]] = HassKey(
f"{DOMAIN}.backup_agent_listeners"
)
12 changes: 12 additions & 0 deletions homeassistant/components/azure_storage/manifest.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
{
"domain": "azure_storage",
"name": "Azure Storage",
"codeowners": ["@zweckj"],
"config_flow": true,
"documentation": "https://www.home-assistant.io/integrations/azure_storage",
"integration_type": "service",
"iot_class": "cloud_polling",
"loggers": ["azure-storage-blob"],
"quality_scale": "bronze",
"requirements": ["azure-storage-blob==12.24.0"]
}
Loading