Merge pull request #20 from BottlecapDave/develop

BottlecapDave · web-flow · commit 1c5bdd404379 · 2025-05-23T06:47:52.000+01:00
Next release
diff --git a/README.md b/README.md
@@ -29,7 +29,7 @@ To install, place the contents of `custom_components` into the `<config director
 
 ## How to setup
 
-You will initially need to setup one or more [data sources](https://bottlecapdave.github.io/HomeAssistant-TargetTimeframes/setup/data_source). You'll then need to setup one or more [target timeframe](https://bottlecapdave.github.io/HomeAssistant-TargetTimeframes/setup/target_timeframe) or [rolling target timeframe](https://bottlecapdave.github.io/HomeAssistant-TargetTimeframes/setup/rolling_target_timeframe) sensors.
+It is recommended to consult the [getting started](https://bottlecapdave.github.io/HomeAssistant-TargetTimeframes/setup/getting_started) guide.
 
 ## Events
 
diff --git a/_docs/assets/target_timeframes_data_source.png b/_docs/assets/target_timeframes_data_source.png
diff --git a/_docs/assets/target_timeframes_sub_menu.png b/_docs/assets/target_timeframes_sub_menu.png
diff --git a/_docs/index.md b/_docs/index.md
@@ -29,7 +29,7 @@ To install, place the contents of `custom_components` into the `<config director
 
 ## How to setup
 
-You will initially need to setup one or more [data sources](./setup/data_source.md). You'll then need to setup one or more [target timeframe](./setup/target_timeframe.md) or [rolling target timeframe](./setup/rolling_target_timeframe.md) sensors.
+It is recommended to consult the [getting started](./setup/getting_started.md) guide.
 
 ## Events
 
diff --git a/_docs/services.md b/_docs/services.md
@@ -6,13 +6,14 @@ There are a few services available within this integration, which are detailed h
 
 ### target_timeframes.update_target_timeframe_data_source
 
-Updates the source data for a given targetframe period. This will replace any existing data for the source.
+Updates the source data for a given target frame. This will update any existing data for the source, but keep any existing data that hasn't been provided. It will also remove any data that is older than 1 day previous to today. For example, if today is 2025-05-17, then all data before 2025-05-16 will be removed.
 
 There are a collection of [blueprints](./blueprints.md#data-sources) available for loading popular data sources.
 
 | Attribute                | Optional | Description                                                                                                           |
 | ------------------------ | -------- | --------------------------------------------------------------------------------------------------------------------- |
 | `target.entity_id`       | `no`     | The name of the [data source last updated](./setup/data_source.md#data-source-last-updated) sensor whose underlying data is to be updated.                                                   |
+| `data.replace_all_existing_data` | `yes` | Determines if the provided data should replace all existing data. If not provided or false, then new data will be added to existing data and existing data will be replaced where start/end times match. |
 | `data.data`      | `no`    | The collection of data to update the data source with. This will override any previous data. For the target rate sensors to work properly, you should target having data for the whole of yesterday, today and tomorrow (e.g. if today is 2025-01-04, you should aim to have data from 2025-01-03T00:00:00 to 2025-01-06T00:00:00).
 
 The structure of the data should match the following
diff --git a/_docs/setup/getting_started.md b/_docs/setup/getting_started.md
@@ -0,0 +1,25 @@
+# Getting started
+
+## Installing the integration
+
+The first thing you need to do is install the new integration. Full instructions can be found on the [home page](../index.md#how-to-install).
+
+## Setting up data sources
+
+Once installed, you'll need to setup a data source that represents data coming from this integration. This can be done by [following the link](https://my.home-assistant.io/redirect/config_flow_start/?domain=target_timeframes) or searching for the integration in your integrations view. You'll need a data source for each unique set of data that your target timeframes will be based upon. This could be something like your electricity rates from an integration like [Octopus Energy](https://bottlecapdave.github.io/HomeAssistant-OctopusEnergy/) or carbon intensity from an integration like [Carbon Intensity](https://bottlecapdave.github.io/HomeAssistant-CarbonIntensity/).
+
+Full details of everything that can be configured can be found in the [Data Source](./data_source.md). An example for Octopus Energy, the name would be something like `Octopus Energy Import` with a source id of `octopus_energy_import`, but you can pick whatever you want as long as it's unique within the integration.
+
+![Data Source window](../assets/target_timeframes_data_source.png)
+
+## Configuring data source data
+
+Once you've configured a data source, you'll need to get data assigned to the data source so any associated target timeframes can be based upon that data. The data is added via an available [service](../services.md#target_timeframesupdate_target_timeframe_data_source). To make things easier, there are also a [range of blueprints](../blueprints.md#data-sources) you can install which add data from external integrations into a Target Timeframe given data source.
+
+## Setting up Target Timeframe sensors
+
+Now we have our data source representing our data and data syncing from an external source into Target Timeframes, it's time to now add a target timeframe sensor. Each type of sensor are added as a sub entry to our data source. You can see below what the menu looks like.
+
+![Integration sub menu](../assets/target_timeframes_sub_menu.png)
+
+For more detail on what each of these sensors provide, you should consult the detailed page on [target timeframe](./target_timeframe.md) and [rolling target timeframe](./rolling_target_timeframe.md).
diff --git a/custom_components/target_timeframes/entities/__init__.py b/custom_components/target_timeframes/entities/__init__.py
@@ -33,6 +33,16 @@ def apply_offset(date_time: datetime, offset: str, inverse = False):
   
   return date_time + timedelta(hours=hours, minutes=minutes, seconds=seconds)
 
+def is_target_timeframe_complete_in_period(current_date: datetime, applicable_time_periods: list | None, target_timeframes: list | None):
+  if applicable_time_periods is None or target_timeframes is None or len(applicable_time_periods) < 1 or len(target_timeframes) < 1:
+    return False
+  
+  return (
+    applicable_time_periods[0]["start"] <= target_timeframes[0]["start"] and 
+    applicable_time_periods[-1]["end"] >= target_timeframes[-1]["end"] and
+    target_timeframes[-1]["end"] <= current_date
+  )
+
 def get_fixed_applicable_time_periods(current_date: datetime, target_start_time: str, target_end_time: str, time_period_values: list, is_rolling_target = True):
   if (target_start_time is not None):
     target_start = parse_datetime(current_date.strftime(f"%Y-%m-%dT{target_start_time}:00%z"))
diff --git a/custom_components/target_timeframes/entities/data_source.py b/custom_components/target_timeframes/entities/data_source.py
@@ -2,7 +2,7 @@
 
 from homeassistant.core import HomeAssistant, callback
 from homeassistant.exceptions import ServiceValidationError
-from homeassistant.util.dt import (utcnow)
+from homeassistant.util.dt import (utcnow, now)
 
 from homeassistant.const import (
     STATE_UNAVAILABLE,
@@ -19,7 +19,7 @@
 from ..utils.attributes import dict_to_typed_dict
 from ..const import DOMAIN, EVENT_DATA_SOURCE
 from ..storage.data_source_data import async_save_cached_data_source_data
-from ..utils.data_source_data import validate_data_source_data
+from ..utils.data_source_data import DataSourceItem, merge_data_source_data, validate_data_source_data
 
 _LOGGER = logging.getLogger(__name__)
 
@@ -82,7 +82,7 @@ async def async_added_to_hass(self):
       _LOGGER.debug(f'Restored state: {self._state}')
 
   @callback
-  async def async_update_target_timeframe_data_source(self, data):
+  async def async_update_target_timeframe_data_source(self, data, replace_all_existing_data = False):
     """Update target timeframe data source"""
     result = validate_data_source_data(data, self._source_id)
     if result.success == False:
@@ -94,9 +94,21 @@ async def async_update_target_timeframe_data_source(self, data):
         },
       )
     
-    await async_save_cached_data_source_data(self._hass, self._source_id, result.data)
+    data_source_data = (
+      result.data 
+      if replace_all_existing_data
+      else merge_data_source_data(
+        now(),
+        result.data,
+        list(map(lambda x: DataSourceItem.parse_obj(x), self._attributes["data"]))
+        if "data" in self._attributes
+        else None
+      )
+    )
+
+    await async_save_cached_data_source_data(self._hass, self._source_id, data_source_data)
     
-    data_dict = list(map(lambda x: x.dict(), result.data))
+    data_dict = list(map(lambda x: x.dict(), data_source_data))
     self._attributes["data"] = data_dict
     self._state = utcnow()
     self.async_write_ha_state()
diff --git a/custom_components/target_timeframes/entities/target_timeframe.py b/custom_components/target_timeframes/entities/target_timeframe.py
@@ -49,6 +49,7 @@
   extract_config,
   get_fixed_applicable_time_periods,
   get_target_time_period_info,
+  is_target_timeframe_complete_in_period,
   should_evaluate_target_timeframes
 )
 
@@ -128,7 +129,8 @@ async def async_update(self):
     # Find the current rate. Rates change a maximum of once every 30 minutes.
     current_date = utcnow()
 
-    should_evaluate = should_evaluate_target_timeframes(current_date, self._target_timeframes, self._config[CONFIG_TARGET_TARGET_TIMES_EVALUATION_MODE] if CONFIG_TARGET_TARGET_TIMES_EVALUATION_MODE in self._config else CONFIG_TARGET_TARGET_TIMES_EVALUATION_MODE_ALL_IN_PAST)
+    evaluation_mode = self._config[CONFIG_TARGET_TARGET_TIMES_EVALUATION_MODE] if CONFIG_TARGET_TARGET_TIMES_EVALUATION_MODE in self._config else CONFIG_TARGET_TARGET_TIMES_EVALUATION_MODE_ALL_IN_PAST
+    should_evaluate = should_evaluate_target_timeframes(current_date, self._target_timeframes, evaluation_mode)
     if should_evaluate:
       _LOGGER.debug(f'{len(self._data_source_data) if self._data_source_data is not None else None} time periods found')
 
@@ -172,7 +174,9 @@ async def async_update(self):
           is_rolling_target
         )
 
-        if applicable_time_periods is not None:
+        is_target_timeframe_complete = is_rolling_target == False and is_target_timeframe_complete_in_period(current_local_date, applicable_time_periods, self._target_timeframes)
+
+        if applicable_time_periods is not None and is_target_timeframe_complete == False:
           number_of_slots = math.ceil(target_hours * 2)
           weighting = create_weighting(self._config[CONFIG_TARGET_WEIGHTING] if CONFIG_TARGET_WEIGHTING in self._config else None, number_of_slots)
 
diff --git a/custom_components/target_timeframes/sensor.py b/custom_components/target_timeframes/sensor.py
@@ -30,6 +30,7 @@ async def async_setup_entry(hass, entry, async_add_entities):
               }
             ],
           ),
+          vol.Optional("replace_all_existing_data"): bool,
         },
         extra=vol.ALLOW_EXTRA,
       ),
diff --git a/custom_components/target_timeframes/services.yaml b/custom_components/target_timeframes/services.yaml
@@ -147,4 +147,9 @@ update_target_timeframe_data_source:
           }
         ]
       selector:
-        object:
+        object:
+    replace_all_existing_data:
+      name: Replace data
+      description: Determines if the provided data should replace all existing data. If not provided or false, then new data will be added to existing data and existing data will be replaced where start/end times match.
+      selector:
+        boolean:
diff --git a/custom_components/target_timeframes/utils/data_source_data.py b/custom_components/target_timeframes/utils/data_source_data.py
@@ -1,4 +1,4 @@
-from datetime import datetime
+from datetime import datetime, timedelta
 from typing import Any
 
 from pydantic import BaseModel
@@ -78,4 +78,26 @@ def _validate_time(value: datetime, key: str, index: int):
   if value.second != 0 or value.microsecond != 0:
     return f"{key} second and microsecond must equal 0 at index {index}"
   
-  return None
+  return None
+
+def merge_data_source_data(current: datetime, new_data: list[DataSourceItem], current_data: list[DataSourceItem]):
+  merged_data: list[DataSourceItem] = []
+  merged_data.extend(new_data)
+  earliest_item = current.replace(hour=0, minute=0, second=0, microsecond=0) - timedelta(days=1)
+
+  if current_data is not None:
+    for item in current_data:
+      if item.end <= earliest_item:
+        continue
+
+      exists = False
+      for merged_item in merged_data:
+        if merged_item.start == item.start and merged_item.end == item.end:
+          exists = True
+          break
+
+      if exists == False:
+        merged_data.append(item)
+
+  merged_data.sort(key=lambda x: x.start)
+  return merged_data
diff --git a/mkdocs.yml b/mkdocs.yml
@@ -5,6 +5,7 @@ docs_dir: _docs
 nav:
   - Home: index.md
   - Setup:
+    - ./setup/getting_started.md
     - ./setup/data_source.md
     - ./setup/target_timeframe.md
     - ./setup/rolling_target_timeframe.md
diff --git a/tests/unit/target_rates/test_is_target_timeframe_complete_in_period.py b/tests/unit/target_rates/test_is_target_timeframe_complete_in_period.py
@@ -0,0 +1,162 @@
+from datetime import datetime, timedelta
+import pytest
+
+from custom_components.target_timeframes.entities import is_target_timeframe_complete_in_period
+
+@pytest.mark.asyncio
+async def test_when_both_lists_are_none_then_false_is_returned():
+  # Arrange
+  current_date = datetime.now()
+  applicable_time_periods = None
+  target_timeframes = None
+
+  # Act
+  result = is_target_timeframe_complete_in_period(current_date, applicable_time_periods, target_timeframes)
+
+  # Assert
+  assert result is False
+
+@pytest.mark.asyncio
+async def test_when_applicable_time_periods_is_empty_then_false_is_returned():
+  # Arrange
+  current_date = datetime.now()
+  applicable_time_periods = []
+  target_timeframes = [{"start": datetime.now(), "end": datetime.now() + timedelta(hours=1)}]
+
+  # Act
+  result = is_target_timeframe_complete_in_period(current_date, applicable_time_periods, target_timeframes)
+
+  # Assert
+  assert result is False
+
+@pytest.mark.asyncio
+async def test_when_target_timeframes_is_empty_then_false_is_returned():
+  # Arrange
+  current_date = datetime.now()
+  applicable_time_periods = [{"start": datetime.now(), "end": datetime.now() + timedelta(hours=1)}]
+  target_timeframes = []
+
+  # Act
+  result = is_target_timeframe_complete_in_period(current_date, applicable_time_periods, target_timeframes)
+
+  # Assert
+  assert result is False
+
+@pytest.mark.asyncio
+async def test_when_target_timeframe_is_within_applicable_periods_and_complete_then_true_is_returned():
+  # Arrange
+  current_date = datetime.now()
+  start_time = current_date - timedelta(hours=2)
+  end_time = current_date - timedelta(minutes=30)
+  
+  applicable_time_periods = [
+    {"start": start_time - timedelta(minutes=30), "end": start_time + timedelta(minutes=30)},
+    {"start": start_time + timedelta(minutes=30), "end": start_time + timedelta(minutes=60)},
+    {"start": start_time + timedelta(minutes=60), "end": end_time + timedelta(minutes=30)}
+  ]
+  
+  target_timeframes = [
+    {"start": start_time, "end": start_time + timedelta(minutes=30)},
+    {"start": start_time + timedelta(minutes=30), "end": end_time}
+  ]
+
+  # Act
+  result = is_target_timeframe_complete_in_period(current_date, applicable_time_periods, target_timeframes)
+
+  # Assert
+  assert result is True
+
+@pytest.mark.asyncio
+async def test_when_target_timeframe_start_before_applicable_periods_then_false_is_returned():
+  # Arrange
+  current_date = datetime.now()
+  start_time = current_date - timedelta(hours=2)
+  end_time = current_date - timedelta(minutes=30)
+  
+  applicable_time_periods = [
+    {"start": start_time, "end": start_time + timedelta(minutes=30)},
+    {"start": start_time + timedelta(minutes=30), "end": start_time + timedelta(minutes=60)},
+    {"start": start_time + timedelta(minutes=60), "end": end_time}
+  ]
+  
+  target_timeframes = [
+    {"start": start_time - timedelta(minutes=30), "end": start_time + timedelta(minutes=30)},
+    {"start": start_time + timedelta(minutes=30), "end": end_time}
+  ]
+
+  # Act
+  result = is_target_timeframe_complete_in_period(current_date, applicable_time_periods, target_timeframes)
+
+  # Assert
+  assert result is False
+
+@pytest.mark.asyncio
+async def test_when_target_timeframe_end_after_applicable_periods_then_false_is_returned():
+  # Arrange
+  current_date = datetime.now()
+  start_time = current_date - timedelta(hours=2)
+  end_time = current_date - timedelta(minutes=30)
+  
+  applicable_time_periods = [
+    {"start": start_time, "end": start_time + timedelta(minutes=30)},
+    {"start": start_time + timedelta(minutes=30), "end": start_time + timedelta(minutes=60)},
+    {"start": start_time + timedelta(minutes=60), "end": end_time}
+  ]
+  
+  target_timeframes = [
+    {"start": start_time, "end": start_time + timedelta(minutes=30)},
+    {"start": start_time + timedelta(minutes=30), "end": end_time + timedelta(minutes=30)}
+  ]
+
+  # Act
+  result = is_target_timeframe_complete_in_period(current_date, applicable_time_periods, target_timeframes)
+
+  # Assert
+  assert result is False
+
+@pytest.mark.asyncio
+async def test_when_target_timeframe_end_not_in_past_then_false_is_returned():
+  # Arrange
+  current_date = datetime.now()
+  start_time = current_date - timedelta(hours=2)
+  end_time = current_date + timedelta(minutes=30)
+  
+  applicable_time_periods = [
+    {"start": start_time - timedelta(minutes=30), "end": start_time + timedelta(minutes=30)},
+    {"start": start_time + timedelta(minutes=30), "end": start_time + timedelta(minutes=60)},
+    {"start": start_time + timedelta(minutes=60), "end": end_time + timedelta(minutes=30)}
+  ]
+  
+  target_timeframes = [
+    {"start": start_time, "end": start_time + timedelta(minutes=30)},
+    {"start": start_time + timedelta(minutes=30), "end": end_time}
+  ]
+
+  # Act
+  result = is_target_timeframe_complete_in_period(current_date, applicable_time_periods, target_timeframes)
+
+  # Assert
+  assert result is False
+
+@pytest.mark.asyncio
+async def test_when_target_timeframe_is_exactly_equal_to_applicable_periods_and_complete_then_true_is_returned():
+  # Arrange
+  current_date = datetime.now()
+  start_time = current_date - timedelta(hours=2)
+  end_time = current_date - timedelta(minutes=30)
+  
+  applicable_time_periods = [
+    {"start": start_time, "end": start_time + timedelta(minutes=30)},
+    {"start": start_time + timedelta(minutes=30), "end": end_time}
+  ]
+  
+  target_timeframes = [
+    {"start": start_time, "end": start_time + timedelta(minutes=30)},
+    {"start": start_time + timedelta(minutes=30), "end": end_time}
+  ]
+
+  # Act
+  result = is_target_timeframe_complete_in_period(current_date, applicable_time_periods, target_timeframes)
+
+  # Assert
+  assert result is True
diff --git a/tests/unit/utils/test_merge_data_source_data.py b/tests/unit/utils/test_merge_data_source_data.py

Original file line number	Diff line number	Diff line change
`@@ -30,6 +30,7 @@ async def async_setup_entry(hass, entry, async_add_entities):`
`30`	`30`	`}`
`31`	`31`	`],`
`32`	`32`	`),`
	`33`	`+ vol.Optional("replace_all_existing_data"): bool,`
`33`	`34`	`},`
`34`	`35`	`extra=vol.ALLOW_EXTRA,`
`35`	`36`	`),`