Continuous updates (experimental feature)

[jeroenterheerdt]

Let's discuss the continuous updates feature here!

[jeroenterheerdt]

#363 and #436 let's move discussion about how to handle various sensors / aggregates to this conversation please!

[jeroenterheerdt]

Two things come to mind:

1. How about precipitation? Shouldn't that be the actual rain rate and not the last/max of daily rain rate when continuous updates are on?
2. Should we be using reimann integral for solrad sensor? If so, how?

[jata1]

@jeroenterheerdt - thanks for introducing this feature. I have it switched on on prod and test environments so happy to help investigate and test.

solar radiation is tricky and I am not getting bucket calculations like I would expect but it is winter here in Oz so that is a factor.

I think there is something not right with how I setup my reimann integral sensor - see below.

raw data from the solar rad sensor on my weather station:

daily solar rad using a reimann integral (curve shape is good but daily total is out maybe by a factor of 100 or 1000):

[jata1]

So as a next step, I will create a new riemann sensor x100 and one x1000 to see if this is closer.

For reference - before I used a riemann I had a senor that simply added up the sol rad sensor and it looks like this - but even this was underestimating watering so this approach did not work either (for me).

[jata1]

Tried changing the riemann to kilo but it only changes the unit so not helpful (for me)

[jata1]

I think I might have found my issue with solar rad being under reported. I found this on the interweb...

The daily irradiation in Wh/m2 will be obtained as the sum of all hourly values in W/m2. For instance, if the irradiance is constant at 100 W/m2 during 10 hours, the daily total irradiation is 1000 Wh/m2=1 kWh/m2.

The same principle applies to 10-min values, but then the total needs to be divided by 6. So if the irradiance is constant at 100 W/m2 during 10 hours or 60 10-min periods, the daily irradiation is 100x60/6=1 kWh/m2 again, as expected.

So I think you need to setup the riemann integral as an hourly calculation (rather than daily) - this means that my original setup was under reporting by a factor of 24. So

I will see what happens now

[pumabaer]

Hi,
Can you share your configuration for your Riemann sum integral sensor?
Did you solve it using a helper or in the configuration.yaml?

[posixx]

Just to understand continous updates; if using sensors only including ET, continuous updates are not possible? What is the advantage of continuous updates?

[jeroenterheerdt]

If you use sensors only you should be able to use continuous updates. If you rely on a weather service we block you right now. Continuous updates update as soon as your weatherstation gives new values so should be highly accurate to your weather as it's up to the minute or second updates depending on how quickly your weatherstation operates..

[posixx]

Ok so this also works using passthrough module with ET sensor available?

[jeroenterheerdt]

it should.

[posixx]

Nice. It would be nice you can fix #430 (comment) so my log isn’t flooded between calculation time and 1st ET sensor update

[pumabaer]

Hello,

I have enabled continuous calculation.
Now, when it calculates in the evening at 11:50 PM, the time increases again. However, the time then looks much higher than during the continuous calculation.
Example:
Over the day, the bucket empties to -0.5. In the evening, it then jumps to -2.5.

Does the evening calculation need to be turned off?

[jeroenterheerdt]

It should not be necessary to turn it off but there is a bug right now that seems to cause this

[pumabaer]

Ah okay.
Let me know if you need any more information.

[jeroenterheerdt]

I think I solved the bug in the latest update (v2024.7.10) so you should be able to keep both continuous updates and automatic updates on.

[pumabaer]

[jeroenterheerdt]

please open an issue for this.

[pumabaer]

Hi,
How should the "solar radiation" sensor be configured?
Is "average" the right option?

[jata1]

Hi, How should the "solar radiation" sensor be configured? Is "average" the right option?

I have made good progress on calculated daily solar radiation. Here is my setup but you will need to adjust for whatever your raw/source solar rad sensor is outputting.

I am using 2 helpers to get this working. One calculates the riemann integral for solar radiation hourly as my sensor in in watts (watts hrs). The other is a utility meter that resets the the riemann sum data every day so you have a good daily calculation of total solar radiation.

In smart irrigation setup you use the utility meter and set average as the calculation method.

sol rad sensor output units is W/M2:

Riemann helper:

Utility meter:

Here is what the solar output daily looks like from my system based on the above. Just plug this sensor into smart irrigation.

[jeroenterheerdt]

@jata1 how easy was this to set up? should we even try to implement this all in the integration itself?

[pumabaer]

Thanks for sharing!
I will test it as well.
If it works as expected, it would be good to incorporate it directly into the integration.

[jata1]

@jata1 how easy was this to set up? should we even try to implement this all in the integration itself?

The root issue in my case is related to the weather station integration/data as it provides real time solar radiation in W/M2. This data needs to be manipulated to be used in the ETO calcs in smart irrigation.

Now I know what I am doing it is easy but took a bit of working through (as you know).

I will contact the guys that maintain the integration for my weather station to see if I can get them to add an additional sensor for daily accumulated solar radiation. solar radiation needs to be handled the same as rain/precipitation.

[pumabaer]

Hi, How should the "solar radiation" sensor be configured? Is "average" the right option?

I have made good progress on calculated daily solar radiation. Here is my setup but you will need to adjust for whatever your raw/source solar rad sensor is outputting.

I am using 2 helpers to get this working. One calculates the riemann integral for solar radiation hourly as my sensor in in watts (watts hrs). The other is a utility meter that resets the the riemann sum data every day so you have a good daily calculation of total solar radiation.

In smart irrigation setup you use the utility meter and set average as the calculation method.

sol rad sensor output units is W/M2:

Riemann helper:

Utility meter:

Here is what the solar output daily looks like from my system based on the above. Just plug this sensor into smart irrigation.

Are you sure that "average" is the right setting for solar radiation?

If I have a counter, the average value is always below the actual (last counted) measured value, right?
Value 1 + value 2, etc. / 2

Just a thought, I might be wrong about this.

[jata1]

@jeroenterheerdt - the good news with all of this work is that I finally have smart irrigation working properly as I am handling solar radiation correctly so the daily ETO calcs are working. The sensor for total daily solar radiation looks correct when comparing to the solar radiation raw data sensor.

[jeroenterheerdt]

Love it. Would you be willing to write up a step by step process of what you did to configure your sensor group and input sensors for continuous updates? I am planning to rewrite the readme (move to wiki) and I think it would be good to have what you learned in there!

[jata1]

Happy to do a write up but not quite there yet.

My testing over the last week has not been using continuous updates as I wanted to test one thing at a time. I now have daily calcs working correctly with my solar radiation sensors. Now I am switching to continuous and I will see what I find and update this thread.

[jata1]

After switching to continuous updates, I have some new errors in the log and a couple of observations.

My setup has two zones (front and back) each with its own sensor group. The sensor groups are identical except one has precipitation set as a static value of 0 (as this zone is under cover). However, the amount of data collected for the two zones are very different - see image below from the last hour since I switched to continuous.

Diagnostic file attached...
config_entry-smart_irrigation-01J3NXJFTGQR2QDP3BMYX2SF83.json

[jeroenterheerdt]

Wait, that's weird! The sensor groups are the same except for the one static? I am wondering if somehow the static zone gets skipped?

[jata1]

So I have been playing with continuous updates and there is something strange going on. I have a feeling it is related to having two sensor groups that use the mostly the same sensors.

I had it working but only one zone was actually updating continuously. So I deleted the sensor group that I assumed was not working and set it up new.

Now this one works but the other zone is not updating. It's very confusing.

My next test is to use the same sensor group on both zones. I will report back

[jeroenterheerdt]

Is there anything in the logs that can help?

[jata1]

Nothing obvious in the logs. I don't understand why there is much more data being collected for one sensor group compared to the other. I think this is a clue to an underlying issue.

Right now I am running a different test. I have set both zones to use the same (newly created) sensor group so I will see what happens with this and report back.

[jata1]

I'm on 2025.5.4 and I think that is latest apart from the 2025.6 betas.

I will re-download 2025.5 and stop/start HA container to clear cache etc

[jeroenterheerdt]

Sorry if I wasn't clear. Please install the latest beta, turn on debug logs and set up the aggregations as follows:

1. Disable continuous updates and use riemann sum as agg for solar radiation. Confirm riemann sum is calculated correctly.
2. Enable continuous updates. Make no further changes. Check logs.
3. Now map current precipitation. Check logs.

[jata1]

OK here are results of test 1.

• version: 2025.6.0-beta5
• continuous: off
• drainage rate: blank/null
• solrad sensor: riemann sum energy in (daily total) = 3026 Wh/m2 - This was used in the calcs and converted to MJ/day/m2
• smart irrigation solrad conversion: 3026 Wh/m2 = 261 MJ/day/m2 - This looks correct to me (1Wh = 0.0036 MJ x 24 hrs)

Note that I turned off continuous and reset all data in the afternoon so this is not a full days data. I will run with these settings again today and share the new log tomorrow.

At the moment there is something wrong as I am not getting enough watering (bucket is -0.5mm - so just a few seconds of watering). Ever since drainage rate has been introduced to smart irrigation I have had issues.

Log:
ha-smart-test1.log

config:
config_entry-smart_irrigation-01J3NXJFTGQR2QDP3BMYX2SF83 (2).json

[jata1]

OK here are results of test 1 second day.

• version: 2025.6.0-beta5
• continuous: off
• drainage rate: blank/null
• solrad sensor: riemann sum energy (daily total) - Issue with the data being used not correct. Solar Energy total was 3,100 Wh/m2 but data used by smart irrigation was 261 - so it looks like it is converting from Wh/m2 to MJday/m2
• solrad sensor units: W/m2 - Note this is an issue as this is power (no time dimension) and my data is Wh/m2
• solrad sensor agg: last - This makes sense for my setup as my solrad sensor is energy total for the day
• smart irrigation solrad conversion: With these settings there is no conversion in the logs

With this config I see two strange things happening as mentioned above

1. there is something weird happening to the source sensor vs. smart irrigation (3100 sensor, 261 smart irrigation) - converted to MJDay?
2. I am not seeing any conversion on the solrad data but this might be intentional and related to point 1

It is different behaviour to what happened yesterday so I think I must have changed config from Average to Last?

Today I will keep my settings exactly the same to see if I get the same results

Log:
ha-smart-test2.log

config:
config_entry-smart_irrigation-01J3NXJFTGQR2QDP3BMYX2SF83 (2).json

[jata1]

Test 1 third day.

• version: 2025.6.0-beta5
• continuous: off
• drainage rate: blank/null
• solrad sensor config: W/m2 Last (note there is no setting for Wh/m2)
• solrad used by smart: 246 (MJday)
• solrad input sensor: 2850 (Whday)

So same behaviour as yesterday and I think I know what is going on.

1. smart irrigation sensor config is is set to W/m2. This is not correct for my setup as my sensor is Wh total for the day (riemann sum)
2. smart irrigation converts the last value of sensor 2850 to 246 by multiplying by 0.0036 to MJ then multiplying by 24hrs in a day
3. this value of 246 is then used in the ETo calculations

But I don't think this is correct as the 2850 is already total solar energy for the day in Wh/M2. The conversion is multiplying by 24x incorrectly.

So I think there is a need to add / change the solrad sensor config so there is an option for Wh/m2.

With current config, my understanding is that the ETo calculation is using a solrad input that is overestimated by 24x.

But I am not getting enough watering and that does not make sense given solrad is overestimated.

Here is a summary of my solrad sensor over the last 3 days. Left chart is solar radiation (power) and right chart is daily total solar radiation (energy) using a riemann sum.

Log:
ha-smart-test3.log

config:
n/a - same as yesterday

[clarejor]

I think what we need to check is whether the new built-in riemann sum matches your manually created total radiation sum. Try using your radiation in W/m2 (left chart) as input to smart irrigation, with the aggregation type set to riemann sum (continuous off).

[jata1]

OMG - I totally missed there was a riemann sum option for the sensor config. I will start testing again - this time with correct config

[clarejor]

As far as the low watering, your max temperature is ~16C in the latest log you posted. The calculated ETo was 3.3mm, and there was 1.7mm of precipitation, bringing the total bucket delta to 1.7-3.3=-1.6. Your solrad value is also too high due to the *24 factor, so your ETo should actually be even lower. But I think those values you're getting are reasonable. For comparison, my max temps have been around 30C and I generally get ETo values of about 4.0mm/day, resulting in irrigation being triggered every 3-4 days (my automation only triggers irrigation when bucket values are <= -15mm)

[jata1]

OK. Good to know. Maybe I have been overwatering for the last year or so. haha

[clarejor]

I also noticed the time range for your calculation was only 15 hours instead of 24:
first_retrieved_at: 2025-06-09 07:02:54.470446, last_retrieved_at: 2025-06-09 22:11:45.584359
Is that because you triggered a manual calculation at 07:02? This makes the resulting values smaller because they are multiplied by the fractional day range (15/24 = 0.63)

[jata1]

not intentionally but HA had an issue and was smashing my server so I stopped/deleted and recreated the docker container so it is possible smart irrigation reset

[jata1]

Test 2 day 1

Note: Not a complete set of daily data

setup:

• version: 2025.6.0-beta5
• continuous: off
• drainage rate: blank/null
• solrad sensor config: W/m2
• solrad sensor agg: Riemann sum
• update interval: hourly

looks like:

• every hour my sensor in W/m2 (point in time) is converted to MJday/m2
• at 11pm the converted hourly data (in MJday/m2) is then used in a reimann sum calculation
• the reimann sum output is then used for the daily ETo calculation

thoughts on this approach are:

• Overall I think it makes sense although i'm not sure about the conversion from W to MJday
• this approach will not be accurate as the sampling rate of hourly not enough (especially when weather is sun/cloud)

I will report back tomorrow with a full days data

Log:
ha-smart-test4.log

config:
config_entry-smart_irrigation-01J3NXJFTGQR2QDP3BMYX2SF83 (2).json

[clarejor]

yeah, looks like the unit conversion is incorrect. It's converting to MJ/day/m2 at the time the data is collected, then multiplying by time in seconds during the riemann sum aggregation. It should be storing the values in W/m2 and applying the riemann on that, which effectively makes it J/m2, then dividing by 1e6 to get MJ/m2. So it's a bug.

[jata1]

Test 2 day 2

With a complete set of daily data.
Weather conditions = full sun, no cloud. mid winter

setup:

• version: 2025.6.0-beta5
• continuous: off
• drainage rate: blank/null
• solrad sensor config: W/m2
• solrad sensor agg: Riemann sum
• update interval: hourly

findings/summary:

• same setup as yesterday with a full set of data for the day
• smart irrigation calculated solrad = 267 MJday
• separate sensor calculated solrad = 3088 Whday
• converting sensor to MJday = 3088 x 0.0036 = 11.12 MJday
• multiply sensor value by 24hr = 11.12 x 24 = 267
• Confirming bug in smart irrigation where solrad is over estimated by 24x
• Also confirming that without the calc bug then very close between sensor and smart irrigation

Other comments:

• I am getting a few errors in my logs. Have included in the log file. Not sure if these are helpful
• looks like on a day where weather is consistent (sunny or cloudy), the calculated solrad reimann will work ok
• I still have concerns with its accuracy when weather is variable

Log:
ha-smart-test5.log

config:
same as yesterday

[jata1]

Also sharing some data that compares solar power and solar energy using to different sensors/integrations (weather station and roof solar.

What this is confirming is that:

• I do have good solrad input data in either W/m2 (power) or Wh/m2 (energy)
• reimann sum is calculating correctly as it is same as roof solar energy sensor

[clarejor]

what I'm doing currently for maximum accuracy is using continuous mode with solrad in W/m2 and aggregation average. my weather station updates once per minute so there's usually only a single data point per update, making the aggregation selection not really matter anyway.

[jata1]

Sounds good. Do you have enough info from me with current config?

I will switch to continuous mode and monitor for the next few days.

My continuous config will be using Last as the aggregation for all sensors

[clarejor]

yes, there's enough info here for a fix. thanks for all the test data!

[jata1]

I have just changed my sensor group config to one I setup for continuous. It uses Last for aggregation for all sensors.

As soon as I do this change. I am getting a big problem with HA saying the recorder database is corrupt. The only fix I have found is to restore from a backup.

So this is a serious issue. Triggered by making a simple change to my smart irrigation setup - changing sensor group for a zone.

Can you reproduce?

[clarejor]

I'd make a separate issue for it with attached debug logs.