History of the Alarmo

[MarcelS1988]

It would be great if there is a alarm history where we can see all the action about the Alarmo.
something like this:

So here you can always see what happend, when the alarm is armed and by wo.
Which sensor are got alarm and so on....

[nielsfaber]

I think this is already integrated in HA in the logbook integration.
The only limitation of the logbook page (via the sidebar) is that you can filter on a single entity at a time, so either your alarm entity or any sensor.
There is however a logbook card for which you could define which entities are visible.
Wouldn't this be sufficient for your use?

[MarcelS1988]

Hi, thanks for the answers.
But is not really what I mean.

There is only visible from wich device the alarm is armed of disarmed.
It would be great to read out that there is a name wo has input the code you created inside Alarmo.

And if we have a next tab here:

That we hare can see all the action about Alarmo.

• Wo has armed/disarmed
• When there is an alarm wich sensors?
  So, you have an easy overview about it.

Hope you liked the idea, and hope it is possible to do :)

[canakoglu]

logbook card

+1

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 7 days

[MarcelS1988]

Do I need to make a feature request?

[Casadomoticap]

This would be great! :)
My support on this

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 7 days

[snickers2k]

+1
i had 2 false-alarms and i dont even have any idea how to look in the logbook for what the heck did trigger the alarm...

i tried to search in logbook for alarmo entity, but there is no logbook entry at all ...

we absolutely need

• history of alarms (time+date) and the reason (triggering device)
• history of users+time+date arming/unarming alarmo

probably one of the most important things missing in alarmo. no chance an alarm-system is missing that O_o

so to answer your question @nielsfaber , no it's not sufficient for our use :/ ^^

[f18m]

+1 also from me

I I'm in the same situation of @snickers2k. I have some faulty sensor triggering false alarms and I need to create a history of these false alarms to understand what I need to tune.

@nielsfaber I read about your suggestion to use the logbook (https://www.home-assistant.io/integrations/logbook/). However I believe that if I setup a filter on that component to include "only" all my alarm sensors, then I won't be able to see any more events related to non-alarm stuff that happen in my house, right?

I'm interested in retaining the default logbook behavior of HA (record and show all entity statuses) but it would be tremendously useful to have a logbook specific to the alarm system only.

Btw if you are open to contributions I might be able to help developing such feature since I'm kind of confident with Python and asyncio (although not that much with the HA codebase)... probably I would need some guidance though.

@MarcelS1988 would you mind re-opening this issue? thanks!

[MarcelS1988]

@nielsfaber Please reconsider this feature.

[nielsfaber]

I hereby reconsider it 🙂
It's definitely a useful addition, but I think the effort involved is quite significant.
Unfortunately I have a long to-do list and very few time to spare lately.

@f18m Any help is welcome! Both in cleaning up existing code or adding new features.
If you have ideas about some storage method (outside of the alarmo configuration storage file I would say) to register any sensor/alarm state changes for last X days (I guess also auto-wiping older data), that would be very helpful.
I think the frontend/visualization part is quite easy to add once the data is there.

[f18m]

@f18m Any help is welcome! Both in cleaning up existing code or adding new features. If you have ideas about some storage method (outside of the alarmo configuration storage file I would say) to register any sensor/alarm state changes for last X days (I guess also auto-wiping older data), that would be very helpful.

I've been thinking a bit about this topic.
My idea would be to use SQLite. Having a look around, I see it's used also by other HomeAssistant integrations/addons. For example Frigate is using it (see https://github.com/blakeblackshear/frigate/blob/dev/frigate/models.py) through the library pee-wee (https://docs.peewee-orm.com/en/latest/).
This software layer allows for some abstraction from the specific database (e.g. pee-wee supports also Postgres, MariaDB, etc). I don't think Alarmo needs will ever require a database more "powerful" than SQLite, but you never know.
SQLite is of course also the default database of HomeAssistant.

@nielsfaber what do you think about the stack pee-wee+SQLite for the "history" feature?

I think the frontend/visualization part is quite easy to add once the data is there.

That's really good. I'm mostly used to coding backends. Frontends are something I have done only sporadically and never created one from scratch... so good to know it should be easy :-)

[nielsfaber]

@f18m
I would like to stress that this should remain a small feature for Alarmo so we should avoid adding lots of code over multiple files for implementing it. Less code means less maintenance, given that the backend of alarmo currently is <5000 lines we should limit ourselves to a 500-1000 lines implementation for the feature.

HA already has its own classes for handling storage. I was hoping no database libraries would have to be added for implementing this feature, just include some methods from HA. I know that HA uses a mix of SQL database and JSON text files for storage, and if I remember correctly these classes only cover the JSON storage method,
Looking into the implementation of the history/recorder component of HA I see a lot of code here related to database handling, this seems to be HA's SQL implementation. If some hook could be made towards this component to provide database access for this feature it would be preferable.

Otherwise the method used in the Frigate component could be a useful alternative, but the implementation seems to contain quite some extra code related to storage handling which scares me off a bit.

[f18m]

@f18m I would like to stress that this should remain a small feature for Alarmo so we should avoid adding lots of code over multiple files for implementing it. Less code means less maintenance, given that the backend of alarmo currently is <5000 lines we should limit ourselves to a 500-1000 lines implementation for the feature.

yeah right, good point.

HA already has its own classes for handling storage. I was hoping no database libraries would have to be added for implementing this feature, just include some methods from HA. I know that HA uses a mix of SQL database and JSON text files for storage, and if I remember correctly these classes only cover the JSON storage method, Looking into the implementation of the history/recorder component of HA I see a lot of code here related to database handling, this seems to be HA's SQL implementation. If some hook could be made towards this component to provide database access for this feature it would be preferable.

Well I had a quick look and it seems to me that HomeAssistant is using SQLAlchemy (https://www.sqlalchemy.org/) which seems very similar (at least conceptually) to the pee-wee solution adopted by Frigate.
Since I never used any of these, I don't have any personal experience and I'm fine with both.
Probably SQLAlchemy is the most HA-way.

Regarding the code at https://github.com/home-assistant/core/blob/dev/homeassistant/helpers/storage.py, it seems to be that, as you mentioned, it's mostly concerned about reading/writing Python dictionaries (in form of versioned JSON blobs).
An example of use is e.g. https://github.com/home-assistant/core/blob/dev/homeassistant/components/google/store.py#L49
However my feeling (still looking around) is that this approach does not allow to perform queries on the storage based on the properties of the Python dictionary.
Do you think it would be useful/feasible to let the user search/query for e.g. sensor, date, action type (alarm trigger, alarm activation, e.g.)?

[f18m]

hi @nielsfaber ,
any further comment on this topic?
Time permitting I would like to start an implementation draft this or next week... thanks!

[f18m]

hi @nielsfaber ,
sorry to bother you. As I have some spare time, I would like to help on this topic / implementation. Can you provide me some hints about where to start ?

In particular I had a first look at the Alarmo source code and I wonder: would the EventHandler class be a good place to hook the storage of the event itself in some place (I would like to log to a file to get started?)

[nielsfaber]

@f18m I don't really have much pointers to help you to get started.
You can use async_dispatcher_connect(hass, "alarmo_state_updated", some_callback_function) to receive state change information of any alarmo entity. Something similar would have to be added to receive information when a sensor state changes.
This can be in a separate class, with some initializer that is called from the __init__ file (same as all the other classes).
I believe the complexity is in the database/storage side and keeping it lightweight. I think storing in a text file is not optimal due to many read/write cycles. HA uses text files for configuration only, not historic info. I think we should follow the same principles.

As an alternative to storing state changes in a database, perhaps the relevant info could be retrieved from the HA logbook or history component. To me it seems this feature will/could look pretty similar to the HA logbook, but with multiple entities (all alarm + sensor entities).

[f18m]

hi @nielsfaber ,
sorry for long delay, I've been sick and sidetracked by other stuff.

@f18m I don't really have much pointers to help you to get started. You can use async_dispatcher_connect(hass, "alarmo_state_updated", some_callback_function) to receive state change information of any alarmo entity. Something similar would have to be added to receive information when a sensor state changes. This can be in a separate class, with some initializer that is called from the init file (same as all the other classes).

ok thanks, that's already a very good start. I'd like to start simple and then grow with features step by step. So I'll start by storing somewhere just the alarmo state changes.

I believe the complexity is in the database/storage side and keeping it lightweight. I think storing in a text file is not optimal due to many read/write cycles. HA uses text files for configuration only, not historic info. I think we should follow the same principles.

Totally agree with that. I mentioned storing in a text file just as initial debug step.

As an alternative to storing state changes in a database, perhaps the relevant info could be retrieved from the HA logbook or history component. To me it seems this feature will/could look pretty similar to the HA logbook, but with multiple entities (all alarm + sensor entities).

That's a possibility I guess. I'm not sure however how smart can be the queries launched on the logbook database (actually it looks like the integration doing the recording is the 'recorder', see https://www.home-assistant.io/integrations/recorder ).
Let me explain it better.
I think the main use cases we should try to address are basically those listed in #642 (comment):

1. history of alarms (time+date) and the reason (triggering device)
2. history of users+time+date arming/unarming alarmo

To address use case #1 using the standard HA recorder / logbook, we need to query the state of the sensors ONLY when the alarm was armed (the condition "alarm is armed" will create a series of time windows) and find out which sensor did trigger the alarm in those time windows. I'm not sure if the standard recorder/logbook is well-suited for such kind of queries.

Addressing use case #2 is probably easier. If I go to the HA logbook today it's easy to find the time+date and see the arming/unarming state of Alarmo. The user information (who armed / disarmed) is missing though.

Final point: it might be a nice feature to have a custom retention time, different from the standard HA recorder/logbook retention time (the 'purge_keep_days' setting).

For all reasons above I believe that having a separate (lightweight) DB for Alarmo does make a lot of sense.

Thoughts?

Then I have a final question for you: I'm experimenting with the code changes in my fork. What's the best way to test the changes for the Alarmo component? Should I scp/rsync/just-copy somehow the modified Python files to the /usr/share/hassio/homeassistant/custom_components/alarmo folder ?
Any suggestion?

Thanks again

[vincenzocoletta]

Hi, I'd quote the request and also signal what strangely happened to me: last night the Alarmo suddenly changed in "disarmed" around 3.30am and nobody of the two people having codes (me and my wife) have disarmed it (and unless we discover to be sleepwalkers we didn't...).
Now the question is, how can we signal a critical issues (in case it is) to help the development in case it disarmed without any human insertion from the owners of the codes? I'd then kindly quote the request but also ask if any help can be provided in order to be sure that we can sleep safe and nothing critical to the system happened and or can happen in the future. I mean, let me please know if there are any logs to verify or special condition to know (ie.: it automatically disarm when... or after... or in case of repetitive events...) to identify the cause of those sudden, mysterious disarming or, at least, esclude an issue of the system.
Many, many thanks and regards,
V!

[f18m]

Hi @nielsfaber ,

Then I have a final question for you: I'm experimenting with the code changes in my fork. What's the best way to test the changes for the Alarmo component? Should I scp/rsync/just-copy somehow the modified Python files to the /usr/share/hassio/homeassistant/custom_components/alarmo folder ? Any suggestion?

I'm back on this topic (history feature).
Today I managed after some pain to setup my dev environment.
I'm posting here the steps I took in the hope they will be useful to others:

• I started HomeAssistant core devcontainer (following https://developers.home-assistant.io/docs/development_environment/)
• I added a mount path in ,devcontainer/devcontainer.json like:

  "mounts": [
    "source=/home/fmontorsi/Documents/alarmo/custom_components/alarmo,target=/workspaces/ha-core/homeassistant/components/alarmo,type=bind,consistency=cached"
  ]

• I created an entry for 'alarmo' to homeassistant/generated/integrations.json
• I launched HomeAssistant inside the devcontainer
• I logged into localhost:8123 and added "Alarmo" custom integration

After these changes I was able to see my "print()" commands inside the Alarmo code.
Now my next investigation steps will be:

1. how to mount inside the HomeAssistant devcontainer the Alarmo card
2. how to simulate binary sensors inside the HomeAssistant devcontainer
   since these 2 things will help me to configure and setup a realistic scenario to test my own changes.

Perhaps it could be worth adding a doc in this repo with the above procedure to explain how to test changes before submitting PRs ?
Do you have any suggestion related to points 1 and 2 ?

[f18m]

Update: about point #1: it turns out to be much simpler than I expected. Apparently Alarmo frontend is 1 single Javascript file (cool!) so I just ran inside the devcontainer:

mkdir -p /workspaces/ha-core/config/custom_components/alarmo/frontend/dist/

ln -sf /workspaces/ha-core/homeassistant/components/alarmo/frontend/dist/alarm-panel.js  /workspaces/ha-core/config/custom_components/alarmo/frontend/dist/alarm-panel.js

and now when I go to http://localhost:8123/alarmo I do see the Alarmo UI!! :)

[f18m]

Update: about point #2: I actually found that I can successfully create an MQTT binary_sensor and use it in Alarmo configuration to test my changes.
Here's the steps I did:

• I added inside the devcontainer in config/configuration.yaml the snippet:

mqtt:
  - binary_sensor:
      state_topic: "fake/sensor1"
      name: "fake sensor1"
      device_class: door

• Then from outside the devcontainer I started an MQTT broker in the same docker network used by VSCode devcontainer, which is the default docker network named "bridge":

docker run -d --publish 1883:1883 --network=bridge \
		--volume /tmp/my-mosquitto.conf:/mosquitto/config/mosquitto.conf \
		eclipse-mosquitto:latest

where the file /tmp/my-mosquitto.conf contains:

# see https://mosquitto.org/man/mosquitto-conf-5.html

protocol mqtt
user root
log_dest stdout
allow_anonymous true

log_type error
log_type warning
log_type notice
log_type information

log_timestamp_format %Y-%m-%d %H:%M:%S
persistence true
persistence_location /data/

listener 1883
protocol mqtt

sys_interval 1

• log into the Mosquitto container and take note of its IP address:

docker exec -ti <mosquitto_broker_container_id> sh
ip a

• start HA inside devcontainer
• Add 'MQTT' as integration and indicate as MQTT broker the IP address of the Mosquitto container (172.17.0.xxx typically)
• check that the "fake sensor1" is now available inside HA
• using 'MQTT explorer' utility: I attached to the MQTT broker using the exposed port (so I indicated 'localhost:1883') and then published on the topic 'fake/sensor1' the payload 'ON' or 'OFF'
• check that inside HA the fake sensor reacts to the publish command from MQTT explorer

I'm writing this down in the hope that this helps other to provide contributions to Alarmo!

[f18m]

@nielsfaber ,
I have a basic question for you: subscribing to the bus "alarmo_event" I'm able to print all the relevant Alarmo events on stdout. Now I'm trying to move to some implementation where those events get stored on disk.
I tried first with a basic dump to file (no database, just a plain file). I now get this warning from Home Assistant: Detected blocking call to open inside the event loop by integration alarmo at .... .

Should I be using instead hass.async_create_task() and asyncio framework?
I believe so but better to double check with you..

[f18m]

I think the frontend/visualization part is quite easy to add once the data is there.

@nielsfaber I have created a first implementation of the History backend. In my local testing it seems to be working just fine... would you be able to help for the frontend part?