SWAP usage running HA OS

[bcutter]

Since switching hardware from Pi 4 to Pi 5 (NVMe SSD) I noticed HA OS prefers to use almost all available SWAP space instead of using RAM (like before on the Pi 4 with a quite slow disk storage).

In general I am wondering whats the reason for this behavior. I mean there's plenty of disk space and it is quite fast, but also there's plenty of (even faster) RAM available not used at all.

During nightly backup it uses even all available SWAP space (100 % usage).

HA OS 14.2.

[Impact123]

You can run this via developer SSH or keyboard & monitor to see what uses the SWAP. I got most of the command from here.

for file in /proc/*/status ; do awk '/VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; done | sort -k 2 -n | tail -n50 | column -t

Does this actually cause any problems? SWAP usage itself is not necessarily an issue.

[Impact123]

I suppose all you can do is wait until they see our conversation here. They are usually notified about every message hence why I want to keep my messages limited.

[bcutter]

They are usually notified about every message hence why I want to keep my messages limited.

I'm not sure this is the case for discussions. For issues quite sure, but this is a discussion so... 🤷‍♂️

[Impact123]

I actually did not notice this was a discussion 👀

[bcutter]

Yeah so... back to the original question: how to have a (HA OS) dev have a look at this?

[bcutter]

3 weeks later... no one had a look...

On HA OS 15.2 meanwhile. Still put my bets on Studio Code Server.

[bcutter]

As some crazy things pop up recently, like (within the last hour)

• InfluxDB addon stopping unexpectedly, entering an endless restart loop
• Studio Code Server addon complaining about a repeatedly failed Home Assistant Language Server

(so @Impact123 answering your

Does this actually cause any problems?

YES, it seems like it creates some (serious) issues meanwhile ⚠️😭)

https://community.home-assistant.io/t/influxdb-restarting/485508/7

So I'd like to urently rule out the 100 % SWAP usage is a root cause for this.

(host not rebooted since 2025-02-17)

How to make any progress here?

Is it maybe a known issue in HA OS 14.2 with Raspberry Pi 5?

[Impact123]

My recommendation is still the same. Set up developer SSH or connect a keyboard and monitor and run my command to see what uses the swap. If you visit the discord server I can help you with that.

[bcutter]

Give me a hint please (discord server). Setting up dev. SSH is probably more interesting.

[Impact123]

You can find the discord server here: https://www.home-assistant.io/help/#communication-channels
The developer SSH instructions here: https://developers.home-assistant.io/docs/operating-system/debugging/
A workaround to get to the OS through the Advanced SSH Addon here https://gist.github.com/Impact123/fb086b391f7d14cb3515144fcbe4785e#full-access-to-files

[bcutter]

Ah great link! I use the Advanced SSH addon already for a while and I'm used to e. g. enter a specific docker container, but wasn't aware this is also possible to gain "direct access" on a host/OS level.

[bcutter]

Running the docker run --privileged --pid=host -it alpine:latest nsenter -t 1 -m -u -n -i sh command gave me this, which makes be feel a bit uncomfortable:

Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine
6e771e15690e: Already exists 
Digest: sha256:a8560b36e8b8210634f77d9f7f9efd7ffa463e380b75e2e74aff4511df3ef88c
Status: Downloaded newer image for alpine:latest

What did happen here? Something's got replaced by that command? It seems we're operating at the core of HA OS' heart here so I want to make as sure as possible nothing gets screwed up. Thanks for explanation @Impact123.

The image is now reported to be created 8 minutes ago (by that command), container name is gifted_tu which I don't know if this is the default or an indicator of something bad already.

Edit: after exiting, the container status now is

[Impact123]

This should not cause any issues but you can also remove the container again as we just need it temporarily. I'm somewhat sure that supervisor removes it during its housekeeping procedures anyways.
I forgot to add --rm to the command to remove it automatically but it's there now. If you need help not directly related to the issue please come to discord. I don't want to pollute this issue any more than it is. Note that portainer isn't really supported or recommended.
Neither is what we do here but still.

[bcutter]

Yes. For debug reasons obviously it was/is necessary (even the dev SSH access is the official but also quite hacky way to achieve this).

See you @ Discord.

[rct]

I just stumbled across this, two comments:

@Impact123 - Thanks for that gist, there was a bunch of interesting bits in there. I hadn't see gdu before (was using ncdu) and didn't realize nsenter could be used from the advanced SSH add-on.

@bcutter - I've had a problem with InfluxDB (via frenck's add-on) in the past where the DB gets too large for the available virtual memory, so it keeps trying to allocate more memory until it fails and exits. The auto-restart on the add-on container just starts the process again. I wasn't able to determine why it was trying to load the whole DB into memory, it might have been trying to recover from an unclean shutdown or the DB just go too big because there was no compaction/resampling/expiration set up by default. Long term statistics came along and I gave up on InfluxDB on Home Assistant.

[bcutter]

That's interesting as my latest 2 trigger for having a look at the SWAP topic were because of InfluxDB (#3956 (comment)). But the database size (~ 15 GB) outperforms the available memory (8 GB) by far already for a long time. And HA internal LTS still has its "things" which is why I continue to keep my long-term data in the InfluxDB.

[bcutter]

Until someone (HA dev/expert) can explain why udisksd is consuming so much virtual memory/swap space...

#3956 (reply in thread)

...I can add a latest experience and current suspicion: it feels like the Studio Code Server addon might be the root cause. The addon went crazy yesterday when/after using it on few HA Companion app iOS devices (iPhone, iPad) yesterday and the consumed SWAP space went down back to normal once the addon was stopped (started again but not used directly afterwards):

Now everything's "normal" again/currently (system):

And as you can see, the Studio Code Server addon memory consumption directly correlates with the system SWAP usage:

---

Interestingly I came across a GitHub issue on the addons' behaviour related to memory consumption, but can't find that issue anymore. It's not at https://github.com/home-assistant/addons/issues. But there are other user reports regarding this like https://community.home-assistant.io/t/studio-code-server-add-on-memory-usage/829842

As this is just a (quite likely or based on the data shown above pretty likely) assumption, I still tend to clarify about the udisksd topic.

[Impact123]

The Code Server addon is not part of the HA addons repository. You can find the memory related bug report here: hassio-addons/addon-vscode#565