Key-value database became readonly after backup restore.

[LeeShan87]

I'm upgrading a very old (1.1.1) eKuiper to the newest (v2.2.0). In our test environment I've noticed a weird bug when the backup system restored the backups. The sqliteKV.db became readonly and the eKuiper service doesn't try to reconnect when this happens.

This produce more bugs. But I need time to collect them.

I've attached an http file which can help you reproduce the issue. (just remove the md suffix)
requests.http.md

An example request:

###
DELETE {{api}}/streams/{{streamId}} HTTP/1.1

# Response
# HTTP/1.1 400 Bad Request
# Content-Type: text/plain; charset=utf-8
# X-Content-Type-Options: nosniff
# Date: Wed, 09 Jul 2025 12:16:11 GMT
# Content-Length: 92
# Connection: close

# {
#   "error": 3000,
#   "message": "delete stream error: attempt to write a readonly database (1032)"
# }

Environment:

• eKuiper version (e.g. 1.3.0): v2.2.0

What happened and what you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

[ngjaying]

What backup system do you mean? It looks like the backup/restore process itself not setting correct permissions.

[LeeShan87]

We use an in-house backup system. I had to take time to figure out how the backup system works. It's nothing fancy - it's similar to:

/kuiper $ tar czf /tmp/backup.tar.gz -C /kuiper ./plugins ./data ./etc
/kuiper $ tar xzf /tmp/backup.tar.gz -C /kuiper

I made an updated gist as a PoC: PoC of eKuiper sqliteKV.db turn into readonly after backup restore.

The backup system uses the same user as the eKuiper service. The restored file permissions are the same as the original one.

The issue is with sqlite. If I recall correctly, when we open a new sqlite database connection, in the background we create a file descriptor to the concrete file. When we try to restore a backup, for example with tar or with cp, this file descriptor become invalid.

Sqlite won't be able to update the database because of this and falls back to read-only mode. All DB write operations will fail at this point. This means when I delete a stream or a rule I can still describe them and they will return the original rule or stream, but they won't be working.

This behavior was introduced in #921

Workaround is to restart the eKuiper service after backup restore. Alternatively, you could introduce an inode watch on the sqliteKV.db and restart the kuiperd, if change happened.