wazuh-manager fails to start

[paul8046]

To reproduce:

1. Create a new container with fresh mount points (either docker volumes or local filesystem mounts).
2. Start the container

Symptoms:

Error during the database migration. Restoring the previous database file
Error details: (sqlite3.OperationalError) unable to open database file
(Background on this error at: https://sqlalche.me/e/20/e3q8)
Traceback (most recent call last):
  File "/var/ossec/framework/python/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 145, in __init__
    self._dbapi_connection = engine.raw_connection()
  File "/var/ossec/framework/python/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3292, in raw_connection
    return self.pool.connect()
  File "/var/ossec/framework/python/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 452, in connect
    return _ConnectionFairy._checkout(self)
  File "/var/ossec/framework/python/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 1269, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/var/ossec/framework/python/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 716, in checkout
    rec = pool._do_get()
  File "/var/ossec/framework/python/lib/python3.10/site-packages/sqlalchemy/pool/impl.py", line 169, in _do_get
    with util.safe_reraise():
  File "/var/ossec/framework/python/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/var/ossec/framework/python/lib/python3.10/site-packages/sqlalchemy/pool/impl.py", line 167, in _do_get
    return self._create_connection()
  File "/var/ossec/framework/python/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 393, in _create_connection
    return _ConnectionRecord(self)
  File "/var/ossec/framework/python/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 678, in __init__
    self.__connect()
  File "/var/ossec/framework/python/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 902, in __connect
    with util.safe_reraise():
  File "/var/ossec/framework/python/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/var/ossec/framework/python/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 898, in __connect
    self.dbapi_connection = connection = pool._invoke_creator(self)
  File "/var/ossec/framework/python/lib/python3.10/site-packages/sqlalchemy/engine/create.py", line 637, in connect
    return dialect.connect(*cargs, **cparams)
  File "/var/ossec/framework/python/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 616, in connect
    return self.loaded_dbapi.connect(*cargs, **cparams)
sqlite3.OperationalError: unable to open database file

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/var/ossec/framework/scripts/create_user.py", line 72, in <module>
    check_database_integrity()
  File "/var/ossec/framework/python/lib/python3.10/site-packages/wazuh/rbac/orm.py", line 3184, in check_database_integrity
    raise e
  File "/var/ossec/framework/python/lib/python3.10/site-packages/wazuh/rbac/orm.py", line 3170, in check_database_integrity
    db_manager.create_database(DB_FILE)
  File "/var/ossec/framework/python/lib/python3.10/site-packages/wazuh/rbac/orm.py", line 2707, in create_database
    _Base.metadata.create_all(self.engines[database])
  File "/var/ossec/framework/python/lib/python3.10/site-packages/sqlalchemy/sql/schema.py", line 5828, in create_all
    bind._run_ddl_visitor(
  File "/var/ossec/framework/python/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3242, in _run_ddl_visitor
    with self.begin() as conn:
  File "/var/ossec/framework/python/lib/python3.10/contextlib.py", line 135, in __enter__
    return next(self.gen)
  File "/var/ossec/framework/python/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3232, in begin
    with self.connect() as conn:
  File "/var/ossec/framework/python/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3268, in connect
    return self._connection_cls(self)
  File "/var/ossec/framework/python/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 147, in __init__
    Connection._handle_dbapi_exception_noconnection(
  File "/var/ossec/framework/python/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2430, in _handle_dbapi_exception_noconnection
    raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
  File "/var/ossec/framework/python/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 145, in __init__
    self._dbapi_connection = engine.raw_connection()
  File "/var/ossec/framework/python/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3292, in raw_connection
    return self.pool.connect()
  File "/var/ossec/framework/python/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 452, in connect
    return _ConnectionFairy._checkout(self)
  File "/var/ossec/framework/python/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 1269, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/var/ossec/framework/python/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 716, in checkout
    rec = pool._do_get()
  File "/var/ossec/framework/python/lib/python3.10/site-packages/sqlalchemy/pool/impl.py", line 169, in _do_get
    with util.safe_reraise():
  File "/var/ossec/framework/python/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/var/ossec/framework/python/lib/python3.10/site-packages/sqlalchemy/pool/impl.py", line 167, in _do_get
    return self._create_connection()
  File "/var/ossec/framework/python/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 393, in _create_connection
    return _ConnectionRecord(self)
  File "/var/ossec/framework/python/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 678, in __init__
    self.__connect()
  File "/var/ossec/framework/python/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 902, in __connect
    with util.safe_reraise():
  File "/var/ossec/framework/python/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/var/ossec/framework/python/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 898, in __connect
    self.dbapi_connection = connection = pool._invoke_creator(self)
  File "/var/ossec/framework/python/lib/python3.10/site-packages/sqlalchemy/engine/create.py", line 637, in connect
    return dialect.connect(*cargs, **cparams)
  File "/var/ossec/framework/python/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 616, in connect
    return self.loaded_dbapi.connect(*cargs, **cparams)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) unable to open database file
(Background on this error at: https://sqlalche.me/e/20/e3q8)

[vcerenu]

Hello @paul8046

Could you add the command you used for the deployment?

Keep in mind that the deployment requires you to use Docker volumes, and that the files you mount on /var/ossec/ are recommended to be mounted within /wazuh-config-mount/ and then copied to their location at the start of the container.

[paul8046]

I am using ansible (with podman) to manage my container deployments. Here is my ansible task:

- name: wazuh-manager container is created
  containers.podman.podman_container:
    name: wazuh-manager
    image: docker.io/wazuh/wazuh-manager:4.9.1
    network: host
    state: created
    ulimit:
      - memlock=-1:-1
      - nofile=655360:655360
    volumes:
      - wazuh_api_configuration:/var/ossec/api/configuration
      - wazuh_etc:/var/ossec/etc
      - wazuh_logs:/var/ossec/logs
      - wazuh_queue:/var/ossec/queue
      - wazuh_var_multigroups:/var/ossec/var/multigroups
      - wazuh_integrations:/var/ossec/integrations
      - wazuh_active_response:/var/ossec/active-response/bin
      - wazuh_agentless:/var/ossec/agentless
      - wazuh_wodles:/var/ossec/wodles
      - wazuh_filebeat_etc:/etc/filebeat
      - wazuh_filebeat_var:/var/lib/filebeat
      - /usr/local/share/ca-certificates/root-ca.crt:/etc/ssl/root-ca.pem:ro
      - "{{ wazuh_manager_path }}/wazuh-manager.pem:/etc/ssl/filebeat.pem:ro"
      - "{{ wazuh_manager_path }}/wazuh-manager-key.pem:/etc/ssl/filebeat.key:ro"
      - "{{ wazuh_manager_path }}/wazuh_manager.conf:/wazuh-config-mount/etc/ossec.conf"
      - "{{ wazuh_manager_path }}/api.yaml:/wazuh-config-mount/api/configuration/api.yaml"
    env:
      INDEXER_URL: "https://{{ wazuh_indexer_url }}:{{ wazuh_indexer_https }}"
      INDEXER_USERNAME: "{{ wazuh_indexer_username }}"
      INDEXER_PASSWORD: "{{ wazuh_indexer_password }}"
      FILEBEAT_SSL_VERIFICATION_MODE: full
      SSL_CERTIFICATE_AUTHORITIES: /etc/ssl/root-ca.pem
      SSL_CERTIFICATE: /etc/ssl/filebeat.pem
      SSL_KEY: /etc/ssl/filebeat.key
      API_USERNAME: "{{ wazuh_api_username }}"
      API_PASSWORD: "{{ wazuh_api_password }}"
    stop_timeout: 30

If I mount a custom api.yaml to /wazuh-config-mount/api/configuration/api.yaml wherein host: is set (I have tried 0.0.0.0, :: and [::]) OR I don't mount anything to /wazuh-config-mount/api/configuration/api.yaml then I get the error:

api.api_exception.APIError: 2000 - Some parameters are not expected in the configuration file (WAZUH_PATH/api/configuration/api.yaml). Please check the documentation for further details: https://documentation.wazuh.com/4.9/user-manual/api/configuration.html#api-configuration-options: '0.0.0.0' is not of type 'array'.

If I mount a custom api.yaml to /wazuh-config-mount/api/configuration/api.yaml but don't set host: I get the following error:

Error during the database migration. Restoring the previous database file
Error details: (sqlite3.OperationalError) unable to open database file

[slhotellier]

Hello
I have the same problem on linux out of docker for agent and server since 4.9.0.
This is because /var is a noexec filesystem.
In my opinion this is a regretion or a security major faillure.
Because on cis rules /var must be noexec.
Best regards

[paul8046]

Interestingly I have tested versions 4.8.x, 4.9.1 and 4.10.0-alpha all with the same result. 4.7.5 produces the error
yaml.scanner.ScannerError: mapping values are not allowed here in "/var/ossec/api/configuration/api.yaml", line 1, column 8

[slhotellier]

Hello

I admit, I have not tested with versions prior to 4.8 and even for 4.8 I had not installed my OS with high security requirements. But it shocked me when I did it in 4.9 on my last servers, I could not start the server components or the agents. The only immediate explanation I found was the /var which was in noexec which corresponds to CIS rule 5009.
There may be other more elegant methods to solve the problem than simply removing the noexec from /var. for example deport the installation to mounts where executions are authorized (/ust, /opt, ...)

Best regards

[vcerenu]

Hello @slhotellier

Regarding the error you get when mounting the api.yaml file, I recommend you check the parameters in our documentation, since several have had changes in 4.9:
https://documentation.wazuh.com/current/user-manual/api/configuration.html#wazuh-server-api-configuration-file