Custom Log Integration Upgrade Fails

[samuel809]

Version

2.4.160

Installation Method

Security Onion ISO image

Description

configuration

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

16

RAM

32Gb

Storage for /

163Gb

Storage for /nsm

1.7Tb

Network Traffic Collection

span port

Network Traffic Speeds

Less than 1Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

Yes, there are additional clues in /opt/so/log/ (please provide detail below)

Detail

Would appreciate any help to solve this issue, it seems it is related to an index patter that can match two index templates.

i have try the list below, but since the template is managed by fleet; the changes are overwritten.

1. changing the priority, it changes back to 200.
2. deleting the custom integration,
3. deleting the indices, component table and index templates.
4. modifying priority via Dev Tools.

so-status

so-checkin; no errors.

Custom Integration Upgrade

docker logs so-kibana
[2025-06-27T18:00:37.869+00:00][INFO ][plugins.securitySolution.telemetry_events.sender.task] outdated task [2025-06-27T18:00:37.872+00:00][INFO ][plugins.securitySolution.telemetry_events.sender.task] Telemetry is not opted-in [2025-06-27T18:04:37.437+00:00][ERROR][plugins.fleet] Download Source 'f6fb8090-37c4-11ef-a8ab-0b3709c83f72' already exists with name 'FleetServer_carahate' [2025-06-27T18:05:03.487+00:00][INFO ][plugins.fleet.fleet:unenroll-inactive-agents-task:1.0.0] [runTask()] started [2025-06-27T18:05:03.500+00:00][INFO ][plugins.fleet.fleet:unenroll-inactive-agents-task:1.0.0] [UnenrollInactiveAgentsTask] runTask ended: success [2025-06-27T18:05:37.519+00:00][INFO ][plugins.securitySolution.telemetry_events.sender.task] outdated task [2025-06-27T18:05:37.522+00:00][INFO ][plugins.securitySolution.telemetry_events.sender.task] Telemetry is not opted-in [2025-06-27T18:05:38.031+00:00][ERROR][plugins.fleet] Error running external callbacks for packagePolicyPostUpdate: [2025-06-27T18:05:38.032+00:00][ERROR][plugins.fleet] Error: [elasticsearch.index_template.mappings]: definition for this key is missing [2025-06-27T18:05:38.032+00:00][ERROR][plugins.fleet] Error: [elasticsearch.index_template.mappings]: definition for this key is missing [2025-06-27T18:07:04.634+00:00][INFO ][plugins.fleet] Install with state machine - Starting installation of [email protected] from registry [2025-06-27T18:07:10.302+00:00][WARN ][plugins.fleet] Error during execution of state "install_index_template_pipelines" with status "failed": illegal_argument_exception [2025-06-27T18:07:11.298+00:00][WARN ][plugins.fleet] Failure to install package [log]: [ResponseError: illegal_argument_exception [2025-06-27T18:07:12.314+00:00][ERROR][plugins.fleet] Rolling back to log-2.3.0 after error installing log-2.3.3 [2025-06-27T18:07:12.328+00:00][INFO ][plugins.fleet] Install with state machine - Starting installation of [email protected] from registry [2025-06-27T18:07:17.613+00:00][WARN ][plugins.fleet] Error during execution of state "install_index_template_pipelines" with status "failed": illegal_argument_exception [2025-06-27T18:07:18.624+00:00][WARN ][plugins.fleet] Failure to install package [log]: [ResponseError: illegal_argument_exception [2025-06-27T18:07:19.665+00:00][ERROR][plugins.fleet] ResponseError: illegal_argument_exception [2025-06-27T18:08:33.689+00:00][INFO ][plugins.securitySolution.telemetry_events.sender.task] Telemetry is not opted-in [2025-06-27T18:10:03.778+00:00][INFO ][plugins.fleet] Fleet Usage: {"agents_enabled":true,"agents":{"total_enrolled":68,"healthy":61,"unhealthy":7,"offline":0,"inactive":4,"unenrolled":0,"total_all_statuses":68,"updating":0},"fleet_server":{"total_enrolled":1,"healthy":1,"unhealthy":0,"offline":0,"updating":0,"inactive":2,"unenrolled":0,"total_all_statuses":3,"num_host_urls":2},"license_issued_to":"securityonion"} [2025-06-27T18:10:26.310+00:00][INFO ][plugins.fleet] Beginning fleet setup [2025-06-27T18:10:26.310+00:00][INFO ][plugins.fleet] Cleaning old indices [2025-06-27T18:10:26.511+00:00][ERROR][plugins.fleet] Failed to fetch latest version of synthetics from registry: Cannot read properties of null (reading 'version') [2025-06-27T18:10:26.788+00:00][INFO ][plugins.fleet] Fleet setup completed [2025-06-27T18:10:37.732+00:00][INFO ][plugins.securitySolution.telemetry_events.sender.task] outdated task [2025-06-27T18:10:37.734+00:00][INFO ][plugins.securitySolution.telemetry_events.sender.task] Telemetry is not opted-in [2025-06-27T18:15:03.601+00:00][INFO ][plugins.fleet.fleet:unenroll-inactive-agents-task:1.0.0] [runTask()] started [2025-06-27T18:15:03.616+00:00][INFO ][plugins.fleet.fleet:unenroll-inactive-agents-task:1.0.0] [UnenrollInactiveAgentsTask] runTask ended: success [2025-06-27T18:15:37.618+00:00][INFO ][plugins.securitySolution.telemetry_events.sender.task] outdated task [2025-06-27T18:15:37.621+00:00][INFO ][plugins.securitySolution.telemetry_events.sender.task] Telemetry is not opted-in [2025-06-27T18:19:55.475+00:00][ERROR][plugins.fleet] Download Source 'f6fb8090-37c4-11ef-a8ab-0b3709c83f72' already exists with name 'FleetServer_carahate' [2025-06-27T18:20:36.809+00:00][ERROR][plugins.fleet] Error running external callbacks for packagePolicyPostUpdate: [2025-06-27T18:20:36.810+00:00][ERROR][plugins.fleet] Error: [elasticsearch.index_template.mappings]: definition for this key is missing [2025-06-27T18:20:36.810+00:00][ERROR][plugins.fleet] Error: [elasticsearch.index_template.mappings]: definition for this key is missing [2025-06-27T18:20:37.709+00:00][INFO ][plugins.securitySolution.telemetry_events.sender.task] outdated task [2025-06-27T18:20:37.711+00:00][INFO ][plugins.securitySolution.telemetry_events.sender.task] Telemetry is not opted-in [2025-06-27T18:22:28.947+00:00][INFO ][plugins.fleet] Install with state machine - Starting installation of [email protected] from registry [2025-06-27T18:22:33.825+00:00][WARN ][plugins.fleet] Error during execution of state "install_index_template_pipelines" with status "failed": illegal_argument_exception [2025-06-27T18:22:34.288+00:00][WARN ][plugins.fleet] Failure to install package [log]: [ResponseError: illegal_argument_exception [2025-06-27T18:22:35.312+00:00][ERROR][plugins.fleet] Rolling back to log-2.3.0 after error installing log-2.3.3 [2025-06-27T18:22:35.329+00:00][INFO ][plugins.fleet] Install with state machine - Starting installation of [email protected] from registry [2025-06-27T18:22:41.017+00:00][WARN ][plugins.fleet] Error during execution of state "install_index_template_pipelines" with status "failed": illegal_argument_exception [2025-06-27T18:22:41.412+00:00][WARN ][plugins.fleet] Failure to install package [log]: [ResponseError: illegal_argument_exception [2025-06-27T18:22:42.430+00:00][ERROR][plugins.fleet] ResponseError: illegal_argument_exception [2025-06-27T18:23:19.296+00:00][INFO ][plugins.fleet] Beginning fleet setup [2025-06-27T18:23:19.297+00:00][INFO ][plugins.fleet] Cleaning old indices [2025-06-27T18:23:19.525+00:00][ERROR][plugins.fleet] Failed to fetch latest version of synthetics from registry: Cannot read properties of null (reading 'version') [2025-06-27T18:23:19.803+00:00][INFO ][plugins.fleet] Fleet setup completed [2025-06-27T18:23:22.474+00:00][INFO ][plugins.securitySolution.telemetry_events.sender.task] outdated task [2025-06-27T18:23:22.475+00:00][WARN ][plugins.actions] Missing required project id while running actions:connector_usage_reporting [2025-06-27T18:23:22.476+00:00][INFO ][plugins.securitySolution.telemetry_events.sender.task] Telemetry is not opted-in [2025-06-27T18:23:22.476+00:00][INFO ][plugins.securitySolution.telemetry_events.sender.task] Telemetry is not opted-in [2025-06-27T18:23:22.476+00:00][INFO ][plugins.securitySolution.telemetry_events.sender.task] Telemetry is not opted-in [2025-06-27T18:23:22.477+00:00][INFO ][plugins.securitySolution.telemetry_events.sender.task] Telemetry is not opted-in [2025-06-27T18:23:22.922+00:00][INFO ][plugins.fleet.fleet:delete-unenrolled-agents-task:1.0.0] [runTask()] started [2025-06-27T18:23:22.926+00:00][INFO ][plugins.fleet] Running Fleet Usage telemetry send task [2025-06-27T18:23:22.944+00:00][INFO ][plugins.fleet.fleet:delete-unenrolled-agents-task:1.0.0] [DeleteUnenrolledAgentsTask] runTask ended: success [2025-06-27T18:25:03.514+00:00][INFO ][plugins.fleet.fleet:unenroll-inactive-agents-task:1.0.0] [runTask()] started [2025-06-27T18:25:03.528+00:00][INFO ][plugins.fleet.fleet:unenroll-inactive-agents-task:1.0.0] [UnenrollInactiveAgentsTask] runTask ended: success [2025-06-27T18:25:03.561+00:00][INFO ][plugins.fleet] Fleet Usage: {"agents_enabled":true,"agents":{"total_enrolled":68,"healthy":62,"unhealthy":6,"offline":0,"inactive":4,"unenrolled":0,"total_all_statuses":68,"updating":0},"fleet_server":{"total_enrolled":1,"healthy":1,"unhealthy":0,"offline":0,"updating":0,"inactive":2,"unenrolled":0,"total_all_statuses":3,"num_host_urls":2},"license_issued_to":"securityonion"} [2025-06-27T18:25:34.288+00:00][INFO ][plugins.fleet] Install with state machine - Starting installation of [email protected] from registry [2025-06-27T18:25:37.558+00:00][INFO ][plugins.securitySolution.telemetry_events.sender.task] outdated task [2025-06-27T18:25:37.561+00:00][INFO ][plugins.securitySolution.telemetry_events.sender.task] Telemetry is not opted-in [2025-06-27T18:25:39.468+00:00][WARN ][plugins.fleet] Error during execution of state "install_index_template_pipelines" with status "failed": illegal_argument_exception [2025-06-27T18:25:40.406+00:00][WARN ][plugins.fleet] Failure to install package [log]: [ResponseError: illegal_argument_exception [2025-06-27T18:25:41.423+00:00][ERROR][plugins.fleet] Rolling back to log-2.3.0 after error installing log-2.3.3 [2025-06-27T18:25:41.440+00:00][INFO ][plugins.fleet] Install with state machine - Starting installation of [email protected] from registry [2025-06-27T18:25:47.225+00:00][WARN ][plugins.fleet] Error during execution of state "install_index_template_pipelines" with status "failed": illegal_argument_exception [2025-06-27T18:25:48.122+00:00][WARN ][plugins.fleet] Failure to install package [log]: [ResponseError: illegal_argument_exception [2025-06-27T18:25:49.135+00:00][ERROR][plugins.fleet] ResponseError: illegal_argument_exception [2025-06-27T18:26:15.968+00:00][ERROR][plugins.fleet] Download Source 'f6fb8090-37c4-11ef-a8ab-0b3709c83f72' already exists with name 'FleetServer_carahate' [2025-06-27T18:27:23.215+00:00][ERROR][plugins.fleet] Error running external callbacks for packagePolicyPostUpdate: [2025-06-27T18:27:23.215+00:00][ERROR][plugins.fleet] Error: [elasticsearch.index_template.mappings]: definition for this key is missing [2025-06-27T18:27:23.216+00:00][ERROR][plugins.fleet] Error: [elasticsearch.index_template.mappings]: definition for this key is missing [2025-06-27T18:27:25.323+00:00][INFO ][plugins.fleet] Install with state machine - Starting installation of [email protected] from registry [2025-06-27T18:27:30.915+00:00][WARN ][plugins.fleet] Error during execution of state "install_index_template_pipelines" with status "failed": illegal_argument_exception [2025-06-27T18:27:31.275+00:00][WARN ][plugins.fleet] Failure to install package [log]: [ResponseError: illegal_argument_exception [2025-06-27T18:27:32.295+00:00][ERROR][plugins.fleet] Rolling back to log-2.3.0 after error installing log-2.3.3 [2025-06-27T18:27:32.308+00:00][INFO ][plugins.fleet] Install with state machine - Starting installation of [email protected] from registry [2025-06-27T18:27:37.976+00:00][WARN ][plugins.fleet] Error during execution of state "install_index_template_pipelines" with status "failed": illegal_argument_exception [2025-06-27T18:27:38.414+00:00][WARN ][plugins.fleet] Failure to install package [log]: [ResponseError: illegal_argument_exception [2025-06-27T18:27:39.437+00:00][ERROR][plugins.fleet] ResponseError: illegal_argument_exception [2025-06-27T18:28:46.451+00:00][INFO ][plugins.fleet] Beginning fleet setup [2025-06-27T18:28:46.452+00:00][INFO ][plugins.fleet] Cleaning old indices [2025-06-27T18:28:46.637+00:00][ERROR][plugins.fleet] Failed to fetch latest version of synthetics from registry: Cannot read properties of null (reading 'version') [2025-06-27T18:28:46.907+00:00][INFO ][plugins.fleet] Fleet setup completed [2025-06-27T18:28:47.510+00:00][INFO ][plugins.fleet] Install with state machine - Starting installation of [email protected] from registry [2025-06-27T18:28:53.626+00:00][WARN ][plugins.fleet] Error during execution of state "install_index_template_pipelines" with status "failed": illegal_argument_exception [2025-06-27T18:28:54.589+00:00][WARN ][plugins.fleet] Failure to install package [log]: [ResponseError: illegal_argument_exception [2025-06-27T18:28:55.607+00:00][ERROR][plugins.fleet] Rolling back to log-2.3.0 after error installing log-2.3.3 [2025-06-27T18:28:55.623+00:00][INFO ][plugins.fleet] Install with state machine - Starting installation of [email protected] from registry [2025-06-27T18:29:00.786+00:00][WARN ][plugins.fleet] Error during execution of state "install_index_template_pipelines" with status "failed": illegal_argument_exception [2025-06-27T18:29:01.660+00:00][WARN ][plugins.fleet] Failure to install package [log]: [ResponseError: illegal_argument_exception [2025-06-27T18:29:02.673+00:00][ERROR][plugins.fleet] ResponseError: illegal_argument_exception [2025-06-27T18:30:37.911+00:00][INFO ][plugins.securitySolution.telemetry_events.sender.task] outdated task [2025-06-27T18:30:37.913+00:00][INFO ][plugins.securitySolution.telemetry_events.sender.task] Telemetry is not opted-in [2025-06-27T18:35:03.591+00:00][INFO ][plugins.fleet.fleet:unenroll-inactive-agents-task:1.0.0] [runTask()] started [2025-06-27T18:35:03.600+00:00][INFO ][plugins.fleet.fleet:unenroll-inactive-agents-task:1.0.0] [UnenrollInactiveAgentsTask] runTask ended: success [2025-06-27T18:35:37.937+00:00][INFO ][plugins.securitySolution.telemetry_events.sender.task] outdated task [2025-06-27T18:35:37.940+00:00][INFO ][plugins.securitySolution.telemetry_events.sender.task] Telemetry is not opted-in

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[cm-ops]

Are you reusing the index template for logs-xmlpc?

[samuel809]

I have deleted all custom logs integrations but still have same issue. the index template logs-xmlrpc event if i deleted it; stills manage to be recreated; I'm supposing that is because is being managed by fleet. where can i unmanage it?

[cm-ops]

Do you see the template in Kibana > Stack Management > Index Management > Index Templates?

[samuel809]

Yes, i do see them there, i have tried to modify them, deleting the component tables, changing the index pattern, changing the priority and also deleting them.

[cm-ops]

Have you tried adjusting the priority in the template from 200?

[samuel809]

yes, via gui and dev console.
it comes back to 200.