Events generated by agent must comply with the common schema

[sdvendramini]

Parent Issue: #241

Description

The events generated by the agent must adhere to the common schema for consistency and compatibility across systems.

Details

• Each index mapping is defined in the repository: wazuh-indexer-plugins
• Each field is documented according to the ECS and can be referenced in the Elastic documentation.

Format body stateless and stateful

Stateless

{
  "agent": {
      "id": "2887e1cf-9bf2-431a-b066-a46860080f56",
      "name": "agent1",
      "type": "endpoint",
      "version": "5.0.0",
      "groups": ["group1", "group2"],
      "host": {
          "hostname": "myhost",
          "os": {
              "name": "Amazon Linux 2"
              "platform": "Linux"
          },
          "ip": ["192.168.1.2"],
          "architecture": "x86_64"
      }
  }
}
{
  "module": "logcollector",
  "type": "file"
}
{
  "log": {
    "file": {
      "path": "string"
    }
  },
  "tags": ["string"],
  "event": {
    "original": "string",
    "ingested": "string",
    "module": "string",
    "provider": "string"
  }
}
{
  "module": "inventory",
  "type": "package"
}
{
  "log": {
    "file": {
      "path": "string"
    }
  },
  "tags": ["string"],
  "event": {
    "original": "string",
    "ingested": "string",
    "module": "string",
    "provider": "string"
  }
}

Stateful

{
  "agent": {
      "id": "2887e1cf-9bf2-431a-b066-a46860080f56",
      "name": "agent1",
      "type": "endpoint",
      "version": "5.0.0",
      "groups": ["group1", "group2"],
      "host": {
          "hostname": "myhost",
          "os": {
              "name": "Amazon Linux 2"
              "platform": "Linux"
          },
          "ip": ["192.168.1.2"],
          "architecture": "x86_64"
      }
  }
}
{
  "module": "inventory",
  "type": "package",
  "operation": "modified",
  "id": "lskdjf023984902358"
}
{
  "scan_time": "2024-10-28T18:26:10.634Z",
  "package": {
    "architecture": "string",
    "description": "string",
    "installed": "2024-10-28T18:26:10.634Z",
    "name": "string",
    "path": "string",
    "size": 0,
    "type": "string",
    "version": "string"
  }
}
{
  "module": "inventory",
  "type": "network",
  "operation": "add",
  "id": "lskdjf023984902358"
}
{
  "scan_time": "2024-10-28T18:26:10.634Z",
  "package": {
    "architecture": "string",
    "description": "string",
    "installed": "2024-10-28T18:26:10.634Z",
    "name": "string",{ 
      "agent": { 
        "uuid": "UUID", 
        "groups": [ ], 
        "os": "Amazon Linux 2", 
        "platform": "Linux", 
        "type": "Endpoint", 
        "version": "5.0.0", 
        "ip": "192.168.1.2" } 
    }
    "path": "string",
    "size": 0,
    "type": "string",
    "version": "string"
  }
}
{
  "module": "inventory",
  "type": "network",
  "operation": "delete",
  "id": "asdfsdfkdsj98237498325"
}

Tasks

• Refactor Message Output for Common Schema Compliance #291
  • Queue update.
  • Update Logcollector module to support new Stateless message structure.
  • Update agent message output so that the body objects comply with the new defined format, similar to a JSON stream.
• Update Inventory Module for Common Schema Compliance #292
  • Update Hardware Inventory Format #296
  • Update System Inventory Format #293
  • Update Processes Inventory Format #295
  • Update Packages Inventory Format #297
  • Update Hotfixes Inventory Format #298
  • Update Ports Inventory Format #299
  • Refactor network interfaces and protocols inventory tables #284
  • Update Network Inventory Format #294

[LucioDonda]

Hi @sdvendramini While I'm looking for them:
Have you detected which fields or in which situation did the agent generate any non-ECS compliant event field?
Where they part of any particular module ?
TIA

[GGP1]

@LucioDonda the ECS templates have been modified recently, it is highly likely that the agent is generating events with an outdated format.

I've been working on Update stateful events data models #26568 which covers the same case but for the Communications API POST /events/stateful endpoint.

Here are some of the structures we are accepting in JSON format.

FIM

{
  "agent": {
	"id": "string",
	"groups": []
  },
  "file": {
	"attributes": [
  	"string"
	],
	"name": "string",
	"path": "string",
	"gid": 0,
	"group": "string",
	"inode": "string",
	"mtime": "2024-10-28T18:26:10.634Z",
	"mode": "string",
	"size": 0,
	"target_path": "string",
	"type": "string",
	"uid": 0,
	"owner": "string",
	"hash": {
  	"md5": "string",
  	"sha1": "string",
  	"sha256": "string"
	}
  },
  "registry": {
	"key": "string",
	"value": "string"
  }
}

Inventory package

{
  "agent": {
	"id": "string",
	"groups": []
  },
  "scan_time": "2024-10-28T18:26:10.634Z",
  "package": {
	"architecture": "string",
	"description": "string",
	"installed": "2024-10-28T18:26:10.634Z",
	"name": "string",
	"path": "string",
	"size": 0,
	"type": "string",
	"version": "string"
  }
}

Inventory processes

{
  "agent": {
	"id": "string",
	"groups": []
  },
  "scan_time": "2024-10-28T18:26:10.634Z",
  "process": {
	"pid": 0,
	"name": "string",
	"parent": {
  	"pid": 0
	},
	"command_line": "string",
	"args": [
  	"string"
	],
	"user": {
  	"id": "string"
	},
	"real_user": {
  	"id": "string"
	},
	"saved_user": {
  	"id": "string"
	},
	"group": {
  	"id": "string"
	},
	"real_group": {
  	"id": "string"
	},
	"saved_group": {
  	"id": "string"
	},
	"start": "2024-10-28T18:26:10.635Z",
	"thread": {
  	"id": "string"
	}
  }
}

Inventory system

{
  "agent": {
	"id": "string",
	"groups": []
  },
  "scan_time": "2024-10-28T18:26:10.635Z",
  "host": {
	"architecture": "string",
	"hostname": "string",
	"os": {
  	"kernel": "string",
  	"full": "string",
  	"name": "string",
  	"platform": "string",
  	"version": "string",
  	"type": "string"
	}
  }
}

Vulnerability

{
  "agent": {
	"id": "string",
	"groups": []
	"name": "string",
	"type": "string",
	"version": "string"
  },
  "host": {
	"os": {
  	"kernel": "string",
  	"full": "string",
  	"name": "string",
  	"platform": "string",
  	"version": "string",
  	"type": "string"
	}
  },
  "package": {
	"architecture": "string",
	"build_version": "string",
	"checksum": "string",
	"description": "string",
	"install_scope": "string",
	"installed": "2024-10-28T18:26:10.635Z",
	"license": "string",
	"name": "string",
	"path": "string",
	"reference": "string",
	"size": 0,
	"type": "string",
	"version": "string"
  },
  "scanner": {
	"source": "string",
	"vendor": "string"
  },
  "score": {
	"base": 0,
	"environmental": 0,
	"temporal": 0,
	"version": "string"
  },
  "category": "string",
  "classification": "string",
  "description": "string",
  "detected_at": "2024-10-28T18:26:10.635Z",
  "enumeration": "string",
  "id": "string",
  "published_at": "2024-10-28T18:26:10.635Z",
  "reference": "string",
  "report_id": "string",
  "severity": "string",
  "under_evaluation": true
}

Command result

{
  "document_id": "string",
  "result": {
      "code": "string",
      "message": "string",
      "data": "string"
  }
}

At the same time, those objects have to be inside the data field of a wrapper object that also includes a module field. For example, an inventory package event would look like this:

{
  "data": {
    "agent": {
      "id": "string",
      "groups": []
    },
    "scan_time": "2024-10-28T18:26:10.634Z",
    "package": {
      "architecture": "string",
      "description": "string",
      "installed": "2024-10-28T18:26:10.634Z",
      "name": "string",
      "path": "string",
      "size": 0,
      "type": "string",
      "version": "string"
    }
  },
  "module": "inventory_package"
}

If you have any doubts or comments, we can arrange a meeting to discuss this further.

[vikman90]

Module format

• module.name: Name of the module
  • command
  • fim
  • inventory
  • vulnerability
  • sca
• module.type: Data type, depending on the module (optional)
  • hotfix
  • network
  • package
  • port
  • process
  • system

Examples

{
  "module": {
    "name": "inventory",
    "type": "package"
  },
  "data": { ... }
}

{
  "module": {
    "name": "vulnerability"
  },
  "data": { ... }
}

{
  "module": {
    "name": "data"
  },
  "data": { ... } 
}

[vikman90]

Stateless: Logcollector

{
  "module": { "name": "logcollector" },
  "data": {
    "file": { "path": "/var/log/syslog" },
    "event": { "original": "2024-10-31T16:21:25.198579+01:00 Rocket systemd-resolved[176]: Clock change detected. Flushing caches." }
  }
}

[cborla]

Inventory analysis

The following analysis is based on the following sources.

• Source from which the indexer-supported structures are obtained: indexer/models/events.py or wazuh-indexer-ecs
• ECS source: Official link also could be use engine-schema.json

The indexer (master) currently supports the following data structures for inventory.

@dataclass
class OS:
    """OS data model."""
    kernel: str
    full: str
    name: str
    platform: str
    version: str
    type: str
    family: str


@dataclass
class Host:
    """Host data model."""
    architecture: str
    hostname: str
    os: OS


@dataclass
class ProcessHash:
    md5: str


@dataclass
class Process:
    """Process data model."""
    hash: ProcessHash


@dataclass
class InventoryEvent(BaseModel):
    """Inventory events data model."""
    host: Host
    process: Process

    def get_index_name(self) -> str:
        """Get the index name for the event type.
        
        Returns
        -------
        str
            Index name.
        """
        return INVENTORY_INDEX
       

From the above classes of the Inventory Stateful event, we can obtain the following diagram.

InventoryEvent
│
├── Host
│   ├── architecture : str
│   ├── hostname : str
│   └── os : OS
│       ├── kernel : str
│       ├── full : str
│       ├── name : str
│       ├── platform : str
│       ├── version : str
│       ├── type : str
│       └── family : str
│
└── Process
    └── hash : ProcessHash
        └── md5 : str

There is a very big difference in the amount of data and data structures being shared. Currently the inventory gets 9 types of structures, with their corresponding information.

• network_iface
• network_protocol
• network_address
• packages
• hotfixes
• ports
• processes
• osinfo
• hwinfo

As a first development, it can be adapted to the structure model proposed by the indexer.

[cborla]

Inventory analysis

New sources.

• Create ECS compliant index templates
• Indexer PR

[cborla]

Update 1/11

• New queue field included (modul_type) to support new packet format for both stateless events and stateful evnets, which is included in the package to be shipped in case it exists.

{
  "module": {
    "name": "inventory",
    "type": "package"
  },
  "data": { ... }
}

{
  "module": {
    "name": "vulnerability"
  },
  "data": { ... }
}

• Update of the format of the messages to be sent to the server from logcollector.

{
  "module": { "name": "logcollector" },
  "data": {
    "file": { "path": "/var/log/syslog" },
    "event": { "original": "2024-10-31T16:21:25.198579+01:00 Rocket systemd-resolved[176]: Clock change detected. Flushing caches." }
  }
}

• Updating of inventory tables and normalisation of fields. (WIP)