[Bug] Trying to export rules from kibana but validationError

[OzzyKampha]

Describe the Bug

Tried to export the elastic defend rules from the detections_rules, then it failed. i have no problem importing rules.

Malicious File - Prevented - Elastic Defend - {'rule': [ValidationError({'type': ['Must be equal to threshold.'], 'threshold': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to eql.'], 'language': ['Must be equal to eql.']}), ValidationError({'type': ['Must be equal to esql.'], 'language': ['Must be equal to esql.']}), ValidationError({'type': ['Must be equal to threat_match.'], 'threat_mapping': ['Missing data for required field.'], 'threat_index': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to machine_learning.'], 'anomaly_threshold': ['Missing data for required field.'], 'machine_learning_job_id': ['Missing data for required field.']}), ValidationError({'_schema': ['Configuration error: Rule Malicious File - Prevented - Elastic Defend - f87e6122-ea34-11ee-a417-f661ea17fbce should not contain rules withversion and revision` set.']}), ValidationError({'type': ['Must be equal to new_terms.'], 'new_terms': ['Missing data for required field.']})]}

• Suspicious Inter-Process Communication via Outlook - {'rule': [ValidationError({'type': ['Must be equal to threshold.'], 'threshold': ['Missing data for required field.']}), ValidationError({'_schema': ['Configuration error: Rule Suspicious Inter-Process Communication via Outlook - 1dee0500-4aeb-44ca-b24b-4a385d7b6ba1 should not contain rules with version and revision set.']}), ValidationError({'type': ['Must be equal to esql.'], 'language': ['Must be equal to esql.']}), ValidationError({'type': ['Must be equal to threat_match.'], 'threat_mapping': ['Missing data for required field.'], 'threat_index': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to machine_learning.'], 'anomaly_threshold': ['Missing data for required field.'], 'machine_learning_job_id': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to query.']}), ValidationError({'type': ['Must be equal to new_terms.'], 'new_terms': ['Missing data for required field.']})]}
• Malicious File - Detected - Elastic Defend - {'rule': [ValidationError({'type': ['Must be equal to threshold.'], 'threshold': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to eql.'], 'language': ['Must be equal to eql.']}), ValidationError({'type': ['Must be equal to esql.'], 'language': ['Must be equal to esql.']}), ValidationError({'type': ['Must be equal to threat_match.'], 'threat_mapping': ['Missing data for required field.'], 'threat_index': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to machine_learning.'], 'anomaly_threshold': ['Missing data for required field.'], 'machine_learning_job_id': ['Missing data for required field.']}), ValidationError({'_schema': ['Configuration error: Rule Malicious File - Detected - Elastic Defend - f2c3caa6-ea34-11ee-a417-f661ea17fbce should not contain rules with version and revision set.']}), ValidationError({'type': ['Must be equal to new_terms.'], 'new_terms': ['Missing data for required field.']})]}
• Ransomware - Detected - Elastic Defend - {'rule': [ValidationError({'type': ['Must be equal to threshold.'], 'threshold': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to eql.'], 'language': ['Must be equal to eql.']}), ValidationError({'type': ['Must be equal to esql.'], 'language': ['Must be equal to esql.']}), ValidationError({'type': ['Must be equal to threat_match.'], 'threat_mapping': ['Missing data for required field.'], 'threat_index': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to machine_learning.'], 'anomaly_threshold': ['Missing data for required field.'], 'machine_learning_job_id': ['Missing data for required field.']}), ValidationError({'_schema': ['Configuration error: Rule Ransomware - Detected - Elastic Defend - 0c74cd7e-ea35-11ee-a417-f661ea17fbce should not contain rules with version and revision set.']}), ValidationError({'type': ['Must be equal to new_terms.'], 'new_terms': ['Missing data for required field.']})]}
• Ransomware - Prevented - Elastic Defend - {'rule': [ValidationError({'type': ['Must be equal to threshold.'], 'threshold': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to eql.'], 'language': ['Must be equal to eql.']}), ValidationError({'type': ['Must be equal to esql.'], 'language': ['Must be equal to esql.']}), ValidationError({'type': ['Must be equal to threat_match.'], 'threat_mapping': ['Missing data for required field.'], 'threat_index': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to machine_learning.'], 'anomaly_threshold': ['Missing data for required field.'], 'machine_learning_job_id': ['Missing data for required field.']}), ValidationError({'_schema': ['Configuration error: Rule Ransomware - Prevented - Elastic Defend - 10f3d520-ea35-11ee-a417-f661ea17fbce should not contain rules with version and revision set.']}), ValidationError({'type': ['Must be equal to new_terms.'], 'new_terms': ['Missing data for required field.']})]}
• Behavior - Prevented - Elastic Defend - {'rule': [ValidationError({'type': ['Must be equal to threshold.'], 'threshold': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to eql.'], 'language': ['Must be equal to eql.']}), ValidationError({'type': ['Must be equal to esql.'], 'language': ['Must be equal to esql.']}), ValidationError({'type': ['Must be equal to threat_match.'], 'threat_mapping': ['Missing data for required field.'], 'threat_index': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to machine_learning.'], 'anomaly_threshold': ['Missing data for required field.'], 'machine_learning_job_id': ['Missing data for required field.']}), ValidationError({'_schema': ['Configuration error: Rule Behavior - Prevented - Elastic Defend - eb804972-ea34-11ee-a417-f661ea17fbce should not contain rules with version and revision set.']}), ValidationError({'type': ['Must be equal to new_terms.'], 'new_terms': ['Missing data for required field.']})]}
• Endpoint Security (Elastic Defend) - {'rule': [ValidationError({'type': ['Must be equal to threshold.'], 'threshold': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to eql.'], 'language': ['Must be equal to eql.']}), ValidationError({'type': ['Must be equal to esql.'], 'language': ['Must be equal to esql.']}), ValidationError({'type': ['Must be equal to threat_match.'], 'threat_mapping': ['Missing data for required field.'], 'threat_index': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to machine_learning.'], 'anomaly_threshold': ['Missing data for required field.'], 'machine_learning_job_id': ['Missing data for required field.']}), ValidationError({'_schema': ['Configuration error: Rule Endpoint Security (Elastic Defend) - 9a1a2dae-0b5f-4c3d-8305-a268d404c306 should not contain rules with version and revision set.']}), ValidationError({'type': ['Must be equal to new_terms.'], 'new_terms': ['Missing data for required field.']})]}
• Memory Threat - Prevented- Elastic Defend - {'rule': [ValidationError({'type': ['Must be equal to threshold.'], 'threshold': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to eql.'], 'language': ['Must be equal to eql.']}), ValidationError({'type': ['Must be equal to esql.'], 'language': ['Must be equal to esql.']}), ValidationError({'type': ['Must be equal to threat_match.'], 'threat_mapping': ['Missing data for required field.'], 'threat_index': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to machine_learning.'], 'anomaly_threshold': ['Missing data for required field.'], 'machine_learning_job_id': ['Missing data for required field.']}), ValidationError({'_schema': ['Configuration error: Rule Memory Threat - Prevented- Elastic Defend - 06f3a26c-ea35-11ee-a417-f661ea17fbce should not contain rules with version and revision set.']}), ValidationError({'type': ['Must be equal to new_terms.'], 'new_terms': ['Missing data for required field.']})]}
• Memory Threat - Detected - Elastic Defend - {'rule': [ValidationError({'type': ['Must be equal to threshold.'], 'threshold': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to eql.'], 'language': ['Must be equal to eql.']}), ValidationError({'type': ['Must be equal to esql.'], 'language': ['Must be equal to esql.']}), ValidationError({'type': ['Must be equal to threat_match.'], 'threat_mapping': ['Missing data for required field.'], 'threat_index': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to machine_learning.'], 'anomaly_threshold': ['Missing data for required field.'], 'machine_learning_job_id': ['Missing data for required field.']}), ValidationError({'_schema': ['Configuration error: Rule Memory Threat - Detected - Elastic Defend - 017de1e4-ea35-11ee-a417-f661ea17fbce should not contain rules with version and revision set.']}), ValidationError({'type': ['Must be equal to new_terms.'], 'new_terms': ['Missing data for required field.']})]}
• Behavior - Detected - Elastic Defend - {'rule': [ValidationError({'type': ['Must be equal to threshold.'], 'threshold': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to eql.'], 'language': ['Must be equal to eql.']}), ValidationError({'type': ['Must be equal to esql.'], 'language': ['Must be equal to esql.']}), ValidationError({'type': ['Must be equal to threat_match.'], 'threat_mapping': ['Missing data for required field.'], 'threat_index': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to machine_learning.'], 'anomaly_threshold': ['Missing data for required field.'], 'machine_learning_job_id': ['Missing data for required field.']}), ValidationError({'_schema': ['Configuration error: Rule Behavior - Detected - Elastic Defend - 0f615fe4-eaa2-11ee-ae33-f661ea17fbce should not contain rules with version and revision set.']}), ValidationError({'type': ['Must be equal to new_terms.'], 'new_terms': ['Missing data for required field.']})]}`

To Reproduce

No response

Expected Behavior

No response

Screenshots

No response

Desktop - OS

None

Desktop - Version

No response

Additional Context

No response

[Mikaayenson]

👋 @OzzyKampha Can you share the command that failed for you?

[OzzyKampha]

@Mikaayenson sure

detection_rules kibana --ignore-ssl-errors true export-rules -d test-export

Running 9.2.0, using elastic-package Stack command

[eric-forte-elastic]

@OzzyKampha Can you try running the following to reduce some of the errors?

detection_rules kibana --ignore-ssl-errors true export-rules --strip-version -d test-export

One of the errors for instance is Rule Malicious File - Prevented - Elastic Defend - f87e6122-ea34-11ee-a417-f661ea17fbce should not contain rules with version and revision set.']})`. This error is expected if you have both version and revision information in your rules from Kibana, and it looks like a number of the rules in your example errors are running into this issue.

[0xBAADF0OD]

@eric-forte-elastic Thank you for that, it got past the initial validation error for me since I had the same problem as OP, but it looks like there are additional errors downstream.

Unknown field
event.dataset: "azure.identity_protection" and
    event.action: "User Risk Detection" and
    azure.identityprotection.properties.activity: "user"
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Try adding event.module or data_stream.dataset to specify beats module

beat_types: [filebeat]

stack: 9.2.0, beats: 9.2.0, ecs: 9.2.0

I believe this is an out-of-the-box elastic rule

[OzzyKampha]

@eric-forte-elastic when using your command im able to export rules. what does --strip-version - do

[eric-forte-elastic]

@0xBAADF0OD what command are you running that is resulting in that error? And on what file contents? If from Kibana, could you provide the ndjson? Thanks!

[eric-forte-elastic]

@OzzyKampha providing --strip-version strips the version fields from all rules when exporting them from Kibana. This version would get set based on what Kibana thinks is the version of the rule. Kibana also populates a revision field if the rule has been modified in Kibana. In our code, we enforce having either the version or the revision set and not both (whereas Kibana allows both).

The purpose for this is to prevent rules from inadvertently overriding (or reverting) their own version without a rule contents change. This can occur if you are bringing in your rules from multiple Kibana instances depending on your governance model. In short, one would want to use this flag if you are not wanting the Kibana version field to be authoritative (which in our case if you are using the detection rules repo, this is generally the case as it is expected that you are using git with it and using git as a version control system).

However, this is a flag instead of default as this is only necessary if you are making revisions in Kibana, as opposed to making revisions in git. If revisions are coming from Kibana and being pushed into git, then one cannot inherently trust the version number as authoritative, and as such it should be removed via the flag.