Skip to content

Commit 4e89588

Browse files
civilx64aothms
civilx64
authored and
aothms
committed
Add Developer documentation for validation outcome severities
1 parent db45eda commit 4e89588

File tree

1 file changed

+151
-0
lines changed

1 file changed

+151
-0
lines changed

Diff for: ‎docs/dev/outcome_severities.md

+151
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,151 @@
1+
# Outcome Severities
2+
3+
Note: This documentation was developed in Nov 2024 based on the codebase at version 0.6.8.
4+
5+
## Overview - Database model enumerations
6+
7+
Severities are an enumeration of possible values for the severity of a
8+
[Validation Outcome](https://github.com/buildingSMART/ifc-validation-data-model/blob/main/models.py#L978).
9+
10+
| String Value | Integer Enumeration Value |
11+
|----------------|---------------------------|
12+
| EXECUTED | 1 |
13+
| PASSED | 2 |
14+
| WARNING | 3 |
15+
| ERROR | 4 |
16+
| NOT_APPLICABLE | 0 |
17+
18+
## Severity Usage
19+
20+
### Schema Check
21+
22+
Schema checks are always applicable and therefore the only possible enumeration values are `PASSED` and `ERROR`.
23+
[Passing outcomes for schema checks are stored to the database](https://github.com/buildingSMART/validate/blob/8f04bcc6d1f400240485a33b2c81e2f7d0edbeab/backend/apps/ifc_validation/tasks.py#L607).
24+
25+
### Syntax Check
26+
27+
Syntax checks are also always applicable and therefore the only possible enumeration values are `PASSED` and `ERROR`.
28+
[Passing outcomes for syntax checks are stored to the database](https://github.com/buildingSMART/validate/blob/8f04bcc6d1f400240485a33b2c81e2f7d0edbeab/backend/apps/ifc_validation/tasks.py#L607).
29+
30+
### Gherkin Rules (Normative Rules and Industry Best Practices)
31+
32+
These rules are generally processed in similar fashion.
33+
Unlike syntax and schema checks, a gherkin rule may or may not be applicable to a given model depending on the schema
34+
and content of the model.
35+
For example, alignment rules are only applicable to models with schema `IFC4X3_ADD2` as those entities were not part of
36+
the
37+
`IFC2X3` or `IFC4` schema versions.
38+
39+
Therefore, both sets of validation checks can potentially also return outcomes with severity of `NOT_APPLICABLE`.
40+
41+
#### Normative Rules
42+
43+
Normative rules enforce requirements from implementer agreements and informal propositions.
44+
Therefore, the potential outcomes are:
45+
46+
- `NOT_APPLICABLE`
47+
- `ERROR`
48+
- `PASSED`
49+
50+
#### Industry Best Practices
51+
52+
Industry Best Practice checks enforce items that are not required,
53+
but rather represent the preferred or most idiomatic way of implementing the IFC standard.
54+
55+
Therefore, the potential outcomes are:
56+
57+
- `NOT_APPLICABLE`
58+
- `WARNING`
59+
- `PASSED`
60+
61+
#### Gherkin Rule processing
62+
63+
Currently there are no individual instance outcomes from gherkin rules stored with severity=`PASSED`.
64+
The initial idea was to pass them to the DB but it was quickly flooded with outcomes of this severity
65+
and is currently [commented out](https://github.com/buildingSMART/ifc-gherkin-rules/blob/b363041433f252fc1b9e043ee3aac0bd6fcfad3d/features/steps/validation_handling.py#L254-L268).
66+
67+
_(potentially remove the following paragraph to avoid confusion...)_
68+
69+
When processing gherkin rules (*.feature files) with `behave`, Severity=`PASSED` is used only for `Given` statements.
70+
A `PASSED` outcome is added to the temporary outcomes when the conditions of a `Given` statement are met.
71+
72+
The entire processing loop is as follows:
73+
74+
1. Severity=`NOT_APPLICABLE` is applied to all entity instances
75+
2. The `Given` statements are executed
76+
3. If there is *at least one* entity instance that meets the requirement of *ALL* `Given` statements,
77+
the rule is considered to be "activated" and severity=`EXECUTED` is applied to these instance(s).
78+
This is an 'all or nothing' situation where all `Given` statements must be "activated" in order for the rule
79+
to be considered "activated".
80+
4. Those instance(s) are then tested against the `Then` statements.
81+
5. Severity=`ERROR` is applied to each instance that fails a requirement of a `Then` statement.
82+
83+
If there is at least one instance with an `ERROR` outcome then an aggregated status for the rule is returned as follows:
84+
85+
- Severity=`ERROR` for normative rules (implementer agreements and informal propositions)
86+
- Severity=`WARNING` for industry best practices.
87+
88+
This status is used to colour the "block" for each rule in the validation report.
89+
90+
## Outcome display and reporting
91+
92+
### Individual rules (Normative and Industry Best Practices)
93+
94+
Outcomes are always reported in the web UI based on aggregated status of all instances
95+
activated by a given rule.
96+
97+
| Aggregated Value of `Severity` for a rule | Display colour | Label reported in `Severity` column |
98+
|-------------------------------------------|----------------|-------------------------------------|
99+
| `NOT_APPLICABLE` | grey | 'N/A' |
100+
| `EXECUTED` | green | 'Applicable' |
101+
| `PASSED` | (not used) | |
102+
| `WARNING` | yellow | 'Warning' |
103+
| `ERROR` | red | 'Error' |
104+
105+
Display colour is determined by the
106+
[statusToColor](https://github.com/buildingSMART/validate/blob/development/frontend/src/mappings.js#L1)
107+
mapping function.
108+
109+
Display label is determined by the
110+
[statusToLabel](https://github.com/buildingSMART/validate/blob/development/frontend/src/mappings.js#L10)
111+
mapping function.
112+
113+
### Overall status
114+
115+
A single overall status for each type of check (syntax, schema, normative rules, best practices)
116+
is displayed on the Validation Service dashboard for each model.
117+
118+
The overall status of each
119+
[ValidationTask](https://github.com/buildingSMART/ifc-validation-data-model/blob/main/models.py#L778)
120+
is captured in a different data structure of
121+
[Model.Status](https://github.com/buildingSMART/ifc-validation-data-model/blob/main/models.py#L324)
122+
with the following possible enumeration values:
123+
124+
| Value of `Model.Status` | Display colour | Symbol |
125+
|-------------------------|----------------|------------------------------------------------------------------------------|
126+
| `VALID` | green | CheckCircleIcon |
127+
| `INVALID` | red | WarningIcon |
128+
| `NOT_VALIDATED` | grey | HourglassBottomIcon |
129+
| `WARNING` | yellow | ErrorIcon |
130+
| `NOT_APPLICABLE` | grey | BrowserNotSupportedIcon (technically possible but doesn't occur in practice) |
131+
132+
The options for overall status of each category of ValidationTask are as follows:
133+
134+
- Syntax and Schema
135+
- `VALID`
136+
- `INVALID`
137+
138+
- Normative Rules
139+
- `VALID`
140+
- `INVALID`
141+
- `NOT_APPLICABLE`
142+
143+
- Best Practices
144+
- `VALID`
145+
- `WARNING`
146+
- `NOT_APPLICABLE`
147+
148+
The overall status of a normative rule ValidationTask is determined by
149+
[taking the highest severity](https://github.com/buildingSMART/ifc-validation-data-model/blob/f32164ab762fc695690d380e12e87c815b641912/models.py#L948)
150+
of outcomes for all the rules contained in that task.
151+

0 commit comments

Comments
 (0)