Skip to content

Commit f2b02a1

Browse files
committed
resolved merge conflicts
2 parents 512b04f + 9f6aa67 commit f2b02a1

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

48 files changed

+2386
-1186
lines changed

.github/workflows/ci.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ jobs:
1818
runs-on: ubuntu-latest
1919
strategy:
2020
matrix:
21-
python: ["3.10", "3.11", "3.12"]
21+
python: ["3.10", "3.11", "3.12", "3.13"]
2222

2323
steps:
2424
- uses: actions/checkout@v4

CHANGELOG.md

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,32 @@ and this project adheres to
88

99
## [Unreleased]
1010

11+
- Documenting support for python 3.13. (#86)
12+
13+
## [0.8.0] - 2024-11-12
14+
15+
- Support linting of sources.
16+
- **Breaking**: Renamed modules: `dbt_score.model_filter` becomes
17+
`dbt_score.rule_filter`
18+
- **Breaking**: Renamed filter class and decorator: `@model_filter` becomes
19+
`@rule_filter` and `ModelFilter` becomes `RuleFilter`.
20+
- **Breaking**: Config option `model_filter_names` becomes `rule_filter_names`.
21+
- **Breaking**: CLI flag naming fixes: `--fail_any_model_under` becomes
22+
`--fail-any-item-under` and `--fail_project_under` becomes
23+
`--fail-project-under`.
24+
25+
## [0.7.1] - 2024-11-01
26+
27+
- Fix mkdocs.
28+
29+
## [0.7.0] - 2024-11-01
30+
31+
- **Breaking**: The rule `public_model_has_example_sql` has been renamed
32+
`has_example_sql` and applies by default to all models.
33+
- **Breaking**: Remove `dbt-core` from dependencies. Since it is not mandatory
34+
for `dbt-score` to execute `dbt`, remove the dependency.
35+
- **Breaking**: Stop using `MultiOption` selection type.
36+
1137
## [0.6.0] - 2024-08-23
1238

1339
- **Breaking**: Improve error handling in CLI. Log messages are written in

README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@
1111

1212
## What is `dbt-score`?
1313

14-
`dbt-score` is a linter for dbt model metadata.
14+
`dbt-score` is a linter for dbt metadata.
1515

1616
[dbt][dbt] (Data Build Tool) is a great framework for creating, building,
1717
organizing, testing and documenting _data models_, i.e. data sets living in a

docs/configuration.md

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ rule_namespaces = ["dbt_score.rules", "dbt_score_rules", "custom_rules"]
1818
disabled_rules = ["dbt_score.rules.generic.columns_have_description"]
1919
inject_cwd_in_python_path = true
2020
fail_project_under = 7.5
21-
fail_any_model_under = 8.0
21+
fail_any_item_under = 8.0
2222

2323
[tool.dbt-score.badges]
2424
first.threshold = 10.0
@@ -51,8 +51,8 @@ The following options can be set in the `pyproject.toml` file:
5151
- `disabled_rules`: A list of rules to disable.
5252
- `fail_project_under` (default: `5.0`): If the project score is below this
5353
value the command will fail with return code 1.
54-
- `fail_any_model_under` (default: `5.0`): If any model scores below this value
55-
the command will fail with return code 1.
54+
- `fail_any_item_under` (default: `5.0`): If any model or source scores below
55+
this value the command will fail with return code 1.
5656

5757
#### Badges configuration
5858

@@ -70,7 +70,7 @@ All badges except `wip` can be configured with the following option:
7070

7171
- `threshold`: The threshold for the badge. A decimal number between `0.0` and
7272
`10.0` that will be used to compare to the score. The threshold is the minimum
73-
score required for a model to be rewarded with a certain badge.
73+
score required for a model or source to be rewarded with a certain badge.
7474

7575
The default values can be found in the
7676
[BadgeConfig](reference/config.md#dbt_score.config.BadgeConfig).
@@ -86,7 +86,7 @@ Every rule can be configured with the following option:
8686
- `severity`: The severity of the rule. Rules have a default severity and can be
8787
overridden. It's an integer with a minimum value of 1 and a maximum value
8888
of 4.
89-
- `model_filter_names`: Filters used by the rule. Takes a list of names that can
89+
- `rule_filter_names`: Filters used by the rule. Takes a list of names that can
9090
be found in the same namespace as the rules (see
9191
[Package rules](package_rules.md)).
9292

docs/create_rules.md

Lines changed: 56 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,9 @@
11
# Create rules
22

3-
In order to lint and score models, `dbt-score` uses a set of rules that are
4-
applied to each model. A rule can pass or fail when it is run. Based on the
5-
severity of the rule, models are scored with the weighted average of the rules
6-
results. Note that `dbt-score` comes bundled with a
3+
In order to lint and score models or sources, `dbt-score` uses a set of rules
4+
that are applied to each item. A rule can pass or fail when it is run. Based on
5+
the severity of the rule, items are scored with the weighted average of the
6+
rules results. Note that `dbt-score` comes bundled with a
77
[set of default rules](rules/generic.md).
88

99
On top of the generic rules, it's possible to add your own rules. Two ways exist
@@ -21,7 +21,7 @@ The `@rule` decorator can be used to easily create a new rule:
2121
from dbt_score import Model, rule, RuleViolation
2222

2323
@rule
24-
def has_description(model: Model) -> RuleViolation | None:
24+
def model_has_description(model: Model) -> RuleViolation | None:
2525
"""A model should have a description."""
2626
if not model.description:
2727
return RuleViolation(message="Model lacks a description.")
@@ -31,6 +31,21 @@ The name of the function is the name of the rule and the docstring of the
3131
function is its description. Therefore, it is important to use a
3232
self-explanatory name for the function and document it well.
3333

34+
The type annotation for the rule's argument dictates whether the rule should be
35+
applied to dbt models or sources.
36+
37+
Here is the same example rule, applied to sources:
38+
39+
```python
40+
from dbt_score import rule, RuleViolation, Source
41+
42+
@rule
43+
def source_has_description(source: Source) -> RuleViolation | None:
44+
"""A source should have a description."""
45+
if not source.description:
46+
return RuleViolation(message="Source lacks a description.")
47+
```
48+
3449
The severity of a rule can be set using the `severity` argument:
3550

3651
```python
@@ -45,15 +60,23 @@ For more advanced use cases, a rule can be created by inheriting from the `Rule`
4560
class:
4661

4762
```python
48-
from dbt_score import Model, Rule, RuleViolation
63+
from dbt_score import Model, Rule, RuleViolation, Source
4964

50-
class HasDescription(Rule):
65+
class ModelHasDescription(Rule):
5166
description = "A model should have a description."
5267

5368
def evaluate(self, model: Model) -> RuleViolation | None:
5469
"""Evaluate the rule."""
5570
if not model.description:
5671
return RuleViolation(message="Model lacks a description.")
72+
73+
class SourceHasDescription(Rule):
74+
description = "A source should have a description."
75+
76+
def evaluate(self, source: Source) -> RuleViolation | None:
77+
"""Evaluate the rule."""
78+
if not source.description:
79+
return RuleViolation(message="Source lacks a description.")
5780
```
5881

5982
### Rules location
@@ -91,30 +114,48 @@ def sql_has_reasonable_number_of_lines(model: Model, max_lines: int = 200) -> Ru
91114
)
92115
```
93116

94-
### Filtering models
117+
### Filtering rules
95118

96-
Custom and standard rules can be configured to have model filters. Filters allow
97-
models to be ignored by one or multiple rules.
119+
Custom and standard rules can be configured to have filters. Filters allow
120+
models or sources to be ignored by one or multiple rules if the item doesn't
121+
satisfy the filter criteria.
98122

99123
Filters are created using the same discovery mechanism and interface as custom
100124
rules, except they do not accept parameters. Similar to Python's built-in
101-
`filter` function, when the filter evaluation returns `True` the model should be
125+
`filter` function, when the filter evaluation returns `True` the item should be
102126
evaluated, otherwise it should be ignored.
103127

104128
```python
105-
from dbt_score import ModelFilter, model_filter
129+
from dbt_score import Model, RuleFilter, rule_filter
106130

107-
@model_filter
131+
@rule_filter
108132
def only_schema_x(model: Model) -> bool:
109133
"""Only applies a rule to schema X."""
110134
return model.schema.lower() == 'x'
111135

112-
class SkipSchemaY(ModelFilter):
136+
class SkipSchemaY(RuleFilter):
113137
description = "Applies a rule to every schema but Y."
114138
def evaluate(self, model: Model) -> bool:
115139
return model.schema.lower() != 'y'
116140
```
117141

142+
Filters also rely on type-annotations to dictate whether they apply to models or
143+
sources:
144+
145+
```python
146+
from dbt_score import RuleFilter, rule_filter, Source
147+
148+
@rule_filter
149+
def only_from_source_a(source: Source) -> bool:
150+
"""Only applies a rule to source tables from source X."""
151+
return source.source_name.lower() == 'a'
152+
153+
class SkipSourceDatabaseB(RuleFilter):
154+
description = "Applies a rule to every source except Database B."
155+
def evaluate(self, source: Source) -> bool:
156+
return source.database.lower() != 'b'
157+
```
158+
118159
Similar to setting a rule severity, standard rules can have filters set in the
119160
[configuration file](configuration.md/#tooldbt-scorerulesrule_namespacerule_name),
120161
while custom rules accept the configuration file or a decorator parameter.
@@ -123,7 +164,7 @@ while custom rules accept the configuration file or a decorator parameter.
123164
from dbt_score import Model, rule, RuleViolation
124165
from my_project import only_schema_x
125166

126-
@rule(model_filters={only_schema_x()})
167+
@rule(rule_filters={only_schema_x()})
127168
def models_in_x_follow_naming_standard(model: Model) -> RuleViolation | None:
128169
"""Models in schema X must follow the naming standard."""
129170
if some_regex_fails(model.name):

docs/get_started.md

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -11,8 +11,8 @@ Installation of `dbt-score` is simple:
1111
pip install dbt-score
1212
```
1313

14-
If a virtual environment is used to run dbt, make sure to install `dbt-score` in
15-
the same environment.
14+
In order to run `dbt-score` with all its features, be sure to install
15+
`dbt-score` in the same environment as `dbt-core`.
1616

1717
## Usage
1818

@@ -40,8 +40,8 @@ It's also possible to automatically run `dbt parse`, to generate the
4040
dbt-score lint --run-dbt-parse
4141
```
4242

43-
To lint only a selection of models, the argument `--select` can be used. It
44-
accepts any
43+
To lint only a selection of models or sources, the argument `--select` can be
44+
used. It accepts any
4545
[dbt node selection syntax](https://docs.getdbt.com/reference/node-selection/syntax):
4646

4747
```shell

docs/index.md

Lines changed: 15 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -2,41 +2,42 @@
22

33
`dbt-score` is a linter for [dbt](https://www.getdbt.com/) metadata.
44

5-
dbt allows data practitioners to organize their data in to _models_. Those
6-
models have metadata associated with them: documentation, tests, types, etc.
5+
dbt allows data practitioners to organize their data in to _models_ and
6+
_sources_. Those models and sources have metadata associated with them:
7+
documentation, tests, types, etc.
78

89
`dbt-score` allows to lint and score this metadata, in order to enforce (or
910
encourage) good practices.
1011

1112
## Example
1213

1314
```
14-
> dbt-score lint --show-all
15-
🥇 customers (score: 10.0)
15+
> dbt-score lint --show all
16+
🥇 M: customers (score: 10.0)
1617
OK dbt_score.rules.generic.has_description
17-
OK dbt_score.rules.generic.has_owner: Model lacks an owner.
18+
OK dbt_score.rules.generic.has_owner
1819
OK dbt_score.rules.generic.sql_has_reasonable_number_of_lines
1920
Score: 10.0 🥇
2021
```
2122

2223
In this example, the model `customers` scores the maximum value of `10.0` as it
2324
passes all the rules. It also is awarded a golden medal because of the perfect
24-
score. By default a passing model with or without rule violations will not be shown,
25-
unless we pass the `--show-all` flag.
25+
score. By default a passing model with or without rule violations will not be
26+
shown, unless we pass the `--show-all` flag.
2627

2728
## Philosophy
2829

29-
dbt models are often used as metadata containers: either in YAML files or
30-
through the use of `{{ config() }}` blocks, they are associated with a lot of
30+
dbt models/sources are often used as metadata containers: either in YAML files
31+
or through the use of `{{ config() }}` blocks, they are associated with a lot of
3132
information. At scale, it becomes tedious to enforce good practices in large
32-
data teams dealing with many models.
33+
data teams dealing with many models/sources.
3334

3435
To that end, `dbt-score` has 2 main features:
3536

36-
- It runs rules on models, and displays rule violations. Those can be used in
37-
interactive environments or in CI.
38-
- Using those run results, it scores models, as to give them a measure of their
39-
maturity. This score can help gamify model metadata improvements, and be
37+
- It runs rules on dbt models and sources, and displays any rule violations.
38+
These can be used in interactive environments or in CI.
39+
- Using those run results, it scores items, to ascribe them a measure of their
40+
maturity. This score can help gamify metadata improvements/coverage, and be
4041
reflected in data catalogs.
4142

4243
`dbt-score` aims to:

docs/programmatic_invocations.md

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,7 @@ $ dbt-score lint --format json
3232
"severity": "medium",
3333
"message": "Model lacks an owner."
3434
},
35-
"dbt_score.rules.generic.public_model_has_example_sql": {
35+
"dbt_score.rules.generic.has_example_sql": {
3636
"result": "OK",
3737
"severity": "low",
3838
"message": null
@@ -61,9 +61,9 @@ When `dbt-score` terminates, it exists with one of the following exit codes:
6161
project being linted either doesn't raise any warning, or the warnings are
6262
small enough to be above the thresholds. This generally means "successful
6363
linting".
64-
- `1` in case of linting errors. This is the unhappy case: some models in the
65-
project raise enough warnings to have a score below the defined thresholds.
66-
This generally means "linting doesn't pass".
64+
- `1` in case of linting errors. This is the unhappy case: some models or
65+
sources in the project raise enough warnings to have a score below the defined
66+
thresholds. This generally means "linting doesn't pass".
6767
- `2` in case of an unexpected error. This happens for example if something is
6868
misconfigured (for example a faulty dbt project), or the wrong parameters are
6969
given to the CLI. This generally means "setup needs to be fixed".

0 commit comments

Comments
 (0)