Merge branch 'master' into release

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,35 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: "[BUG] "
+labels: ''
+assignees: DVAlexHiggs
+
+---
+
+**Describe the bug**
+A clear and concise description of what the bug is.
+
+**Versions**
+
+dbt: 
+dbtvault: 
+
+**To Reproduce**
+Steps to reproduce the behavior:
+1. Go to '...'
+2. Click on '....'
+3. Scroll down to '....'
+4. See error
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+**Screenshots**
+If applicable, add screenshots to help explain your problem.
+
+**Log files**
+If applicable, provide dbt log files which include the problem.
+
+**Additional context**
+Add any other context about the problem here.

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,20 @@
+---
+name: Feature request
+about: Suggest an idea for this project
+title: "[FEATURE]"
+labels: ''
+assignees: DVAlexHiggs
+
+---
+
+**Is your feature request related to a problem? Please describe.**
+A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+
+**Describe the solution you'd like**
+A clear and concise description of what you want to happen.
+
+**Describe alternatives you've considered**
+A clear and concise description of any alternative solutions or features you've considered.
+
+**Additional context**
+Add any other context or screenshots about the feature request here.

.gitignore

@@ -9,4 +9,5 @@
 invoke.yml
 Pipfile
 Pipfile.lock
-tasks.py
+tasks.py
+.idea/

CONTRIBUTING.md

@@ -1,14 +1,16 @@
+## Contributing new features
+
+Please refer to our contribution guidelines over on our [development repository](https://github.com/Datavault-UK/dbtvault-dev/blob/master/CONTRIBUTING.md)
+
 ## We'd love to hear from you
 
 dbtvault is very much a work in progress – we’re constantly adding quality of life improvements and will be adding
 new table types regularly.
 
-We know that it deserves new features, that the code base can be tidied up and the SQL better tuned. 
-
-Rest assured we’re working on it for future releases – [our roadmap contains information on what’s coming](roadmap.md).
+Rest assured we’re working on future releases – [our roadmap contains information on what’s coming](https://dbtvault.readthedocs.io/en/latest/roadmap/).
 
 If you spot anything you’d like to bring to our attention, have a request for new features, have spotted an improvement we could make, 
-or want to tell us about a typo or bug, then please don’t hesitate to let us know via [Github](https://github.com/Datavault-UK/dbtvault/issues). 
+or want to tell us about a typo or bug, then please don’t hesitate to let us know via [github](https://github.com/Datavault-UK/dbtvault/issues). 
 
 We’d rather know you are making active use of this package than hearing nothing from all of you out there! 
 
@@ -30,8 +32,4 @@ We'd love to add new features to make this package even more useful for the comm
 please feel free to submit ideas and thoughts!
 
 ### If it's an idea, feedback or a general inquiry
-Create a post with as much detail as possible; We'll be happy to reply and work with you.
-
-## Pull requests
-If you've developed something which we can add via a pull request, we're more than happy to consider it, but we'd
-like to discuss the changes first.
+Create a post with as much detail as possible; We'll be happy to reply and work with you.

README.md

@@ -3,15 +3,18 @@
 </p>
 
 <p align="center">
-  <a href="https://dbtvault.readthedocs.io/en/latest/?badge=stable"><img
+  <a href="https://dbtvault.readthedocs.io/en/stable/?badge=stable"><img
     src="https://readthedocs.org/projects/dbtvault/badge/?version=stable" 
     alt="Documentation Status"
   /></a>
   <a href="https://join.slack.com/t/dbtvault/shared_invite/enQtODY5MTY3OTIyMzg2LWJlZDMyNzM4YzAzYjgzYTY0MTMzNTNjN2EyZDRjOTljYjY0NDYyYzEwMTlhODMzNGY3MmU2ODNhYWUxYmM2NjA"><img
     src="https://img.shields.io/badge/Slack-Join-yellow?style=flat&logo=slack" 
     alt="Join our slack"
   /></a>
-
+  <a href="https://circleci.com/gh/Datavault-UK/dbtvault-dev"><img
+    src="https://circleci.com/gh/Datavault-UK/dbtvault-dev/tree/master.svg?style=shield" 
+    alt="CircleCI"
+  /></a>
 </p>
 
 
@@ -44,20 +47,8 @@ Learn quickly with our worked example:
 
 ## Installation
 
-Add the following to your `packages.yml`
-
-
-```yaml
-packages:
-
-  - git: "https://github.com/Datavault-UK/dbtvault"
-    revision: v0.7.0 # Latest stable version
-```
-
-And run 
-`dbt deps`
-
-[Read more on package installation](https://docs.getdbt.com/docs/building-a-dbt-project/package-management/#git-packages)
+Check [dbt Hub](https://hub.getdbt.com/datavault-uk/dbtvault/latest/) for the latest installation instructions, 
+or [read the docs](https://docs.getdbt.com/docs/building-a-dbt-project/package-management/) for more information on installing packages.
 
 ## Usage
 

dbt_project.yml

@@ -1,10 +1,9 @@
 name: 'dbtvault'
-version: '0.7.0'
+version: '0.7.2'
 require-dbt-version: [">=0.18.0", "<0.19.0"]
+config-version: 2
 
-profile: 'dbtvault'
-
-source-paths: ["models", "models_test"]
+source-paths: ["models"]
 analysis-paths: ["analysis"] 
 test-paths: ["tests"]
 data-paths: ["data"]
@@ -13,9 +12,8 @@ docs-paths: ["docs"]
 
 target-path: "target"
 clean-targets:
-    - "target"
-    - "dbt_modules"
+  - "target"
+  - "dbt_modules"
 
-models:
-  vars:
-    hash: MD5
+vars:
+  hash: MD5

docs/materialisations/helpers/get_period_filter_sql.md

@@ -1,6 +1,6 @@
 {% docs macro__get_period_filter_sql %}
 
-A wrapper around the `replace_placeholder_with_filter` macro which creates a query designed to
+A wrapper around the `replace_placeholder_with_period_filter` macro which creates a query designed to
 build a temporary table, to select the necessary records for the given load cycle. 
 
 {% enddocs %}

docs/materialisations/helpers/replace_placeholder_with_filter.md

@@ -1,12 +1,12 @@
-{% docs macro__replace_placeholder_with_filter %}
+{% docs macro__replace_placeholder_with_period_filter %}
 
 Replace the `__PERIOD_FILTER__` string present in the given SQL, with a `WHERE` clause which filters data by a
 specific `period` of time, `offset` from the `start_date`.
 
 {% enddocs %}
 
 
-{% docs arg__replace_placeholder_with_filter__core_sql %}
+{% docs arg__replace_placeholder_with_period_filter__core_sql %}
 
 SQL string containing the `__PERIOD_FILTER__` string.
 

docs/staging.md

@@ -3,7 +3,7 @@
 A macro to aid in generating a staging layer for the raw vault. Allows users to:
 
 - Create new columns from already existing columns (Derived columns)
-- Create new hashed columns from already existing columns (Hashed columns)
+- Create new hashed columns from already existing columns and provided derived columns (Hashed columns)
 
 [Read more online](https://dbtvault.readthedocs.io/en/latest/macros/#stage)
 

macros/internal/alias.sql

@@ -1,14 +1,14 @@
 {%- macro alias(alias_config=none, prefix=none) -%}
 
-    {{- adapter.dispatch('alias', packages = ['dbtvault'])(alias_config=alias_config, prefix=prefix) -}}
+    {{- adapter.dispatch('alias', packages = dbtvault.get_dbtvault_namespaces())(alias_config=alias_config, prefix=prefix) -}}
 
 {%- endmacro %}
 
 {%- macro default__alias(alias_config=none, prefix=none) -%}
 
-{%- if alias_config -%}
+{%- if alias_config is defined and alias_config is not none and alias_config -%}
 
-    {%- if alias_config is iterable and alias_config is not string -%}
+    {%- if alias_config is mapping -%}
 
         {%- if alias_config['source_column'] and alias_config['alias'] -%}
 

macros/internal/alias_all.sql

@@ -1,12 +1,12 @@
 {%- macro alias_all(columns=none, prefix=none) -%}
 
-    {{- adapter.dispatch('alias_all', packages = ['dbtvault'])(columns=columns, prefix=prefix) -}}
+    {{- adapter.dispatch('alias_all', packages = dbtvault.get_dbtvault_namespaces())(columns=columns, prefix=prefix) -}}
 
 {%- endmacro %}
 
 {%- macro default__alias_all(columns, prefix) -%}
 
-{%- if columns is iterable and columns is not string -%}
+{%- if dbtvault.is_list(columns) -%}
 
     {%- for column in columns -%}
         {{ dbtvault.alias(alias_config=column, prefix=prefix) }}
@@ -17,6 +17,12 @@
 
 {{ dbtvault.alias(alias_config=columns, prefix=prefix) }}
 
-{%- endif -%}
+{%- else -%}
+
+    {%- if execute -%}
+        {{ exceptions.raise_compiler_error("Invalid columns object provided. Must be a list or a string.") }}
+    {%- endif %}
+
+{%- endif %}
 
 {%- endmacro -%}

macros/internal/as_constant.sql

@@ -1,12 +1,12 @@
 {%- macro as_constant(column_str=none) -%}
 
-    {{- adapter.dispatch('as_constant', packages = ['dbtvault'])(column_str=column_str) -}}
+    {{- adapter.dispatch('as_constant', packages = dbtvault.get_dbtvault_namespaces())(column_str=column_str) -}}
 
 {%- endmacro %}
 
 {%- macro default__as_constant(column_str) -%}
 
-    {% if column_str is not none %}
+    {% if column_str is not none and column_str is string and column_str %}
 
         {%- if column_str | first == "!" -%}
 
@@ -17,6 +17,10 @@
             {{- return(column_str) -}}
 
         {%- endif -%}
+    {%- else -%}
+        {%- if execute -%}
+            {{ exceptions.raise_compiler_error("Invalid columns_str object provided. Must be a string and not null.") }}
+        {%- endif %}
     {%- endif -%}
 
 {%- endmacro -%}

macros/internal/expand_column_list.sql

@@ -8,34 +8,41 @@
 
 {%- set col_list = [] -%}
 
-{%- if columns is iterable -%}
+{%- if dbtvault.is_list(columns) -%}
 
     {%- for col in columns -%}
 
         {%- if col is string -%}
 
-            {%- set _ = col_list.append(col) -%}
+            {%- do col_list.append(col) -%}
 
         {#- If list of lists -#}
-        {%- elif col is iterable and col is not string -%}
+        {%- elif dbtvault.is_list(col) -%}
 
-            {%- if col is mapping -%}
+            {%- for cols in col -%}
 
-                {%- set _ = col_list.append(col) -%}
+                {%- do col_list.append(cols) -%}
 
-            {%- else -%}
+            {%- endfor -%}
+        {%- elif col is mapping -%}
 
-                {%- for cols in col -%}
+            {%- do col_list.append(col) -%}
 
-                    {%- set _ = col_list.append(cols) -%}
+        {%- else -%}
 
-                {%- endfor -%}
-
-            {%- endif -%}
+            {%- if execute -%}
+                {{ exceptions.raise_compiler_error("Invalid columns object provided. Must be a list of lists, dictionaries or strings.") }}
+            {%- endif %}
 
         {%- endif -%}
 
     {%- endfor -%}
+{%- else -%}
+
+    {%- if execute -%}
+        {{ exceptions.raise_compiler_error("Invalid columns object provided. Must be a list.") }}
+    {%- endif %}
+
 {%- endif -%}
 
 {% do return(col_list) %}

macros/internal/get_package_namespaces.sql

@@ -0,0 +1,4 @@
+{%- macro get_dbtvault_namespaces() -%}
+    {%- set override_namespaces = var('adapter_packages', []) -%}
+    {%- do return(override_namespaces + ['dbtvault']) -%}
+{%- endmacro -%}

macros/internal/is_checks.sql

@@ -0,0 +1,33 @@
+{%- macro is_list(obj, empty_is_false=false) -%}
+
+    {%- if obj is iterable and obj is not string and obj is not mapping -%}
+        {%- if obj is none and obj is undefined and not obj and empty_is_false -%}
+            {%- do return(false) -%}
+        {%- endif -%}
+
+        {%- do return(true) -%}
+    {%- else -%}
+        {%- do return(false) -%}
+    {%- endif -%}
+
+{%- endmacro -%}
+
+{%- macro is_nothing(obj) -%}
+
+    {%- if obj is none or obj is undefined or not obj -%}
+        {%- do return(true) -%}
+    {%- else -%}
+        {%- do return(false) -%}
+    {%- endif -%}
+
+{%- endmacro -%}
+
+{%- macro is_something(obj) -%}
+
+    {%- if obj is not none and obj is defined and obj -%}
+        {%- do return(true) -%}
+    {%- else -%}
+        {%- do return(false) -%}
+    {%- endif -%}
+
+{%- endmacro -%}

macros/internal/multikey.sql

@@ -1,6 +1,6 @@
 {%- macro multikey(columns, prefix=none, condition=none, operator='AND') -%}
 
-    {{- adapter.dispatch('multikey', packages = ['dbtvault'])(columns=columns, prefix=prefix, condition=condition, operator=operator) -}}
+    {{- adapter.dispatch('multikey', packages = dbtvault.get_dbtvault_namespaces())(columns=columns, prefix=prefix, condition=condition, operator=operator) -}}
 
 {%- endmacro %}
 
@@ -20,7 +20,7 @@
             {%- if not loop.last %} {{ operator }} {% endif %}
         {% endfor -%}
     {%- else -%}
-        {%- if columns is iterable and columns is not string -%}
+        {%- if dbtvault.is_list(columns) -%}
             {%- for col in columns -%}
                 {{ prefix[0] ~ '.' if prefix }}{{ col }} {{ condition if condition else '' }}
                 {%- if not loop.last -%} {{ "\n    " ~ operator }} {% endif -%}