Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

payments: expose remaining elavon batch types and modify row access policy #3419

Merged
merged 1 commit into from
Aug 18, 2024

Conversation

charlie-costanzo
Copy link
Member

@charlie-costanzo charlie-costanzo commented Aug 16, 2024

Description

This PR creates new unique intermediate tables for the two remaining batch types of Elavon transactions found in stg_elavon__transactions that are not yet exposed in tables

New batch types and tables:

  • A found in new table int_elavon__adjustment_transactions
  • C found in new table int_elavon__chargeback_transactions

the YML file for these tables have also been updated in the dbt project

This PR also adds the service account for metabase-payments-team to the create_row_access_policy macro to more easily facilitate access to payments tables with row access policies by payments data team members

Resolves #3411

As some background:
we used batch types B and D when creating the fct_elavon__transactions table. When constructing that, we tried to incorporate these other two batch types for reconciliation (the batch types above, A and C ) but they are aggregated at a different grain so we were unable to use them for those calculations.
However, as this batch data is valuable on it's own, once this PR is merged they will be available as their own tables in the warehouse and Metabase to be used for analysis.

Type of change

  • New feature

How has this been tested?

locally with dbt run and dbt test

Post-merge follow-ups

  • Actions required (specified below)
  • [ ]
    Metabase will need to be configured to expose these two new tables in the staging scheme to the Payments Team Warehouse in Metabase

We will also need to verify that the payments team service account is successfully accessing the payments tables protected by the row access policy

Copy link

Warehouse report 📦

Checks/potential follow-ups

Checks indicate the following action items may be necessary.

  • For new models, do they all have a surrogate primary key that is tested to be not-null and unique?

New models 🌱

calitp_warehouse.staging.payments.elavon.int_elavon__adjustment_transactions

calitp_warehouse.staging.payments.elavon.int_elavon__chargeback_transactions

DAG

Legend (in order of precedence)

Resource type Indicator Resolution
Large table-materialized model Orange Make the model incremental
Large model without partitioning or clustering Orange Add partitioning and/or clustering
View with more than one child Yellow Materialize as a table or incremental
Incremental Light green
Table Green
View White

@charlie-costanzo charlie-costanzo changed the title payments: expose all elavon transaction data batch types and modify row access policy payments: expose remaining elavon batch types and modify row access policy Aug 16, 2024
Copy link
Contributor

@erikamov erikamov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even though I don't know much yet about the Elavon data, it is easy to understand the change!
Great job!

@charlie-costanzo charlie-costanzo merged commit 796a3e1 into main Aug 18, 2024
4 checks passed
@charlie-costanzo charlie-costanzo deleted the expose-all-elavon-batches branch August 18, 2024 22:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Elavon Batch Data
2 participants