[Package] Semantic Types (Typed Entities)

[kyleaton]

What Are Semantic Types?

That's a great question! Semantic Types categorize data by the type of information it represents. For example, consider a column with STRING data types. How do you know what the data means? You could be looking at a list of streets, cities, counties, etc. Without background information, it can be hard to know what you're looking at. That's where Semantic Types come in. The entity’s Semantic Type defines how it should be interpreted.

Why should we test on Semantic Types?

Testing on these Semantic Types allows us to create more explicit, fit-for-purpose tests, unlocking questions like Does this column contain US State abbreviations? instead of asking Does this column contain strings in the set [AL, AL, AR, AZ...]

How do you know when data entities are good candidates for Semantic Types testing?

Any data that you can tie to a real-world category or reference is ideal for this kind of test, e.g., phone numbers, ZIP codes, countries, coordinates, URLs, email addresses, etc.

Linked below are a number of Expectations on Semantic Types that have already been contributed to Great Expectations!

[austiezr]

• Expect Column Values Are In Language
• Expect Column Values To Be A Non-Bot User Agent
• Expect Column Values To Be ASCII
• Expect Column Values To Be Secure Passwords
• Expect Column Values To Be US Zipcode Within Radius Of Given Zipcode
• Expect Column Values To Be Valid Degree-Decimal Coordinates
• Expect Column Values To Be Valid URLs
• Expect Column Values To Be Valid UUID
• Expect Column Values To Be Valid Wikipedia Articles
• Expect Column Values To Contain Valid Email

[joshua-stauffer]

I'm implementing a metric which calculates an input against the Luhn algorithm, and will contribute expect_column_values_to_be_valid_luhn_number and expect_column_values_to_be_valid_credit_card_number, which will both use the metric. There are a number of other identifiers which use the algorithm, so I might get around to some of them as well.

[voidforall]

Hello, I am new to join the hackathon and really interested in contributing to the Semantic Types!

I have submitted a PR yesterday to expect column values to be valid IPv4 address. It's a trivial feature and I am also trying to learn the contribution process from it. I guess it could exist many problems and your review can help me understand and fix them. @austiezr @kyleaton

[austiezr]

Hey Hackathon People!

As we come into the final hours of the Great Expectations Hackathon, we wanted to briefly highlight what we believe would be some quick wins in the Semantic Types category enabled by Regex-Based Column Map Expectations and Set-Based Column Map Expectations.

Regex-Based Column Map Expectations

• expect_column_values_to_be_valid_msisdn
• expect_column_values_to_be_valid_upc_a
• expect_column_values_to_be_valid_upc_e
• expect_column_values_to_be_valid_ean_13
• expect_column_values_to_be_valid_isbn_10

Set-Based Column Map Expectations

• expect_column_values_to_be_valid_top_level_domain
• expect_column_values_to_be_valid_http_method
• expect_column_values_to_be_valid_iso_639_1
• expect_coolumn_values_to_be_valid_zipcode_prefixes_in_new_hampshire
• expect_column_values_to_be_valid_canadian_administrative_divisions

---

Creating expectations of these types can be as simple as identifying a valid regex or value-set and writing a couple of tests; the new RegexBased and SetBasedColumnMapExpectations will take care of the rest!

[szecsip]

@austiezr Thank you for the suggestions. I implemented most barcode validating in one expectation with a third-party package and the users can set the barcode type. (PR #4664)

[Derekma73]

@austiezr can you clear up what the intent behind posting the expect_coolumn_values_to_be_valid_zipcode_prefixes_in_new_hampshire as an expectation was? It was my understanding that that level of granularity for an expectation was acceptable with the idea that all custom expectations could be further organized in the contrib folder. When thinking about the end-user, my thought was that it would be easier for them to assign an extremely specific custom expectation to their column rather than spend time configuring the expectation (more clicks now = less clicks for them later).