IDEA: plot cumulative_num_comparisons_from_blocking_rules_chart in the style of https://upset.app/

[NickCrews]

When I'm looking at blocking rules, I might be curious how many pairs are

• in blocking rule A, and no other rules
• in both rules B and C
• in rule D and not E

In the implementation, the order of the blocking rules doesn't matter, since we do a DISTINCT on them. But when visualizing them, the order that you specify the rules matters. Thus, if you re-order your rules you can get vastly different impressions of which rules generate a lot of records, and which ones generate a small amount.

Really what we are interested in is visualizing the amount of intersect between a lot of different sets. It looks like https://upset.app/ is made for this. Not immediately needed, but this would be a cool goal to work towards I think. Not sure what the python/altair story is here.

[NickCrews]

There is this, which looks sorta complex and I wouldn't want to maintain, but perhaps they would be open to turning it into a library: https://github.com/hms-dbmi/upset-altair-notebook

[RobinL]

Interesting - I agree those charts look very useful.

This is also pertinent to the work on this pr, where I'm trying to autodetect

• blocking rules for prediction; and
• blocking rules for EM training

One aspect of this work is I'm trying to find a cost function that is able to evaluate/compare 'how good' different sets of blocking rules are. See here. Such a cost function would have different weights depending on whether you're looking for training or prediction blocking rules.

At the moment this cost function does not take account of the size of the intersection, but ideally it would.

The biggest challenge here is that we have a new function name tbc with pr here that is very fast at finding the number of comparisons generated by a blocking rule. But the speed comes from avoiding generating the comparisons (and hence you also can't use it to compute intersections).

At the moment, of the main things the cost function is looking for is how much freedom different columns get from the set of blocking rules:
e.g.

"l.first_name  = r.first_name and l.dob = r.dob",
"l.first_name  = r.first_name and l.surname = r.surname"

Doesn't allow first_name to vary at all. surname and dob both have 'freedom' of 1. All other fields have freedom of 2.

Re: the upset.app charts, I was actually working on a inferior but somewhat similar version of these charts the other day (that deal with the combinations, but not intersections) :

Here's the script that makes that chart:

```python import pandas as pd from splink.duckdb.linker import DuckDBLinker import altair as alt import itertools

df = pd.read_parquet(

"/Users/robinlinacre/Documents/data_linking/splink/synthetic_50k_clean.parquet"

)

df = pd.read_parquet(
"/Users/robinlinacre/Documents/data_linking/splink/synthetic_1m_clean.parquet"
)
df.head()

settings = {"link_type": "dedupe_only"}
linker = DuckDBLinker(df, settings)

import itertools
import time

num_rows = len(df)

if len(df) > 1e6:
fields = [ "first_name", "surname", "dob", "birth_place", "postcode_fake", "gender", "occupation", ]
else:
fields = [ "first_name", "middle_names", "surname", "dob", "birth_place", "postcode", "lat", "lng", "gender", "occupation", ]

threshold_maximum = 20e6

results = []
for r in range(1, 3):
for combination in itertools.combinations(fields, r):
br = " AND ".join([f"l.{item} = r.{item}" for item in combination])
count = linker._count_num_comparisons_from_blocking_rule_pre_filter_conditions(
br
)
succinct = ", ".join(combination)
result = {"blocking_rule": br, "count": count, "succinct": succinct}

    # One-hot encode the fields in the combination
    for field in fields:
        result[field] = 1 if field in combination else 0

    results.append(result)

results_df = pd.DataFrame(results)
print(results_df.head(2).to_markdown(index=False))

tick_values = [10**i for i in range(1, 15)]

Creating the bar chart

chart = (
alt.Chart(results_df)
.mark_bar()
.encode(
y=alt.Y("blocking_rule:N", title="", axis=alt.Axis(labels=False), sort="-x"),
x=alt.X(
"count:Q",
title="Count",
scale=alt.Scale(type="symlog", constant=1e5),
axis=alt.Axis(values=tick_values),
),
tooltip=["blocking_rule:N", alt.Tooltip("count:Q", format=",.0f")],
)
).properties(width=600)

Add the text on top of the bar

text = chart.mark_text(align="left", baseline="middle", dx=3).encode(
text=alt.Text("succinct:N"),
x=alt.value(1.1),
y=alt.Y(
"blocking_rule:N",
title="",
axis=alt.Axis(labels=False),
sort=alt.EncodingSortField(field="count", order="descending"),
),
)

Render the chart

(chart + text)

Can we plot a grid on the rhs with ticks and crosses according to the blocking vars?

df_melted = results_df.melt(
id_vars=["blocking_rule", "count", "succinct"],
var_name="attribute",
value_name="value",
)

Create the chart

chart2 = (
alt.Chart(df_melted)
.mark_rect()
.encode(
x="attribute:N",
y=alt.Y(
"blocking_rule:N",
title="",
axis=alt.Axis(labels=False),
sort=alt.EncodingSortField(field="count", order="descending"),
),
color="value:N",
tooltip=["blocking_rule:N", alt.Tooltip("count:Q", format=",.0f")],
)
)

Display the chart

c = (chart2 | (chart + text))
c

</details>

[NickCrews]

Thanks Robin, that looks cool. I forgot that I already started this discussion, and remade an issue for this 🤭 at #1614. Maybe we track it there in a proper issue?