Block size, block size statistics and distributions

[illeamb]

The comparison numbers generated for different blocking rules are really nice for example from the tutorial:

blocking_rule_1 = "substr(l.first_name,1,1) = substr(r.first_name,1,1) and l.surname = r.surname"
count = linker.count_num_comparisons_from_blocking_rule(blocking_rule_1)
print(f"Number of comparisons generated by '{blocking_rule_1}': {count:,.0f}")

Is there a way to get some information on the block size and some block size statistics (i.e min block size, max block size, average block size, distribution of the blocks etc.) easily?

[RossKen]

Hi @illeamb, we don't have anything that gives you summary stats for blocking out of the box, unfortunately.

There is a function that works in the background of the blocking rules chart that may be of use. Using the data from that tutorial:

from splink.analyse_blocking import cumulative_comparisons_generated_by_blocking_rules

blocking_rules = [blocking_rule_1, blocking_rule_2]

cumulative_comparisons_generated_by_blocking_rules(linker,blocking_rules)

Gives

[{'row_count': 473,
  'rule': 'substr(l.first_name,1,1) = substr(r.first_name,1,1) and l.surname = r.surname',
  'cumulative_rows': 473,
  'cartesian': 499500,
  'reduction_ratio': 'The rolling reduction ratio with your given blocking rule(s) is 0.999. \nThis represents the reduction in the total number of comparisons due to your rule(s).',
  'start': 0},
 {'row_count': 1165,
  'rule': 'l.surname = r.surname',
  'cumulative_rows': 1638,
  'cartesian': 499500,
  'reduction_ratio': 'The rolling reduction ratio with your given blocking rule(s) is 0.997. \nThis represents the reduction in the total number of comparisons due to your rule(s).',
  'start': 473}]

Which gets you part of the way there. It does feel like this could be a useful addition so I will have raised an issue #1106. I can't guarantee we will get round to it anytime soon so if you make any progress with it please let us know and it would be great to include anything you create into splink!