multiple phonenumbers / emailadresses per customer

[lightningdave]

Hi, I just starting to look into splink for a MDM challenge.

When I have the MDM data from 10 sources.
And a customer-record can have multiple phonenumbers and emailadresses (up to 6 for some syststems!).

How do I deal with this? Do I need to pivot the data into phonenumber1, phonenumber2, phonenumber3, etc.?
And then make matching conditions for all combinations of phonenumbers?

customterPK	phone1	phone2	phone3
1	12341	12342	12343
2	12344	12345	12346

Of do I store the unpivoted with all combinations of email/phonenumber for a customer?

customterPK	phone
1	12341
1	12342
1	12343
2	12344
2	12345
2	12346

(next to phone number there are postal adresses, customernames, emailadresses, company names, but the above shows the problem i am having how to load multiple 'columns' of the same informationtype).

Or is there a different way to go about this?

Thanks!

[aalexandersson]

As a new user of splink, I do not have an answer other than to mention that, yes, there is a different way to go about this:

1. For each unpivoted dataset, de-duplicate the dataset.
2. Link the two de-duplicated datasets.

The pivoted (wide) form violates the assumption of independent variables.
The unpivoted (long) form violates the assumption of independent observations.
An additional complication is that deduplication in splink is hard to use:

A. There are two splink settings for de-duplication: dedupe_only and link_and_dedupe. Which setting is best here?
B. splink finds but does not remove duplicates. You will need extra non-splink code for that.

[adelinor]

Hi,

I faced the same issue and started by creating columns as you suggest phone1, phone2. This though does not work well for writing comparison on the multi-valued attribute: the phone in your case.

The approach I took was to create the list of values as an array type under a single column: let say phones in your example. In comparisons, we verify that at least one value is common between left and right: see #1053 .

If you need to apply blocking rules on this multi-valued data, you may want to evaluate the percentage of records having a single value. If that proportion is significant you could apply the blocking rule, say l.phones = r.phones when the length is one and let through all cases where left or right have more than one entry.

Kind regards