User-Defined Function Raised Exception

[patrick-lee-warren]

I tried to run the python package on a set of about 40k tweets and got this error:

I tried some smaller subsamples... same thing. Any ideas?

[patrick-lee-warren]

FYI: Python version 3.9.12, sqlite version 3.38.2.

[deimosnz]

Hi,

I'm still looking into this, sorry. I'll get back to you.

Rob

[riozhu-GZ]

The error arises because
len(set_1 | set_2) might be 0, where the denominator can't be equal to 0.

Please ensure that the dataset does not contain any data with empty tweet text. Or the code can be changed to:

def similarity(tokens_1, tokens_2):
    set_1 = set(tokens_1.split())
    set_2 = set(tokens_2.split())
    if len(set_1 | set_2) == 0:
        return 0
    else:
        return len(set_1 & set_2) / len(set_1 | set_2)