error creating linker object in pyspark

[cvandenende]

Hi,

I have been trying to initialize a linker object in a pyspark environment following the steps described in: Workshop Materials/notebooks/Splink in Databricks - Expert Answer.py

I am trying to keep the environment as simple as possible:

%pip install splink --quiet

Importing the SparkLinker:

from splink.spark.spark_linker import SparkLinker

This throws up an error: ImportError: initialization failed

The traceback leads to a line which appears to import duckdb:

<command-3308449202875057> in <module>
----> 1 from splink.spark.spark_linker import SparkLinker
      2 from splink.databricks.enable_splink import enable_splink
      3 import splink.spark.spark_comparison_library as cl
      4 import splink.spark.spark_comparison_level_library as cll
      5 

/local_disk0/.ephemeral_nfs/envs/pythonEnv-de522752-c270-41ee-827b-8caaf4b6c787/lib/python3.7/site-packages/splink/spark/spark_linker.py in <module>
     13 from pyspark.sql.utils import AnalysisException
     14 
---> 15 from ..linker import Linker
     16 from ..splink_dataframe import SplinkDataFrame
     17 from ..term_frequencies import colname_to_tf_tablename

/local_disk0/.ephemeral_nfs/envs/pythonEnv-de522752-c270-41ee-827b-8caaf4b6c787/lib/python3.7/site-packages/splink/linker.py in <module>
     24 from .blocking import block_using_rules_sql, BlockingRule
     25 from .comparison_vector_values import compute_comparison_vector_values_sql
---> 26 from .em_training_session import EMTrainingSession
     27 from .misc import bayes_factor_to_prob, prob_to_bayes_factor, ensure_is_list
     28 from .predict import predict_from_comparison_vectors_sqls

/local_disk0/.ephemeral_nfs/envs/pythonEnv-de522752-c270-41ee-827b-8caaf4b6c787/lib/python3.7/site-packages/splink/em_training_session.py in <module>
      5 import logging
      6 
----> 7 from .expectation_maximisation import expectation_maximisation
      8 from .misc import bayes_factor_to_prob, prob_to_bayes_factor
      9 from .parse_sql import get_columns_used_from_sql

/local_disk0/.ephemeral_nfs/envs/pythonEnv-de522752-c270-41ee-827b-8caaf4b6c787/lib/python3.7/site-packages/splink/expectation_maximisation.py in <module>
      4 from typing import TYPE_CHECKING
      5 import time
----> 6 import duckdb
      7 
      8 from .predict import predict_from_comparison_vectors_sqls

I am not sure if I am overlooking something, or if there may be a conflict with another package?

[RobinL]

I don't have access to databricks so difficult to test. What happens if you try to run import duckdb directly? What error message do you get if any?

[NickCrews]

Also, run the pip install not in quiet mode and post the output

[cvandenende]

Thanks for the responses, when I import duckdb directly I continue getting the same error message:

ImportError                               Traceback (most recent call last)
<command-3308449202875032> in <module>
----> 1 import duckdb
      2 from splink.spark.spark_linker import SparkLinker
      3 settings = { "link_type": "dedupe_only", "unique_id_column_name": "report_id" }
      4 linker = SparkLinker(df, settings)

ImportError: generic_type: type "DuckDBPyRelation" is already registered!

pip install splink output not quiet:

Python interpreter will be restarted.
Collecting splink
  Downloading splink-3.6.0-py3-none-any.whl (2.4 MB)
Collecting jsonschema<5.0,>=3.2
  Downloading jsonschema-4.17.3-py3-none-any.whl (90 kB)
Requirement already satisfied: pandas<2.0.0,>=1.0.0 in /databricks/python3/lib/python3.7/site-packages (from splink) (1.0.1)
Collecting Jinja2<4.0.0,>=3.0.3
  Downloading Jinja2-3.1.2-py3-none-any.whl (133 kB)
Collecting duckdb<0.7.0,>=0.6.0
  Downloading duckdb-0.6.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (14.4 MB)
Collecting sqlglot>=5.1.0
  Downloading sqlglot-11.2.3-py3-none-any.whl (233 kB)
Collecting altair<5.0.0,>=4.2.0
  Downloading altair-4.2.2-py3-none-any.whl (813 kB)
Collecting typing-extensions; python_version < "3.8"
  Downloading typing_extensions-4.5.0-py3-none-any.whl (27 kB)
Collecting pkgutil-resolve-name>=1.3.10; python_version < "3.9"
  Downloading pkgutil_resolve_name-1.3.10-py3-none-any.whl (4.7 kB)
Collecting pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0
  Downloading pyrsistent-0.19.3-py3-none-any.whl (57 kB)
Collecting importlib-resources>=1.4.0; python_version < "3.9"
  Downloading importlib_resources-5.12.0-py3-none-any.whl (36 kB)
Collecting importlib-metadata; python_version < "3.8"
  Downloading importlib_metadata-6.0.0-py3-none-any.whl (21 kB)
Collecting attrs>=17.4.0
  Downloading attrs-22.2.0-py3-none-any.whl (60 kB)
Requirement already satisfied: python-dateutil>=2.6.1 in /databricks/python3/lib/python3.7/site-packages (from pandas<2.0.0,>=1.0.0->splink) (2.8.1)
Requirement already satisfied: numpy>=1.13.3 in /databricks/python3/lib/python3.7/site-packages (from pandas<2.0.0,>=1.0.0->splink) (1.18.1)
Requirement already satisfied: pytz>=2017.2 in /databricks/python3/lib/python3.7/site-packages (from pandas<2.0.0,>=1.0.0->splink) (2019.3)
Collecting MarkupSafe>=2.0
  Downloading MarkupSafe-2.1.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB)
Requirement already satisfied: entrypoints in /databricks/python3/lib/python3.7/site-packages (from altair<5.0.0,>=4.2.0->splink) (0.3)
Collecting toolz
  Downloading toolz-0.12.0-py3-none-any.whl (55 kB)
Collecting zipp>=3.1.0; python_version < "3.10"
  Downloading zipp-3.15.0-py3-none-any.whl (6.8 kB)
Requirement already satisfied: six>=1.5 in /databricks/python3/lib/python3.7/site-packages (from python-dateutil>=2.6.1->pandas<2.0.0,>=1.0.0->splink) (1.14.0)
Installing collected packages: typing-extensions, pkgutil-resolve-name, pyrsistent, zipp, importlib-resources, importlib-metadata, attrs, jsonschema, MarkupSafe, Jinja2, duckdb, sqlglot, toolz, altair, splink
Successfully installed Jinja2-3.1.2 MarkupSafe-2.1.2 altair-4.2.2 attrs-22.2.0 duckdb-0.6.1 importlib-metadata-6.0.0 importlib-resources-5.12.0 jsonschema-4.17.3 pkgutil-resolve-name-1.3.10 pyrsistent-0.19.3 splink-3.6.0 sqlglot-11.2.3 toolz-0.12.0 typing-extensions-4.5.0 zipp-3.15.0
WARNING: You are using pip version 20.0.2; however, version 23.0.1 is available.
You should consider upgrading via the '/local_disk0/.ephemeral_nfs/envs/pythonEnv-9182b1ef-d592-4aa1-9d6c-4b6e3e93e3cf/bin/python -m pip install --upgrade pip' command.
Python interpreter will be restarted.

[cvandenende]

Thanks both for your input. This problem appears to be related to our local databricks cluster and we will pursue it with them, there is no problem when importing the spark linker on the databricks community cluster.

[RobinL]

Thanks for confirming!