VS Code extension disable hook fail to initialise when use with other Kedro plugin

[noklam]

Description

Split out from #51.

Part of settings.py:

from kedro_pandera.framework.config.resolvers import (  # noqa: E402
    resolve_dataframe_model,
    resolve_interpolated_yaml_schema,
    resolve_yaml_schema,
)

CONFIG_LOADER_ARGS = {
    "base_env": "base",
    "default_run_env": "local",
    "custom_resolvers": {
        # For vscode-kedro extension to work we need to import kedro_pandera resolvers manually
        "pa.dict": resolve_yaml_schema,
        "pa.yaml": resolve_interpolated_yaml_schema,
        "pa.python": resolve_dataframe_model,
    },
}

which is basically this - https://github.com/Galileo-Galilei/kedro-pandera/blob/4fa5ca3c8bbb31f2fcbed00e3dbcd03594ff4a01/kedro_pandera/framework/hooks/pandera_hook.py#L46

The error is thrown during initialization of the extension

Context

The reason for this to happen is because VS Code extension disable all the hooks. The failure is because the kedro-pandera resolver is registered via hook (and is now missing), thus the initialisation step fails when we try to get catalog config.

An example can be found here:

example_iris_data:
  type: pandas.CSVDataset
  filepath: data/01_raw/iris.csv
  metadata:
    pandera:
      schema: ${pa.yaml:_example_iris_data_schema}

Steps to Reproduce

Checklist

• Have you read the "Requirements" section already?
• Have you make sure you are running the extension with the correct virtual environment? Cmd + P and Python: Select Interpeter

Expected Result

Actual Result

-- If you received an error, place it here.

-- Separate them if you have more than one.

Your Environment

• Kedro version used (pip show kedro or kedro -V):
• Kedro VSCode version used
• Python version used (python -V):
• Operating system and version:

[Calychas]

Answering #51 (comment):

Just need the import error to go away ;). Any highlighting and Go to definition of a schema would be a major + though

[astrojuanlu]

vscode-kedro/bundled/tool/lsp_server.py

Lines 112 to 114 in 2523edf

	# todo: less hacky way to override session hook manager
	# avoid initialise spark hooks etc
	session._hook_manager = _NullPluginManager()

[astrojuanlu]

@noklam Do you recall what the original motivation of disabling hooks was?

[noklam]

Hooks are usually not needed. For example you don't want the LS running a spark session in the background.

Maybe a more reasonable approach is let users to configure which hooks to enable/disable (though I believe most people will not touch any settings, so the default should work with most cases)

[rashidakanchwala]

hey @noklam, just checking in — is this still an active issue? Haven’t seen much discussion on it recently. Do we already have a proposed solution, or should we spike this further?

[noklam]

@rashidakanchwala It was originally mentioned in #51. The issues is similar to Kedro-Viz where there is a lite mode that spin up with minimal dependencies and a more heavy mode that will use hooks. Occasionally people may create dataset/pipelines from hooks (not 100% sure if that's possible now in 1.0), so if hooks is not used, they will not be able to see this. On the other side, the extension (language server) should really be low resource, you don't want to trigger a Spark hook for auto-completion. So the challenge is how to keep this flexible while avoiding this to happen.

I don't recalled we have a proposed solution, this is somewhat a rare case so it wasn't prioritised.

[rashidakanchwala]

We’ve seen the same issue in Viz, and it only comes up with the Spark hook. Right now, we only have DISABLE_HOOKS_FOR_PLUGINS in settings.py which doesn't help much because it disables entirely for that plugin. Do you think introducing something along those lines would help, since this problem really revolves around Spark?

DISABLE_SPARK_HOOKS = True
DISABLE_HOOKS = ["spark"]

[noklam]

@rashidakanchwala I don't think we should have a specific config just for Spark. (Also, not sure how is it feasible given that often you have a spark hook inside a project, not necessarily a plugin).

I think fundamentally the problem is. These are runtime behavior but currently it's only possible to change it in settings.py.

i.e. you may want to have the hook enable when you do kedro run, but you want the VSCocde extension to run it without the hook. The current way of doing this requires changing settings.py unlike Kedro CLI which you can passed as an extra argument. Note this is true for all other Kedro plugin CLI as well. i.e with data validation, you may want to do something like kedro run --no-validation but this is not possible right now because the Kedro CLI is not easily extendible.