[bug] Compiling pipeline parameters with NoneType raises protobuf error

[jimhortle]

Environment

• How do you deploy Kubeflow Pipelines (KFP)?
  Deployed on Vertex AI by compiling to YAML and then calling PipelineJob.submit(). Currently failing at compilation step, however.

• KFP version:

• KFP SDK version: 2.5, 2.7, 2.13.0 (Python SDK)

Steps to reproduce

We have the following compilation function, which we use to convert a YAML file to a dataclass for parameter sanitisation, which is then turned into a dictionary for compilation.

from pathlib import Path
from dataclasses import asdict
from kfp.compiler import Compiler
from kfp.components import BaseComponent

def compile_pipeline(
    path_to_yaml_config: Path | str, 
    pipeline_config_dataclass: type[PipelineConfig], 
    pipeline_func: BaseComponent, 
    pipeline_package_path: Path | str | None = None
):
    [... sanity checks etc]

    with path_to_yaml_config.open() as fp:
          yaml_dict = yaml.safe_load(fp)
    pipeline_config = pipeline_config_dataclass(**yaml_dict)

     [...]

     Compiler().compile(
        pipeline_func=pipeline_func,
        package_path=...,
        pipeline_parameters=asdict(pipeline_config)
     )

When the YAML config file contains null values (usually nested inside a mapping or sequence, but also top-level), these get converted to Python None values when loading in the YAML file. These values have previously compiled and submitted without error.

This now leads to a protobuf error within the recursive call to kfp.compiler.pipeline_spec_builder.to_protobuf_value() :

... line 79 in to_protobuf_value
struct_value=struct_pb2.Struct(
TypeError: bad argument type for built-in operation

Our version of protobuf hasn't changed and I can see that there is a _NullValue type in the protobuf types file in google.protobuf.struct_pb2. Removing all the Python Nones and YAML nulls causes the error to change but is still occurring in the recursive calls to to_protobuf_value().

I think the if-else isinstance checks in to_protobuf_value() are missing the NoneType case, or alternatively the compiler has changed in some way that causes NoneType to raise an exception.

Depending on what nested structures are in the YAML file, sometimes those nested structures are unpacked and lead to:

ValueError('Value must be one of the following types...')
which is from the else-block of the isinstance() checks in to_protobuf_value().

I would like to know what has changed between these minor versions of KFP that could have caused this change in behaviour.

Expected result

The pipeline compiles. We have successfully compiled and submitted pipelines in exactly the same fashion on KFP 2.5, 2.7 and 2.13. The behaviour appears to have changed recently but I can't see any recent changes to the compiler or protobuf that would affect this.

Materials and reference

Labels

/area sdk

---

Impacted by this bug? Give it a 👍.