feat: Use pyproject.toml for python packages within the repo

[zazulam]

Chore description

Given the number of python packages we host in this repo:

• kfp
• kfp-server-api
• kfp-pipeline-spec
• kfp-kubernetes

Each of them use a setup.py for their packaging. As an end user I would like to have a pyproject.toml used for defining the packages' details for their build requirements.

The pyproject.toml has become the recommended way for configuring package related tools/information for targeting different build systems in light of PEP 518 and PEP 621 .

I believe this modern aspect of packaging would ease maintenance by having a central location for dependencies and tool specific settings i.e. pytest, ruff, uv etc.

I would assume there would need to be some sort of hierarchical pyproject.toml structure or either independent ones for each package.

Labels

/area sdk

---

Love this idea? Give it a 👍.

[CollinHowland]

I would be down to investigate/take a stab at this.

Thinking ahead based on some of the reasoning behind this issue: uv env management could be a little tricky because it expects one source folder. I think there are some configurations to work around this but that could inform the direction of the implementation of this if you all want to move to uv.

[aniketpati1121]

@zazulam I’d like to pick this up. Do you prefer having independent pyproject.toml files for each package, or a single hierarchical pyproject.toml at the repo root?

[imadnansyed]

Hi @zazulam I’d like to pick this up and work on migrating the packages to use pyproject.toml as described.

I’ll start by evaluating whether a single hierarchical pyproject.toml or independent files per package makes the most sense, and will open a PR once I have a proposal.

[CollinHowland]

/assign

[amanscisingh]

@zazulam

Hi, I was also interested in working on this migration. I was experimenting with pyproject.toml migration for kfp-kubernetes. I noticed there hasn’t been an update in a while — would it be okay if I help move this forward?

I have also validated the migration for kubernetes_platform setup.py.

[CollinHowland]

I have a commit I want to create a PR for so that I can get feedback on how to approach the rest of the packages. However, I'm having some issues getting things committed with some of the pre-commit hooks + I'm on Windows which is having a complaint about what kind of wheels it can install. Let me see if I can get it up tomorrow after messing around with it more.