[Core Feature][Not ready] Allow to configure agent status update frequency

[popojk]

Tracking issue

Related to #5733

Why are the changes needed?

In the current Flyte backend, a single connector plugin is registered to handle all connector task types. The configuration file allows setting a global cache resync interval and rate limit for the connector plugin. However, these settings apply uniformly across all connector task types, limiting flexibility.

To improve configurability, we propose initializing a separate connector plugin instance for each configured connector task type. This approach enables fine-grained control, allowing distinct cache resync intervals and rate limits per task type.

The proposed system architecture is illustrated below:

In Flyte config file, we may configure cache resync interval and rate limit for each connector deployment

What changes were proposed in this pull request?

1. In Propeller TaskHandler, add a watcher to watch message from connector global watcher to update/create defaultPlugins in TaskHandler.
2. In PluginMachinery registry, add a pluginChan channel for connector global watcher to send message.
3. In webapi connector, add global watcher. The global watcher will watch all deployments defined in config file on routine basis to retrieve supported task types from each deployment. A connector plugin will be registered for a task type of each deployment.

How was this patch tested?

Labels

Please add one or more of the following labels to categorize your PR:

• added: For new features.

This is important to improve the readability of release notes.

Setup process

Screenshots

Check all the applicable boxes

• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.

Related PRs

Docs link

Summary by Bito

This pull request significantly enhances the Flyte backend's connector plugin system, allowing for individual configuration of cache resync intervals and rate limits for each task type. It introduces a global watcher for dynamic management of connector deployments and refines the plugin registration process, improving overall flexibility and performance.

[codecov]

Codecov Report

❌ Patch coverage is 77.77778% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 59.66%. Comparing base (ecc61c0) to head (73f07cb).

Files with missing lines	Patch %	Lines
flytestdlib/cache/in_memory_auto_refresh.go	77.77%	3 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6552      +/-   ##
==========================================
+ Coverage   58.69%   59.66%   +0.97%     
==========================================
  Files         938      551     -387     
  Lines       71613    37780   -33833     
==========================================
- Hits        42031    22541   -19490     
+ Misses      26394    13577   -12817     
+ Partials     3188     1662    -1526     

Flag	Coverage Δ
unittests-datacatalog	59.03% <ø> (ø)
unittests-flyteadmin	56.11% <ø> (ø)
unittests-flytecopilot	39.56% <ø> (ø)
unittests-flytectl	64.72% <ø> (ø)
unittests-flyteidl	76.12% <ø> (ø)
unittests-flyteplugins	?
unittests-flytepropeller	?
unittests-flytestdlib	64.07% <77.77%> (+0.03%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[flyte-bot]

Bito Review Skipped - No Changes Detected

Bito didn't review this pull request because we did not detect any changes in the pull request to review.