Add bandwidth limit feature for HTTP and S3 datasources

[Copilot]

This PR adds bandwidth limiting capability to prevent data downloads from consuming all available bandwidth and affecting other services in production environments.

Changes

Core Implementation

• HTTP Datasource: Added bandwidthLimit option support that adds --bwlimit flag to rclone commands
• S3 Datasource: Added bandwidthLimit option support that adds --bwlimit flag to rclone commands
• Documentation: Updated API documentation to include the new option for both HTTP and S3 sources

Usage

Users can now specify a bandwidth limit in their Dataset configurations:

apiVersion: dataset.baizeai.io/v1alpha1
kind: Dataset
metadata:
  name: my-dataset
spec:
  source:
    type: HTTP  # or S3
    uri: http://example.com/data/
    options:
      bandwidthLimit: "10M"  # Limit to 10 MiB/s

Bandwidth Limit Format

The bandwidth limit follows rclone's format:

• Plain numbers are interpreted as KiB/s
• Suffixes can be used: B, K, M, G, T, P
• Examples: "1M" (1 MiB/s), "500K" (500 KiB/s), "2G" (2 GiB/s)

Testing

• Added comprehensive tests for both HTTP and S3 datasources with bandwidth limits
• Verified that the --bwlimit flag is correctly added to rclone commands
• Created sample configuration files demonstrating usage

Implementation Details

The feature leverages rclone's built-in --bwlimit flag for bandwidth throttling:

1. Parse the bandwidthLimit option from Dataset configuration
2. Store it in the respective loader options struct
3. Add --bwlimit <value> to rclone command arguments when present
4. rclone handles the actual bandwidth limiting during download

This approach is minimal, surgical, and leverages existing rclone capabilities for reliability.

Fixes #3.

---

💬 Share your feedback on Copilot coding agent for the chance to win a $200 gift card! Click here to start the survey.

[kebe7jun]

Do you have any solutions for other types of Datasets?

[kebe7jun]

Do you have any solutions for other types of Datasets?

[Copilot]

Same as above - bandwidth limiting can be extended to other dataset types beyond HTTP and S3, though implementation complexity varies. The current approach leverages rclone's built-in --bwlimit flag, which is why it's straightforward for HTTP and S3. Other types would require external bandwidth limiting tools like trickle or traffic control.

[nekomeowww]

Suggested change

	secretRef: s3-credentials
	secretRef: s3-credentials

[Copilot]

Do you have any solutions for other types of Datasets?

Yes, bandwidth limiting can be extended to other dataset types, though the implementation complexity varies:

Currently supported: HTTP and S3 (leveraging rclone's built-in --bwlimit)

Possible future extensions:

• Git: Could use external tools like trickle or tc (traffic control) to wrap git commands, though Git doesn't have native bandwidth limiting
• HuggingFace/ModelScope: Could similarly use external bandwidth limiting tools to wrap the CLI downloads
• Conda: Package manager downloads could be wrapped with external tools

Not applicable: PVC, NFS, REFERENCE (no network downloads involved)

The HTTP/S3 implementation is straightforward because rclone has built-in bandwidth limiting. For other types, we'd need external tools which adds complexity and dependencies. Would you like me to implement any specific dataset type?

[kebe7jun]

Do you have any solutions for other types of Datasets?

Yes, bandwidth limiting can be extended to other dataset types, though the implementation complexity varies:

Currently supported: HTTP and S3 (leveraging rclone's built-in --bwlimit)

Possible future extensions:

• Git: Could use external tools like trickle or tc (traffic control) to wrap git commands, though Git doesn't have native bandwidth limiting
• HuggingFace/ModelScope: Could similarly use external bandwidth limiting tools to wrap the CLI downloads
• Conda: Package manager downloads could be wrapped with external tools

Not applicable: PVC, NFS, REFERENCE (no network downloads involved)

The HTTP/S3 implementation is straightforward because rclone has built-in bandwidth limiting. For other types, we'd need external tools which adds complexity and dependencies. Would you like me to implement any specific dataset type?

Help improve Copilot by leaving feedback using the 👍 or 👎 buttons

Can you help me add the feature to limit bandwidth using trickle? In this PR.

[Copilot]

Do you have any solutions for other types of Datasets?

Yes, bandwidth limiting can be extended to other dataset types, though the implementation complexity varies:
...

I've implemented bandwidth limiting using trickle for Git, HuggingFace, and ModelScope dataset types!

The implementation:

• Git: Wraps git clone and git pull commands with trickle
• HuggingFace: Wraps huggingface-cli download commands with trickle
• ModelScope: Wraps modelscope download commands with trickle

The bandwidth limit format is consistent with the existing HTTP/S3 implementation (KiB/s with B|K|M|G|T|P suffixes) and automatically converts to trickle's format.

Usage example:

spec:
  source:
    type: GIT
    uri: https://github.com/user/repo.git
    options:
      bandwidthLimit: "5M"  # Limit to 5 MiB/s

The feature is fully tested and includes sample configuration files. Commit: ebdb19d

[kebe7jun]

Is it possible to integrate trickle directly into the entrypoint of the data-loader Docker image? In this way, there is no need to add wrap when executing the command. The speed limit value can be passed through the environment variable.

[Copilot]

Is it possible to integrate trickle directly into the entrypoint of the data-loader Docker image? In this way, there is no need to add wrap when executing the command. The speed limit value can be pas...

Yes! I've successfully integrated trickle directly into the data-loader Docker entrypoint. The implementation:

• Dockerfile: Now installs trickle and uses a new entrypoint script
• Entrypoint script: Checks for BANDWIDTH_LIMIT environment variable and wraps data-loader with trickle when set
• Controller: Passes bandwidthLimit option as environment variable instead of command argument
• Datasources: Removed all individual command wrapping logic for cleaner code

The bandwidth limit value is automatically converted from rclone format (KiB/s with B|K|M|G|T|P suffixes) to trickle's KB/s format. Users still specify bandwidth limits the same way in their Dataset configurations - the change is transparent.

Benefits:

• Simpler, cleaner implementation
• More reliable (trickle controls all container network traffic)
• Consistent behavior across all datasource types
• No dependencies on external bandwidth limiting tools in datasource code

Commit: 333b1ef