Skip to content

[ENH] Improve performance by 2000% by directly batching without using dataloader #1846

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jobs-git opened this issue May 26, 2025 · 0 comments · May be fixed by #1850
Open

[ENH] Improve performance by 2000% by directly batching without using dataloader #1846

jobs-git opened this issue May 26, 2025 · 0 comments · May be fixed by #1850
Labels
enhancement New feature or request

Comments

@jobs-git
Copy link

Is your feature request related to a problem? Please describe.
Data Loader from pytorch was designed for accessing files from Disk, however, this results to a very slow and very low utilization of high performance accelerators resulting to slower completion and inefficient training.

This can be alleviated by manually batching via for loop, and can result to about 2000% speed up! So that is very much performance improvement.

Describe the solution you'd like
De-couple the pytorch from pytorch lighting to enable manual batching

Describe alternatives you've considered
n/a

Additional context
n/a

@jobs-git jobs-git changed the title [ENH] 2000% performance improvement by directly without using data loader [ENH] 2000% performance improvement by directly batching without using dataloader May 26, 2025
@jobs-git jobs-git changed the title [ENH] 2000% performance improvement by directly batching without using dataloader [ENH] Improve performance by 2000% by directly batching without using dataloader May 26, 2025
@jobs-git jobs-git linked a pull request May 27, 2025 that will close this issue
4 tasks
@fkiraly fkiraly added the enhancement New feature or request label May 27, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants