You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
Data Loader from pytorch was designed for accessing files from Disk, however, this results to a very slow and very low utilization of high performance accelerators resulting to slower completion and inefficient training.
This can be alleviated by manually batching via for loop, and can result to about 2000% speed up! So that is very much performance improvement.
Describe the solution you'd like
De-couple the pytorch from pytorch lighting to enable manual batching
Describe alternatives you've considered
n/a
Additional context
n/a
The text was updated successfully, but these errors were encountered:
jobs-git
changed the title
[ENH] 2000% performance improvement by directly without using data loader
[ENH] 2000% performance improvement by directly batching without using dataloader
May 26, 2025
jobs-git
changed the title
[ENH] 2000% performance improvement by directly batching without using dataloader
[ENH] Improve performance by 2000% by directly batching without using dataloader
May 26, 2025
Is your feature request related to a problem? Please describe.
Data Loader from pytorch was designed for accessing files from Disk, however, this results to a very slow and very low utilization of high performance accelerators resulting to slower completion and inefficient training.
This can be alleviated by manually batching via for loop, and can result to about 2000% speed up! So that is very much performance improvement.
Describe the solution you'd like
De-couple the pytorch from pytorch lighting to enable manual batching
Describe alternatives you've considered
n/a
Additional context
n/a
The text was updated successfully, but these errors were encountered: