Search results

Iterating over a HF dataset with <code>num_workers&gt;0</code> after iterating over a HF dataset with <code>num_workers=0</code> hangs

bugSomething isn't workingSomething isn't working

help wantedExtra attention is neededExtra attention is needed

Status: Open.

#636 In Lightning-AI/litdata;

· philgzl opened on Jun 25, 2025

Command to profile &amp; clear the cache

enhancementNew feature or requestNew feature or request

questionFurther information is requestedFurther information is requested

Status: Open.

#626 In Lightning-AI/litdata;

· robmarkcole opened on Jun 13, 2025

Explore Optimizing and Running Tests in Parallel for Faster CI in <code>litdata</code>

ci / tests

help wantedExtra attention is neededExtra attention is needed

Status: Open.

#612 In Lightning-AI/litdata;

· bhimrazy opened on Jun 4, 2025

Optimize fails with StreamingDataloader as inputs for &gt;=5 workers while optimizing hf dataset

bugSomething isn't workingSomething isn't working

help wantedExtra attention is neededExtra attention is needed

Status: Open.

Bug

#599 In Lightning-AI/litdata;

· bhimrazy opened on May 20, 2025

Add example for hugging face dataset optimization and benchmarks reports

documentationImprovements or additions to documentationImprovements or additions to documentation

Status: Open.

#558 In Lightning-AI/litdata;

· bhimrazy opened on Apr 17, 2025

<code>OutOfBoundsError</code> when streaming parquet files with <code>low_memory=True</code>

bugSomething isn't workingSomething isn't working

help wantedExtra attention is neededExtra attention is needed

waiting on authorWaiting for user input or feedback.Waiting for user input or feedback.

Status: Open.

Bug

#553 In Lightning-AI/litdata;

· kyoungrok0517 opened on Apr 13, 2025

Cycle option for StreamingDataLoader

enhancementNew feature or requestNew feature or request

waiting on authorWaiting for user input or feedback.Waiting for user input or feedback.

Status: Open.

Feature

#524 In Lightning-AI/litdata;

· Aceticia opened on Mar 24, 2025

Local cache dir not fully clearing in DDP multi-node training.

bugSomething isn't workingSomething isn't working

help wantedExtra attention is neededExtra attention is needed

Status: Open.

#512 In Lightning-AI/litdata;

· JackUrb opened on Mar 12, 2025

Add batch sampler for StreamDataloader to enable flexiable training strategy.

enhancementNew feature or requestNew feature or request

Status: Open.

#509 In Lightning-AI/litdata;

· xinsir6 opened on Mar 11, 2025

How to optimimize dataset for pretraining from HuggingFace

bugSomething isn't workingSomething isn't working

questionFurther information is requestedFurther information is requested

Status: Open.

#482 In Lightning-AI/litdata;

· TheLukaDragar opened on Feb 21, 2025

CI error: <code>All chunks should've been deleted</code> keeps coming back

bugSomething isn't workingSomething isn't working

help wantedExtra attention is neededExtra attention is needed

Status: Open.

#437 In Lightning-AI/litdata;

· deependujha opened on Dec 20, 2024

Restart training with new data, mid-epoch

enhancementNew feature or requestNew feature or request

won't fix

Status: Open.

#436 In Lightning-AI/litdata;

· schopra8 opened on Dec 17, 2024