GitHub · Where software is built

Labels Milestones New issue

Training loss will increase drastically when resume from ckpts after adopting self-defined sampler in multi-GPU experiments [Fabric]

accelerator: cuda

#21312

· Moondok opened

on Oct 24, 2025

Abnormally slow both single-gpu & DDP training, what is the problem here?

#20702

· Mollylulu opened

on Apr 7, 2025

Validation stuck when trainers have different data size

trainer: validate

#20561

· btian opened

on Jan 23, 2025

Multi-gpu training with slurm times out

environment: slurm

#20434

· nightingal3 opened

on Nov 19, 2024

Model Checkpointing + FSDP causes Cuda OOM

callback: model checkpoint

#20312

· profPlum opened

on Oct 1, 2024

<code>_update_dataloader</code> improperly copies state of subclassed dataloader with attribute names that differ from <code>init</code> parameters.

#20265

· spenceforce opened

on Sep 8, 2024

Weights are misshappen when using model's forward in on_fit_end() hook with FSDP

#20255

· QuentinAndre11 opened

on Sep 6, 2024

Unexpected Behavior: <code>Fabric.load</code> operates out-of-place on nested states

#20208

· Markus28 opened

on Aug 17, 2024

7x slower training speed when switching from lightning 1.0 to 2.0

#20201

· MaiBe-ctrl opened

on Aug 14, 2024

Checkpoint callback run before validation step - stale or none monitor values considered for validation metrics

callback: model checkpoint

#20185

· PheelaV opened

on Aug 10, 2024