Stalling while training.

[BlueEyedFoxUnderscore]

I downloaded my dataset and have started training my method using this command:
!yolo task=detect mode=train model=yolov8s.pt batch=8 data=/data.yaml epochs=25 imgsz=128 plots=True device=0 verbose=True

Instead of actually training the model, it makes a "runs" folder with one attempt directory, generates a bunch of threads, eats through about half of my memory, and then stalls (not necessarily in that order). I made sure it wasn't actually doing anything silently -- it's been running for 90 mins approx. and no prints, new image files, weights, or anything, the only things it generated were an args.yaml file and a file named "events.out.tfevents..Omega..0". The .zip file has one events.out.tf... file and one args.yaml file.

Any ideas? Things I did wrong? Thanks!

examples.zip

[glenn-jocher]

@Pagzma09 hello! It sounds like you're encountering a stalling issue during the training process. Let's try to troubleshoot this step by step:

1. Check Dataset: Ensure your dataset is correctly formatted and accessible at the specified location. The data.yaml should correctly reference the train and val datasets with the correct paths.

2. System Resources: Verify that your system has enough resources (RAM, GPU memory) to handle the training process. Training can be resource-intensive, and insufficient memory might cause stalling.

3. Verbose Output: Since you've enabled verbose=True, you should be seeing output logs. If there's no output, it might indicate an early stall before training begins.

4. Hardware Acceleration: Confirm that your GPU is being utilized. You've specified device=0, which should use the first GPU. If you're not on a GPU-enabled system, this could cause issues.

5. Software Versions: Make sure you're using the latest version of the Ultralytics YOLOv8 repo and that your environment meets all the necessary dependencies.

6. Event File: The events.out.tfevents file is a TensorBoard log file. It's normal for this file to be generated, but it won't help with the stalling issue.

If after checking these points the issue persists, you may want to simplify your command (e.g., remove plots=True and verbose=True) to see if a more basic setup works. If the problem continues, please provide more details about your system configuration and any error messages you might be receiving.

For further guidance, you can refer to our documentation on the Train mode, which might help clarify any configuration issues. Remember, the community and the Ultralytics team are here to help! 🚀