Question about the sort/order of repeating training images

[bjspi]

Hi all,

I have 5000 reg images and 50 training images of myself.
Right now, I'm using 100 repeats so that I can make use of all 5k reg images, because 50*100 gives me 5k training samples.

In total, this yields 501002 = 10k steps.

I noticed, that quite early, I'd say at step 2-3k, the models are quite good already.
To avoid overfitting, I want to use the "earliest" checkpoint (least number of steps) which fits my needs w.r.t. reproducing my model.

Now the question is:
let's assume the earliest modell which fits my needs is already at around step 2000 and I chose this one.
It means, 1k steps were reg-images, 1k steps are training samples.
Do those 1k samples effectively contain ALL my 50 training images (only repeated for *20 times)?
Or is every training image repeated for 100 times first and than the next image is trained, which would mean, at step 1k, only 10 samples have been used for training with 100 repeats each?

Thanks a lot!

[5KilosOfCheese]

1 epoch means going through all the images once (including repeats)
So if you have 10 images with 2 repeats, one epoch is same as 2 imags.

Also I'd recommend that instead of 100 repeats, you'd do 10 repeats and 10 epochs.

I don't train anything for more than 3000-4000 steps which is with my batch size of 4-6 less than 1000 steps. Because if it isn't working with that, it ain't working and I need to rethink the whole things. Most things I train I get done at like 2500 steps.

Here is a thought thou. Give this a try, which is what I do.
Start with the 50 images you go, add repeat of 2 so each epoch is 100 steps (at batch size 1). Don't add regularisation images yet. Then do 10 epochs, and see where that gets you. If you likeness is trained to a decent level keep the repeats and such at that. If not add repeats and epochs. I usually add repeats by 2 at a time and epochs 5 at a time. Once you get a model which gets your looks, then keep those. Now add good variety of reg images for your next training run. Start with a calm ratio of 1:1 and expand as needed until the model gets the properties you want.

I personally am firm believer in "You don't need reg images by default", this is because by testing and talking with people who also train a lot... We been just able to get things done without them - almost always. Only in some specific cases where the training is failing to catch on I add regularisation - but I generally have bad luck with getting them to work. I have realised that if it doesn't work without, it probably wont work with them and this is because I am doing something fundamentally wrong.

And I'm nore sure if I understoof your question right. But the order of images in training is set by the seed.

[bjspi]

I guess what you meant in your second sentence is: "same as 20 images".

And again about my question: I do use the feature to "save every N steps" - so actually even if I only use 1 epoch, I save intermediate checkpoints. And that's where my curiosity starts. E.g.: If I take 50 images and 100 repeats, it means I have 5k steps. So my question is:
What type of data is trained e.g. let's say at step 2000? Is it: All my 50 pictures have been trained with 40 repeats (50*40 = 2k)? OR is it like that: The first 20 images have been trained with 100 repeats, but all other 30 images have not yet been trained at all.

Thats actually my question.
That's why I'm talking about the "order / sort" of train.

[5KilosOfCheese]

You can use "save every epoch" also.

The order the images are presented to the training system is based on the seed. All the pictures are put into a long list that the system goes through, including repeats so if training images have repeat of 5 the are added to the list 5 times. At the end of the list the system has passed 1 epoch. However if you use buckets then things get bit more complex, which can lead to epoch and step counter (if you got batch size of more than 1) showing that less steps than youd expect is needed.

So there is a risk that if you don't use full epoch, your training pictures might not be fully used during the training. Because the system never gets to the end of the list (all pictures show). Since the seed determines the order, there is no way you can know whether all the images been shows.

If you have 100 pictures and 10 repeats for total of 1000 steps. Then you stop training at 500 steps, there is a risk that some images have not been shown at all. (This is the answer I assume you are looking for). Only way to be sure that all images have been used in the training is to use epochs.