Draft: possible way of adding validation loop to training script

[jetjodh]

I wanted a validation loop by the way of creating a split from training data by withholding some percent of training data and using it to validate current model training by calculating loss on the holdout data every n steps defined in config. This is a quick and dirty try to implement this mechanism without changing much of the codebase.
I would like some pointers if my current way is sound or there is a much better way (probably is), and how I should go about it.

[jetjodh]

@jaretburkett a quick review?