README.txt

I wrapped my Hessian-free code in a generic class, usable as a black-box to train your models if you can provide the cost function as a Theano expression.

It includes all the details in Martens (ICML 2010) and Martens & Sutskever (ICML 2011) crucial to make it work:
- Tikhonov damping with the Levenberg-Marquardt heuristics,
- Gauss-Newton matrix products (you specify an Theano expression `s` to section your computational graph in 2),
- Proper handling of batches and mini-batches (an example SequenceDataset class is provided for variable-length input)
- Conjugate gradient (CG) with information sharing, backtracking, preconditioning and terminations conditions.
- Structural damping for RNNs.

It relies heavily on the Rop. In practice, I could make it work without hassle for a feed-forward network, an RNN with different objectives, NADE (Larochelle) and a more complex model (RNN-NADE) that ties two scans together, so it seems reasonably flexible.
Only the gradients and Gauss-Newton matrix products (95% of the computation) are in Theano, CG and the training logic is in python. It runs on GPU, but for the models I tried, it was a bit slower.
Hessian-free is slow, you need CG batch sizes of 1000+ (don't skimp on this), but you can get really better results than SGD from it with almost zero tweaking.

There is an option to save and recover a checkpoint of training and do early stopping.

I included an RNN example that can memorize an input for 100 time steps (example_RNN). Launch it on 4 cores, come back in 8 hours, and you should have at least one nice solution with 0 error on the validation set.
In comparison, SGD can solve this problem about 0.0% of the time.

It is available here:
https://github.com/boulanni/theano-hf


Author: Nicolas Boulanger-Lewandowski
University of Montreal, 2012