Stochastic version of Prox/FW splitting for separable variables

[GeoffNN]

Stochastic version of PR #52

[fabianp]

that would indeed be cool. Is there any published work on this? Perhaps @gideonite knows

[GeoffNN]

None that I've found. Very probably this is just applying momentum in the right place(s). :)

The setting I'm looking at is \min_{X\in C, Y} f(X + Y) + g(Y), where f is an expectation of smooth functions, g is non smooth and we can compute an LMO on C. Ideally I want to avoid variance reduction to apply this to neural nets: typically f will be non convex.

[gideonite]

Thanks for the quick summary, @GeoffNN, very helpful =)

To be honest, I don't think this is relevant, but:
there is some work on stochastic FW for f + g where g is non smooth and either f or (f and g) are stochastic, e.g. Vladarean et al 2020. But these works generally use some kind of variance reduction and assume convexity. They also don't assume split operators so that the argument to g is also assumed to be in C. We will also have some work coming out soon on this topic which I will link to as soon as possible.

I am curious, why not variance reduction for neural nets?

[GeoffNN]

Thanks for the refs!

On VR for Neural Nets: https://arxiv.org/abs/1812.04529