What is a Transformation?

[tbreloff]

I was about to start writing code, but I think it's worth having a discussion first. If we're going to realize my eventual vision of autonomous creation of valid "directed graphs of transformations", then we need to approach the problem systematically. Every transformation should have the same type signature, otherwise we're going to have too many special cases for the framework to be useful.

A Transformation should:

• Know what input shape it expects (is it generative? if so, maybe nothing is the input. does it always get a scalar? or a vector?)
• Know the input domain (can take any numbers? only reals? only probabilities?)
• Likewise, it should know the output shape and domain. (Sigmoid produces probabilities, for example)

Should this info be part of the type signature as parameters? It might have to be, though I'm going to attempt to solve this through "query functions" that produce traits to be dispatched on. If I'm successful, we don't need the wrappers for Distributions at all... we just need the generic:

output_shape(::Distributions.Sampleable{Univariate}) = 0  # scalar
output_shape(::Distributions.Sampleable{Multivariate}) = 1 # vector
# etc

For generative/stochastic transformations (generating distributions):

I wonder if we should think of the input as "randomness". i.e. we take "randomness" as input, and generate rand(t, output_dims). I don't know the best way to represent this... maybe just immutable Randomness end

The transform should, I think, take in nothing, and output the center. Think about if you have a generative process:

T := y = wx + N(mu,sigma)

where T is a Transformation. What does it mean to transform x into y? I think it probably means to give our best estimate of y given x, so: transform(T, x) == w * x + mu. If that's the case, then transform(N) == mu.

What does it mean to generate an output from T? We're walking into the land of Bayesians here... I started to answer this but need more time to think it through.

What does it mean to learn the parameters w? I think: learn(T, x, y) means to learn w, mu, and sigma in parallel.

Do we agree on these definitions? If so then we need some drastic changes to the first PR.

[Evizero]

To give a short reply I can list a few requirements that I would need one way or the other

• A Linear transformation with and without a bias/intercept
• A way to dispatch on a bias being present. Maybe name the types Linear... vs Affine...
• a function that allows me to transform my input (number, vector, or matrix)
• a function to compute the derivative/grad

Some discussion topics:

• The bias should be part of the coefficient vector I think.
• for performance reasons a collumn should denote an observation. @joshday mentioned in his juliacon talk that there might be a efficient way to have rows for observations now?
• We should have a clear rule if the bias is the first or last element of the coef vector

[tbreloff]

Additionally, what would be really powerful is if we didn't need to subtype from Transformation at all, as long as the query functions were implemented for the type, and anything implied by the traits was implemented. What if we re-think the type tree to be the type tree of traits? Then Transformation is the abstract type of the trait, not the actual transformation object. Let me think out loud:

# I/O are the shapes (num dims) of the input and output
abstract TransformationType{I, O}
    immutable Transformable{I,O} <: TransformationType{I,O} end
    immutable NotTransformable{I,O} <: TransformationType{I,O} end

transform(nt::NotTransformable, args...) = error()

typealias Generative{O} Transformable{Void, O}

# this is the object which will do an affine transformation: yhat = wx + b
immutable Affine{W, B}
    w::W
    b::B
end

transform(aff::Affine, x) = aff.w * x + aff.b  # actual code would be more efficient

# these methods allow us to "plug in" an Affine object in with other "Transformable" objects,
#    even though they can have their own disparate type systems
# note: can't be exactly like this to maintain type stability, but something similar
input_shape(aff::Affine) = (size(w,2), size(b,2))
output_shape(aff::Affine) = size(b)
transformation_type(aff::Affine) = Transformation{1,1}


# we could similarly do `rand` implementations using a different trait.
# hopefully you understand the pattern I'm trying for:

abstract Randomness
    immutable Static <: Randomness end
    immutable Stochastic <: Randomness end

# everything is static, unless we specify
randomness(x) = Static()

generate(t, args...) = generate(randomness(t), t, args...)
generate(::Static, args...) = error()
generate(::Stochastic, t, args...) = rand(t, args...)

randomness(d::Distribution) = Stochastic()

# now we can "plug in" Distribution objects, as long as they implement `rand`
# we don't need wrapper types

[tbreloff]

A way to dispatch on a bias being present. Maybe name the types Linear... vs Affine...

If the type is parameterized, then typealias Linear{W} Affine{W,Void}?

for performance reasons a collumn should denote an observation.

I'm on board with this. I regret the decision in OnlineStats.

@joshday mentioned in his juliacon talk that there might be a efficient way to have rows for observations now?

Yes... Tim Holy's PermutedDimsArray. They are apparently hidden in Base 0.5. I think this should be the method of using a row-based table.

[joshday]

for performance reasons a collumn should denote an observation.

This isn't universally better. It's great for SGD, but bad for coordinate descent. Enforcing one or the other may be enough to keep people from buying in to JuliaML. I would prefer the default match notation (observations in rows), but I could be convinced otherwise.

[ahwillia]

The transform should, I think, take in nothing, and output the center.

I think we need at least three (maybe four) verbs here:

• transform should produce the exact output, represented as a Distribution type.
  • Note that we won't be able to support this in all cases -- it is easy to come up with transformations that won't have closed form solutions. Still, there are cases where we can do it (e.g. GLMs) and I think it should be supported.
  • Edit: we could also consider pdf as a name for this
• rand should produce a random sample from the output distribution.
• mean should produce the expected value of the output distribution.
• mode should produce the mode.

Also, I have a nitpick about this example:

T := y = wx + N(mu,sigma)

Should instead be:

T := y = wx + N(0,sigma)

(unless mu is the intercept in the model). The more important point I want to make is that it is actually better to think about the above model like this:

T := y = N(wx,sigma)

The difference is subtle, but important, for GLMs. Basically you think of the total transformation as a linear predictor wx that specifies the first moment of the noise distribution (after an inverse link function is applied). For Poisson regression there is no second moment:

T := y = Poisson(exp(wx))

From a practical standpoint, if you want a generative model, you need to specify sigma in the first example, but no additional parameters for Poisson.

[ahwillia]

I would prefer the default match notation (observations in rows), but I could be convinced otherwise.

+1, though I could also be convinced otherwise.

[ahwillia]

One more thought. If we are really going the Bayes Net route, we should talk with @sisl and maybe sync up with their BayesNets.jl package. cc: @tawheeler

[tbreloff]

T := y = N(wx,sigma)

I like this, if we can figure out the right way to do it. It should probably be T := y = N(wx + b, sigma)

[Evizero]

For SVMs to be fast i need an observation to be adjacent in memory

[tbreloff]

Yeah I guess we should have our convention be y = x'w + b, which is common in Matlab workflows

[tbreloff]

I pushed up some of my experiments... I think they're pretty promising. Check out the main Transformations.jl in the tom branch.

The gist is that there's a immutable Transformation which wraps an arbitrary object, and holds input/output data dimensions in the parameter signature. Then you implement a few "info methods" and one version of transform!, learn!, etc (whichever methods are appropriate), and then you've "linked" your object into the Transformation abstraction. We can build generic methods which dispatch on the results of queries (like is_learnable(o)).

Here's an example from runtests which wraps an OnlineStats.Means object:

# "link" this object into the transformations world
Transformations.input_size(m::Means) = size(m.value)
Transformations.output_size(m::Means) = size(m.value)
Transformations.is_learnable(m::Means) = true
Transformations.transform!(y, m::Means, x) = (y[:] = x - m.value)
Transformations.learn!(m, x) = OnlineStats.fit!(m, x)

# instantiate an OnlineStats.Means, which computes an online mean of a vector
m = Means(5)

# wrap the object in a Transformation, which stores input/output dimensions in its type,
# and allows for the common functionality to apply to the Transformation object.
t = transformation(m)

# update/learn the parameters for this transformation
#   (at this point, we don't care what the transformation is!)
learn!(t, 1:5)

# center the data by applying the transform. this dispatches generically.
# we only need to define a single `transform!` method
y = transform(t, 10ones(5))

# make sure everything worked
@test y == 10ones(5) - (1:5)

[tawheeler]

Hey! Happy to discuss BayesNets.
So my initial understanding is that you want to create a sort of computational graph (without the backprop)?

[tbreloff]

Hi Tim. Yes computational graph is part of it. I do care about backprop,
though not everyone will. It's more an attempt to generalize and unify
different models and approaches. We'd love your comments on the discussion
so far.

On Friday, July 1, 2016, Tim Wheeler notifications@github.com wrote:

Hey! Happy to discuss BayesNets.
So my initial understanding is that you want to create a sort of
computational graph (without the backprop)?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#2 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AA492nwC0yoGKdsrQqKYvfcGWRmx3KaBks5qRZaLgaJpZM4JDHaw
.

[ahwillia]

One thing I'm curious about is whether we can use the same set of abstractions to specify a computation graph as well as a generative model (more or less as a Bayes net). At the moment, I am interested in the latter for my day job, but I might be in the minority.

This has made me think it isn't so crazy to support both: http://edwardlib.org/

P.S. @tawheeler - Are you in the Bay Area for the summer? I'm at Stanford as well, but am in Livermore until October. Would love to meet and chat about BayesNets.jl in person.

[tawheeler]

This sounds like a very neat concept. I would think that forward and backward would be natural function names to use.
BayesNets.jl has each node know its own name and the names of its parents in order to allow it to pull assignments from dictionaries. I don't know whether that would necessarily be best for transformations.jl; nodes may be better off directly pulling from their parents. Nevertheless, one would want to make the design decision as to whether each transformation node works as a stand-alone or whether it only makes sense in the context of a type containing the DAG info.
Is this package meant to always output distributions? Note that computational graphs like in TensorFlow do not, in general, produce distributions.

@ahwillia I am, but will be heading to Germany from Oct through the end of the year. Happy to connect you with SISL though, or meet thereafter.