Textmodel integration

[Chandu-4444]

Can do the following:

lm = FastText.LanguageModel(true)
classifier = FastText.TextClassifier(lm)
FastText.train_classifier!(classifier) # Would throw an error as I haven't fully enabled the model to work with FastAI's data container.

[lorenzoh]

I think you've readded some files from TextModels.jl that we don't need, could you remove those? 🙂

[Chandu-4444]

Sure! Will clean up in the next commit.

[Chandu-4444]

• Deleted the files from TextModels.jl that are unnecessary.
• Add a utility function for padding the batches for sequence data (Very naively implemented, as a starting point though)

julia> batches = FastText.load_batchseq(data, task)
julia> batches[1][1]
92-element Vector{Vector{Int64}}:
 [25000, 25000, 25000, 25000, 25000, 25000, 25000, 25000]
 [633779, 633779, 633779, 633779, 633779, 633779, 633779, 633779]
 [2731, 34, 315, 354, 2087, 2209, 70, 1307]
 [44047, 435, 633779, 633779, 6589, 633779, 633779, 205]
 ⋮
 [0, 0, 0, 0, 0, 213, 0, 0]
 [0, 0, 0, 0, 0, 25, 0, 0]
 [0, 0, 0, 0, 0, 1778, 0, 0]

julia> batches[1][2]
8-element Vector{Int64}:
 1
 1
 1
 1
 1
 0
 1
 1

[Chandu-4444]

• fastai's way of encoding data doesn't include the removal of stop words (and Jeremy recommends it). So, I removed the stop word removal step.
• I've added a vocab_size keyword argument to TextClassificationSingle.
• Added <unk>, <pad> to the vocabulary.

Next:

• A batch loader for textdata with padding length = max(sentence length in a batch).

[ToucheSir]

Should the vocab CSV files be checked in? I would've assumed they would be artifacts or DataDeps as well.

[ToucheSir]

Porting the TextModels code is going to be a big task, but for now the priority is just to make things work e2e :)

[ToucheSir]

Likewise, I think this repeated lambda should be pulled out into a helper function of some sort.

[ToucheSir]

@lorenzoh I don't believe we have any utils for this in Flux or FastAI, right? Haven't kept up with the list of callbacks.

[lorenzoh]

I don’t think so. This also seems very model-specific so it’s maybe best to keep it as part of the model.

[Chandu-4444]

julia> data, blocks = load(datarecipes()["imdb"])
((mapobs(loadfile, ObsView(::MLDatasets.FileDataset{typeof(identity), String}, ::Vector{Int64})), mapobs(parentname, ObsView(::MLDatasets.FileDataset{typeof(identity), String}, ::Vector{Int64}))), (Paragraph(), Label{String}(["neg", "pos"])))

julia> task = TextClassificationSingle(blocks, data)
SupervisedTask(Paragraph -> Label{String})

julia> model = FastAI.taskmodel(task, FastText.LanguageModel(false, task))
#90 (generic function with 1 method)

julia> batches = FastText.load_batchseq(data, task)
6250-element Vector{Tuple{Vector{Vector{Int64}}, WARNING: both Losses and NNlib export "ctc_loss"; uses of it in module Flux must be qualified
Flux.OneHotArray{UInt32, 2, 1, 2, Vector{UInt32}}}}:
 ([[35, 35, 35, 35], [3, 3, 3, 9], [40, 18025, 15, 14], [224, 10, 3541, 3040], [737, 34, 24, 505], [49, 7, 809, 3], [4, 4, 221, 3836], [1927, 104, 4, 
3], [7, 16, 629, 28440], [6, 351, 7, 17]  …  [2, 2, 2, 44], [2, 2, 2, 3], [2, 2, 2, 9839], [2, 2, 2, 17], [2, 2, 2, 1041], [2, 2, 2, 27], [2, 2, 2, 3], [2, 2, 2, 3836], [2, 2, 2, 3], [2, 2, 2, 28440]], [0 0 1 1; 1 1 0 0])

julia> using FluxTraining

julia> td, vd = splitobs(batches, at=0.9)

julia> using Flux

julia> learner = Learner(model, Flux.Losses.logitcrossentropy, callbacks=[Metrics(accuracy)]; data=(td, vd))
Learner()

julia> fit!(learner, 1)
Epoch 1 TrainingPhase():   0%|█                                                                                               |  ETA: 4 days, 3:35:31 

[darsnack]

Just reviewing the weight dropped LSTM for now

[darsnack]

These are now replaced by state0 and stored in Recur

[darsnack]

This comment is still outstanding.

[darsnack]

More changes from the call

[darsnack]

Why does it only mask part of the bias @Chandu-4444?

[Chandu-4444]

Since it is ported from TextModels.jl I'm not completely sure of this. Even Flux.jl has this line this.
Would be helpful if someone could explain the reason.

[darsnack]

Overload Flux.reset! here

[darsnack]

Bumping this comment

[ToucheSir]

As promised, here's the proposal from our call today:

mutable struct VarDropCell{F}
    p::F
    active::Union{Bool, Nothing} # matches other norm layers
end
@functor VarDropCell

VarDropCell(p::Real=0.0) = VarDropCell(p, nothing)

function (vd::VarDropCell)((has_mask, mask), x)
    if Flux._isactive(vd)
        mask = has_mask ? mask : drop_mask(x, vd.p)
        return (true, mask), x .* mask
    elseif !has_mask
        return (has_mask, mask), x
    else
        error("Mask set but layer is in test mode. Call `reset!` to clear the mask.")
    end
end

# The single-element array keeps Recur happy.
# Limitation: typeof(p) must == typeof(<inputs>)
VarDrop(p::Real) = Recur(VarDropCell(p), ones(typeof(p), 1, 1))

Plus bits to make testmode!, _isactive and reset! work properly.

[Chandu-4444]

The changes have been merged to #258.

[darsnack]

Try this

[darsnack]

This comment is still outstanding.

[lorenzoh]

I don’t think so. This also seems very model-specific so it’s maybe best to keep it as part of the model.

[darsnack]

Some of the issues that you posted on Zulip should be resolved by this

[darsnack]

Suggested change

	maskWi = Flux.dropout_mask(Flux.rng_from_array(), layer.cell.Wi, layer.cell.p)
	maskWh = Flux.dropout_mask(Flux.rng_from_array(), layer.cell.Wh, layer.cell.p)
	maskWi = Flux.dropout_mask(Flux.rng_from_array(layer.cell.Wi), layer.cell.Wi, layer.cell.p)
	maskWh = Flux.dropout_mask(Flux.rng_from_array(layer.cell.Wh), layer.cell.Wh, layer.cell.p)

[darsnack]

Bumping this comment