[LRNRQ] Add FastAI Learners

[mb706]

Algorithm

Neural Networks

Package

https://github.com/EagerAI/fastai

Why do I think this is a useful learner?

AutoGluon uses this, so it's probably not that bad.

[AnnaNzrv]

Here is an example step by step idea walkthrough of how this could be implemented (or at least the train method of the learner):

Step 1. Get our task

task = tsk("sonar")
df = task$data()
ids = partition(task)

Step 2. Load data into a FastAi-compatible format
We need to divide column names into Categorical columns and Continuous columns.
We can also define several data preprocessors procs = list(FillMissing(),Categorify(),Normalize()) and pass them to TabularDataTable.
We can provide our own train, test, validation splits. If no splits are provided, no validation is done and the model is trained on the entire training set.

df_fai = fastai::TabularDataTable(df, cat_names = cat_cols, cont_names = num_cols,
                                  y_names = task$target_names, splits=ids)

dl = fastai::dataloaders(df_fai, bs=50)
# These params are not documented by fastai-r but only by fastai-py:
ps_dataloaders = ps(
  bs          = p_int(defualt = NULL, tags="train"), # how many samples per batch to load
  num_workers = p_int(default = 0L, tags = "train"),  # how many subprocesses to use for data loading
  pin_memory  = p_lgl(default = TRUE, tags = "train"),  # If True, the data loader will copy Tensors into CUDA pinned memory before returning them.
  shuffle     = p_lgl(default = FALSE, tags = "train"),  # If True, then data is shuffled every time dataloader is fully read/iterated.
  drop_last   = p_lgl(default = FALSE, tags = "train")  #  If True, then the last incomplete batch is dropped.
)

Step 3. Define a set of configs for the FastAI learner

config = fastai::tabular_config(embed_p = 0.3, use_bn = FALSE)

ps_config = ps(
  ps       = p_uty(default = NULL, tags = "train"),  # Sequence of dropout probabilities
  embed_p  = p_dbl(lower = 0L, upper = 1L, default = 0L, tags = "train"),  # Dropout probability for Embedding layer
  y_range  = p_uty(default = NULL, tags = "train"),  # Low and high for SigmoidRange activation (see below)
  use_bn   = p_lgl(default = TRUE, tags = "train"),  # Use BatchNorm1d in LinBnDrop layers
  bn_final = p_lgl(default = FALSE, tags = "train"),  # Use BatchNorm1d on final layer
  bn_cont  = p_lgl(default = TRUE, tags = "train"),  # Use BatchNorm1d on continuous variables
  act_cls  = p_uty(tags = "train")  # Activation type for LinBnDrop layers, e.g., fastai::nn()$ReLU(inplace = TRUE)
)

# SigmoidRange: torch.sigmoid(x) * (high - low) + low
#
# LinBnDrop is a module grouping BatchNorm1d, Dropout and Linear layers:
#
# lin_first is not set in fastai-r and defaults to True
# class LinBnDrop(nn.Sequential):
#   def __init__(self, n_in, n_out, bn=True, p=0., act=None, lin_first=False):
#   layers = [BatchNorm(n_out if lin_first else n_in, ndim=1)] if bn else []
#   if p != 0: layers.append(nn.Dropout(p))
#   lin = [nn.Linear(n_in, n_out, bias=not bn)]
#   if act is not None: lin.append(act)
#   layers = lin+layers if lin_first else layers+lin
#   super().__init__(*layers)

Step 4. Define and fit the tabular learner

model = fastai::tabular_learner(dl, layers=c(200,100,100,200),
                                config = config,
                                metrics = list(accuracy, RocAucBinary(),
                                               Precision(), Recall(),
                                               F1Score()))
fit(model, 5, lr = 0.005)

ps_learner = ps(
  layers     = p_uty(tags="train"),  # Sequence of ints used to specify the input and output size of each LinBnDrop layer
  emb_szs    = p_int(lower = 0, default=NULL, tags="train"),  # Sequence of (num_embeddings, embedding_dim) for each categorical variable
  n_out      = p_int(lower = 1, default = 2, tags="train"),  # Number of outputs for final LinBnDrop layer
  loss_func  = p_uty(tags="train"),
  opt_func   = p_uty(tags="train"),  # Optimizer created when Learner.fit is called. E.g., fastai::Adam()
  lr         = p_int(lower=0, default = 0.001, tags = "train"),  # Learning rate
  metrics    = p_uty(tags = "train"),  # optional list of metrics, e.g, fastai::Precision() or fastai::accuracy()
  train_bn   = p_lgl(default = TRUE, tags = "train"),  # controls if BatchNorm layers are trained
  wd_bn_bias = p_lgl(default = FALSE, tags = "train")  # controls if weight decay is applied to BatchNorm layers and bias
)

I'm currently working on integrating FastAi here using this approach. I need to add a few more parameters and combine everything into a single parameter set.

[mb706]

Regarding ids = partition(task): We have some machinery for internal validation within Learners:

https://mlr3book.mlr-org.com/chapters/chapter15/predsets_valid_inttune.html

so you could do something like rbind the task$data() and the task$internal_valid_task$data(), and provide splits as the indices that correspond to the training or validation part of this.