Prediction/preprocessing helper

[lorenzoh]

Is there interest in a prediction or preprocessing helper that helps people perform on (batches) of data for (Image-Net) pretrained models?

Thinking something like

function preprocess(img::AbstractArray{U, N}, sz = (224, 224); T = Float32, C = RGB{N0f8} )
    item = DataAugmentation.Image{N}(img)
    tfm = ToEltype(C) |> CenterResizeCrop(sz) |> ToTensor(T)  |> Normalize(IMAGENET_STATS)
    return apply(tfm, item) |> itemdata
end

preprocess(imgs::AbstractVector{<:AbstractArray}) = ... 

This would require adding a dep to DataAugmentation.jl, though.

[theabhirath]

I'd love to help in whatever way I can....is there a reason that Metalhead doesn't have DataAugmentation built into it though? I've noticed that the Julian way is to keep several lightweight packages and compose them as and when needed but as a torchvision user I often find myself appreciating the fact that everything is in one place 😅 I hope I'm not phrasing this the wrong way, I'm just curious regarding the thought process - I was thinking of writing CV stuff like object detection and semantic segmentation pipelines and was wondering if this would be the right repo to collect everything

[lorenzoh]

I'm gathering a lot of common computer vision functionality in FastAI.jl's Vision submodule. FastAI.jl is the batteries-included deep learning package that brings together many of the packages that you end up needing doing deep learning. Unfortunately the documentation for everything but the very high-level cases and interfaces is still lacking; I am planning to tackle this soon, though, to make it easier to contribute to.

For now, if you want to have a preprocessing function like above, try out ProjectiveTransforms and ImagePreprocessing from FastAI.jl, which end up doing something very similar as the above snippet.

[ToucheSir]

One practical reason for splitting packages up currently is import latency. using Flux, is already 8-10s, plus another few if you add Metalhead on top of it. If that became negligible then I think something like this would be a no-brainer.

[darsnack]

I totally agree that we should have these built-in pipelines, but I don't think Metalhead.jl is the right place for them. I would suggest MLDatasets.jl instead, since generally the pre-processing pipeline is tied to a dataset, not a model.

We could make a FluxVision.jl umbrella if that's desired (though I expect a lot of that group would either be happy loading MLDatasets and Metalhead separately or just use FastAI?). Just cause there are separate usings doesn't mean the packages shouldn't be designed to work together seamlessly. Proper documentation is lacking in this regard. Since the redesign, Metalhead is usable as a dep for downstream packages. Keeping the overall load time low is important in this case.

[CarloLucibello]

But MLDatasets contains generic datasets, while Metalhead explicitly targets vision. Since vision models are commonly used with preprocessing, I think it would make more sense to have some preprocessing pipeline in here. Or at the very least document how do it using packages from the ecosystem.

[darsnack]

I was thinking that all the datasets in MLDatasets (including the vision ones) should have pre-processing helpers. If I have a custom model that I plan training with CIFAR-100, then I will load MLDatasets.jl but not necessarily Metalhead.jl. I do agree that we should document both here how to use the pre-processing pipelines.

[CarloLucibello]

Maybe we should have a VisionDatasets.jl package then

[theabhirath]

We could make a FluxVision.jl umbrella

Even if not in terms of packages, maybe I could come up with docs that document the most common usecases - but TBVH, the doubt I always have is that when I search image augmentation in Julia, Augmentor.jl pops up first. While that probably works just fine because of how well Julia packages compose, there are probably packages that target Flux and Metalhead more specifically, and given the issues with latency, gradients etc. I'm always unsure how well things will end up working 😅 Dunno, kinda new to this so I'm not too sure

VisionDatasets.jl sounds like a good idea purely because there's a lot of stuff like COCO, KITTY and CelebA that's not been ported yet (in a nice wrapper like manner, at least) to Julia and having the datasets available easily means that porting models and testing them will be easier too. I'm happy to help wherever I can 😄

[darsnack]

I'm okay with that, as well as merging the pipelines into Metalhead if that's what folks want.

Is the concern here about MLDatasets deps? It already provides the common vision datasets except ImageNet, so I think I'm missing why vision should be separate.

Also, if we do VisionDatasets, then will CIFAR, etc. move out of MLDatasets?

[ToucheSir]

Even if not in terms of packages, maybe I could come up with docs that document the most common usecases - but TBVH, the doubt I always have is that when I search image augmentation in Julia, Augmentor.jl pops up first. While that probably works just fine because of how well Julia packages compose, there are probably packages that target Flux and Metalhead more specifically, and given the issues with latency, gradients etc. I'm always unsure how well things will end up working sweat_smile Dunno, kinda new to this so I'm not too sure

You're not the only one. We've a dearth of higher-level documentation like "cookbooks" that are more complex than tutorials. Augmentor.jl is a bit of an unfortunate case because it took a very generic name for a domain specific library (ref. FluxML/FastAI.jl#68). The "coordination" part of https://github.com/FluxML/ML-Coordination-Tracker was meant to have us link all these disparate ecosystems together (JuliaText is another example), but the sheer amount of work required to get FluxML itself up to speed kind of put the rest on the backburner 😅

[CarloLucibello]

Is the concern here about MLDatasets deps?

yes

It already provides the common vision datasets except ImageNet, so I think I'm missing why vision should be separate.

If we want to add vision-specific preprocessing pipelines I think we should factor out the vision datasets into a specialized package. What MLDatasets.jl is right now is just a centralized repo for downloading generic ML datasets basically. I just wish people would add more datasets to that. Maybe some dev docs and some homogenization would help.

What about higher having level standard preprocessing function, e.g. for ImageNet and CIFAR10, directly in DataAugmentation.jl?

[lorenzoh]

What about higher having level standard preprocessing function, e.g. for ImageNet and CIFAR10

Once we do this for multiple datasets, this would become a bit unwieldy and maybe too much of a specialized function? There should definitely be a tutorial/cookbook in DataAugmentation.jl's docs for how to build common image pipelines. How about we add this and then link to it from Metalhead.jl and MLDatasets.jl?

[darsnack]

I think that is an easy first step we can do until we split off the vision datasets like Carlo suggested.

[ToucheSir]

What about higher having level standard preprocessing function, e.g. for ImageNet and CIFAR10, directly in DataAugmentation.jl?

Would it be easy to do this as an "alias" system? Assuming the custom pipelines are sequestered in their own module, that could reduce the impact on the rest of the code base and allow for reuse across similar datasets (e.g. variants of ImageNet).

Barring that, one step we could take now would be to add metadata for data augmentations in MLDatasets. Then there's no explicit dependency, but given a sufficiently general set of batteries-included pipelines like #117 (comment), one could pass in just the dataset-specific parts like stats.