v3 Roadmap

[JimBobSquarePants]

For v3 I'd like to focus more over streamlining the codebase over implementing new features.
By doing so we can really focus on making the library as fast as possible while easier to maintain.

Maintenance and performance

• Update License VNext License Changes #2151.
• Remove legacy polyfills and compiler conditions.
• Introduce Nullable Reference Types
• Revisit SIMD pipelines and simplify/optimize (Are we overusing Unsafe API?).
• Performance pass for RGBA compatible converters.
• Add ARM SIMD support plus CI pipeline.
• Optimize pixel blending with integer arithmetics Optimize pixel blending with integer arithmetics #1433

New Features

• Decode to target size Jpeg downscaling decoding #2076
• Jpeg encoder complete rewrite Jpeg encoder complete rewrite #2120
• Expose convolution API and reimplement auto matrix linear decomposition
• Investigate OriginalImageFormat Add an OriginalImageFormat property onto ImageMetadata #2090
• Color conversion with ICC profiles Color conversion with ICC profiles #1567
• WrapMemory enhancements Add memory size to WrapMemory(void* ...); #1489

Please note anything else you think I've missed that you feel is important.

[brianpopow]

I was thinking about adding encoding for animated webp for v3, but first I am not sure if I will find time soon to do so and second I am not sure if it's that important. Maybe this can wait also for after v3.

Add ARM SIMD support plus CI pipeline.

While I really would like to have that, I am not 100% convinced that the current PoC in #1994 with QEMU is a very satisfying solution. It will take more the than 1h to run the test just for ARM. Maybe we can find a more clever way to trigger this pipeline only when there are ARM changes or only run ARM specific tests in that pipeline to speed things up. A third option would be to run the pipeline on a custom CI runner with a ARM CPU. Never done that with github, not sure how good that will work, but it for sure will be faster.

[JimBobSquarePants]

We can trigger the ARM64 tests when a PR has a certain label.

https://stackoverflow.com/a/62331521

[brianpopow]

yeah maybe, that could do it.

[antonfirsov]

Decode to target size

For me this is much more than just #2076 where we are landing an advanced, jpeg-specific feature. What I wish to see is a general LoadResize() API with a memory footprint close to the footprint of vips and magicscaler. ImageSharp has to be among the absolute winners.

[JimBobSquarePants]

Don't they both use an entirely different pipeline approach though to the other libraries? That would mean significant refactoring.

Do we have updated numbers for that test anywhere. We must be close to the top 3 now surely?

I agree though that I want the API to be multi format. A jpeg specific API wouldn't match our overall philosophy.

[antonfirsov]

Do we have updated numbers for that test anywhere.

We can produce some with LoadResizeSaveParallelMemoryStress but it's not a trivial job since you have to run every library on beefy multi-core desktop or server and analyze graphs manually with WPT.

From what I can recall from my 10 core runs (which was actually stressing parallelism more than Clinton's run with 6 cores):
we went down from 7-10 GB to something 1.3 GB-ish, but vips & magicscaler was around 200 MBs. (But my memory can terribly fail me.)

Don't they both use an entirely different pipeline approach though to the other libraries

Yes, but we don't have to go fully-pipelined. It's enough if the decoders can somehow utilize the resizer internally, and produce downscaled output as they walk the scanlines.

[JimBobSquarePants]

Yes, but we don't have to go fully-pipelined. It's enough if the decoders can somehow utilize the resizer internally, and produce downscaled output as they walk the scanlines.

I like this. 👍

[br3aker]

JFYI we can get really close to libvips in terms of implementation. Jpeg decoding is done via abstract 'convert stride' method, nothing stops us from doing decode-downscale-encode in special converter implementation. That way we won't even need to allocate image itself. But I don't think we actually need to do it this way. IMO LoadResize is enough for a general use case, for special use cases people can use netvips.