CASSANDRA-20975 Add capability to load pluggable compression service providers by shyla226 · Pull Request #4513 · apache/cassandra

shyla226 · 2025-12-04T17:57:58Z

Updated the code based on feedback from #4420.

Code now does not need any changes in cassandra.yaml file
Added a new api in CompressionParams, getDecoratedSstableCompressor so that plugin compressor does not change the compression metadata

smiklosovic · 2025-12-15T22:09:07Z

src/java/org/apache/cassandra/schema/CompressionParams.java

+     * @param options compression options of baseCompressor
+     * @return returns a decorated compressor, if service available, otherwise baseCompressor
+     */
+    private static ICompressor decorateCompressor(ICompressor baseCompressor, Map<String, String> options)


I think this might be way simplified

private static ICompressor decorateCompressor(ICompressor baseCompressor, Map<String, String> options) { return Optional.ofNullable(baseCompressor) .flatMap(c -> ServiceLoader.load(ICompressorFactory.class) .stream() .map(ServiceLoader.Provider::get) .filter(factory-> c.getClass().getSimpleName() .equals(factory.getSupportedCompressorName())) .findFirst()) .flatMap(factory -> factory.createCompressor(options)) .map(pluginCompressor -> new CompressorDecorator(baseCompressor, pluginCompressor)) .orElse(null); }

We can get rid of getServiceProviderFactory completely. Just return Optional<ICompressor> from factory's createCompressor. You are just catching that exception here and logging, you can do same in the implementation and return empty optional instead if not possible to instantiate.

smiklosovic · 2025-12-15T22:10:27Z

src/java/org/apache/cassandra/schema/CompressionParams.java

        this.otherOptions = ImmutableMap.copyOf(otherOptions);
        this.minCompressRatio = minCompressRatio;
        this.maxCompressedLength = maxCompressedLength;
+	this.decoratedSstableCompressor = decorateCompressor(sstableCompressor, otherOptions);


smiklosovic · 2025-12-15T22:11:12Z

src/java/org/apache/cassandra/schema/CompressionParams.java


 public final class CompressionParams
 {
+    private static final Logger logger = LoggerFactory.getLogger(CompressionParams.class);


if you log in the implementation as suggested then this might go away (with imports too)

smiklosovic · 2025-12-15T22:12:23Z

src/java/org/apache/cassandra/io/compress/ICompressorFactory.java

+
+public interface ICompressorFactory
+{
+    public static final ImmutableMap<String, String> COMPRESSOR_NAME_MAP = ImmutableMap.of(


this is unused and can be removed, also not correct as such, we would need to be way more robust than this in general.

public static final redundant too. Would you mind to shed some light at the logic behind this?

We would need to use some reflection to scan classes in org.apache.cassandra.io.compress package which are not abstract and implement ICompressor otherwise we would need to update this every time we add a new compressor implementation. (you are already missing ZstdDictionaryCompressor here)

I am also not completely sure if all such compression algorithms support your hardware speedup. In that case we would need to some filter them more. Not sure, waiting on your feedback.

Intention behind using the map was to have a simple name to represent the algorithm like, deflate, lz4 etc. and map them to concrete classes in Cassandra. Another way I can think of is make the plugin factory return concrete class names like DeflateCompressor when getSupportedCompressorName is invoked, to specify which compressor it is accelerating. Will that be alright?

src/java/org/apache/cassandra/io/compress/ICompressorFactory.java

smiklosovic · 2025-12-15T22:24:46Z

src/java/org/apache/cassandra/io/compress/CompressorDecorator.java

+/**
+ * This class uses a plugin compressor to perform compress/decompress, if available, otherwise reverts to default SstableCompressor.
+ */
+public class CompressorDecorator extends AbstractCompressorDecorator


I do not think that controlling the execution flow with try/catch in below methods is a good idea. Perhaps you might decide if it is possible to use your plugin as you are getting it from the factory? This is very uncomfortable and probably also not performance friendly when we do uncompress on plugin every single time and then falling back to base compressor.

Based on the discussion on ML it seems we will go with the fallback logic after all. In that case we need to log (in a non-spamming manner) + introduce metrics and update them on each failure.

sounds good. I will look into adding the metrics

smiklosovic · 2025-12-15T22:29:20Z

src/java/org/apache/cassandra/io/compress/CompressedSequentialWriter.java

                            .bufferType(parameters.getSstableCompressor().preferredBufferType())
                            .finishOnClose(option.finishOnClose())
                            .build());
-        ICompressor compressor = parameters.getSstableCompressor();


alignement, plus we should most probably not use it like this. We should still use getSstableCompressor() as it was before. It is implementation detail if the compressor is decorated or not.

Agree. Reasoning for doing it this way, sstableCompressor being final it can be changed only in the createCompressor function. But if I change it there, CompressionMetadata also gets updated and stored.
Any thoughts how I could overcome this issue?

smiklosovic

i think we need to iron out some design issues

smiklosovic · 2025-12-16T15:53:58Z

src/java/org/apache/cassandra/io/compress/CompressionMetadata.java

            if (result == null)
            {
-                result = resolveCompressor(parameters.getSstableCompressor(), compressionDictionary);
+                result = resolveCompressor(parameters.getDecoratedSstableCompressor(), compressionDictionary);


same as above

smiklosovic · 2025-12-16T15:57:46Z

src/java/org/apache/cassandra/schema/CompressionParams.java

        this.otherOptions = ImmutableMap.copyOf(otherOptions);
        this.minCompressRatio = minCompressRatio;
        this.maxCompressedLength = maxCompressedLength;
+	this.decoratedSstableCompressor = decorateCompressor(sstableCompressor, otherOptions);


do we actually need decoratedSstableCompressor? I propose to do something like

this.sstableCompressor = maybeDecorateCompressor(sstableCompressor, otherOptions);

Then sstableCompressor would be either decorated or not.

Ah..I will try this. I think you answered one of my questions

shyla226 · 2026-02-04T00:36:52Z

@smiklosovic, I have updated the code using CryptoProvider as reference. Please take a look

Adding support for plugin compressors

5a14f5f

shyla226 mentioned this pull request Dec 4, 2025

CASSANDRA-20975 Add capability to load pluggable compression service providers #4420

Closed

smiklosovic self-requested a review December 15, 2025 21:38

smiklosovic reviewed Dec 15, 2025

View reviewed changes

src/java/org/apache/cassandra/io/compress/ICompressorFactory.java Outdated Show resolved Hide resolved

smiklosovic reviewed Dec 15, 2025

View reviewed changes

smiklosovic requested changes Dec 15, 2025

View reviewed changes

smiklosovic reviewed Dec 16, 2025

View reviewed changes

shyla226 added 2 commits February 3, 2026 16:21

Merge branch 'apache:trunk' into CASSANDRA-20975

cd049d4

Incorporated code review feedback

70a81dd

shyla226 requested a review from smiklosovic February 4, 2026 00:31

Fixed checkstyle issue

3faea11

Conversation

shyla226 commented Dec 4, 2025

Uh oh!

smiklosovic Dec 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

smiklosovic Dec 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

smiklosovic Dec 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

smiklosovic left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

smiklosovic Dec 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shyla226 commented Feb 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

smiklosovic Dec 15, 2025 •

edited

Loading

smiklosovic Dec 15, 2025 •

edited

Loading

smiklosovic Dec 15, 2025 •

edited

Loading

smiklosovic Dec 16, 2025 •

edited

Loading