CSHARP-3222: Add LINQ support for median and percentile accumulators/window functions #1743

adelinowona · 2025-07-29T11:03:56Z

This PR introduces the capability to calculate the median and percentile of numeric values in the MongoDB aggregation pipeline for $group and $setWindowFields stages.

…ow expressions

rstam

Initial quick review. Will review more thoroughly after next commit.

rstam · 2025-07-29T17:30:08Z

...tionExpressionTranslators/MethodTranslators/MedianMethodToAggregationExpressionTranslator.cs

+            var method = expression.Method;
+            var arguments = expression.Arguments;
+
+            if (IsMedianMethod(method))


Normally we do if (method.IsOneOf(__medianMethods)).

Do we want to do it differently here?

I noticed that pattern but I just felt that would be too much boilerplate kind of code and there's an easier way to do it. Plus I noticed the StandardDeviationMethodsToAggregationExpressionTranslator follows a similar pattern already.

There is some boilerplate in setting up the __medianMethods static field.

There isn't any boilerplate in the if statement. It's roughly the same.

One advantage of using __medianMethods is that it is VERY precise. There is no danger of false hits like there is with the IsMedianMethod approach.

Plus I noticed the StandardDeviationMethodsToAggregationExpressionTranslator follows a similar pattern already.

Yes. That's older code that predates the newer practice of being more precise.

...tionExpressionTranslators/MethodTranslators/MedianMethodToAggregationExpressionTranslator.cs

rstam

Changes requested

src/MongoDB.Driver/Linq/Linq3Implementation/Ast/Expressions/AstComplexAccumulatorExpression.cs

src/MongoDB.Driver/Linq/Linq3Implementation/Ast/Visitors/AstNodeVisitor.cs

...xpressionTranslators/MethodTranslators/MedianMethodToAggregationExpressionTranslatorTests.cs

...xpressionTranslators/MethodTranslators/WindowMethodToAggregationExpressionTranslatorTests.cs

...er.Tests/Linq/Linq3ImplementationWithLinq2Tests/Translators/AggregateGroupTranslatorTests.cs

rstam · 2025-07-29T18:53:35Z

...tionExpressionTranslators/MethodTranslators/MedianMethodToAggregationExpressionTranslator.cs

+            var method = expression.Method;
+            var arguments = expression.Arguments;
+
+            if (IsMedianMethod(method))


There is some boilerplate in setting up the __medianMethods static field.

There isn't any boilerplate in the if statement. It's roughly the same.

One advantage of using __medianMethods is that it is VERY precise. There is no danger of false hits like there is with the IsMedianMethod approach.

Plus I noticed the StandardDeviationMethodsToAggregationExpressionTranslator follows a similar pattern already.

Yes. That's older code that predates the newer practice of being more precise.

...xpressionTranslators/MethodTranslators/MedianMethodToAggregationExpressionTranslatorTests.cs

src/MongoDB.Driver/Linq/Linq3Implementation/Ast/Visitors/AstNodeVisitor.cs

rstam · 2025-07-30T01:41:10Z

src/MongoDB.Driver/Linq/Linq3Implementation/Ast/Expressions/AstComplexAccumulatorExpression.cs

+
+namespace MongoDB.Driver.Linq.Linq3Implementation.Ast.Expressions
+{
+    internal sealed class AstComplexAccumulatorExpression : AstAccumulatorExpression


This is clever but I don't like the idea of a Dictionary<string, AstExpression>. A class that can represent anything in this way is not easy to work with.

I would prefer you just create two new classes AstMedianAccumulatorExpression and AstPercentileAccumulatorExpression.

That would mirror AstMedianExpression and AstPercentileExpression.

It would also avoid the messiness in the VisitComplexAccumulatorExpression method. Instead we could just have two simple methods VisitMedianAccumulatorExpression and VisitPercentileAccumulatorExpression.

The whole point of the Ast classes is to have type-safe representations of MQL, and the use of a Dictionary<string, AstExpression> throws type-safety out the window.

That's fair. I'll revert this then

src/MongoDB.Driver/Linq/Linq3Implementation/Ast/Optimizers/AstGroupingPipelineOptimizer.cs

...tionExpressionTranslators/MethodTranslators/MedianMethodToAggregationExpressionTranslator.cs

...ExpressionTranslators/MethodTranslators/PercentileMethodToAggregationExpressionTranslator.cs

...tionExpressionTranslators/MethodTranslators/WindowMethodToAggregationExpressionTranslator.cs

rstam · 2025-08-01T02:51:27Z

src/MongoDB.Driver/Linq/MongoEnumerable.cs

+        /// </summary>
+        /// <param name="source">The sequence of values.</param>
+        /// <returns>The median value.</returns>
+        public static double Median(this IEnumerable<decimal> source)


I'm still concerned that the return type is double instead of decimal.

Same below.

discussed on slack. The Server team currently has $median returning doubles regardless of input type but there is a ticket for improving accuracy in the future to return the correct types. So I'll change the return type here to decimal.

rstam · 2025-08-01T02:54:47Z

src/MongoDB.Driver/Linq/MongoEnumerable.cs

+        /// </summary>
+        /// <param name="source">The sequence of values.</param>
+        /// <returns>The median value.</returns>
+        public static double Median(this IEnumerable<float> source)


Enumerable.Average(IEnumerable<float>) returns float.

Should we also?

rstam · 2025-08-01T02:55:42Z

src/MongoDB.Driver/Linq/MongoEnumerable.cs

+        /// <param name="source">A sequence of values to calculate the percentiles of.</param>
+        /// <param name="percentiles">The percentiles to compute (each between 0.0 and 1.0).</param>
+        /// <returns>The percentiles of the sequence of values.</returns>
+        public static double[] Percentile(this IEnumerable<decimal> source, IEnumerable<double> percentiles)


Should we return decimal[]

Should the percentiles be decimal also to match?

rstam · 2025-08-01T02:56:48Z

src/MongoDB.Driver/Linq/MongoEnumerable.cs

+        /// <param name="source">A sequence of values to calculate the percentiles of.</param>
+        /// <param name="percentiles">The percentiles to compute (each between 0.0 and 1.0).</param>
+        /// <returns>The percentiles of the sequence of values.</returns>
+        public static double[] Percentile(this IEnumerable<float> source, IEnumerable<double> percentiles)


Should we return float[]?

SHould the percentiles web IEnumerable<flost> to match?

rstam · 2025-08-01T02:57:59Z

src/MongoDB.Driver/Linq/ISetWindowFieldsPartitionExtensions.cs

+        /// <param name="selector">The selector that selects a value from the input document.</param>
+        /// <param name="window">The window boundaries.</param>
+        /// <returns>The median of the selected values.</returns>
+        public static double Median<TInput>(this ISetWindowFieldsPartition<TInput> partition, Func<TInput, decimal> selector, SetWindowFieldsWindow window = null)


Should we return decimal?

rstam

Very close to done!

rstam · 2025-08-07T01:57:59Z

src/MongoDB.Driver/Linq/Linq3Implementation/Ast/Expressions/AstExpression.cs

+            return new AstMedianExpression(input);
+        }
+
+        public static AstMedianAccumulatorExpression MedianAccumulator(AstExpression input)


Consider a return type of AstAccumulatorExpression.

In most (all?) methods in this class we return the most general class possible. This allows the factory method to sometimes return different types depending on the parameters.

rstam · 2025-08-07T01:58:08Z

src/MongoDB.Driver/Linq/Linq3Implementation/Ast/Expressions/AstExpression.cs

+            return new AstMedianAccumulatorExpression(input);
+        }
+
+        public static AstMedianWindowExpression MedianWindowExpression(AstExpression input, AstWindow window)


Consider a return type of AstWindowExpression.

In most (all?) methods in this class we return the most general class possible. This allows the factory method to sometimes return different types depending on the parameters.

rstam · 2025-08-07T01:58:36Z

src/MongoDB.Driver/Linq/Linq3Implementation/Ast/Expressions/AstExpression.cs

@@ -597,6 +597,21 @@ public static AstExpression Max(AstExpression arg1, AstExpression arg2)
            return new AstNaryExpression(AstNaryOperator.Max, [arg1, arg2]);
        }

+        public static AstExpression Median(AstExpression input)


Consider a return type of AstExpression.

In most (all?) methods in this class we return the most general class possible. This allows the factory method to sometimes return different types depending on the parameters.

rstam · 2025-08-07T01:58:48Z

src/MongoDB.Driver/Linq/Linq3Implementation/Ast/Expressions/AstExpression.cs

@@ -653,6 +668,21 @@ public static AstExpression Or(params AstExpression[] args)
            return new AstNaryExpression(AstNaryOperator.Or, flattenedArgs);
        }

+        public static AstPercentileExpression Percentile(AstExpression input, AstExpression percentiles)


Consider a return type of AstExpression.

In most (all?) methods in this class we return the most general class possible. This allows the factory method to sometimes return different types depending on the parameters.

rstam · 2025-08-07T01:58:57Z

src/MongoDB.Driver/Linq/Linq3Implementation/Ast/Expressions/AstExpression.cs

+            return new AstPercentileExpression(input, percentiles);
+        }
+
+        public static AstPercentileAccumulatorExpression PercentileAccumulator(AstExpression input, AstExpression percentiles)


Consider a return type of AstAccumulatorExpression.

In most (all?) methods in this class we return the most general class possible. This allows the factory method to sometimes return different types depending on the parameters.

rstam · 2025-08-07T02:02:21Z

src/MongoDB.Driver/Linq/Linq3Implementation/Ast/Optimizers/AstGroupingPipelineOptimizer.cs

-                bool TryOptimizeAccumulatorOfElements(out AstExpression optimizedExpression)
+            private bool IsMappedElementsField(AstExpression expression, out AstExpression rewrittenArg)
+            {
+                if (expression is AstMapExpression map && IsElementsField(map.Input))


Rename map to mapExpression.

rstam · 2025-08-07T02:02:39Z

src/MongoDB.Driver/Linq/Linq3Implementation/Ast/Optimizers/AstGroupingPipelineOptimizer.cs

-                    optimizedExpression = null;
-                    return false;
-                }
+            private AstExpression CreateGetAccumulatorFieldExpression(AstAccumulatorExpression accumulator)


Rename accumulator to accumulatorExpression.

rstam · 2025-08-07T02:03:17Z

src/MongoDB.Driver/Linq/Linq3Implementation/Reflection/EnumerableMethod.cs

@@ -112,6 +112,26 @@ internal static class EnumerableMethod
        private static readonly MethodInfo __maxSingle;
        private static readonly MethodInfo __maxSingleWithSelector;
        private static readonly MethodInfo __maxWithSelector;
+        private static readonly MethodInfo __medianDecimal;


All these new fields and properties should be in MongoEnumerableMethod.cs, not here.

rstam · 2025-08-07T02:07:13Z

...tionExpressionTranslators/MethodTranslators/MedianMethodToAggregationExpressionTranslator.cs

+                {
+                    var selectorLambda = (LambdaExpression)arguments[1];
+                    var selectorParameter = selectorLambda.Parameters[0];
+                    var selectorParameterSerializer = ArraySerializerHelper.GetItemSerializer(sourceTranslation.Serializer);


Rename selectorParameterSerializer to sourceItemSerializer.

I think it's better to name the variable for what it IS, rather than where it is used.

Consider reordering lines 80-85 like this:

var sourceItemSerializer = ArraySerializerHelper.GetItemSerializer(sourceTranslation.Serializer); var selectorLambda = (LambdaExpression)arguments[1]; var selectorParameter = selectorLambda.Parameters[0]; var selectorParameterSymbol = context.CreateSymbol(selectorParameter, sourceItemSerializer); var selectorContext = context.WithSymbol(selectorParameterSymbol); var selectorTranslation = ExpressionToAggregationExpressionTranslator.Translate(selectorContext, selectorLambda.Body);

to extract the sourceItemSerializer from the source serializer earlier.

rstam · 2025-08-07T02:07:35Z

...ExpressionTranslators/MethodTranslators/PercentileMethodToAggregationExpressionTranslator.cs

+
+                if (method.IsOneOf(__percentileWithSelectorMethods))
+                {
+                    var selectorLambda = (LambdaExpression)arguments[1];


Rename selectorParameterSerializer to sourceItemSerializer.

And reorder lines as suggested above.

adelinowona added 6 commits July 29, 2025 06:48

Add enumerable extensions for median and percentile

1837037

add setWindowFields extensions for median and percentile

b5e24ab

Add AstExpression support for median and percentile accumulators/wind…

0235684

…ow expressions

Implement translators for median and percentile methods

ef6c98b

Update grouping pipeline optimizer

3697ac4

add tests

84d9e39

adelinowona requested review from rstam and sanych-sun July 29, 2025 11:03

adelinowona added the improvement label Jul 29, 2025

adelinowona requested a review from a team as a code owner July 29, 2025 11:03

fix some tests failures

91efb8b

rstam approved these changes Jul 29, 2025

View reviewed changes

rstam requested changes Jul 29, 2025

View reviewed changes

sanych-sun requested changes Jul 29, 2025

View reviewed changes

rstam requested changes Jul 29, 2025

View reviewed changes

...xpressionTranslators/MethodTranslators/MedianMethodToAggregationExpressionTranslatorTests.cs Outdated Show resolved Hide resolved

sanych-sun added feature and removed improvement labels Jul 29, 2025

adelinowona added 4 commits July 29, 2025 16:48

rearrange the methods in MongoEnumerable.cs

dc6d059

add median feature and undo accidental change

cafdd06

use reflection pattern to recognize methods in translators

2604f8a

fix complex accumulator visitor bug

61824fd

adelinowona requested review from rstam and sanych-sun July 29, 2025 23:38

sanych-sun requested changes Jul 30, 2025

View reviewed changes

src/MongoDB.Driver/Linq/Linq3Implementation/Ast/Visitors/AstNodeVisitor.cs Outdated Show resolved Hide resolved

rstam requested changes Jul 30, 2025

View reviewed changes

adelinowona added 2 commits July 31, 2025 20:13

use separate classes for median and percentile accumulator ASTs

691b8aa

use StandardSerializers

0cc714d

rstam requested changes Aug 1, 2025

View reviewed changes

update return types and tests

f878f7b

adelinowona requested review from rstam and sanych-sun August 4, 2025 21:46

rstam requested changes Aug 7, 2025

View reviewed changes

CSHARP-3222: Add LINQ support for median and percentile accumulators/window functions #1743

Are you sure you want to change the base?

CSHARP-3222: Add LINQ support for median and percentile accumulators/window functions #1743

Conversation

adelinowona commented Jul 29, 2025

Uh oh!

rstam left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

rstam left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rstam left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!