SERVER-25957 Optimize $filter + $arrayElemAt to avoid scanning entire array by KevinCybura · Pull Request #1236 · mongodb/mongo

KevinCybura · 2018-04-26T18:16:56Z

No description provided.

cswanson310 · 2018-04-26T22:54:38Z

src/mongo/db/pipeline/expression.cpp

 }

+intrusive_ptr<Expression> ExpressionArrayElemAt::optimize() {
+    // If ExpressionArrayElemAt is passed an ExpressionFilter as its first arugment


Similar to what you did in the ExpressionIndexOf optimization, I think you should first call ExpressionNary::optimize():

mongo/src/mongo/db/pipeline/expression.cpp

Lines 2809 to 2813 in c799c6c

// This will optimize all arguments to this expression.

auto optimized = ExpressionNary::optimize();

if (optimized.get() != this) {

return optimized;

}

(Also, while you're there, it looks like the comment right below that got wrapped at a strange place, can you fix that?)

cswanson310 · 2018-04-26T22:57:58Z

src/mongo/db/pipeline/expression.cpp

+                    str::stream() << getOpName() << "'s second argument must be a numeric value,"
+                                  << " but is " 
+                                  << typeName(indexArg.getType()),
+                    indexArg.numeric());


I think this check is redundant with the one below. If something is integral it must be numeric. https://github.com/mongodb/mongo/blob/master/src/mongo/db/pipeline/value.cpp#L1010

cswanson310 · 2018-04-26T22:58:13Z

src/mongo/db/pipeline/expression.cpp

+                                  << " a 32-bit integer: " 
+                                  << indexArg.coerceToDouble(),
+                    indexArg.integral());
+            auto ind = indexArg.coerceToInt();


Use full names, index

cswanson310 · 2018-04-26T22:58:46Z

src/mongo/db/pipeline/expression.cpp

+                    str::stream() << "Second argument to $slice must be a numeric value,"
+                                  << " but is of type: "
+                                  << typeName(secondVal.getType()),
+                    secondVal.numeric());


Again, I don't think you need the assertion about numeric().

cswanson310 · 2018-04-26T22:59:34Z

src/mongo/db/pipeline/expression.cpp

+                if (auto thirdArg = dynamic_cast<ExpressionConstant*>(vpOperand[2].get())) {
+                    auto thirdVal = thirdArg->getValue();
+
+                    uassert(50799,


I think you can condense these three asserts into 'uassertIfNotIntegralAndNonNegative': https://github.com/mongodb/mongo/blob/master/src/mongo/db/pipeline/expression.cpp#L2704

cswanson310 · 2018-04-26T23:41:19Z

src/mongo/db/pipeline/expression.cpp

+
+                    int arg3 = thirdVal.coerceToInt();
+                    if (arg2 >= 0) {
+                        // If ExpressionSlice is given three arguments set limit to 'firstArg' +


Can you elaborate a little bit more here? Maybe something like If we have three arguments, then we can stop the $filter once we have seen 'firstArg' + 'secondArg' + 1. We need to skip 'firstArg' elements, then include 'secondArg' things after that. For example, if we want to slice starting from 2 with length 3, we skip the first 2, then include 3 after that.

Wait, I think we don't need the +1 here? Is that right?

cswanson310 · 2018-04-26T23:47:33Z

src/mongo/db/pipeline/expression.h

    boost::intrusive_ptr<Expression> _input;
    // The expression determining whether each element should be present in the result array.
    boost::intrusive_ptr<Expression> _filter;
+    // For expression ArrayElemAt and Slice we can optimize filter by setting a limit to end the


I'd make it clearer that this optimization applies when $filter is being used as an argument to either $slice or $arrayElemAt. Also, feel free to use the $slice name instead of ExpressionSlice here.

cswanson310 · 2018-04-26T23:47:56Z

src/mongo/db/pipeline/expression_test.cpp


 }  // namespace Object

+TEST(ExpressionFilter, ExpressionFilterWithASetLimitShouldReturnAArrayNoGreaterThanTheLimit) {


Use AnArray instead of AArray.

cswanson310 · 2018-04-26T23:50:25Z

src/mongo/db/pipeline/expression_test.cpp

+    auto arrayElemAtSpec = BSON("$arrayElemAt" << BSON_ARRAY(filterSpec << 2));
+
+    auto expArrayElemAt = ExpressionArrayElemAt::parse(expCtx, arrayElemAtSpec.firstElement(), vps);
+    auto xpArrayElemAt = dynamic_cast<ExpressionArrayElemAt*>(expArrayElemAt.get())->optimize();


Use a better variable name, maybe 'optimized'?

cswanson310 · 2018-04-26T23:50:46Z

src/mongo/db/pipeline/expression_test.cpp

+    auto val = xpArrayElemAt->evaluate(Document());
+    ASSERT_VALUE_EQ(val, Value(6));
+
+    auto eA = ExpressionArrayElemAt::parse(expCtx, BSON("$arrayElemAt" << BSON_ARRAY(BSON_ARRAY(1 << 2 <<3 << 4 <<5) << 1)).firstElement(), vps);


You can re-use the variable names from before, I think that'd be clearer than two-letter variable names.

Same with below. Alternatively, you could separate each one into it's own test.

… for it

kelly-cs · 2025-10-03T21:49:01Z

Hey @KevinCybura!

I was checking through old PR's and saw that SERVER-25957 is actually still open, as this wasn't merged. I am poking that ticket to see where it's at.

If you are still looking to get these changes in, can you please resolve the merge conflicts and get this caught up with master? I saw you already signed our contributor's agreement. I know it's been a few years, so it's okay if that's not on the table. (I'm trying to clean up our pull request queue right now)

KevinCybura added 5 commits April 20, 2018 11:11

optimizeFilter 04/20/2018: test complete ready for pull request

d6e3fe5

optimizeFilter 04/20/2018: ready for pull request

5c95848

optimizeFilter 04/25/2018: ready for pull request

9971faf

optimizeFilter 04/25/2018: ready for pull request

ed42718

optimizeFilter 04/26/2018: ready for pull request

f2df3ca

cswanson310 requested changes Apr 26, 2018

View reviewed changes

cswanson310 self-assigned this Apr 27, 2018

KevinCybura added 3 commits April 27, 2018 15:22

optimizeFilter 04/27/2018: requested changes made

e856658

optimizeFilter 05/1/2018: added ExpressionNary optimize and unittests…

385438b

… for it

optimizeFilter: final changes

11eabed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SERVER-25957 Optimize $filter + $arrayElemAt to avoid scanning entire array#1236

SERVER-25957 Optimize $filter + $arrayElemAt to avoid scanning entire array#1236
KevinCybura wants to merge 8 commits intomongodb:masterfrom
KevinCybura:optimizeFilter

KevinCybura commented Apr 26, 2018

Uh oh!

cswanson310 Apr 26, 2018

Uh oh!

cswanson310 Apr 26, 2018

Uh oh!

cswanson310 Apr 26, 2018

Uh oh!

cswanson310 Apr 26, 2018

Uh oh!

cswanson310 Apr 26, 2018

Uh oh!

cswanson310 Apr 26, 2018

Uh oh!

cswanson310 Apr 26, 2018

Uh oh!

cswanson310 Apr 26, 2018

Uh oh!

cswanson310 Apr 26, 2018

Uh oh!

cswanson310 Apr 26, 2018

Uh oh!

cswanson310 Apr 26, 2018

Uh oh!

kelly-cs commented Oct 3, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	// This will optimize all arguments to this expression.
	auto optimized = ExpressionNary::optimize();
	if (optimized.get() != this) {
	return optimized;
	}


		} // namespace Object

		TEST(ExpressionFilter, ExpressionFilterWithASetLimitShouldReturnAArrayNoGreaterThanTheLimit) {

Conversation

KevinCybura commented Apr 26, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kelly-cs commented Oct 3, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants