SourceIncludes does not appear to work and is not including specified Fields in the query

[ahazelwood]

Elastic.Clients.Elasticsearch version: 8.15.10

Elasticsearch version: 8.15.1

.NET runtime version: 8.0

Operating system version: Windows 12

Description of the problem including expected versus actual behavior:
A clear and concise description of what the bug is.

In the previous NEST client, I could issue a query like:

                search = search.Source(s => s
                    .Includes(i => i
                        .Fields(
                            f => f.Id,
                            f => f.Title
                        )
                    )
                );

and the resulting json would look like:

{
    "aggs": {
        "category": {
            "terms": {
                "field": "category",
                "min_doc_count": 0,
                "size": 2147483647
            }
        },
        "stream": {
            "terms": {
                "field": "stream",
                "min_doc_count": 0,
                "size": 2147483647
            }
        },
        "entity": {
            "terms": {
                "field": "entity",
                "min_doc_count": 0,
                "size": 2147483647
            }
        },
        "folder": {
            "terms": {
                "field": "folder",
                "min_doc_count": 0,
                "size": 2147483647
            }
        },
        "label": {
            "terms": {
                "field": "label",
                "min_doc_count": 0,
                "size": 2147483647
            }
        },
        "language": {
            "terms": {
                "field": "language",
                "min_doc_count": 0,
                "size": 2147483647
            }
        },
        "metadata": {
            "terms": {
                "field": "metadata",
                "min_doc_count": 0,
                "size": 2147483647
            }
        },
        "rating": {
            "terms": {
                "field": "rating",
                "min_doc_count": 0,
                "size": 2147483647
            }
        },
        "tag": {
            "terms": {
                "field": "tag",
                "min_doc_count": 0,
                "size": 2147483647
            }
        }
    },
    "from": 0,
    "query": {
        "query_string": {
            "default_operator": "and",
            "query": "projectId:2334"
        }
    },
    "size": 1000,
    "sort": [
        {
            "approved": {
                "order": "desc"
            }
        },
        {
            "_score": {
                "order": "desc"
            }
        },
        {
            "rating": {
                "order": "desc"
            }
        }
    ],
    "_source": {
        "includes": [
            "id",
            "title"
        ]
    },
    "track_total_hits": true
}

In the new ElasticSearch client (after searching through various issues, as well as the code), I am trying to replicate this like the following, but am not getting the includes in the resulting json.

                search = search.SourceIncludes(Fields.FromFields([
                    Infer.Field<SearchAlert>(f => f.Id), Infer.Field<SearchAlert>(f => f.Title)
                ]));

{
    "aggregations": {
        "category": {
            "terms": {
                "field": "category",
                "min_doc_count": 0,
                "size": 2147483647
            }
        },
        "stream": {
            "terms": {
                "field": "stream",
                "min_doc_count": 0,
                "size": 2147483647
            }
        },
        "entity": {
            "terms": {
                "field": "entity",
                "min_doc_count": 0,
                "size": 2147483647
            }
        },
        "folder": {
            "terms": {
                "field": "folder",
                "min_doc_count": 0,
                "size": 2147483647
            }
        },
        "label": {
            "terms": {
                "field": "label",
                "min_doc_count": 0,
                "size": 2147483647
            }
        },
        "language": {
            "terms": {
                "field": "language",
                "min_doc_count": 0,
                "size": 2147483647
            }
        },
        "metadata": {
            "terms": {
                "field": "metadata",
                "min_doc_count": 0,
                "size": 2147483647
            }
        },
        "rating": {
            "terms": {
                "field": "rating",
                "min_doc_count": 0,
                "size": 2147483647
            }
        },
        "tag": {
            "terms": {
                "field": "tag",
                "min_doc_count": 0,
                "size": 2147483647
            }
        }
    },
    "from": 0,
    "query": {
        "query_string": {
            "default_operator": "and",
            "query": "projectId:2423"
        }
    },
    "size": 1000,
    "sort": [
        {
            "approved": {
                "order": "desc"
            }
        },
        {
            "_score": {
                "order": "desc"
            }
        },
        {
            "rating": {
                "order": "desc"
            }
        }
    ],
    "track_total_hits": true
}

Any help would be appreciated as this is one of the last pieces of our client that needs to be updated, and we don't want to bring back all fields from our Search document, but still want to rely upon it for reference within various other Expressions.

[flobernd]

Hi @ahazelwood,

the SourceIncludes value gets serialized in the request url and not the request body. If you - for some reason - need the includes to be in the body, please use this method instead:

.Source(new SourceConfig(new SourceFilter
{
    Includes = f
})

Both methods are causing the correct filters to be applied.

Please let me know, if that solved your issue.

[ahazelwood]

Yes. That worked great. Looking forward to more documentation that details how to effectively use all of the capabilities that exist in the new client.

[alikleitcr7]

Hi @flobernd , how did you compose the fields f ? given that its usually an Action to take the .Field from.

SearchResponse<MyDocument> searchResponse = await client.SearchAsync<MyDocument>(s => s
.Index(index_name)
.From(page)
//.Source(new Elastic.Clients.Elasticsearch.Core.Search.SourceConfig(new SourceFilter()
//{
//    Includes = new string[] { "", "" }  does not work
//})
.Size(countPerPage)

or 
.Source(new Elastic.Clients.Elasticsearch.Core.Search.SourceConfig(new SourceFilter()
{
   Includes = new List<Field>
   {
       "id",
       "title"      .. nope 
   }
})

or 
.Source(new Elastic.Clients.Elasticsearch.Core.Search.SourceConfig(new SourceFilter()
{
    Includes = new Fields(new[]     ...nope
    {
        new Field("id"),
        new Field("postId"),      
        new Field("title")
    })
})

I cannot seem to find the Fields.FromFields ... would it them be the right way to use what I see @ahazelwood is using, the Infer in this case?

@ahazelwood how did it work out for you?

From what I understand, to use Field, is like this

 if (request.SomeId.HasValue)
 {
     mustStatemenets.Add((m) =>
     {
         m.Bool(mb =>
         {
             mb.Must(s =>
             {
                 s.Match(t => t.Field(f => f.SomeId).Query(request.SomeId.Value.ToString()));
             });
         });
     });
 }

My open discussion: Here

---

UPDATE, OK after digging in to the Infer - it seems this is the way to go ?

Infer.Fields<MyDocument>(x => x.SomeField1, x => x.SomeField2)

Version 8.9.3

[flobernd]

@alikleitcr7 Using Infer here is indeed the correct approach 🙂

[ahazelwood]

We haven't upgraded yet because there were still a few issues related to aggregations the last time I looked several months ago. So not sure yet.