MongoDB custom filter

The river settings support a new filter attribute which can be used as a "trigger" for the document that should be indexed.

Let"s say there is a document { "name": "richard", "needs_idexing": 0 } stored in MongoDB. This document should be only indexed by ES if needs_indexing is 1. The filter attribute could be used for this scenario. The filter value will be: {"o.needs_indexing": 1}. So the river setting will be:

{
 "type": "mongodb",
 "mongodb": {
  "servers": [
   {
    "host": "localhost",
    "port": "27017"
   }
  ],
  "db": "testriver",
  "collection": "documents",
  "gridfs": "false",
  "filter": "{\"o.score\" : 1}"
 },
 "index": {
  "name": "documentsindex",
 }
}

Filter has been only tested with "MongoDB dot notation" [1].

Filter value should be prefix by o. because of the structure of the document in oplog.rs collection. More details available here [2].

Current limitation: The indexed document will not be remove from ES index if the filter condition does not match anymore. In our example needs_indexing has been set to 1 so the document is indexed by ES. A new update in MongoDB set needs_indexing to 0 the document will not be remove from ES.

fn1. "Dot Notation (Reaching into Objects)":http://www.mongodb.org/display/DOCS/Dot+Notation+(Reaching+into+Objects)

fn2. "Replication Internals":http://www.kchodorow.com/blog/2010/10/12/replication-internals/