FileSystem River for Elasticsearch

Welcome to the FS River Plugin for Elasticsearch

This river plugin helps to index documents from your local file system.

WARNING: If you use this river in a multinode mode on differents servers, you need to ensure that the river can access files on the same mounting point. If not, when a node stop, the other node will think that your local dir is empty and will erase all your docs.

WARNING: starting from 0.0.3, you need to have the Attachment Plugin. It's not included anymore in the distribution.

Versions

FS River Plugin	ElasticSearch	Attachment Plugin
master (0.0.3)	0.19.8	1.4.0
0.0.2	0.19.8	1.4.0
0.0.1	0.19.4	1.4.0

Build Status

Thanks to cloudbees for the build status :

Getting Started

Installation

Just type :

$ bin/plugin -install dadoonet/fsriver/0.0.2

This will do the job...

-> Installing dadoonet/fsriver/0.0.2...
Trying https://github.com/downloads/dadoonet/fsriver/fsriver-0.0.2.zip...
Downloading ...DONE
Installed fsriver

Creating a FS river

We create first an index to store our documents :

$ curl -XPUT 'localhost:9200/mydocs/' -d '{}'

We create the river with the following properties :

• FS URL : /tmp
• Update Rate : every 15 minutes (15 * 60 * 1000 = 900000 ms)
• Get only docs like *.doc and *.pdf
• Don't index resume*

$ curl -XPUT 'localhost:9200/_river/mydocs/_meta' -d '{
  "type": "fs",
  "fs": {
	"name": "My tmp dir",
	"url": "/tmp",
	"update_rate": 900000,
	"includes": "*.doc,*.pdf",
	"excludes": "resume"
  }
}'

Adding another FS river

We add another river with the following properties :

• FS URL : /tmp2
• Update Rate : every hour (60 * 60 * 1000 = 3600000 ms)
• Get only docs like *.doc, *.xls and *.pdf

By the way, we define to index in the same index/type as the previous one:

• index: docs
• type: doc

$ curl -XPUT 'localhost:9200/_river/mynewriver/_meta' -d '{
  "type": "fs",
  "fs": {
	"name": "My tmp2 dir",
	"url": "/tmp2",
	"update_rate": 3600000,
	"includes": [ "*.doc" , "*.xls", "*.pdf" ]
  },
  "index": {
  	"index": "mydocs",
  	"type": "doc",
  	bulk_size: 50
  }
}'

Searching for docs

This is a common use case in elasticsearch, we want to search for something ;-)

$ curl -XGET http://localhost:9200/docs/doc/_search -d '{
  "query" : {
    "text" : {
        "_all" : "I am searching for something !"
    }
  }
}'

Advanced

Autogenerated mapping

When the FSRiver detect a new type, it creates automatically a mapping for this type.

{
  "doc" : {
    "properties" : {
      "file" : {
        "type" : "attachment",
        "path" : "full",
        "fields" : {
          "file" : {
            "type" : "string",
            "store" : "yes",
            "term_vector" : "with_positions_offsets"
          },
          "author" : {
            "type" : "string"
          },
          "title" : {
            "type" : "string",
            "store" : "yes"
          },
          "name" : {
            "type" : "string"
          },
          "date" : {
            "type" : "date",
            "format" : "dateOptionalTime"
          },
          "keywords" : {
            "type" : "string"
          },
          "content_type" : {
            "type" : "string"
          }
        }
      },
      "name" : {
        "type" : "string",
        "analyzer" : "keyword"
      },
      "pathEncoded" : {
        "type" : "string",
        "analyzer" : "keyword"
      },
      "postDate" : {
        "type" : "date",
        "format" : "dateOptionalTime"
      },
      "rootpath" : {
        "type" : "string",
        "analyzer" : "keyword"
      },
      "virtualpath" : {
        "type" : "string",
        "analyzer" : "keyword"
      }
    }
  }
}

Creating your own mapping (analyzers)

If you want to define your own mapping to set analyzers for example, you can push the mapping before starting the FS River.

{
  "doc" : {
    "properties" : {
      "file" : {
        "type" : "attachment",
        "path" : "full",
        "fields" : {
          "file" : {
            "type" : "string",
            "store" : "yes",
            "term_vector" : "with_positions_offsets",
            "analyzer" : "french"
          },
          "author" : {
            "type" : "string"
          },
          "title" : {
            "type" : "string",
            "store" : "yes"
          },
          "name" : {
            "type" : "string"
          },
          "date" : {
            "type" : "date",
            "format" : "dateOptionalTime"
          },
          "keywords" : {
            "type" : "string"
          },
          "content_type" : {
            "type" : "string"
          }
        }
      },
      "name" : {
        "type" : "string",
        "analyzer" : "keyword"
      },
      "pathEncoded" : {
        "type" : "string",
        "analyzer" : "keyword"
      },
      "postDate" : {
        "type" : "date",
        "format" : "dateOptionalTime"
      },
      "rootpath" : {
        "type" : "string",
        "analyzer" : "keyword"
      },
      "virtualpath" : {
        "type" : "string",
        "analyzer" : "keyword"
      }
    }
  }
}

To send mapping to Elasticsearch, refer to the Put Mapping API

Meta fields

FS River creates some meta fields :

Field	Description	Example
name	Original file name	mydocument.pdf
pathEncoded	BASE64 encoded file path (for internal use)	112aed83738239dbfe4485f024cd4ce1
postDate	Indexing date	1312893360000
rootpath	BASE64 encoded root path (for internal use)	112aed83738239dbfe4485f024cd4ce1
virtualpath	Relative path	mydir/otherdir

Advanced search

You can use meta fields to perform search on.

$ curl -XGET http://localhost:9200/docs/doc/_search -d '{
  "query" : {
    "term" : {
        "name" : "mydocument.pdf"
    }
  }
}'

Behind the scene

How it works ?

TO BE COMPLETED

License

This software is licensed under the Apache 2 license, quoted below.

Copyright 2011-2012 David Pilato

Licensed under the Apache License, Version 2.0 (the "License"); you may not
use this file except in compliance with the License. You may obtain a copy of
the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations under
the License.