Elasticlunr.js


Contributor Welcome!!!

As I'm now focusing on new domain, hope that someone who interested on this project could help to maintain this repository.

If anyone want to help to maintain this repo, please leave a message and I could add you to the admin account. Thanks very much~~~ :)

Milestone

Thanks that so many people use and love Elasticlunr.js, it has received more than 1K stars.

Elasticlunr.js

Elasticlunr.js is a lightweight full-text search engine developed in JavaScript for browser search and offline search.
Elasticlunr.js is developed based on Lunr.js, but more flexible than lunr.js. Elasticlunr.js provides Query-Time boosting, field search, more rational scoring/ranking methodology, fast computation speed and so on.
Elasticlunr.js is a bit like Solr, but much smaller and not as bright, but also provide flexible configuration, query-time boosting, field search and other features.

Why You Need Lightweight Offline Search?

1. In some systems, you don't want to deploy any complex full-text search engine (such as Lucene, Elasticsearch, Sphinx, etc.), you only want to provide some static web pages and provide search functionality, then you could build index in previous and load index in client side (such as Browser).
2. Provide offline search functionality. For some documents, the user usually downloads these documents, you could build index and put index in the documents package, then provide offline search functionality.
3. For limited or restricted networks, such WAN or LAN, offline search is a better choice.
4. For mobile devices, Iphone or Android phones, network traffic may be very expensive, so offline search is a good choice.
5. If you want to provide search functionality in your Node.js system, and you don't want to use a complex system, or you only need to support thousands of documents, then Elasticlunr.js is what you want to use.

Key Features Comparing with Lunr.js

1. Query-Time Boosting, you don't need to set up boosting weight in index building procedure, Query-Time Boosting make it more flexible so you could try different boosting schemes.
2. More Rational Scoring Mechanism, Elasticlunr.js uses a similar scoring mechanism as Elasticsearch, and also this scoring mechanism is used by Lucene.
3. Field-Search, you can choose which field to index and which field to search.
4. Boolean Model, you can set which field to search and the boolean model for each query token, such as "OR", "AND".
5. Combined Boolean Model, TF/IDF Model and the Vector Space Model, makes the results ranking more reliable.
6. Fast, Elasticlunr.js removed TokenCorpus and Vector from lunr.js, by using combined model there is no need to compute the vector of a document and query string to compute similarity of query and matched document, this improves the search speed significantly.
7. Small Index Size, Elasticlunr.js did not store TokenCorpus because there is no need to compute query vector and document vector, then the index file is small, and also user can choose if they need to store the origianl JSON doc, if user cares more about the index size, they can choose not store the original JSON doc. This can reduce the index size significantly. This is especially helpful when elasticlunr.js is used as offline search. The index size is about half size of lunr.js index file.

Example

A very simple search index can be created using the following scripts:

``javascript
var index = elasticlunr(function () {
this.addField('title');
this.addField('body');
this.setRef('id');
});
`

Adding documents to the index is as simple as:

`javascript
var doc1 = {
"id": 1,
"title": "Oracle released its latest database Oracle 12g",
"body": "Yesterday Oracle has released its new database Oracle 12g, this would make more money for this company and lead to a nice profit report of annual year."
}

var doc2 = {
"id": 2,
"title": "Oracle released its profit report of 2015",
"body": "As expected, Oracle released its profit report of 2015, during the good sales of database and hardware, Oracle's profit of 2015 reached 12.5 Billion."
}

index.addDoc(doc1);
index.addDoc(doc2);
`

Then searching is as simple:

`javascript
index.search("Oracle database profit");
`

Also, you could do query-time boosting by passing in a configuration.

`javascript
index.search("Oracle database profit", {
fields: {
title: {boost: 2},
body: {boost: 1}
}
});
`

This returns a list of matching documents with a score of how closely they match the search query:

`javascript
[{
"ref": 1,
"score": 0.5376053707962494
},
{
"ref": 2,
"score": 0.5237481076838757
}]
`

You can also query on different terms for each field.

`javascript
index.search({
title: 'database',
body: 'profit',
});
`

If user doesn't want to store the original JSON documents, they can use the following setting:
`javascript
var index = elasticlunr(function () {
this.addField('title');
this.addField('body');
this.setRef('id');
this.saveDocument(false);
});
`

Then elasticlunr.js will not store the JSON documents, this will reduce the index size, but also bring some inconvenience such as update a document or delete a document by document id or reference. Actually most of the time user will not udpate or delete a document from index.

API documentation is available, as well as a full working example.

Description

Elasticlunr.js is developed based on Lunr.js, but more flexible than lunr.js. Elasticlunr.js provides Query-Time Boosting, Field Search, more rational scoring/ranking methodology, flexible configuration and so on.
A bit like Solr, but much smaller and not as bright, but also provide flexible configuration, query-time boosting, field search, etc.

Installation

Simply include the elasticlunr.js source file in the page that you want to use it. Elasticlunr.js is supported in all modern browsers.

Browsers that do not support ES5 will require a JavaScript shim for Elasticlunr.js to work. You can either use Augment.js, ES5-Shim or any library that patches old browsers to provide an ES5 compatible JavaScript environment.

Documentation

This part only contain important apects of elasticlunr.js, for the whole documentation, please go to API documentation.

1. Build Index

When you first create a index instance, you need to specify which field you want to index. If you did not specify which field to index, then no field will be searchable for your documents.
You could specify fields by:
`javascript
var index = elasticlunr(function () {
this.addField('title');
this.addField('body');
this.setRef('id');
});
`

You could also set the document reference by this.setRef('id'), if you did not set document ref, elasticlunr.js will use 'id' as default.

You could do the above index setup as followings:
`javascript
var index = elasticlunr();
index.addField('title');
index.addField('body');
index.setRef('id');
`

Also you could choose not store the original JSON document to reduce the index size by:
`javascript
var index = elasticlunr();
index.addField('title');
index.addField('body');
index.setRef('id');
index.saveDocument(false);
`




2. Add document to index

Add document to index is very simple, just prepare you document in JSON format, then add it to index.

`javascript
var doc1 = {
"id": 1,
"title": "Oracle released its latest database Oracle 12g",
"body": "Yestaday Oracle has released its new database Oracle 12g, this would make more money for this company and lead to a nice profit report of annual year."
}

var doc2 = {
"id": 2,
"title": "Oracle released its profit report of 2015",
"body": "As expected, Oracle released its profit report of 2015, during the good sales of database and hardware, Oracle's profit of 2015 reached 12.5 Billion."
}

index.addDoc(doc1);
index.addDoc(doc2);
`

If your JSON document contains field that not configured in index, then that field will not be indexed, which means that field is not searchable.

3. Remove document from index

Elasticlunr.js support remove a document from index, just provide JSON document to elasticlunr.Index.prototype.removeDoc() function.

For example:
`javascript
var doc = {
"id": 1,
"title": "Oracle released its latest database Oracle 12g",
"body": "Yestaday Oracle has released its new database Oracle 12g, this would make more money for this company and lead to a nice profit report of annual year."
}

index.removeDoc(doc);
`

Remove a document will remove each token of that document's each field from field-specified inverted index.

4. Update a document in index

Elasticlunr.js support update a document in index, just provide JSON document to elasticlunr.Index.prototype.updateDoc() function.

For example:
`javascript
var doc = {
"id": 1,
"title": "Oracle released its latest database Oracle 12g",
"body": "Yestaday Oracle has released its new database Oracle 12g, this would make more money for this company and lead to a nice profit report of annual year."
}

index.updateDoc(doc);
`

5. Query from Index

Elasticlunr.js provides flexible query configuration, supports query-time boosting and Boolean logic setting.
You could setup a configuration tell elasticlunr.js how to do query-time boosting, which field to search in, how to do the boolean logic.
Or you could just use it by simply provide a query string, this will aslo works perfectly because the scoring mechanism is very efficient.

$3

Because elasticlunr.js has a very perfect scoring mechanism, so for most of your requirement, simple search would be easy to meet your requirement.

`javascript
index.search("Oracle database profit");
`

Output is a results array, each element of results array is an Object contain a ref field and a score field.
ref is the document reference.
score is the similarity measurement.

Results array is sorted descent by score.

$3

#### 5.2.1 Query-Time Boosting

Setup which fields to search in by passing in a JSON configuration, and setup boosting for each search field.
If you setup this configuration, then elasticlunr.js will only search the query string in the specified fields with boosting weight.

The scoring mechanism used in elasticlunr.js is very complex, please goto details for more information.


`javascript
index.search("Oracle database profit", {
fields: {
title: {boost: 2},
body: {boost: 1}
}
});
`

#### 5.2.2 Boolean Model

Elasticlunr.js also support boolean logic setting, if no boolean logic is setted, elasticlunr.js use "OR" logic defaulty. By "OR" default logic, elasticlunr.js could reach a high Recall.

`javascript
index.search("Oracle database profit", {
fields: {
title: {boost: 2},
body: {boost: 1}
},
bool: "OR"
});
`

Boolean model could be setted by global level such as the above setting or it could be setted by field level, if both global and field level contains a "bool" setting, field level setting will overwrite the global setting.
`javascript
index.search("Oracle database profit", {
fields: {
title: {boost: 2, bool: "AND"},
body: {boost: 1}
},
bool: "OR"
});
`
The above setting will search title field by AND model and other fields by "OR" model.
Currently if you search in multiply fields, resutls from each field will be merged together to give the query results. In the future elasticlunr will support configuration that user could set how to combine the results from each field, such as "most_field" or "top_field".

#### 5.2.3 Token Expandation
Sometimes user want to expand a query token to increase RECALL, then user could set expand model to true by configuration, default is false.
For example, user query token is "micro", and assume "microwave" and "microscope" are in the index, then is user choose expand the query token "micro" to increase RECALL, both "microwave" and "microscope" will be returned and search in the index.
The query results from expanded tokens are penalized because they are not exactly the same as the query token.
`javascript
index.search("micro", {
fields: {
title: {boost: 2, bool: "AND"},
body: {boost: 1}
},
bool: "OR",
expand: true
});
`

Field level expand configuration will overwrite global expand configuration.
`javascript
index.search("micro", {
fields: {
title: {
boost: 2,
bool: "AND",
expand: false
},
body: {boost: 1}
},
bool: "OR",
expand: true
});
`

6. Add customized stop words

Elasticlunr.js contains some default stop words of English, such as:
* a
* about
* an
* all
* also
* and
* any
* but
* the
* ...

Defaultly elasticlunr.js contains 120 stop words, user could decide not use these default stop words or add customized stop words.

$3

You could remove default stop words simply as:
`javascript
elasticlunr.clearStopWords();
`

$3

User could add a list of customized stop words.
`javascript
var customized_stop_words = ['an', 'hello', 'xyzabc'];
elasticlunr.addStopWords(customized_stop_words);
`

7. Use elasticlunr in Node.js

Elasticlunr support Node.js, you could use elastilunr in node.js as a node-module.

Install elasticlunr by:
`javascript
npm install elasticlunr
`

then in your node.js project or in node.js console:
`javascript
var elasticlunr = require('elasticlunr');

var index = elasticlunr(function () {
this.addField('title')
this.addField('body')
});

var doc1 = {
"id": 1,
"title": "Oracle released its latest database Oracle 12g",
"body": "Yestaday Oracle has released its new database Oracle 12g, this would make more money for this company and lead to a nice profit report of annual year."
}

var doc2 = {
"id": 2,
"title": "Oracle released its profit report of 2015",
"body": "As expected, Oracle released its profit report of 2015, during the good sales of database and hardware, Oracle's profit of 2015 reached 12.5 Billion."
}

index.addDoc(doc1);
index.addDoc(doc2);

index.search("Oracle database profit");
`

8. Save & Load Index

You just need to build index only one time offline, and then save the index to a JSON file, for future usage, you just need to load the index file.

Save the index to JSON file as followings:
`javascript
var elasticlunr = require('./elasticlunr.js'),
fs = require('fs');

var idx = elasticlunr(function () {
this.setRef('id');

this.addField('title');
this.addField('tags');
this.addField('body');
});

fs.readFile('./example/example_data.json', function (err, data) {
if (err) throw err;

var raw = JSON.parse(data);

var questions = raw.questions.map(function (q) {
return {
id: q.question_id,
title: q.title,
body: q.body,
tags: q.tags.join(' ')
};
});

questions.forEach(function (question) {
idx.addDoc(question);
});

fs.writeFile('./example/example_index.json', JSON.stringify(idx), function (err) {
if (err) throw err;
console.log('done');
});
});
`

The above code is the example index builder usage in elasticlunr online example, you only need to pay attention to:
`javascript
fs.writeFile('./example/example_index.json', JSON.stringify(idx), function (err) {
if (err) throw err;
console.log('done');
});
`

The function of elasticlunr.Index.prototype.toJSON is called in JSON.stringify(idx).

If you want load an alreay persisted index file, you could use:
`javascript
var indexDump = JSON.parse(indexDump)
console.time('load')
window.idx = elasticlunr.Index.load(indexDump)
`

indexDump is a file content, other your could load archived JSON index as whatever way you like. I'm not familiar with Javascript, so you are welcome to update this document to provide detailed example.

Other Languages

Default supported language of elasticlunr.js is English, if you want to use elasticlunr.js to index other language documents, then you need to use elasticlunr.js combined with lunr-languages.

Other languages example in Browser


Suppose you are using elasticlunr.js in browser for other languages, you could download the corresponding language support from lunr-languages, then include the scripts as:
`html


`

then, you could use elasticlunr.js as normal:
`javascript
var index = elasticlunr(function () {
// use the language (de)
this.use(elasticlunr.de);
// then, the normal elasticlunr index initialization
this.addField('title')
this.addField('body')
});
`

Pay attention to the special code:
`javascript
this.use(elasticlunr.de);
`

If you are using other language, such as es(Spanish), download the corresponding lunr.es.js file and lunr.stemmer.support.js, and change the above line to:
`javascript
this.use(elasticlunr.es);
`

Other languages example in Node.js

Suppose you are using elasticlunr.js in Node.js for other languages, you can download the corresponding language support from lunr-languages, put the files lunr.es.js file and lunr.stemmer.support.js in your project, then in your Node.js module, use elasticlunr.js as:

`javascript
var elasticlunr = require('elasticlunr');
require('./lunr.stemmer.support.js')(elasticlunr);
require('./lunr.de.js')(elasticlunr);

var index = elasticlunr(function () {
// use the language (de)
this.use(elasticlunr.de);
// then, the normal elasticlunr index initialization
this.addField('title')
this.addField('body')
});
`

For more details, please go to lunr-languages.

Contributing

See the CONTRIBUTING.mdown` file.