Lightweight and performant Lucene-like parser, serializer and search engine.
npm install liqe
Lightweight and performant Lucene-like parser, serializer and search engine.
* Motivation
* Usage
* Query Syntax
* Liqe syntax cheat sheet
* Keyword matching
* Number matching
* Range matching
* Wildcard matching
* Boolean operators
* Serializer
* AST
* Utilities
* Compatibility with Lucene
* Recipes
* Handling syntax errors
* Highlighting matches
* Development
* Tutorials
Originally built Liqe to enable Roarr log filtering via cli. I have since been polishing this project as a hobby/intellectual exercise. I've seen it being adopted by various CLI and web applications that require advanced search. To my knowledge, it is currently the most complete Lucene-like syntax parser and serializer in JavaScript, as well as a compatible in-memory search engine.
Liqe use cases include:
* parsing search queries
* serializing parsed queries
* searching JSON documents using the Liqe query language (LQL)
Note that the Liqe AST is treated as a public API, i.e., one could implement their own search mechanism that uses Liqe query language (LQL).
``ts
import {
filter,
highlight,
parse,
test,
} from 'liqe';
const persons = [
{
height: 180,
name: 'John Morton',
},
{
height: 175,
name: 'David Barker',
},
{
height: 170,
name: 'Thomas Castro',
},
];
`
Filter a collection:
`ts`
filter(parse('height:>170'), persons);
// [
// {
// height: 180,
// name: 'John Morton',
// },
// {
// height: 175,
// name: 'David Barker',
// },
// ]
Test a single object:
`ts`
test(parse('name:John'), persons[0]);
// true
test(parse('name:David'), persons[0]);
// false
Highlight matching fields and substrings:
`ts`
highlight(parse('name:john'), persons[0]);
// [
// {
// path: 'name',
// query: /(John)/,
// }
// ]
highlight(parse('height:180'), persons[0]);
// [
// {
// path: 'height',
// }
// ]
Liqe uses Liqe Query Language (LQL), which is heavily inspired by Lucene but extends it in various ways that allow a more powerful search experience.
`rbsearch for "foo" term anywhere in the document (case insensitive)
foo
field field field, member of name, i.e.$3
Search for word "foo" in any field (case insensitive).
rb
foo
`Search for word "foo" in the
field.`rb
name:foo
`Search for
field values matching /foo/i regex.`rb
name:/foo/i
`Search for
field values matching f*o wildcard pattern.`rb
name:f*o
`Search for
field values matching f?o wildcard pattern.`rb
name:f?o
`Search for phrase "foo bar" in the
field (case sensitive).`rb
name:"foo bar"
`$3
Search for value equal to 100 in the
field.`rb
height:=100
`Search for value greater than 100 in the
field.`rb
height:>100
`Search for value greater than or equal to 100 in the
field.`rb
height:>=100
`$3
Search for value greater or equal to 100 and lower or equal to 200 in the
field.`rb
height:[100 TO 200]
`Search for value greater than 100 and lower than 200 in the
field.`rb
height:{100 TO 200}
`$3
Search for any word that starts with "foo" in the
field.`rb
name:foo*
`Search for any word that starts with "foo" and ends with "bar" in the
field.`rb
name:foo*bar
`Search for any word that starts with "foo" in the
field, followed by a single arbitrary character.`rb
name:foo?
`Search for any word that starts with "foo", followed by a single arbitrary character and immediately ends with "bar" in the
field.`rb
name:foo?bar
`$3
Search for phrase "foo bar" in the
field AND the phrase "quick fox" in the bio field.`rb
name:"foo bar" AND bio:"quick fox"
`Search for either the phrase "foo bar" in the
field AND the phrase "quick fox" in the bio field, or the word "fox" in the name field.`rb
(name:"foo bar" AND bio:"quick fox") OR name:fox
`Serializer
Serializer allows to convert Liqe tokens back to the original search query.
ts
import {
parse,
serialize,
} from 'liqe';const tokens = parse('foo:bar');
// {
// expression: {
// location: {
// start: 4,
// },
// quoted: false,
// type: 'LiteralExpression',
// value: 'bar',
// },
// field: {
// location: {
// start: 0,
// },
// name: 'foo',
// path: ['foo'],
// quoted: false,
// type: 'Field',
// },
// location: {
// start: 0,
// },
// operator: {
// location: {
// start: 3,
// },
// operator: ':',
// type: 'ComparisonOperator',
// },
// type: 'Tag',
// }
serialize(tokens);
// 'foo:bar'
`AST
ts
import {
type BooleanOperatorToken,
type ComparisonOperatorToken,
type EmptyExpression,
type FieldToken,
type ImplicitBooleanOperatorToken,
type ImplicitFieldToken,
type LiteralExpressionToken,
type LogicalExpressionToken,
type RangeExpressionToken,
type RegexExpressionToken,
type TagToken,
type UnaryOperatorToken,
} from 'liqe';
`There are 11 AST tokens that describe a parsed Liqe query.
If you are building a serializer, then you must implement all of them for the complete coverage of all possible query inputs. Refer to the built-in serializer for an example.
Utilities
ts
import {
isSafeUnquotedExpression,
} from 'liqe';/**
* Determines if an expression requires quotes.
* Use this if you need to programmatically manipulate the AST
* before using a serializer to convert the query back to text.
*/
isSafeUnquotedExpression(expression: string): boolean;
`Compatibility with Lucene
The following Lucene abilities are not supported:
* Fuzzy Searches
* Proximity Searches
* Boosting a Term
Recipes
$3
In case of a syntax error, Liqe throws
.`ts
import {
parse,
SyntaxError,
} from 'liqe';try {
parse('foo bar');
} catch (error) {
if (error instanceof SyntaxError) {
console.error({
// Syntax error at line 1 column 5
message: error.message,
// 4
offset: error.offset,
// 1
offset: error.line,
// 5
offset: error.column,
});
} else {
throw error;
}
}
`$3
package to highlight Liqe matches.Development
$3
If you are going to modify parser, then use
npm run watch to run compiler in watch mode.$3
Before making any changes, capture the current benchmark on your machine using
npm run benchmark`. Run benchmark again after making any changes. Before committing changes, ensure that performance is not negatively impacted.