LoopBack Connector for Elasticsearch 6.x and 7.x
npm install loopback-connector-esv6
Elasticsearch(versions 6.x and 7.x) datasource connector for Loopback 3.x.
- Overview
- Install this connector in your loopback app
- Configuring connector
- Required properties
- Recommended properties
- Optional properties
- Sample for copy paste
- Elasticsearch SearchAfter Support
- TotalCount Support for search
- Example
- Troubleshooting
- Contributing
- Frequently Asked Questions
1. lib directory has the entire source code for this connector
1. this is what gets downloaded to your node_modules folder when you run npm install loopback-connector-esv6 --save --save-exact
1. examples directory has a loopback app which uses this connector
1. this is not published to NPM, it is only here for demo purposes
1. it will not be downloaded to your node_modules folder!
1. similarly the examples/server/datasources.json file is there for this demo app to use
1. you can copy their content over to examples/server itself and expect changes to take place in your app!
1. test directory has unit tests
1. it does not reuse the loopback app from the examples folder
1. instead, loopback and ES/datasource are built and injected programatically
1. this directory is not published to NPM.
1. Refer to .npmignore if you're still confused about what's part of the published connector and what's not.
1. You will find the datasources.json files in this repo mention various configurations:
1. elasticsearch-ssl
2. elasticsearch-plain
3. db
4. You don't need them all! They are just examples to help you see the various ways in which you can configure a datasource. Delete the ones you don't need and keep the one you want. For example, most people will start off with elasticsearch-plain and then move on to configuring the additional properties that are exemplified in elasticsearch-ssl. You can mix & match if you'd like to have mongo and es and memory, all three! These are basics of the "connector" framework in loooback and not something we added.
1. Don't forget to edit your model-config.json file and point the models at the dataSource you want to use.
``bash`
cd
npm install loopback-connector-esv6 --save --save-exact
- This connector will only connect to one index per datasource.
- This package is created to support ElasticSearch v6.x and 7.x only.
- docType property is automatically added in mapping properties which is required to differentiate documents stored in index with loopback model data. It stores loopback modelName value. docType: { type: "keyword", index: true }
- name: name of the connector.
- connector: Elasticsearch driver 'esv6'.
- configuration: Elasticsearch client configuraiton object which includes nodes, authetication and ssl coonfiguration. Please refer this official link for more information on configuraiton.
- index: Name of the ElasticSearch index eg: shakespeare.eg: version: 7
- version: specify the major version of the Elasticsearch nodes you will be connecting to. Supported versions: [6, 7]
- mappingType: mapping type for provided index. defaults to . Required only for version: 6mappingType
- mappingProperties: An object with properties for above mentioned
- indexSettings: optional settings object for creating index.
- defaultSize: Search size limit. Default is 50.
1.Edit datasources.json and set:
`javascript
"elastic-search-ssl": {
"name": "elasticsearch-example-index-datasource",
"connector": "esv6",
"version": 7,
"index": "example-index",
"configuration": { // Elastic client configuration
"node": "http://localhost:9200",
"requestTimeout": 30000,
"pingTimeout": 3000,
"auth": {
"username": "test",
"password": "test"
},
"ssl": {
"rejectUnauthorized": true
}
},
"defaultSize": 50,
"indexSettings": { // Elastic index settings
"number_of_shards": 2,
"number_of_replicas": 1
},
"mappingType": "basedata", // not required for verison: 7, will be ignored
"mappingProperties": {
"docType": {
"type": "keyword",
"index": true
},
"id": {
"type": "keyword",
"index": true
},
"seq": {
"type": "integer",
"index": true
},
"name": {
"type": "keyword",
"index": true
},
"email": {
"type": "keyword",
"index": true
},
"birthday": {
"type": "date",
"index": true
},
"role": {
"type": "keyword",
"index": true
},
"order": {
"type": "integer",
"index": true
},
"vip": {
"type": "boolean",
"index": true
},
"objectId": {
"type": "keyword",
"index": true
},
"ttl": {
"type": "integer",
"index": true
},
"created": {
"type": "date",
"index": true
}
}
}
`
2.You can peek at /examples/server/datasources.json for more hints.
- `search_after` feature of elasticsearch is supported in loopback filter.`
- For this, you need to create a property in model called _search_after` with loopback type `["any"]`. This field cannot be updated using in connector.`
- Elasticsearch sort` value will return in this field.`
- You need pass _search_after` value in `searchafter` key of loopback filter.`
- Example filter query for find`.
`json`
{
"where": {
"username": "hello"
},
"order": "created DESC",
"searchafter": [
1580902552905
],
"limit": 4
}
- Example result.
`json`
[
{
"id": "1bb2dd63-c7b9-588e-a942-15ca4f891a80",
"username": "test",
"_search_after": [
1580902552905
],
"created": "2020-02-05T11:35:52.905Z"
},
{
"id": "fd5ea4df-f159-5816-9104-22147f2a740f",
"username": "test3",
"_search_after": [
1580902552901
],
"created": "2020-02-05T11:35:52.901Z"
},
{
"id": "047c0adb-13ea-5f80-a772-3d2a4691d47a",
"username": "test4",
"_search_after": [
1580902552897
],
"created": "2020-02-05T11:35:52.897Z"
}
]
- This is useful for pagination. To go to previous page, change sorting order.
- `total` value from elasticsearch for search queries is now supported in loopback response.`
- For this, you need to create a property in model called _total_count` with loopback type `"number"`. This field cannot be updated using in connector.`
- Example response find`.
`json`
[
{
"id": "1bb2dd63-c7b9-588e-a942-15ca4f891a80",
"username": "test",
"_search_after": [
1580902552905
],
"_total_count": 3,
"created": "2020-02-05T11:35:52.905Z"
},
{
"id": "fd5ea4df-f159-5816-9104-22147f2a740f",
"username": "test3",
"_search_after": [
1580902552901
],
"_total_count": 3,
"created": "2020-02-05T11:35:52.901Z"
},
{
"id": "047c0adb-13ea-5f80-a772-3d2a4691d47a",
"username": "test4",
"_search_after": [
1580902552897
],
"_total_count": 3,
"created": "2020-02-05T11:35:52.897Z"
}
]
1. The examples directory contains a loopback app which uses this connector.
1. You can point this example at your own elasticsearch instance or use the quick instances provided via docker.
1. Do you have both elasticsearch-ssl and elasticsearch-plain in your datasources.json file? You just need one of them (not both), based on how you've setup your ES instance.model-config.json
1. Did you forget to set to point at the datasource you configured? Maybe you are using a different or misspelled name than what you thought you had!version
1. Make sure to configure major version of Elastic in
1. Maybe the version of ES you are using isn't supported by the client that this project uses. Try removing the sub-dependency from folder and then install the latest client:cd
1.
1. then remove && es7 folderrm -rf es6 es7
1. unix/mac quickie:
1.
1. go back to yourApp's root directory
1. unix/mac quickie:
1. And test that you can now use the connector without any issues!
1. These changes can easily get washed away for several reasons. So for a more permanent fix that adds the version you want to work on into a release of this connector, please look into Contributing.
1. Feel free to contribute via PR or open an issue for discussion or jump into the gitter chat room if you have ideas.
1. I recommend that project contributors who are part of the team:
1. should merge master into develop ... if they are behind, before starting the feature branchfeature
1. should create branches from the develop branchfeature
1. should merge into develop then create a release branch to:npm-release x.x.x -m "
1. update the changelog
1. close related issues and mention release version
1. update the readme
1. fix any bugs from final testing
1. commit locally and run
1. merge into both master and develop
1. push and develop to GitHubdevelop
1. For those who use forks:
1. please submit your PR against the branch, if possiblemaster
1. if you must submit your PR against the branch ... I understand and I can't stop you. I only hope that there is a good reason like develop not being up-to-date with master for the work you want to build upon.npm-release
1. may be used to publish. Pubilshing to NPM should happen from the master branch. It should ideally only happen when there is something release worthy. There's no point in publishing just because of changes to test or examples folder or any other such entities that aren't part of the "published module" (refer to .npmignore) to begin with.
1. How do we enable or disable the logs coming from the underlying elasticsearch client? There may be a need to debug/troubleshoot at times.
1. Use the env variable DEBUG=elasticsearch for elastic client logs.DEBUG=loopback:connector:elasticsearch
1. How do we enable or disable the logs coming from this connector?
1. By default if you do not set the following env variable, they are disabled:
1. What are the tests about? Can you provide a brief overview?
1. Tests are prefixed with or 02 etc. in order to run them in that order by leveraging default alphabetical sorting.02.basic-querying.test.js
1. The file uses two models to test various CRUD operations that any connector must provide, like find(), findById(), findByIds(), updateAttributes() etc.User
1. the two models are and Customer
2. their ES mappings are laid out in
3. their loopback definitions can be found in the first block that performs setup in 02.basic-querying.test.js file ... these are the equivalent of a MyModel.json in your real loopback app.id
1. naturally, this is also where we define which property serves as the for the model and if its generated or not_id
1. How do we get elasticserch to take over ID generation?
1. An automatically generated id-like field that is maintained by ES is . Without some sort of es-field-level-scripting-on-index (if that is possible at all) ... I am not sure how we could ask elasticsearch to take over auto-generating an id-like value for any arbitrary field! So the connector is setup such that adding id: {type: String, generated: true, id: true} will tell it to use _id as the actual field backing the id ... you can keep using the doing model.id abstraction and in the background _id values are mapped to it.generated: true
1. Will this work for any field marked as with and id: true?_id
1. No! The connector isn't coded that way right now ... while it is an interesting idea to couple any such field with ES's field inside this connector ... I am not sure if this is the right thing to do. If you had objectId: {type: String, generated: true, id: true} then you won't find a real objectId field in your ES documents. Would that be ok? Wouldn't that confuse developers who want to write custom queries and run 3rd party app against their ES instance? Don't use objectId, use _id` would have to be common knowledge. Is that ok?