LoopBack Connector for Elasticsearch 7.x
npm install loopback-connector-esv7
Basic Elasticsearch datasource connector for Loopback.
Table of Contents generated with DocToc
- Overview
- Install this connector in your loopback app
- Configuring connector
- Required properties
- Recommended properties
- Optional properties
- Sample for copy paste
- About the example app
- Run both example and ES in docker
- Run example locally and ES in docker
- Run example locally
- How to achieve Instant search
- Troubleshooting
- Testing
- Contributing
- Frequently Asked Questions
- Release notes
1. lib directory has the entire source code for this connector
1. this is what gets downloaded to your node_modules folder when you run npm install loopback-connector-es --save --save-exact
1. examples directory has a loopback app which uses this connector
1. this is not published to NPM, it is only here for demo purposes
1. it will not be downloaded to your node_modules folder!
1. similarly the examples/server/datasources.json and examples/server/datasources. files are there for this demo app to use
1. you can copy their content over to examples/server itself and expect changes to take place in your app!
1. test directory has unit tests
1. it does not reuse the loopback app from the examples folder
1. instead, loopback and ES/datasource are built and injected programatically
1. this directory is not published to NPM.
1. Refer to .npmignore if you're still confused about what's part of the published connector and what's not.
1. You will find the datasources.json files in this repo mention various configurations:
1. elasticsearch-ssl
2. elasticsearch-plain
3. db
4. You don't need them all! They are just examples to help you see the various ways in which you can configure a datasource. Delete the ones you don't need and keep the one you want. For example, most people will start off with elasticsearch-plain and then move on to configuring the additional properties that are exemplified in elasticsearch-ssl. You can mix & match if you'd like to have mongo and es and memory, all three! These are basics of the "connector" framework in loooback and not something we added.
1. Don't forget to edit your model-config.json file and point the models at the dataSource you want to use.
``
cd
npm install loopback-connector-esv7 --save --save-exact
. (mandatory)
- apiVersion: specify the major version of the Elasticsearch nodes you will be connecting to.$3
- mappingType: mapping type for provided index. defaults to basedata
- mappingProperties: An object with properties for above mentioned mappingType#### Important Note:
- This package is created to support ElasticSearch v7.x only.
-
property is automatically added in mapping properties which is required to differentiate documents stored in index with loopback model data. It stores loopback modelName value. docType: { type: "keyword", index: true }$3
- log: sets elasticsearch client's logging, you can refer to the docs here
- defaultSize: total number of results to return per page.
- refreshOn optional array with method names you want to set refresh option as true
- requestTimeout: this value is in milliseconds
- ssl: useful for setting up a secure channel
- protocol: can be or https (http is the default if none specified) ... must be https if you're using ssl
- auth: useful if you have access control setup via services like or found or shield
- amazonES: configuration for NOTE: The package needs to be installed in your project. Its not part of this Connector. Version 1.x.x (currently 1.1.3) will have to be used, later versions will not pass through aws configuration.$3
1. Edit datasources.json and set:
`
"db": {
"connector": "es",
"name": "",
"index": "",
"hosts": [
{
"protocol": "http",
"host": "127.0.0.1",
"port": 9200,
"auth": "username:password"
}
],
"apiVersion": "7.0",
"refreshOn": ["save","create", "updateOrCreate"],
"log": "trace",
"defaultSize": ,
"requestTimeout": 30000,
"ssl": {
"ca": "./../cacert.pem",
"rejectUnauthorized": true
},
"amazonES": {
"region": "us-east-1",
"accessKey": "AKID",
"secretKey": "secret"
},
"mappingType": "basedata",
"mappingProperties": {
"id": {
"type": "keyword",
"index": true
},
"docType": {
"type": "keyword",
"index": true
},
"name": {
"type": "text",
"index": true
},
"realm": {
"type": "keyword",
"index": true
},
"username": {
"type": "keyword",
"index": true
},
"description": {
"type": "text",
"index": true
},
"roleId": {
"type": "keyword",
"index": true
}
},
"settings": {}
}
2. You can peek at for more hints.About the example app
1. The
examples directory contains a loopback app which uses this connector.
1. You can point this example at your own elasticsearch instance or use the quick instances provided via docker.$3
As a developer, you may want a short lived ES instance that is easy to tear down when you're finished dev testing. We recommend docker to facilitate this.
Pre-requisites
You will need docker-engine and docker-compose installed on your system.
Step-1
- Set desired versions for node and Elasticsearch
- here are the valid values to use for Node
- here are the valid values to use for Elasticsearch
`
combination of node v0.10.46 with elasticsearch v1
export NODE_VERSION=0.10.46
export ES_VERSION=1
echo 'NODE_VERSION' $NODE_VERSION && echo 'ES_VERSION' $ES_VERSIONsimilarly feel free to try relevant combinations:
of node v0.10.46 with elasticsearch v2
of node v0.12 with elasticsearch v2
of node v0.4 with elasticsearch v2
of node v5 with elasticsearch v2
elasticsearch v5 will probably not work as there isn't an
client for it, as of this writing
etc.
`
Step-2
- Run the setup with commands.`
git clone https://github.com/strongloop-community/loopback-connector-elastic-search.git myEsConnector
cd myEsConnector/examples
npm install
docker-compose up
Step-3
- Visit
and you will find our example loopback app running there.$3
1. Empty out
examples/server/datasources.json so that it only has the following content remaining: {}
1. Set the environment variable on your local/host machine
1. Set the environment variable NODE_ENV=sample-es-plain-1 if you want to use examples/server/datasources.sample-es-plain-1.js
1. Set the environment variable if you want to use examples/server/datasources.sample-es-plain-2.js
1. Set the environment variable if you want to use examples/server/datasources.sample-es-ssl-1.js
1. a sample docker instance for this hasn't been configured yet, so it doesn't work out-of-the-box, use it only as readable (not runnable) reference material for now
1. You can configure your own or datasources. based on what you learn from these sample files.
1. Technically, to run the example, you don't need to set NODE_ENV if you won't be configuring via the . files ... configuring everything within datasources.json is perfectly fine too. Just remember that you will lose the ability to have inline comments and will have to use double-quotes if you stick with .json
1. Start elasticsearch version 1.x and 2.x using:
git clone https://github.com/strongloop-community/loopback-connector-elastic-search.git myEsConnector
cd myEsConnector
docker-compose -f docker-compose-for-tests.yml up # in another terminal window or tab
cd myEsConnector/examples
npm install
DEBUG=boot:test:* node server/server.js
1. Visit and you will find our example loopback app running there.$3
1. Install dependencies and start the example server
`
git clone https://github.com/strongloop-community/loopback-connector-elastic-search.git myEsConnector
cd myEsConnector/examples
npm install
2. Configure the connector
* Don't forget to create an index in your ES instance:
* If you mess up and want to delete, you can use:
* Don't forget to set a valid value for field in examples/server/datasources.json that matches the version of ES you are running.
3. Set up a cacert.pem file for communicating securely (https) with your ES instance. Download the certificate chain for your ES server using this sample (will need to be edited to use your provider) command:
`
cd myEsConnector
openssl s_client -connect my.es.cluster.com:9243 -showcerts | tee cacert.pem
1. The command may not self terminate so you may need to use
2. It will be saved at the base of your cloned project
3. Sometimes extra data is added to the file, you should delete everything after the following lines:
---
No client certificate CA names sent
---
4. Run:
cd myEsConnector/examples
DEBUG=boot:test:* node server/server.js
* The file will automatically populate data for UserModels on your behalf when the server starts.
5. Open this URL in your browser: http://localhost:3000/explorer
* Try fetching all the users via the rest api console
* You can dump all the data from your ES index, via cmd-line too: curl -X POST username:password@my.es.cluster.com/shakespeare/_search -d '{"query": {"match_all": {}}}'
6. To test a specific filter via GET method, use for example: How to achieve Instant search
From version 1.3.4,
option is added which support's instant search after create and update. This option is configurable and one can activate or deactivate it according to their need. By default refresh is true which makes response to come only after documents are indexed(searchable).
To know more about refresh go through this article$3
Datasource File: Pass refreshOn array from datasource file including methods name in which you want this to be true
"es": {
"name": "es",
"refreshOn": ["save","create", "updateOrCreate"],
.....
Model.json file: Configurable on per model and operation level (, false, wait_for)
`
"elasticsearch": {
"create": {
"refresh": false
},
"destroy": {
"refresh": false
},
"destroyAll": {
"refresh": "wait_for"
}
}
###### NOTE:- While a refresh is useful, it still has a performance cost. A manual refresh can be useful, but avoid manual refresh every time you index a document in production; it will hurt your performance. Instead, your application needs to be aware of the near real-time nature of Elasticsearch and make allowances for it.Troubleshooting
1. Do you have both
and elasticsearch-plain in your datasources.json file? You just need one of them (not both), based on how you've setup your ES instance.
1. Did you forget to set model-config.json to point at the datasource you configured? Maybe you are using a different or misspelled name than what you thought you had!
1. Did you forget to set a valid value for apiVersion field in datasources.json that matches the version of ES you are running?
1. Maybe the version of ES you are using isn't supported by the client that this project uses. Try removing the elasticsearch sub-dependency from folder and then install the latest client:
1. cd
1. then remove the folder
1. unix/mac quickie: rm -rf elasticsearch
1.
1. to "academically" prove to yourself that this will work with the new install:
1. on unix/mac you can quickly dump the supported versions to your terminal with:
2. on other platforms, look into the and search for the supported_es_branches json block.
1. go back to yourApp's root directory
1. unix/mac quickie: cd
1. And test that you can now use the connector without any issues!
1. These changes can easily get washed away for several reasons. So for a more permanent fix that adds the version you want to work on into a release of this connector, please look into Contributing.Testing
1. You can edit
to point at your ES instance and then run npm test
1. If you don't have an ES instance and want to leverage docker based ES instances then:
1. If you want to run all tests across all versions in one go, then:
1. Run
1. Then run
1. When you're finished and want to tear down the docker instances, run:
1. You can test a specific version of elasticsearch if you want
1. elasticsearch version 1.x
1. Run
1. Then run
1. To run tests with additional logging, use:
1.
1.
1. Troubleshoot test with node-inspector if the level of details is still not enough:
1.
1.
1.
1. When you're finished and want to tear down the docker instances, run:
1. elasticsearch version 2.x
1. Run
1. Then run
1. To run tests with additional logging, use:
1.
1.
1. Troubleshoot test with node-inspector if the level of details is still not enough:
1.
1.
1.
1. When you're finished and want to tear down the docker instances, run:
1. elasticsearch version 5.x
1. Run
1. Then run
1. To run tests with additional logging, use:
1.
1.
1. Troubleshoot test with node-inspector if the level of details is still not enough:
1.
1.
1.
1. When you're finished and want to tear down the docker instances, run: Contributing
1. Feel free to contribute via PR or open an issue for discussion or jump into the gitter chat room if you have ideas.
1. I recommend that project contributors who are part of the team:
1. should merge
into develop ... if they are behind, before starting the feature branch
1. should create feature branches from the develop branch
1. should merge feature into develop then create a release branch to:
1. update the changelog
1. close related issues and mention release version
1. update the readme
1. fix any bugs from final testing
1. commit locally and run npm-release x.x.x -m "
1. merge into both master and develop
1. push and develop to GitHub
1. For those who use forks:
1. please submit your PR against the develop branch, if possible
1. if you must submit your PR against the master branch ... I understand and I can't stop you. I only hope that there is a good reason like develop not being up-to-date with master for the work you want to build upon.
1. npm-release may be used to publish. Pubilshing to NPM should happen from the master branch. It should ideally only happen when there is something release worthy. There's no point in publishing just because of changes to test or examples folder or any other such entities that aren't part of the "published module" (refer to .npmignore) to begin with.FAQs
1. How do we enable or disable the logs coming from the underlying elasticsearch client? There may be a need to debug/troubleshoot at times.
1. Use the
"log": "trace" field in your datasources file or omit it. You can refer to the detailed docs here and here
1. How do we enable or disable the logs coming from this connector?
1. By default if you do not set the following env variable, they are disabled: DEBUG=loopback:connector:elasticsearch
1. For example, try running tests with and without it, to see the difference:
1. with:
1. without:
1. What are the tests about? Can you provide a brief overview?
1. Tests are prefixed with or 02 etc. in order to run them in that order by leveraging default alphabetical sorting.
1. The 02.basic-querying.test.js file uses two models to test various CRUD operations that any connector must provide, like find(), findById(), findByIds(), updateAttributes() etc.
1. the two models are User and Customer
2. their ES mappings are laid out in
3. their loopback definitions can be found in the first block that performs setup in 02.basic-querying.test.js file ... these are the equivalent of a MyModel.json in your real loopback app.
1. naturally, this is also where we define which property serves as the id for the model and if its generated or not
1. How do we get elasticserch to take over ID generation?
1. An automatically generated id-like field that is maintained by ES is _id. Without some sort of es-field-level-scripting-on-index (if that is possible at all) ... I am not sure how we could ask elasticsearch to take over auto-generating an id-like value for any arbitrary field! So the connector is setup such that adding id: {type: String, generated: true, id: true} will tell it to use _id as the actual field backing the id ... you can keep using the doing model.id abstraction and in the background _id values are mapped to it.
1. Will this work for any field marked as with generated: true and id: true?
1. No! The connector isn't coded that way right now ... while it is an interesting idea to couple any such field with ES's _id field inside this connector ... I am not sure if this is the right thing to do. If you had objectId: {type: String, generated: true, id: true} then you won't find a real objectId field in your ES documents. Would that be ok? Wouldn't that confuse developers who want to write custom queries and run 3rd party app against their ES instance? Don't use objectId, use _id` would have to be common knowledge. Is that ok?