node-mongotools


This project provides 2 wrappers for :
- mongodump,
- mongorestore.

This project also include dropbox integration feature to
- dump and upload onto dropbox,
- restore from a dropbox hosted mongo backup.

There is an autonomous feature called rotation that provide a backup file rotation mechanism
- remove N oldest deprecated backups.

This readme contains some samples. Other samples are under examples/.

You may notice that mongodump and mongorestore binaries are part of database-tools (follow standard installation for common OS). These binaries are mandatory for node-mongotools nodejs project as we are talking here of "wrapper" : a more convenient way to launch command + arguments from NodeJS code).

From a GitHub Actions context, you can install them via action-mongo-tools. There is an example in this project in main workflow that execute tests.

Command line usage

You should decide to install it globally without getting source or by cloning repository.

$3

``bash
npm install -g node-mongotools@latest
`
When using a global install node-mongotools command must be directly available (ie. you can replace node mt by node-mongotools in examples).

You could revert thins by using
`bash
npm uninstall -g node-mongotools
`

$3

`bash
git clone https://github.com/boly38/node-mongotools.git

install dependencies

npm install
`

$3

`bash
#~ setup environment variables
cp env/initEnv.template.sh env/initEnv.dontpush.sh

you must update env/initEnv.dontpush.sh

`

$3

source your options
`bash
. ./env/initEnv.dontpush.sh
`

$3

`bash

create a mongo dump

node mt dump

or if you rely on a global install

node-mongotools dump

apply the same logic for next examples

create a encrypted mongo dump

node mt dumpz

list backups

node mt list

restore a mongo local dump

please note that mt restore use following options : dropBeforeRestore: true, deleteDumpAfterRestore: true

node mt restore backup/myDatabase__2020-11-08_150102.gz

rotate backup files

node mt rotation

Helper : show current options values

node mt options
`

$3

You could play with env default options plus in-line command extra options
`bash

create dump of a given 'shippingprices' collection, provide a target backup name as '2023_01_shippingPrices.gz', and show mongodump command

MT_COLLECTION=shippingprices MT_FILENAME=2023_01_shippingPrices.gz MT_SHOW_COMMAND=true node mt dump

show backup in list

node mt list

using mongo: drop a given collection

mongo myDb --eval "db.shippingprices.drop()"

restore collection

MSYS_NO_PATHCONV=1 MT_COLLECTION=shippingprices MT_SHOW_COMMAND=true node mt restore /backup/2023_01_shippingprices.gz

Note that collection option will produce wildcard in nsInclude arg '--nsInclude myDb.*'

`

$3

You have a dropbox access token in your preferences, (cf. "Mongo tools options")

`

create a mongo dump is the same command

node mt dump

restore a mongo dropbox dump

node mt restore /backup/myDatabase__2020-11-08_150102.gz

git bash for windows users having absolute path issue could use the following command

unalias node
MSYS_NO_PATHCONV=1 node mt restore /backup/myDatabase__2020-11-08_150102.gz

rotate local and dropbox backup files

node mt rotation
`

Library use

$3

You have to import as dependency
`
npm install node-mongotools
`

$3

`
import { MongoTools, MTOptions, MTCommand } from "node-mongotools";

var mongoTools = new MongoTools();
const mtOptions = {
db: 'myDb',
port: 17017,
path: '/opt/myapp/backups',
dropboxToken: process.env.MYAPP_DROPBOX_SECRET_TOKEN
};
`

This project is compatible with ES Module projects that rely on import.
For now it's not compatible with CommonJS (require). But contribution are welcome.

$3

`
var promiseResult = mongoTools.list(mtOptions);
`

$3

`
var promiseResult = mongoTools.mongodump(mtOptions);
`

$3

`
var promiseResult = mongoTools.mongorestore(mtOptions);
`

$3

`
var promiseResult = mongoTools.rotation(mtOptions);
`

Mongo tools options

Each mongotools feature rely on Mongo tools options (aka. MTOption).

Options precedence is the following:
- take options attribute if set,
- else take related environment variable if any,
- else take default value if any,
- else if it's a mandatory option, throw an error.

TIP: you could also show current options by doing:
`
console.log(new MTOptions());
`

$3

These options are used by dump and restore.

Either uri or host/port/db:

| option | env | required | default value | description |
|--------|--------------|----------|---------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------|
| uri | MT_MONGO_URI | true | (none) | mongodump uri, example mongodb+srv://granted-user:MySecretHere@cluster0.xzryx.mongodb.net/tMyDatababse. You could omit database name to dump all databases. |

or

| option | env | required | default value | description |
|------------|------------------|----------|---------------|-------------------------------------------------------------------------------------|
| db | MT_MONGO_DB | true | (none) | mongo database name. For dump only, you could set it to * to dump all databases |
| host | MT_MONGO_HOST | false | 127.0.0.1 | mongo database hostname |
| port | MT_MONGO_PORT | false | 27017 | mongo database port |
| username | MT_MONGO_USER | false | (none) | mongo database username |
| password | MT_MONGO_PWD | false | (none) | mongo database password |
| authDb | MT_MONGO_AUTH_DB | false | admin | mongo auth database |

#### ssl options

Optional ssl related options

| option | env | required | default value | description |
|----------------------|-------------------------------|----------|----------------|-------------------------------------------------------------------|
| ssl | MT_MONGO_SSL | false | (none) | if "1" then add --ssl option |
| sslCAFile | MT_MONGO_SSL_CA_FILE | false | (none) | .pem file containing the root certificate chain |
| sslPEMKeyFile | MT_MONGO_SSL_PEM_KEY_FILE | false | (none) | .pem file containing the certificate and key |
| sslPEMKeyPassword | MT_MONGO_SSL_PEM_KEY_PASSWORD | false | (none) | password to decrypt the sslPEMKeyFile, if necessary |
| sslCRLFile | MT_MONGO_SSL_CRL_FILE | false | (none) | pem file containing the certificate revocation list |
| sslFIPSMode | MT_MONGO_SSL_FIPS | false | (none) | if "1" then use FIPS mode of the installed openssl library |
| tlsInsecure | MT_MONGO_TLS_INSECURE | false | (none) | if "1" then bypass the validation for server's certificate chain |

$3

| option | env | required | default value | description |
|--------------------------|------------------------|----------|-------------------------|------------------------------------------------------------------------------|
| path | MT_PATH | false | backup | dump target directory, created if it doesn't exist |
| dumpCmd | MT_MONGODUMP | false | mongodump | mongodump binary |
| fileName | MT_FILENAME | false | | dump target filename |
| encrypt | | false | false | encrypt the dump using secret |
| secret | MT_SECRET | false | null | secret to use if encrypt is enabled (aes-256-ctr require 32 byte length key) |
| encryptSuffix | | false | .enc | encrypt file suffix |
| includeCollections | | false | (none) | Deprecated - please use collection |
| collection | MT_COLLECTION | false | (none) | Collection to include, if not specified all collections are included |
| excludeCollections | MT_EXCLUDE_COLLECTIONS | false | (none) | Collections to exclude, if not specified all collections are included |
| numParallelCollections | | false | 4 | Number of collections mongodump should export in parallel. |
| viewsAsCollections | | false | false | When specified, mongodump exports read-only views as collections. |

Simple example:
`
import { MongoTools, MTOptions, MTCommand } from "node-mongotools";
var mongoTools = new MongoTools();

mongoTools.mongodump({
db:'myDatabase',
path:'backup',
username:'root', password:'mypass', authDb:'admin'
})
.then((success) => console.info("success", success) )
.catch((err) => console.error("error", err) );
`

$3

| option | env | required | default value | description |
|--------------------------|------------------|----------|----------------|----------------------------------------------------------------|
| dbFrom | MT_MONGO_DB_FROM | false | (none) | name of the source db (if db is not specified)
| dbTo | MT_MONGO_DB_TO | false | (none) | name of the target db (if db is not specified) |
| dumpFile | MT_DUMP_FILE | true | (none) | dump file to restore |
| restoreCmd | MT_MONGORESTORE | false | mongorestore | mongorestore binary |
| dropBeforeRestore | | false | false | set it to true to append --drop option |
| deleteDumpAfterRestore | | false | false | set it to true to remove restored backup file |
| decrypt | | false | false | decrypt the dump using secret. Activated if suffix is detected |
| secret | MT_SECRET | false | null | secret to use if decrypt is enabled |

Simple example:
`
import { MongoTools, MTOptions, MTCommand } from "node-mongotools";
var mongoTools = new MongoTools();

mongoTools.mongorestore({
dumpFile:'backup/myDatabase__2020-11-8_160011.gz',
username:'root', password:'mypass', authDb:'admin'
})
.then((success) => {
console.info("success", success.message);
if (success.stderr) {
console.info("stderr:\n", success.stderr);// mongorestore binary write details on stderr
}
})
.catch((err) => console.error("error", err) );
`

$3

You could create a dropbox app to get a token : cf. https://www.dropbox.com/developers/apps/ "Generated access token"

| option | env | required | default value | description |
|-----------------------|----------------------------|-----------|-----------------|----------------------------------------------------------------------|
| dropboxLocalPath | MT_DROPBOX_LOCAL_PATH | false | "dropbox" | local directory to receive dropbox dump |
| dropboxAppKey | MT_DROPBOX_APP_KEY | false | (none) | (refresh token based) dropbox feature application key. (*1) |
| dropboxAppSecret | MT_DROPBOX_APP_SECRET | false | (none) | (refresh token based) dropbox feature application secret. (*1) |
| dropboxRefreshToken | MT_DROPBOX_REFRESH_TOKEN | false | (none) | (refresh token based) dropbox feature application refreshToken. (*1) |
| dropboxToken (2) | MT_DROPBOX_TOKEN | false | (none) | DEPRECATED - activate dropbox feature if present. (2) |

(*1) dropboxAppKey, dropboxAppSecret and dropboxRefreshToken (long-lived offline refresh token) are required to activate optional dropbox feature. You have some help about how to get them in dropbox-refresh-token dedicated helper repository.
(*2) dropboxToken option is DEPRECATED : This was legacy old-long-lived access-token - this kind of token are no more available from dropbox developers portal (src). Please switch to dropboxAppKey, dropboxAppSecret, and dropboxRefreshToken instead. For now this is still supported.

When a token is set,
- the list feature will list the / + path dropbox directory
- the mongodump feature will upload the dump onto / + path dropbox directory (in addition to spawn it locally),
- the mongorestore feature will use dumpFile as dropbox dump location
and retrieve it into dropboxLocalPath before doing the mongorestore action.

$3

A safe time windows is defined by :
* now - rotationWindowsDays day(s) ===> now
where backups can't be removed.

Backup out of safe time windows are called deprecated backup.

- rotationMinCount: minimum deprecated backups to keep,
- rotationCleanCount: number of (oldest) deprecated backups to delete.

| option | env | required | default value | description |
|-----------------------|--------------------------|----------|---------------|--------------------------------------------------------|
| rotationDryMode | MT_ROTATION_DRY_MODE | false | false | dont do delete actions, just print it |
| rotationWindowsDays | MT_ROTATION_WINDOWS_DAYS | true | 15 | safe time windows in days since now |
| rotationMinCount | MT_ROTATION_MIN_COUNT | true | 2 | minimum deprecated backups to keep. |
| rotationCleanCount | MT_ROTATION_CLEAN_COUNT | true | 10 | number of (oldest first) deprecated backups to delete. |

Simple example:
`
MT_ROTATION_CLEAN_COUNT=2 \
MT_ROTATION_DRY_MODE=true \
MT_ROTATION_WINDOWS_DAYS=3 \ node mt rotation
`
Example details: if there is a backup that is more than 3 days old, keep 2 newer ones and delete the 10 oldest.

Dropbox limits:
- rotation feature will not apply if dropbox backup target directory content contains more than 2000 files.

How to contribute

You're not a dev ? just submit an issue (bug, improvements, questions). Or else:
* Clone
* Install deps
* Then mocha tests
`
git clone https://github.com/boly38/node-mongotools.git
npm install
npm run test
`
* you could also fork, feature branch, then submit a pull request.

$3

| badge | name | description |
|--------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------|:----------------------------------------------------------|
| !CI/CD | Github actions | Continuous tests. |
|  | Github actions | Continuous vulnerability audit. |
|  | Houndci | JavaScript automated review (configured by .hound.yml`) |

$3

!Repobeats

provided by Repobeats