MongoDB bulk data migration for NodeJs

_Mongodb schema migration utility_

About

MongoBulkDataMigration is a 1-liner MongoDb migration utility for your 1-shot schema migrations.
It is fast and resume-able, handles rollback automatically.

$3

- 🚀 fast: built over Mongo Bulk operations to less stress the Mongo socket. Bulk is adapted to massive update as it groups write operations.

- 🏔️ scalable: working on 1 million / 1 billion documents is not an issue.
Backup is done in a dedicated collection. Bulks can be throttled.

- 🔒 safe and easy: Backup and restore is done progressively by bulk.
Whatever is fetched and projected is what is saved as backup. Script built over MBDM is explicit and focused.

- 🔄 resume-able: a script can be resumed if it crashes _(disconnection)_.
By design, it's safe to run it twice in update or rollback mode.

- 💝 minimal: A "unified" extension script is handle by the platform to write less code.
Also, the test util doRollbackAndAssertForInitialState will simplify test writing for Jest and Chai frameworks.

- 🏔️ Aggregate-ready: You can fetch using a simple query or an aggregation pipeline

$3

MBDM fits most needs, but not all needs.
Prerequisite:

- MBDM can't insert new documents
_(you can only update or delete documents)_
- MBDM can only update 1 collection at a time
- MBDM can only update a document once for a same migration ID _(unless you clean backup manually in-between)_
- queries should ideally not match an already updated document _(for resume-ability)_

$3

Biggest features making MBDM powerful:

- MBDM can automatically rollback to the exact same state projected values:
- for $set operations
- for $unset operations
- for document deletion operations
- ... for anything more advanced, you can still rollback using a custom rollback function
- MBDM accepts basic Mongo queries, or aggregation pipelines,
- MBDM can compute a document update operation in an async callback, with a defined maxConcurrentUpdateCalls, allowing you to call a complex asynchronous code.
- MBDM makes chore script writing fast 🚀

$3

MBDM

---

📘 Getting Started

$3

Using npm

``bash
npm install --save-dev @360-l/mongo-bulk-data-migration
`

$3

MBDM expects a connection to your database, a id (string), and a target collection to process.

`ts
import { MongoBulkDataMigration } from "@360-l/mongo-bulk-data-migration";

// connect to db
// handle script input

const migration = new MongoBulkDataMigration({
db: mongoClient, // MongoClient established instance
collection: "myCol", // Collection where there will be an update
id: "uid_migration", // Required to rollback storage
... // see below for query/update
});

// Do the update
await migration.update();

// Or revert updated properties (only) in all updated documents
await migration.rollback();
`

MBDM does not provide CLI tool. You can consider to create some structural abstract code to run it like.
As an example here:

`bash
node ./myscript.ts --action [update|rollback]
`

$3

This migration will set { total: 0 } for every doc not having 0.

_Note: rollback will automatically unset back total._

`ts
new MongoBulkDataMigration({
db,
id: 'scores_set_total',
collectionName: 'scores',
projection: {},
query: { total: { $exists: false } },
update: { $set: { total: 0 } },
});
`

⚙️ Options

`ts
new MongoBulkDataMigration({ ..., options: { ... } })
`

- maxBulkSize (default: 1000): Batch size of of updates to execute. 1000 is a good number. If your updated are huge, consider decreasing it to avoid running high memory. If documents are tiny, 10k might slightly improve performances on huge databases.
- dontCount (default: false): will skip initial filter count() or aggregate $count. Logs won't print the total progression if disabled.
- projectionBackupFilter (default: none): filter properties to save and auto-rollback. Necessary if your update needs virtual fields.
- bypassRollbackValidation and bypassUpdateValidation (default: false): Will set validationLevel to "off" then to "moderate".
- throttle (default: 0): amount of time to sleep between a bulk update. Use this to decrease database stress.
- continueOnBulkWriteError (default: false): will continue the migration on the error in a bulk.

📕 Advanced usages

MBDM has support for async updates query, custom rollback, aggregation pipelines...

$3

This migration will sum 2 projected fields scoreA and scoreB for all documents (no filter).

_Note: rollback will automatically set back scoreA and scoreB and reset total._

`ts
new MongoBulkDataMigration({
db,
id: 'scores_total_new_field',
collectionName: 'scores',
projection: { scoreA: 1, scoreB: 1 },
query: {},
update: (doc) => {
$set: {
total: doc.scoreA + doc.scoreB;
}
},
});
`

$3

This migration will delete doc having negative total.

_Note: rollback will automatically restore full document._

`ts
import { MongoBulkDataMigration, DELETE_OPERATION } from "@360-l/mongo-bulk-data-migration";
...
new MongoBulkDataMigration({
db,
id: "delete_negative_total",
collectionName: "scores",
projection: {}, // Everything needs to be projected
query: { total: { $lt: 0 } },
update: DELETE_OPERATION,
});
`

$3

query expects a query (object), or an aggregation pipeline (array).
Note: projection won't do anything.

_Example: update totalGames of a corresponding table_

`ts
import { MongoBulkDataMigration } from "@360-l/mongo-bulk-data-migration";
...
new MongoBulkDataMigration({
db,
id: "delete_negative_total",
collectionName: "scores",
projection: { "games.value": 1, totalGames: 1 },
query: [
{ $lookup: { as: "games", ... } },
{ $match: { "games.value": "xxx" } },
],
update: (doc) => ({
$set: {
totalGames: doc.games.value
}
}),
options: {
// Necessary to save only totalGames, and not auto-restore uncexisting games field
projectionBackupFilter: ["totalGames"]
}
});
`

$3

The collection will be renamed to the backup collection.
When you rollback, the collection will simply be renamed back

`ts
import { MongoBulkDataMigration, DELETE_COLLECTION } from "@360-l/mongo-bulk-data-migration";
...
const migration status = new MongoBulkDataMigration({
db,
id: "delete_collection_scores",
collectionName: "scores",
operation: DELETE_COLLECTION,
});
console.log(status); // { ok: 1 }
`

🧩 Ease testing

MBDM provides to simplify your tests files. Just writes what you expect, and the util will ensure the database is back to initia state.
It supports for Jest and Chai testing library (expect).

_Note: expectation is done on sorted docs by _id_

`ts
it('should not modify pages with existing translations', async () => {
const docs = [{ _id: new ObjectId(), value: 'invalid' }];
await collection.insertMultiple(docs);

await dataMigration.update();

// Test what you expect
const updatedDocs = await collection.find({}).toArray();
expect(updatedDocs).toEqual([{ _id: new ObjectId(), value: 'valid' }]);

// Test rollback will work
await doRollbackAndAssertForInitialState(dataMigration, docs, { expect });
});
``