change every item in a dynamodb table
npm install ddb-simple-migrate```
yarn
The integration tests use docker-compose to manage dynamo tables,
install it here. If
you would like to run the integration tests, run this first:
``
docker-compose build
unit tests:
``
yarn test
integration tests that serve as examples:
``
docker-compose run test
important! It's critical that before migrating a table the table is placed
in On-Demand billing mode (docs on this matter are here).
Trying to balance provisioned capacity and throughput for a table in Provisioned
billing mode during a migration like this is terrible and I don't recommend it.
``
const { migrate } = require('ddb-simple-migrate')
import { migrate } from 'ddb-simple-migrate'
Basic usage is covered by the examples in test/examples.ts,
covering one or multiple tables, stream or batch mode, and forcing a migration
on a provisioned table.
Most of the time, this library will be used in a node script. The script in
test/exampleScript.js is a good example of what
will generally happen in those scripts.
* TableName the name of the dynamo tablefilterCb
* callback to filter out unneeded items. return true to migrate itemcb
* async callback to change item before writing back to table. is passed the Item,batchCb
the counts object, and the batchLog function. not called in batch mode. return the changed item.
* async callback to change a batch of items. is passed the Items array, thescanDelay
counts object, the batchLog function, the batchWrite function, and the dlq array. only called in
batch mode. does not return, is responsible for writing changed Items to the table.
* ms to wait between scan batches, defaults to 0.writeDelay
* ms to wait between write batches, defaults to 0.mode
* either "batch" or "stream" (default). "stream" calls "cb" for each item indynamoOptions
the table, while "batch" calls "batchCb" for each scan batch, and expects that "batchWrite" is
explicitly called.
* options that are passed to the dynamo client, consists of:region
* the AWS region. defaults to 'us-east-1'endpoint
* for dynamo tables. if not provided, defaults to the AWS default endpoint for "region"accessKeyId
* the AWS access key id, part of AWS credentialssecretAccessKey
* the AWS secret access key, part of AWS credentialscustomCounts
* only valid in "batch" mode. initializes each string provided insaveDlq
"counts", for keeping track of different values. Prints them at the end.
* defaults to true. saves dlq to a json file in the current directory,quiet
including table name, batch requests, and dynamo error. the dlq is also returned from the operation.
* defaults to false. when true, silences all log output.force
* defaults to false. when true, allows migration on provisioned-mode table.asScript
* defaults to true. most of the time, this will be run as part of a node
script, and needs to listen for Ctrl-C to quit.
This library tries to migrate your dynamo table at a single-partition scale, to
keep things simple (Partition splits occur at ~1000 WCU or ~3000 RCU, for more
information on partitions see the docs).
It scans only batches of 25 items at a time to keep read throughput down. If
for whatever reason the read throughput during the migration gets too close to
3000 RCU, you should stop the migration and introduce a scanDelay, generally
between 0-50ms. This is pretty unlikely!
If for whatever reason the write throughput during the migration gets too close
to 1000 WCU (which is more likely), you should stop the migration and start it
again with a writeDelay, generally between 0-50 ms. Try starting the migration
over with a delay, and experiment with higher delay times until you find a time
that keeps the throughput low.
1. Filter out as many items as possible, migrating only the items that absolutely
need it.
2. Introduce a writeDelay` of a few ms, upping that value until the WCU goes down.
3. If writing to two or more tables, delay the writes to the second table for a
few ms.