ddb-simple-migrate

change every item in a dynamodb table

installation

``
yarn
`

The integration tests use docker-compose to manage dynamo tables,
install it here. If
you would like to run the integration tests, run this first:

`
docker-compose build
`

testing

unit tests:

`
yarn test
`

integration tests that serve as examples:

`
docker-compose run test
`

usage

important! It's critical that before migrating a table the table is placed
in On-Demand billing mode (docs on this matter are here).
Trying to balance provisioned capacity and throughput for a table in Provisioned
billing mode during a migration like this is terrible and I don't recommend it.

`
const { migrate } = require('ddb-simple-migrate')
import { migrate } from 'ddb-simple-migrate'
`

Basic usage is covered by the examples in test/examples.ts,
covering one or multiple tables, stream or batch mode, and forcing a migration
on a provisioned table.

Most of the time, this library will be used in a node script. The script in
test/exampleScript.js is a good example of what
will generally happen in those scripts.

$3

* TableName the name of the dynamo table
* filterCb callback to filter out unneeded items. return true to migrate item
* cb async callback to change item before writing back to table. is passed the Item,
the counts object, and the batchLog function. not called in batch mode. return the changed item.
* batchCb async callback to change a batch of items. is passed the Items array, the
counts object, the batchLog function, the batchWrite function, and the dlq array. only called in
batch mode. does not return, is responsible for writing changed Items to the table.
* scanDelay ms to wait between scan batches, defaults to 0.
* writeDelay ms to wait between write batches, defaults to 0.
* mode either "batch" or "stream" (default). "stream" calls "cb" for each item in
the table, while "batch" calls "batchCb" for each scan batch, and expects that "batchWrite" is
explicitly called.
* dynamoOptions options that are passed to the dynamo client, consists of:
* region the AWS region. defaults to 'us-east-1'
* endpoint for dynamo tables. if not provided, defaults to the AWS default endpoint for "region"
* accessKeyId the AWS access key id, part of AWS credentials
* secretAccessKey the AWS secret access key, part of AWS credentials
* customCounts only valid in "batch" mode. initializes each string provided in
"counts", for keeping track of different values. Prints them at the end.
* saveDlq defaults to true. saves dlq to a json file in the current directory,
including table name, batch requests, and dynamo error. the dlq is also returned from the operation.
* quiet defaults to false. when true, silences all log output.
* force defaults to false. when true, allows migration on provisioned-mode table.
* asScript defaults to true. most of the time, this will be run as part of a node
script, and needs to listen for Ctrl-C to quit.

tips

This library tries to migrate your dynamo table at a single-partition scale, to
keep things simple (Partition splits occur at ~1000 WCU or ~3000 RCU, for more
information on partitions see the docs).

It scans only batches of 25 items at a time to keep read throughput down. If
for whatever reason the read throughput during the migration gets too close to
3000 RCU, you should stop the migration and introduce a scanDelay, generally
between 0-50ms. This is pretty unlikely!

If for whatever reason the write throughput during the migration gets too close
to 1000 WCU (which is more likely), you should stop the migration and start it
again with a writeDelay, generally between 0-50 ms. Try starting the migration
over with a delay, and experiment with higher delay times until you find a time
that keeps the throughput low.

$3

1. Filter out as many items as possible, migrating only the items that absolutely
need it.
2. Introduce a writeDelay` of a few ms, upping that value until the WCU goes down.
3. If writing to two or more tables, delay the writes to the second table for a
few ms.