[dynamodb-replicator](https://github.com/mapbox/dynamodb-replicator) offers several different mechanisms to manage redundancy and recoverability on [DynamoDB](http://aws.amazon.com/documentation/dynamodb) tables.
dynamodb-replicator offers several different mechanisms to manage redundancy and recoverability on DynamoDB tables.
- A replicator function that processes events from a DynamoDB stream, replaying changes made to the primary table and onto a replica table. The function is designed to be run as an AWS Lambda function, optionally with deployment assistance from streambot. - An incremental backup function that processes events from a DynamoDB stream, replaying them as writes to individual objects on S3. The function is designed to be run as an AWS Lambda function, optionally with deployment assistance from streambot. - A consistency check script that scans the primary table and checks that each individual record in the replica table is up-to-date. The goal is to double-check that the replicator is performing as is should, and the two tables are completely consistent. - A table dump script that scans a single table, and writes the data to a file on S3, providing a snapshot of the table's state. - A snapshot script that scans an S3 folder where incremental backups have been made, and writes the aggregate to a file on S3, providing a snapshot of the backup's state.
Design
Managing table redundancy and backups involves many moving parts. Please read DESIGN.md for an in-depth explanation.
Utility scripts
dynamodb-replicator provides several CLI tools to help manage your DynamoDB table.
$3
Given two tables and an item's key, this script looks up the record in both tables and checks for consistency.
Given two tables and a set of options, performs a complete consistency check on the two, optionally repairing records in the replica table that differ from the primary.
Options: --repair perform actions to fix discrepancies in the replica table --segment segment identifier (0-based) --segments total number of segments --backfill only scan primary table and write to replica
Log information about discrepancies between the two tables
Usage: replicate-record - primary tableinfo: the primary table to replicate from, specified as
region/tablename - replica tableinfo: the replica table to replicate to, specified as region/tablename - recordkey: the key for the record specified as a JSON object
Copy the state of a record from the primary to the replica table
Options: --jobid assign a jobid to this backup --segment segment identifier (0-based) --segments total number of segments --metric cloudwatch metric namespace. Will provide dimension TableName = the name of the backed-up table.
Writes a backup file to s3://my-bucket/some-prefix//0
Prints each version of a record that is available in an incremental backup folder on S3.
` $ incremental-record-history --help
Usage: incremental-record-history - tableinfo: the table where the record lives, specified as
region/tablename - s3url: s3 folder where the incremental backups live. Table name will be appended - recordkey: the key for the record specified as a JSON object