Dynamo Data Transform is an easy to use data transformation tool for DynamoDB.

It allows performing powerful data transformations using simple Javascript commands, without the risk of breaking your database.
Available as a Serverless plugin, npm package and even as an interactive CLI, Dynamo Data Transform saves you time and keeps you safe with features like dry-running a data transformation and even rolling back your last trasnformation if needed.

Features

- Seemless data transformations management.
- Support for multiple stages.
- History of executed data transformations.
- Dry run option for each command (by suppling --dry flag, the data will be printed instead of stored).
- Safe & Secure preparation data
- Store preparation data in a private s3 bucket. Prepare data for your data transformation

- Quick Start
- Serverless plugin
- Standalone npm package
- Interactive CLI
- Creating your first data transformation
- Usage and command-line options
- What happens behind the scenes
- Examples
- The data transformation process

Quick Start

$3

- Install
``

bash
npm install dynamo-data-transform --save-dev


- Add the tool to your serverless.yml
Run:

bash
npx serverless plugin install -n dynamo-data-transform


Or add manually to your serverless.yml:

YML
plugins:
  - dynamo-data-transform


- Run

bash
sls dynamodt --help


$3

- Install the tool

bash
npm install -g dynamo-data-transform -s


- Run the tool

bash
dynamodt help


Or with the shortcut

bash
ddt help


$3

After installing the npm package, run:

bash
dynamodt -i


!cli gif
Creating your first data transformation

1. Intialize data-transformations folder
Serverless (the plugin reads the table names from the serverless.yml file):

bash
sls dynamodt init --stage


Standalone:

bash
ddt init --tableNames

Open the generated data transformation file 'v1_script-name.js' file and implement the following functions: - transformUp: Executed when runningdynamodt up- transformDown: Executed when runningdynamodt down -t


  - prepare (optional): Executed when running

dynamodt prepare -t

--tNumber

The function parameters: - ddb: The DynamoDB Document client object see DynamoDB Client - isDryRun: Boolean indicating if --dry run supplied. You can use it to print/log the data instead of storing it. - preparationData: if you stored the preparation data usingdynamodt prepare, you can use it here.

2. Run the data transformation`bash dynamodt up`

`Data Transformation Script Format`


  Make sure your script name contains the transformation number, for example: v1_transformation_script

js
const { utils } = require('dynamo-data-transform')
const TABLE_NAME = 'UsersExample'
const transformUp = async ({ ddb, isDryRun, preparationData }) => {
  // your code here... 
  // return { transformed: 50 } // return the number of transformed items
}
const transformDown = async ({ ddb, isDryRun, preparationData }) => {
  // your code here...
  // return { transformed: 50 } // return the number of transformed items
}
const prepare = async ({ ddb, isDryRun }) => {
  // your code here...
  // return { transformed: 50 } // return the number of transformed items
}

module.exports = { transformUp, transformDown, prepare, // optional transformationNumber: 1, }`

`Usage and command-line options`

List available commands: Serverless plugin:`bash sls dynamodt --help`Standalone npm package:`bash dynamodt help`

To list all of the options for a specific command run: Serverless plugin:`bash sls dynamodt --help`

Standalone npm package:`bash dynamodt --help`

`What happens behind the scenes`


- When a data transformation runs for the first time, a record in your table is created. This record is for tracking the executed transformations on a specific table.
Examples

Examples of data transformation code

$3

`js // Seed users data transformation const { utils } = require('dynamo-data-transform'); const { USERS_DATA } = require('../../usersData');

const TABLE_NAME = 'UsersExample';

/** * @param {DynamoDBDocumentClient} ddb - dynamo db document client https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-dynamodb * @param {boolean} isDryRun - true if this is a dry run */ const transformUp = async ({ ddb, isDryRun }) => { return utils.insertItems(ddb, TABLE_NAME, USERS_DATA, isDryRun); };

const transformDown = async ({ ddb, isDryRun }) => { return utils.deleteItems(ddb, TABLE_NAME, USERS_DATA, isDryRun); };

module.exports = { transformUp, transformDown, transformationNumber: 1, };`

`$3`

js
// Adding a "randomNumber" field to each item
const { utils } = require('dynamo-data-transform');
const TABLE_NAME = 'UsersExample';
const transformUp = async ({ ddb, isDryRun }) => {
  const addRandomNumberField = (item) => {
    const updatedItem = { ...item, randomNumber: Math.random() };
    return updatedItem;
  };
  return utils.transformItems(ddb, TABLE_NAME, addRandomNumberField, isDryRun);
};
const transformDown = async ({ ddb, isDryRun }) => {
  const removeRandomNumberField = (item) => {
    const { randomNumber, ...oldItem } = item;
    return oldItem;
  };
  return utils.transformItems(ddb, TABLE_NAME, removeRandomNumberField, isDryRun);
};

module.exports = { transformUp, transformDown, transformationNumber: 2, };``

For more examples of data transformation code, see the examples folder in the repository.

Read how to automate data-transformation process here:
https://www.infoq.com/articles/dynamoDB-data-transformation-safety/

dynamo-data-transform - npm explorer

dynamo-data-transform

v0.1.11

DynamoDB Data Transformation Tool

dynamodb dynamodb data dynamo dynamo data dynamodt transform transformation dynamo transformations dynamo transformer dynamo migration

0/weekUpdated 12 months agoMITUnpacked: 52.7 KB

Published by Guy Braunstain

npm install dynamo-data-transform

Repository Homepage npm

!ddt_graphic1x_tl

Dynamo Data Transform is an easy to use data transformation tool for DynamoDB.

Features

Quick Start

$3

- Install
``

bash
npm install dynamo-data-transform --save-dev


- Add the tool to your serverless.yml
Run:

bash
npx serverless plugin install -n dynamo-data-transform


Or add manually to your serverless.yml:

YML
plugins:
  - dynamo-data-transform


- Run

bash
sls dynamodt --help


$3

- Install the tool

bash
npm install -g dynamo-data-transform -s


- Run the tool

bash
dynamodt help


Or with the shortcut

bash
ddt help


$3

After installing the npm package, run:

bash
dynamodt -i


!cli gif
Creating your first data transformation

1. Intialize data-transformations folder
Serverless (the plugin reads the table names from the serverless.yml file):

bash
sls dynamodt init --stage


Standalone:

bash
ddt init --tableNames


  - prepare (optional): Executed when running

dynamodt prepare -t

--tNumber

2. Run the data transformation`bash dynamodt up`

`Data Transformation Script Format`


  Make sure your script name contains the transformation number, for example: v1_transformation_script

js
const { utils } = require('dynamo-data-transform')
const TABLE_NAME = 'UsersExample'
const transformUp = async ({ ddb, isDryRun, preparationData }) => {
  // your code here... 
  // return { transformed: 50 } // return the number of transformed items
}
const transformDown = async ({ ddb, isDryRun, preparationData }) => {
  // your code here...
  // return { transformed: 50 } // return the number of transformed items
}
const prepare = async ({ ddb, isDryRun }) => {
  // your code here...
  // return { transformed: 50 } // return the number of transformed items
}

module.exports = { transformUp, transformDown, prepare, // optional transformationNumber: 1, }`

`Usage and command-line options`

List available commands: Serverless plugin:`bash sls dynamodt --help`Standalone npm package:`bash dynamodt help`

To list all of the options for a specific command run: Serverless plugin:`bash sls dynamodt --help`

Standalone npm package:`bash dynamodt --help`

`What happens behind the scenes`


- When a data transformation runs for the first time, a record in your table is created. This record is for tracking the executed transformations on a specific table.
Examples

Examples of data transformation code

$3

`js // Seed users data transformation const { utils } = require('dynamo-data-transform'); const { USERS_DATA } = require('../../usersData');

const TABLE_NAME = 'UsersExample';

const transformDown = async ({ ddb, isDryRun }) => { return utils.deleteItems(ddb, TABLE_NAME, USERS_DATA, isDryRun); };

module.exports = { transformUp, transformDown, transformationNumber: 1, };`

`$3`

js
// Adding a "randomNumber" field to each item
const { utils } = require('dynamo-data-transform');
const TABLE_NAME = 'UsersExample';
const transformUp = async ({ ddb, isDryRun }) => {
  const addRandomNumberField = (item) => {
    const updatedItem = { ...item, randomNumber: Math.random() };
    return updatedItem;
  };
  return utils.transformItems(ddb, TABLE_NAME, addRandomNumberField, isDryRun);
};
const transformDown = async ({ ddb, isDryRun }) => {
  const removeRandomNumberField = (item) => {
    const { randomNumber, ...oldItem } = item;
    return oldItem;
  };
  return utils.transformItems(ddb, TABLE_NAME, removeRandomNumberField, isDryRun);
};

module.exports = { transformUp, transformDown, transformationNumber: 2, };``

For more examples of data transformation code, see the examples folder in the repository.

Read how to automate data-transformation process here:
https://www.infoq.com/articles/dynamoDB-data-transformation-safety/

dynamo-data-transform

Table of contents

Quick Start

$3

$3

$3

Creating your first data transformation

`Data Transformation Script Format`

`Usage and command-line options`

`What happens behind the scenes`

Examples

$3

`$3`

dynamo-data-transform

Table of contents

Quick Start

$3

$3

$3

Creating your first data transformation

`Data Transformation Script Format`

`Usage and command-line options`

`What happens behind the scenes`

Examples

$3

`$3`