Data Processing and Transformation Utility
npm install data-point   
> JavaScript Utility for collecting, processing and transforming data.
DataPoint helps you reason with and streamline your data processing layer. With it you can collect, process, and transform data from multiple sources, and deliver the output in a tailored format to your end consumer.
Prerequisites
Node v8 LTS or higher
Installing
``bash`
npm install --save data-point
- Getting Started
- Hello World Example
- Fetching remote services
- API
- create
- createReducer
- resolve
- transform
- addEntities
- addValue
- Accumulator
- Reducers
- function
- object
- entity
- entity-id
- list
- Reducer Helpers
- assign
- map
- filter
- find
- constant
- parallel
- withDefault
- Entities
- Instance Entity
- Registered Entity
- Entity Base API
- Entity type check
- Entity types
- Reducer
- Model
- Entry
- Request
- Hash
- Collection
- Control
- Schema
- Entity ComposeReducer
- Extending Entities
- Global Entity Options
- Middleware
- dataPoint.use
- Custom Entity Types
- Tracing DataPoint calls
- Integrations
- Patterns and Best Practices
- Contributing
- License
DataPoint provides the following mechanisms for transforming data:
- Reducers - these are the simplest transformations; think of them as DataPoint "primitives"
- Entities - these are more complex transformations that are defined using one or more reducers
- Middleware - middleware functions give the user more control when resolving entities; they're useful to implement caching and other metatasks
The following examples demonstrate some of these concepts. For detailed API documentation, you can jump into the DataPoint.create section and move from there.
Additionally, there is a Hello World YouTube tutorial that explains the basics of DataPoint.
Trivial example of transforming a given input with a function reducer.
`js
const DataPoint = require('data-point')
// create DataPoint instance
const dataPoint = DataPoint.create()
// function reducer that concatenates
// accumulator.value with 'World'
const reducer = (input) => {
return input + ' World'
}
// applies reducer to input
dataPoint
.resolve(reducer, 'Hello')
.then((output) => {
// 'Hello World'
console.log(output)
})
`
Example at: examples/hello-world.js
Based on an initial feed, fetch and aggregate results from multiple remote services.
Example
Using the amazing swapi.co service, the example below gets information about a planet and the residents of that planet.
`js
const DataPoint = require('data-point')
// create DataPoint instance
const dataPoint = DataPoint.create()
const {
Request,
Model,
Schema,
map
} = DataPoint
// schema to verify data input
const PlanetSchema = Schema('PlanetSchema', {
schema: {
type: 'object',
properties: {
planetId: {
$id: '/properties/planet',
type: 'integer'
}
}
}
})
// remote service request
const PlanetRequest = Request('Planet', {
// {value.planetId} injects the
// value from the accumulator
// creates: https://swapi.co/api/planets/1/
url: 'https://swapi.co/api/planets/{value.planetId}'
})
const ResidentRequest = Request('Resident', {
// check input is string
inputType: 'string',
url: '{value}'
})
// model entity to resolve a Planet
const ResidentModel = Model('Resident', {
inputType: 'string',
value: [
// hit request:Resident
ResidentRequest,
// extract data
{
name: '$name',
gender: '$gender',
birthYear: '$birth_year'
}
]
})
// model entity to resolve a Planet
const PlanetModel = Model('Planet', {
inputType: PlanetSchema,
value: [
// hit request:Planet data source
PlanetRequest,
// map result to an object reducer
{
// map name key
name: '$name',
population: '$population',
// residents is an array of urls
// eg. https://swapi.co/api/people/1/
// where each url gets mapped
// to a model:Resident
residents: ['$residents', map(ResidentModel)]
}
]
})
const input = {
planetId: 1
}
dataPoint.resolve(PlanetModel, input)
.then((output) => {
console.log(output)
/*
output ->
{
name: 'Tatooine',
population: 200000,
residents:
[
{ name: 'Luke Skywalker', gender: 'male', birthYear: '19BBY' },
{ name: 'C-3PO', gender: 'n/a', birthYear: '112BBY' },
{ name: 'Darth Vader', gender: 'male', birthYear: '41.9BBY' },
...
]
}
*/
})
`
Example at: examples/full-example-instances.js
Static method that creates a DataPoint instance.
SYNOPSIS
`js`
DataPoint.create([options])
Arguments
| Argument | Type | Description |
|:---|:---|:---|
| options | Object (_optional_) | This parameter is optional, as are its properties (values, entities, and entityTypes). You may configure the instance later through the instance's API. |
The following table describes the properties of the options argument.
| Property | Type | Description |
|:---|:---|:---|
| values | Object | Hash with values you want exposed to every Reducer |Object
| entities | | Application's defined entities |Object
| entityTypes | | Custom Entity Types |
RETURNS
DataPoint instance.
#### DataPoint.create example
Create a DataPoint object without configuring options
`js`
const DataPoint = require('data-point')
const dataPoint = DataPoint.create()
Create the DataPoint object and set options argument:
jshello ${input}!!
const DataPoint = require('data-point')
const dataPoint = DataPoint
.create({
values: {
foo: 'bar'
},
entities: {
'reducer:HelloWorld': (input) => {
return
}
}
})
SYNOPSIS
`js`
DataPoint.createReducer(source:*, [options:Object]):Reducer
ARGUMENTS
| Argument | Type | Description |
|:---|:---|:---|
| source | * | Spec for any of the supported Reducer types |Object
| options | | Optional config object |
Options
| Property | Type | Description |
|:---|:---|:---|
| default | * | Default value for the reducer. Setting this value is equivalent to using the withDefault reducer helper. |
Execute a Reducer against an input value. This function supports currying and will be executed when at least the first 2 parameters are provided.
SYNOPSIS
`js`
dataPoint.resolve(reducer:Reducer, input:, options:Object):Promise(output:)
This method returns a Promise with the final output value.
ARGUMENTS
| Argument | Type | Description |
|:---|:---|:---|
| reducer | Reducer | Reducer that manipulates the input. |
| input | * | Input value that you want to transform. If none, pass null or empty object {}. |Object
| options | | Options within the scope of the current transformation. More details available here. |
EXAMPLES:
- Hello World example.
- With options example.
This method is similar to dataPoint.resolve. The differences between the methods are:
- .transform() accepts an optional third parameter for node style callback..transform()
- returns a Promise that resolves to the full Accumulator object instead of accumulator.value. This may come in handy if you want to inspect other values from the transformation.
SYNOPSIS
`js`
// as promise
dataPoint.transform(reducer:Reducer, input:, options:Object):Promise(acc:)
// as nodejs callback function
dataPoint.transform(reducer:Reducer, input:*, options:Object, done:Function)
This method will return a Promise if done is omitted.
ARGUMENTS
| Argument | Type | Description |
|:---|:---|:---|
| reducer | Reducer | Reducer that manipulates the input. |
| input | * | Input value that you want to transform. If none, pass null or empty object {}. |Object
| options | | Options within the scope of the current transformation |function
| done | _(optional)_ | Error-first Node.js style callback with the arguments (error, accumulator). The second parameter is an Accumulator object where accumulator.value is the actual result of the transformation.
The following table describes the properties of the options argument.
| Property | Type | Description |
|:---|:---|:---|
| locals | Object | Hash with values you want exposed to every reducer. See example. |boolean
| trace | | Set this to true to trace the entities and the time each one is taking to execute. Use this option for debugging. |
Execute a Reducer from a provided Accumulator. This function will attempt at resolving a reducer providing an already constructed Accumulator Object. It will take the value provided in the Accumulator object to use as the input.
SYNOPSIS
`js`
dataPoint.resolveFromAccumulator(reducer:Reducer, acc:Accumulator):Promise(acc:Accumulator)
This method returns a Promise with the final output value.
ARGUMENTS
| Argument | Type | Description |
|:---|:---|:---|
| reducer | Reducer | Reducer that manipulates the input. |
| acc | Accumulator | Reducer's accumulator Object. The main property is value, which is the value the reducer will use as its input. |
This method adds new entities to a DataPoint instance.
SYNOPSIS
When defining new entities, must refer to either a built-in type like 'model' or a custom entity type. should be unique for each type; for example, model:foo and hash:foo can both use the foo ID, but an error is thrown if model:foo is defined twice.
`js`
dataPoint.addEntities({
'
'
...
})
OPTIONS
| Part | Type | Description |
|:---|:---|:---|
| EntityType | string | valid entity type |string
| EntityId | | unique entity ID |
Stores any value to be accessible via Accumulator.values. This object can also be set by passing a values property to DataPoint.create.
SYNOPSIS
`js`
dataPoint.addValue(objectPath, value)
ARGUMENTS
| Argument | Type | Description |
|:---|:---|:---|
| objectPath | string | object path where you want to add the new value. Uses _.set to append to the values object |*
| value | | anything you want to store |
This object is passed to reducers and middleware callbacks; it has contextual information about the current transformation or middleware that's being resolved.
The accumulator.value property is the current input data. This property should be treated as a read-only immutable object. This helps ensure that your reducers are pure functions that produce no side effects. If the value is an object, use it as your initial source for creating a new object.
Properties exposed:
| Key | Type | Description |
|:---|:---|:---|
| value | Object | Value to be transformed. |Object
| initialValue | | Initial value passed to the entity. You can use this value as a reference to the initial value passed to your Entity before any reducer was applied. |Object
| values | | Access to the values stored via dataPoint.addValue. |Object
| params | | Value of the current Entity's params property. (for all entities except Reducer) |Object
| locals | | Value passed from the options _argument_ when executing dataPoint.transform. |Object
| reducer | | Information relative to the current Reducer being executed. |Function
| debug | | debug method with scope data-point |
Reducers are used to transform values asynchronously. DataPoint supports the following reducer types:
1. path
2. function
3. object
4. entity
5. entity-id
6. list
A path reducer is a string that extracts a path from the current Accumulator value (which must be an Object). It uses lodash's _.get behind the scenes.
SYNOPSIS
`js`
'$[.|..|
OPTIONS
| Option | Description |
|:---|:---|
| $ | Reference to current accumulator.value. |accumulator
| $.. | Gives full access to properties (i.e. $..params.myParam). |accumulator.value
| $path | Object path notation to extract data from . |[]
| $path[] | Appending will map the reducer to each element of an input array. If the current accumulator value is not an array, the reducer will return undefined.
#### Root path $
EXAMPLES:
Gets the entire value
`js
const DataPoint = require('data-point')
const dataPoint = DataPoint.create()
const input = {
a: {
b: [
'Hello World'
]
}
}
dataPoint
.resolve('$', input)
.then((output) => {
assert.strictEqual(output, input)
})
`
#### Access accumulator reference
Access the reference of the accumulator.
`js
const DataPoint = require('data-point')
const dataPoint = DataPoint.create()
const input = {
a: {
b: [
'Hello World'
]
}
}
dataPoint
.resolve('$..value', input)
.then(output => {
assert.strictEqual(output input)
})
`
#### Object Path
Traverse an object's structure
`js`
const DataPoint = require('data-point')
const dataPoint = DataPoint.create()
const input = {
a: {
b: [
'Hello World'
]
}
}
dataPoint
.resolve('$a.b[0]', input)
.then(output => {
assert.strictEqual(output, 'Hello World')
})
Example at: examples/reducer-path.js
#### Object Map
Map an array by traversing object structures
`js`
const DataPoint = require('data-point')
const dataPoint = DataPoint.create()
const input = [
{
a: {
b: 'Hello World'
}
},
{
a: {
b: 'Hello Solar System'
}
},
{
a: {
b: 'Hello Universe'
}
}
]
dataPoint
.resolve('$a.b[]', input)
.then(output => {
assert.deepStrictEqual(output, ['Hello World', 'Hello Solar System', 'Hello Universe'])
})
A function reducer allows you to use a function to apply a transformation. There are several ways to define a function reducer:
- Synchronous function that returns new valuefunction
- Asynchronous that returns a Promise
- Asynchronous with callback parameterfunction
- Asynchronous through async/await (only if your environment supports it)
IMPORTANT: Be careful with the parameters passed to your function reducer; DataPoint relies on the number of arguments to detect the type of function reducer it should expect.
#### Returning a value
The returned value is used as the new value of the transformation.
SYNOPSIS
`js`
const name = (input:*, acc:Accumulator) => {
return newValue
}
Reducer's arguments
| Argument | Type | Description |
|:---|:---|:---|
| input | * | Reference to acc.value |value
| acc | Accumulator | Current reducer's accumulator Object. The main property is , which is the current reducer's value. |
Function Reducer Example
`js`
const reducer = (input, acc) => {
return input + ' World'
}
dataPoint
.resolve(reducer, 'Hello')
.then((output) => {
assert.strictEqual(output, 'Hello World')
})
Example at: examples/reducer-function-sync.js
#### Returning a Promise
If you return a Promise its resolution will be used as the new value of the transformation. Use this pattern to resolve asynchronous logic inside your reducer.
SYNOPSIS
`js`
const name = (input:*, acc:Accumulator) => {
return Promise.resolve(newValue)
}
Reducer's arguments
| Argument | Type | Description |
|:---|:---|:---|
| input | * | Reference to acc.value |acc.value
| acc | Accumulator | Current reducer's accumulator Object. The main property is , which is the current reducer's value. |
Example
`js`
const reducer = (input, acc) => {
// input is a reference to acc.value
return Promise.resolve(acc.value + ' World')
}
dataPoint
.resolve(reducer, 'Hello')
.then((output) => {
assert.strictEqual(output, 'Hello World')
})
Example at: examples/reducer-function-promise.js
#### With a callback parameter
Accepting a third parameter as a callback allows you to execute an asynchronous block of code. This should be an error-first, Node.js style callback with the arguments (error, value), where value will be the _value_ passed to the _next_ transform; this value becomes the new value of the transformation.
SYNOPSIS
`js`
const name = (input:*, acc:Accumulator, next:function) => {
next(error:Error, newValue:*)
}
Reducer's arguments
| Argument | Type | Description |
|:---|:---|:---|
| input | * | Reference to acc.value |acc.value
| acc | Accumulator | Current reducer's accumulator Object. The main property is , which is the current reducer's value. |Function(error,value)
| next | | Node.js style callback, where value is the value to be passed to the next reducer.
Example
`js`
const reducer = (input, acc, next) => {
next(null, input + ' World')
}
dataPoint
.resolve(reducer, 'Hello')
.then((output) => {
assert.strictEqual(output, 'Hello World')
})
Example at: examples/reducer-function-with-callback.js
Throw an error from the reducer
`js`
const throwError = (error, acc, next) => {
// passing first argument will be
// handled as an error by the transform
next(new Error('oh noes!!'))
}
dataPoint
.transform(throwError, 'Hello')
.catch((error) => {
console.assert(error instanceof Error)
console.log(error.toString()) // 'Error: oh noes!!'
})
Example at: examples/reducer-function-error.js
These are plain objects where the value of each key is a Reducer. They're used to aggregate data or transform objects. You can add constants with the constant reducer helper, which is more performant than using a function reducer:
`js
const { constant } = require('data-point')
const objectReducer = {
// in this case, x and y both resolve to 42, but DataPoint
// can optimize the resolution of the constant value
x: () => 42,
y: constant(42)
}
`
Transforming an object
`js`
const inputData = {
x: {
y: {
z: 2
}
}
}
const objectReducer = {
y: '$x.y',
zPlusOne: ['$x.y.z', (input) => input + 1]
}
// output from dataPoint.transform(objectReducer, inputData):
{
y: {
z: 2
},
zPlusOne: 3
}
Combining multiple requests
`js`
const dataPoint = require('data-point').create()
dataPoint.addEntities({
'request:Planet': {
url: 'https://swapi.co/api/planets/{value}'
}
})
const objectReducer = {
tatooine: ['$tatooine', 'request:Planet'],
alderaan: ['$alderaan', 'request:Planet']
}
const planetIds = {
tatooine: 1,
alderaan: 2
}
dataPoint.resolve(objectReducer, planetIds)
.then(output => {
// do something with the aggregated planet data!
})
Each of the reducers, including the nested ones, are resolved against the same accumulator value. This means that input objects can be rearranged at any level:
Rearranging input data
`js`
const inputData = {
a: 'A',
b: 'B',
c: {
x: 'X',
y: 'Y'
}
})
// some data will move to a higher level of nesting,
// but other data will move deeper into the object
const objectReducer = {
x: '$c.x',
y: '$c.y',
z: {
a: '$a',
b: '$b'
}
}
// output from dataPoint.resolve(objectReducer, inputData):
{
x: 'X',
y: 'Y',
z: {
a: 'A',
b: 'B'
}
}
Each of the reducers might contain more object reducers (which might contain other reducers, and so on). Notice how the output changes based on the position of the object reducers in the two expressions:
Nested Transform Expressions
`js`
const inputData = {
a: {
a: 1,
b: 2
}
}
const objectReducer = {
x: [
'$a',
// this comes second, so it's resolved
// against the output from the '$a' transform
{
a: '$a'
}
],
y: [
// this comes first, so it's resolved
// against the main input to objectReducer
{
a: '$a'
},
'$a'
]
}
// output from dataPoint.resolve(objectReducer, inputData):
{
x: {
a: 1
},
y: {
a: 1,
b: 2
}
}
An empty object reducer will resolve to an empty object:
`js
const reducer = {}
const input = { a: 1 }
dataPoint.resolve(reducer, input) // => {}
`
An entity instance reducer is used to apply a given entity with to the current Accumulator.
See the Entities section for information about the supported entity types.
OPTIONS
| Option | Type | Description |
|:---|:---|:---|
| ? | String | Only execute entity if acc.value is not equal to false, null or undefined. |String
| EntityType | | Valid Entity type. |String
| EntityID | | Valid Entity ID. Appending [] will map the reducer to each element of an input array. If the current accumulator value is not an array, the reducer will return undefined. |
Entity Reducer Example
`js
const {
Model,
Request
} = require('data-point')
const PersonRequest = Request('PersonRequest', {
url: 'https://swapi.co/api/people/{value}'
})
const PersonModel = Model('PersonModel', {
value: {
name: '$name',
birthYear: '$birth_year'
}
})
dataPoint
.resolve([PersonRequest, PersonModel], 1)
.then((output) => {
assert.deepStrictEqual(output, {
name: 'Luke Skywalker',
birthYear: '19BBY'
})
})
`
Example at: examples/reducer-entity-instance.js
An entity reducer is used to execute an entity with the current Accumulator as the input.
Appending [] to an entity reducer will map the given entity to each element of an input array. If the current accumulator value is not an array, the reducer will return undefined.
See the Entities section for information about the supported entity types.
SYNOPSIS
`js`
'[?]
OPTIONS
| Option | Type | Description |
|:---|:---|:---|
| ? | String | Only execute entity if acc.value is not equal to false, null or undefined. |String
| EntityType | | Valid Entity type. |String
| EntityID | | Valid Entity ID. Appending [] will map the reducer to each element of an input array. If the current accumulator value is not an array, the reducer will return undefined. |
Entity Reducer Example
`js
const input = {
a: {
b: 'Hello World'
}
}
const toUpperCase = (input) => {
return input.toUpperCase()
}
dataPoint.addEntities({
'reducer:getGreeting': '$a.b',
'reducer:toUpperCase': toUpperCase,
})
// resolve reducer:getGreeting,reducer:toUpperCase
// pipe value to
dataPoint
.resolve(['reducer:getGreeting | reducer:toUpperCase'], input)
.then((output) => {
assert.strictEqual(output, 'HELLO WORLD')
})
Array Mapping Example
`js
const input = {
a: [
'Hello World',
'Hello Laia',
'Hello Darek',
'Hello Italy',
]
}
const toUpperCase = (input) => {
return input.toUpperCase()
}
dataPoint.addEntities({
'reducer:toUpperCase': toUpperCase
})
dataPoint
.resolve(['$a | reducer:toUpperCase[]'], input)
.then((output) => {
assert.strictEqual(output[0], 'HELLO WORLD')
assert.strictEqual(output[1], 'HELLO LAIA')
assert.strictEqual(output[2], 'HELLO DAREK')
assert.strictEqual(output[3], 'HELLO ITALY')
})
`
A list reducer is an array of reducers where the result of each reducer becomes the input to the next reducer. The reducers are executed serially and asynchronously. It's possible for a list reducer to contain other list reducers.
| List Reducer | Description |
|:---|:---|
| ['$a.b', (input) => { ... }] | Get path a.b, pipe value to function reducer |['$a.b', (input) => { ... }, 'hash:Foo']
| | Get path a.b, pipe value to function reducer, pipe result to hash:Foo |
IMPORTANT: an empty list reducer will resolve to undefined. This mirrors the behavior of empty functions.
`js
const reducer = []
const input = 'INPUT'
dataPoint.resolve(reducer, input) // => undefined
`
Only execute an entity if the accumulator value is not equal to false, null or undefined. If the conditional is not met, the entity will not be executed and the value will remain the same.
Conditionally execute an entity
`js
const people = [
{
name: 'Luke Skywalker',
swapiId: '1'
},
{
name: 'Yoda',
swapiId: null
}
]
dataPoint.addEntities({
'request:getPerson': {
url: 'https://swapi.co/api/people/{value}'
},
'reducer:getPerson': {
name: '$name',
// request:getPerson will only
// be executed if swapiId is
// not false, null or undefined
birthYear: '$swapiId | ?request:getPerson | $birth_year'
}
})
dataPoint
.resolve('reducer:getPerson[]', people)
.then((output) => {
assert.deepStrictEqual(output, [
{
name: 'Luke Skywalker',
birthYear: '19BBY'
},
{
name: 'Yoda',
birthYear: undefined
}
])
})
`
Example at: examples/reducer-conditional-operator.js
Reducer helpers are factory functions for creating reducers. They're accessed through the DataPoint Object:
`js`
const {
assign,
constant,
filter,
find,
map,
parallel,
withDefault
} = require('data-point')
The assign reducer creates a new Object by resolving the provided Reducer and merging the result with the current accumulator value. It uses Object.assign internally.
SYNOPSIS
`js`
assign(reducer:Reducer):Object
Reducer's arguments
| Argument | Type | Description |
|:---|:---|:---|
| reducer | Reducer | Result from this reducer will be merged into the current accumulator.value. By convention, this reducer should return an Object. |
EXAMPLE:
Add a key that references a nested value from the accumulator.
`js
const input = {
a: 1
}
// merges the object reducer with
// accumulator.value
const reducer = DataPoint.assign({
c: '$b.c'
})
dataPoint
.resolve(reducer, input)
.then(output => {
/*
output --> {
a: 1,
b: {
c: 2
},
c: 2
}
*/
})
`
Example at: examples/reducer-helper-assign.js
The map reducer creates a new array with the results of applying the provided Reducer to every element in the input array.
SYNOPSIS
`js`
map(reducer:Reducer):Array
Reducer's arguments
| Argument | Type | Description |
|:---|:---|:---|
| reducer | Reducer | The reducer will get applied to each element in the array. |
EXAMPLE:
Apply a set of reducers to each item in an array
`js
const input = [{
a: 1
}, {
a: 2
}]
// get path a then multiply by 2
const reducer = DataPoint.map(
['$a', (input) => input * 2]
)
dataPoint
.resolve(reducer, input)
.then(output => {
// output -> [2, 4]
})
`
Example at: examples/reducer-helper-map.js
The filter reducer creates a new array with elements that resolve as truthy when passed to the given Reducer.
SYNOPSIS
`js`
filter(reducer:Reducer):Array
Reducer's arguments
| Argument | Type | Description |
|:---|:---|:---|
| reducer | Reducer | Reducer result is used to test for _truthy_ on each element of the array. |
EXAMPLE:
Find objects where path a
is greater than 1
`js
const input = [{ a: 1 }, { a: 2 }]
// filters array elements that are not
// truthy for the given list reducer
const reducer = DataPoint.filter(
['$a', (input) => input > 1]
)
dataPoint
.resolve(reducer, input)
.then(output => {
// output -> [{ a: 2 }]
})
`
Example at: examples/reducer-helper-filter.js
The find reducer returns the first element of an array that resolves to _truthy_ when passed through the provided Reducer. It returns undefined if no match is found.
SYNOPSIS
`js`
find(reducer:Reducer):*
Reducer's arguments
| Argument | Type | Description |
|:---|:---|:---|
| reducer | Reducer | Reducer result is used to test for _truthy_ on each element of the array. |
EXAMPLE:
Find elements where path b
resolves to _truthy_ value
`js
const input = [{ a: 1 }, { b: 2 }]
// the $b reducer is truthy for the
// second element in the array
const reducer = DataPoint.find('$b')
dataPoint
.resolve(reducer, input)
.then(output => {
// output -> { b: 2 }
})
`
Example at: examples/reducer-helper-find.js
The constant reducer always returns the given value. If a reducer is passed it will not be evaluated. This is primarily meant to be used in object reducers.
SYNOPSIS
`js`
constant(value:):
Reducer's arguments
| Argument | Type | Description |
|:---|:---|:---|
| value | * | The value the reducer should return |
EXAMPLE:
returning an object constant
`js
const input = {
a: 1,
b: 2
}
const reducer = {
a: '$a',
b: DataPoint.constant({
a: '$a',
b: 3
})
}
dataPoint
.resolve(reducer, input)
.then(output => {
// {
// a: 1,
// b: {
// a: '$a',
// b: 3
// }
// }
}
})
`
reducers are not evaluated when defined inside of constants
`js
const input = {
b: 1
}
// object reducer that contains a path reducer ('$a')
let reducer = {
a: '$b'
}
dataPoint.resolve(reducer, input) // => { a: 1 }
// both the object and the path will be treated as
// constants instead of being used to create reducers
reducer = DataPoint.constant({
a: '$b'
})
dataPoint.resolve(reducer, input) // => { a: '$b' }
`
This resolves an array of reducers. The output is a new array where each element is the output of a reducer;
this contrasts with list reducers, which return the output from the last reducer in the array.
SYNOPSIS
`js`
parallel(reducers:Array
Reducer's arguments
| Argument | Type | Description |
|:---|:---|:---|
| reducers | Array
EXAMPLE:
resolving an array of reducers with parallel
`js
const reducer = DataPoint.parallel([
'$a',
['$b', (input) => input + 2] // list reducer
])
const input = {
a: 1,
b: 2
}
dataPoint.resolve(reducer, input) // => [1, 4]
`
The withDefault reducer adds a default value to any reducer type. If the reducer resolves to null, undefined, NaN, or '',
the default is returned instead.
SYNOPSIS
`js`
withDefault(source:, value:):*
Reducer's arguments
| Argument | Type | Description |
|:---|:---|:---|
| source | * | Source data for creating a Reducer |
| value | * | The default value to use (or a function that returns the default value) |
The default value is not cloned before it's returned, so it's good practice to wrap any Objects in a function.
EXAMPLE:
`js
const input = {
a: undefined
}
// adds a default to a path reducer
const r1 = DataPoint.withDefault('$a', 50)
dataPoint.resolve(r1, input) // => 50
// passing a function is useful when the default value is
// an object, because it returns a new object every time
const r2 = withDefault('$a', () => {
return { b: 1 }
})
dataPoint.resolve(r2, input) // => { b: 1 }
`
Entities are used to transform data by composing multiple reducers, they can be created as non-registered or registered entities.
- Instance entities - are entity objects created directly with an Entity Factory, they are meant to be used as a entity reducer.
- Registered entities - are entity objects which are registered and cached in a DataPoint instance, they are meant to be used as a registered entity reducer.
Registered entities may be added to DataPoint in two different ways:
1. With the DataPoint.create method (as explained in the setup examples)
2. With the dataPoint.addEntities instance method
See built-in entities for information on what each entity does.
Entities can be created from these factory functions:
`js`
const {
Entry,
Model,
Reducer,
Collection,
Hash,
Request,
Control,
Schema
} = require('data-point')
SYNOPSIS
Each factory has the following signature:
`js`
Factory(name:String, spec:Object):EntityInstance
ARGUMENTS
| Argument | Type | Description |
|:---|:---|:---|
| name | string | The name of the entity; this will be used to generate an entity ID with the format |Object
| spec | | The source for generating the entity |
Example
`js
const DataPoint = require('data-point')
const { Model } = DataPoint
const dataPoint = DataPoint.create()
const HelloWorld = Model('HelloWorld', {
value: input => ({
hello: 'world'
})
})
// to reference it we use the actual entity instance
dataPoint.resolve(HelloWorld, {})
.then(value => {
console.assert(value, {
hello: 'world'
})
})
`
You may register an entity through DataPoint.create or dataPoint.addEntities.
Example
`js
const DataPoint = require('data-point')
dataPoint = DataPoint.create({
entities: {
'model:HelloWorld', {
value: input => ({
hello: 'world'
}
}
}
})
// to reference it we use a string with its registered id
dataPoint.resolve('model:HelloWorld', {})
.then(value => {
console.assert(value, {
hello: 'world'
})
})
`
All entities share a common API (except for Reducer).
`js
{
// type checks the entity's input
inputType: String | Reducer,
// executes --before-- any modifier
before: Reducer,
// executes --after-- any modifier
after: Reducer,
// type checks the entity's output
outputType: String | Reducer,
// executes in case there is an error at any
// point of the entire transformation
error: Reducer,
// this object allows you to store and eventually
// access it at any given time on any reducer
params: Object
}
`
Properties exposed:
| Key | Type | Description |
|:---|:---|:---|
| inputType | String, Reducer | type checks the entity's input value, but does not mutate it |
| before | Reducer | reducer to be resolved before the entity resolution |
| after | Reducer | reducer to be resolved after the entity resolution |
| outputType | String, Reducer | type checks the entity's output value, but does not mutate it |
| error | Reducer | reducer to be resolved in case of an error (including errors thrown from the inputType and outputType reducers) |Object
| params | | user defined Hash that will be passed to every transform within the context of the transform's execution |
You can use inputType and outputType for type checking an entity's input and output values. Type checking does not mutate the result.
Built in type checks:
To use built-in type checks, you may set the value of inputType or outputType to: 'string', 'number', 'boolean', 'function', 'error', 'array', or 'object'.
Check if a model outputs an string
This example uses a Model Entity, for information on what a model is please go to the Model Entity section.
`js
const dataPoint = DataPoint.create()
dataPoint.addEntities({
'model:getName': {
value: '$name',
outputType: 'string'
}
})
const input = {
name: 'DataPoint'
}
dataPoint.resolve('model:getName', input)
.then(output => {
// output -> 'DataPoint'
})
`
Custom type checking:
You may also type check with a Schema Entity, or by creating a Reducer with the createTypeCheckReducer function.
SYNOPSIS
`js`
DataPoint.createTypeCheckReducer(typeCheckFunction, [expectedType])
ARGUMENTS
| Argument | Type | Description |
|:---|:---|:---|
| typeCheckFunction | Function | Return true when the input is valid; otherwise, an error will be thrown. If the function returns a string, that will be appended to the error message. |string
| expectedType | _(optional)_ | The expected type; this will also be used in the error message. |
`js
const DataPoint = require('data-point')
const { createTypeCheckReducer } = DataPoint
const isNonEmptyArray = input => Array.isArray(input) && input.length > 0
const dataPoint = DataPoint.create({
entities: {
'model:get-first-item': {
inputType: createTypeCheckReducer(isNonEmptyArray, 'non-empty-array'),
value: input => input[0]
}
}
})
`
In this example we are using a Schema Entity to check the inputType.
`js
const dataPoint = DataPoint.create()
dataPoint.addEntities({
'model:getName': {
// assume schema:RepoSchema
// exists and checks of the
// existence of name
inputType: 'schema:RepoSchema',
value: '$name'
}
})
const input = {
name: 'DataPoint'
}
dataPoint.resolve('model:getName', input)
.then(output => {
// output -> DataPoint
})
`
DataPoint comes with the following built-in entity types:
- Reducer / Transform
- Model
- Entry
- Request
- Hash
- Collection
- Control
- Schema
A Reducer entity is a 'snippet' that you can re-use in other entities. It does not expose the before/after/error/params API that other entities have.
IMPORTANT: Reducer Entities do not support extension.
SYNOPSIS
`js`
dataPoint.addEntities({
'reducer:
})
For backwards compatibility, the keyword transform can be used in place of reducer:
`js`
dataPoint.addEntities({
'reducer:
})
Reducer Entity Example
`js`
const input = {
a: {
b: {
c: [1, 2, 3]
}
}
}
const getMax = (input) => {
return Math.max.apply(null, input)
}
const multiplyBy = (number) => (input) => {
return input * number
}
dataPoint.addEntities({
'reducer:foo': ['$a.b.c', getMax, multiplyBy(10)]
})
dataPoint
.resolve('reducer:foo', input)
.then((output) => {
assert.strictEqual(output, 30)
})
A Model entity is a generic entity that provides the base methods.
SYNOPSIS
`js`
dataPoint.addEntities({
'model:
inputType: String | Reducer,
before: Reducer,
value: Reducer,
after: Reducer,
outputType: String | Reducer,
error: Reducer,
params: Object
}
})
Properties exposed:
| Key | Type | Description |
|:---|:---|:---|
| inputType | String, Reducer | type checks the entity's input value, but does not mutate it |
| before | Reducer | reducer to be resolved before the entity resolution |
| after | Reducer | reducer to be resolved after the entity resolution |
| outputType | String, Reducer | type checks the entity's output value, but does not mutate it |
| error | Reducer | reducer to be resolved in case of an error |
| params | Object | user defined Hash that will be passed to every transform within the context of the transform's execution |
#### Model.value
Using the value
property to transform an input
`js`
const input = {
a: {
b: {1
c: [1, 2, 3]
}
}
}
const getMax = (input) => {
return Math.max.apply(null, input)
}
const multiplyBy = (number) => (input) => {
return input * number
}
dataPoint.addEntities({
'model:foo': {
value: ['$a.b.c', getMax, multiplyBy(10)]
}
})
dataPoint
.resolve('model:foo', input)
.then((output) => {
assert.strictEqual(output, 30)
})
Example at: examples/entity-model-basic.js
#### Model.before
Checking whether the value passed to an entity is an array
`js`
const toArray = (input) => {
return Array.isArray(input)
? input
: [input]
}
dataPoint.addEntities({
'model:foo': {
before: toArray,
value: '$'
}
})
dataPoint
.resolve('model:foo', 100)
.then((output) => {
assert.deepStrictEqual(output, [100])
})
Example at: examples/entity-model-before.js
#### Model.after
Using after
transform
`js`
const toArray = (input) => {
return Array.isArray(input)
? input
: [input]
}
dataPoint.addEntities({
'model:foo': {
value: '$a.b',
after: isArray()
}
})
const input = {
a: {
b: [3, 15]
}
}
dataPoint
.resolve('model:foo', input)
.then((output) => {
assert.deepStrictEqual(output, [3, 15])
})
#### Model.error
Any error that happens within the scope of the Entity can be handled by the error transform. To respect the API, error reducers have the same API.
Error handling
Passing a value as the second argument will stop the propagation of the error.
EXAMPLES:
Handling Entry Errors
Let's resolve to a non-array value and see how it would be handled, this example will use outputType for type checking.
`js
dataPoint.addEntities({
'model:getArray': {
// points to a NON Array value
value: '$a.b',
outputType: 'isArray',
error: (error) => {
// prints out the error
// message generated by
// isArray type check
console.log(error.message)
console.log('Value is invalid, resolving to empty array')
// passing a value will stop
// the propagation of the error
return []
}
}
})
const input = {
a: {
b: 'foo'
}
}
dataPoint
.resolve('model:getArray', input)
.then((output) => {
assert.deepStrictEqual(output, [])
})
`
Example at: examples/entity-model-error-handled.js
Pass the array to be handled somewhere else
`js
const logError = (error) => {
console.log(error.toString())
throw error
}
dataPoint.addEntities({
'model:getArray': {
value: '$a',
outputType: 'isArray',
error: logError
}
})
const input = {
a: {
b: 'foo'
}
}
dataPoint
.resolve('model:getArray', input)
.catch((error) => {
console.log(error.toString())
})
`
Example at: examples/entity-model-error-rethrow.js
#### Model.params
The params object is used to pass custom data to your entity. This Object is exposed as a property of the Accumulator Object. Which can be accessed via a function reducer, as well as through a path reducer expression.
On a Function Reducer
`js`
const multiplyValue = (input, acc) => {
return input * acc.params.multiplier
}
dataPoint.addEntities({
'model:multiply': {
value: multiplyValue,
params: {
multiplier: 100
}
}
})
dataPoint
.resolve('model:multiply', 200)
.then((output) => {
assert.deepStrictEqual(output, 20000)
})
On a Path Reducer
`js`
dataPoint.addEntities({
'model:getParam': {
value: '$..params.multiplier',
params: {
multiplier: 100
}
}
})
dataPoint.resolve('model:getParam')
.then((output) => {
assert.deepStrictEqual(output, 100)
})
This entity is very similar to the Model entity. Its main difference is that this entity will default to an empty object { } as its initial value if none was passed. As a best practice, use it as your starting point, and use it to call more complex entities.
SYNOPSIS
`js`
dataPoint.addEntities({
'entry:
inputType: String | Reducer,
before: Reducer,
value: Reducer,
after: Reducer,
outputType: String | Reducer,
error: Reducer,
params: Object
}
})
Properties exposed:
| Key | Type | Description |
|:---|:---|:---|
| inputType | String, Reducer | type checks the entity's input value, but does not mutate it |
| before | Reducer | reducer to be resolved before the entity resolution |
| after | Reducer | reducer to be resolved after the entity resolution |
| outputType | String, Reducer | type checks the entity's output value, but does not mutate it |
| error | Reducer | reducer to be resolved in case of an error |
| params | Object | user defined Hash that will be passed to every transform within the context of the transform's execution |
Requests a remote source, using request-promise behind the scenes. The features supported by request-promise are exposed/supported by Request entity.
SYNOPSIS
`js`
dataPoint.addEntities({
'request:
inputType: String | Reducer,
before: Reducer,
value: Reducer,
url: StringTemplate,
options: Reducer,
after: Reducer,
outputType: String | Reducer,
error: Reducer,
params: Object
}
})
Properties exposed:
| Key | Type | Description |
|:---|:---|:---|
| inputType | String, Reducer | type checks the entity's input value, but does not mutate it |
| before | Reducer | reducer to be resolved before the entity resolution |
| value | Reducer | the result of this reducer is the input when resolving url and options
| url | StringTemplate | String value to resolve the request's url |
| options | Reducer | reducer that returns an object to use as request-promise options
| after | Reducer | reducer to be resolved after the entity resolution |
| error | Reducer | reducer to be resolved in case of an error |
| outputType | String, Reducer | type checks the entity's output value, but does not mutate it |
| params | Object | user defined Hash that will be passed to every reducer within the context of the transform function's execution |
#### Request.url
Sets the url to be requested.
NOTE: When Request.url is not defined it will use the current Accumulator.value (if the value is of type string) as the Request's url.
Request.url
Example
`js
dataPoint.addEntities({
'request:getLuke': {
url: 'https://swapi.co/api/people/1/'
}
})
dataPoint.resolve('request:getLuke', {})
.then(output => {
/*
output ->
{
name: 'Luke Skywalker',
height: '172',
...
}
*/
})
`
Example at: examples/entity-request-basic.js
#### Request.url as StringTemplate
StringTemplate is a string that supports a minimal templating system. You may inject any value into the string by enclosing it within {ObjectPath} curly braces. The context of the string is the Request's Accumulator Object, meaning you have access to any property within it.
Using acc.value property to make the url dynamic:
acc.value
Example
`js
dataPoint.addEntities({
'request:getLuke': {
// inject the acc.value.personId
url: 'https://swapi.co/api/people/{value.personId}/'
}
})
const input = {
personId: 1
}
dataPoint.resolve('request:getLuke', input)
.then(output => {
/*
output ->
{
name: 'Luke Skywalker',
height: '172',
...
}
*/
})
`
Example at: examples/entity-request-string-template.js
Using acc.locals property to make the url dynamic:
acc.locals
example
`js
dataPoint.addEntities({
'request:getLuke': {
url: 'https://swapi.co/api/people/{locals.personId}/'
}
})
const options = {
locals: {
personId: 1
}
}
dataPoint.resolve('request:getLuke', {}, options)
.then(output => {
/*
output ->
{
name: 'Luke Skywalker',
height: '172',
...
}
*/
})
`
Example at: examples/entity-request-options-locals.js
For more information on acc.locals: Transform Options and Accumulator Objects.
Using constants in the options reducer:
constants example
`js
const DataPoint = require('data-point')
const dataPoint = DataPoint.create()
dataPoint.addEntities({
'request:searchPeople': {
url: 'https://swapi.co/api/people',
// options is a Reducer, but values
// at any level can be wrapped as
// constants (or just wrap the whole
// object if all the values are static)
options: {
'content-type': DataPoint.constant('application/json')
qs: {
// get path searchTerm from input
// to dataPoint.resolve
search: '$searchTerm'
}
}
}
})
const input = {
searchTerm: 'r2'
}
// the second parameter to transform is the input value
dataPoint
.resolve('request:searchPeople', input)
.then(output => {
assert.strictEqual(output.results[0].name, 'R2-D2')
})
`
Example at: examples/entity-request-options.js
For more examples of request entities, see the Examples, the Integration Examples, and the unit tests: Request Definitions.
#### Inspecting Request Entities
You may inspect a Request entity through the params.inspect property.
note: At the moment this feature is only available on Request entity, PRs are welcome.
SYNOPSIS
`js`
dataPoint.addEntities({
'request:
params: {
inspect: Boolean|Function
}
}
})
Boolean
If params.inspect is true, it will output the entity's information to the console.
Function
If params.inspect is a function, it will be called twice: once before the request is made, and once when the request is resolved. It should have the signature `(accumulator: Object, data: Object)`.
The inspect function is first called just before initiating the request. The first argument is the accumulator, and the second is a data object with these properties:
`js`
{
type: 'request',
// unique ID that is shared with the 'response' object
debugId: Number,
// ex: 'GET'
method: String,
// fully-formed URI
uri: String,
// the value of request.body (or undefined)
[body]: String
}
It's then called when the request succeeds or fails. The data object will have a type property of either 'response' or 'error'. The debugId can be used to match the response with the corresponding request.
`js`
{
type: 'response|error',
// unique ID that is shared with the 'request' object
debugId: Number,
// http status code
statusCode: Number,
}
A Hash entity transforms a _Hash_ like data structure. It enables you to manipulate the keys within a Hash.
To prevent unexpected results, a Hash can only return Plain Objects, which are objects created by the Object constructor. If a hash resolves to a different type, it will throw an error. This type check occurs before the value is passed to the (optional) outputType reducer.
SYNOPSIS
`js`
dataPoint.addEntities({
'hash:
inputType: String | Reducer,
before: Reducer,
value: Reducer,
mapKeys: TransformMap,
omitKeys: String[],
pickKeys: String[],
addKeys: TransformMap,
addValues: Object,
compose: ComposeReducer[],
after: Reducer,
outputType: String | Reducer,
error: Reducer,
params: Object,
}
})
Properties exposed:
| Key | Type | Description |
|:---|:---|:---|
| inputType | String, Reducer | type checks the entity's input value, but does not mutate it |
| before | Reducer | reducer to be resolved before the entity resolution |
| value | Reducer | The value to which the Entity resolves |
| mapKeys | Object Reducer | Map to a new set of key/values. Each value accepts a reducer |
| omitKeys | String[] | Omits keys from acc.value. Internally. |String[]
| pickKeys | | Picks keys from acc.value. Internally. |Object
| addKeys | Object Reducer | Add/Override key/values. Each value accepts a reducer. Internally, this uses the assign reducer helper |
| addValues | | Add/Override hard-coded key/values. Internally, this uses the assign reducer helper |[]
| compose | ComposeReducer | Modify the value of accumulator through an Array of ComposeReducer objects. Think of it as a Compose/Flow Operation), where the result of one operation gets passed to the next one|Object
| after | Reducer | reducer to be resolved after the entity resolution |
| outputType | String, Reducer | type checks the entity's output value, but does not mutate it. Collection only supports custom outputType reducers, and not the built-in types like string, number, etc. |
| error | Reducer | reducer to be resolved in case of an error |
| params | | User-defined Hash that will be passed to every reducer within the context of the transform function's execution |
Hash entities expose a set of optional reducers: mapKeys, omitKeys, pickKeys, addKeys, and addValues. When using more than one of these reducers, they should be defined through the compose property.
#### Hash.value
Resolve accumulator.value to a hash
`js`
const input = {
a: {
b: {
c: 'Hello',
d: ' World!!'
}
}
}
dataPoint.addEntities({
'hash:helloWorld': {
value: '$a.b'
}
})
dataPoint
.resolve('hash:helloWorld', input)
.then((output) => {
assert.deepStrictEqual(output, {
c: 'Hello',
d: ' World!!'
})
})
Example at: examples/entity-hash-context.js
#### Hash.mapKeys
Maps to a new set of key/value pairs through a object reducer, where each value is a Reducer.
Going back to our GitHub API examples, let's map some keys from the result of a request:
Hash.mapKeys Example
`jshttps://github.com/ViacomInc/${_.kebabCase(input)}
const _ = require('lodash')
dataPoint.addEntities({
'hash:mapKeys': {
mapKeys: {
// map to acc.value.name
name: '$name',
// uses a list reducer to
// map to acc.value.name
// and generate a string with
// a function reducer
url: [
'$name',
input => {
return
}
]
}
}
})
const input = {
name: 'DataPoint'
}
dataPoint.resolve('hash:mapKeys', input).then(output => {
assert.deepStrictEqual(output, {
name: 'DataPoint',
url: 'https://github.com/ViacomInc/data-point'
})
})
`
Example at: examples/entity-hash-mapKeys.js
#### Hash.addKeys
Adds keys to the current Hash value. If an added key already exists, it will be overridden.
Hash.addKeys is very similar to Hash.mapKeys, but the difference is that mapKeys will ONLY map the keys you give it, whereas addKeys will ADD/APPEND new keys to your existing acc.value. You may think of addKeys as an _extend_ operation.
Hash.addKeys Example
`js
dataPoint.addEntities({
'hash:addKeys': {
addKeys: {
nameLowerCase: ['$name', input => input.toLowerCase()],
url: () => 'https://github.com/ViacomInc/data-point'
}
}
})
const input = {
name: 'DataPoint'
}
dataPoint.resolve('hash:addKeys', input).then(output => {
assert.deepStrictEqual(output, {
name: 'DataPoint',
nameLowerCase: 'datapoint',
url: 'https://github.com/ViacomInc/data-point'
})
})
`
Example at: examples/entity-hash-addKeys.js
#### Hash.pickKeys
Picks a list of keys from the current Hash value.
The next example is similar to the previous example. However, instead of mapping key/value pairs, this example just picks some of the keys.
Hash.pickKeys Example
`js
dataPoint.addEntities({
'hash:pickKeys': {
pickKeys: ['url']
}
})
const input = {
name: 'DataPoint',
url: 'https://github.com/ViacomInc/data-point'
}
dataPoint.resolve('hash:pickKeys', input).then(output => {
// notice how name is no longer
// in the object
assert.deepStrictEqual(output, {
url: 'https://github.com/ViacomInc/data-point'
})
})
`
Example at: examples/entity-hash-pickKeys.js
#### Hash.omitKeys
Omits keys from the Hash value.
This example will only omit some keys, and let the rest pass through:
Hash.omitKeys Example
`js
dataPoint.addEntities({
'hash:omitKeys': {
omitKeys: ['name']
}
})
// notice how name is no longer in the object
const expectedResult = {
url: 'https://github.com/ViacomInc/data-point'
}
const input = {
name: 'DataPoint',
url: 'https://github.com/ViacomInc/data-point'
}
dataPoint.resolve('hash:omitKeys', input).then(output => {
assert.deepStrictEqual(output, expectedResult)
})
``
Example at: examples/entity-hash-omitKeys.js
##