Reductio: Crossfilter groupings
npm install reductioReductio: Crossfilter grouping
========
Reductio is a library for generating Crossfilter reduce functions and applying them to Crossfilter groups. Crossfilter supports basic count and sum aggregations, but even aggregations as conceptually simple as a group minimum or maximum can be difficult to build correctly and efficiently on a Crossfilter group. Reductio provides helper functions that generate these aggregations in an efficient and composable way, making it easy to use more complex aggregations with Crossfilter and to have more than one aggregation per group without having to worry about designing 2-way reduce functions.
Basic use:
``javascript
var data = crossfilter([
{ foo: 'one', bar: 1 },
{ foo: 'two', bar: 2 },
{ foo: 'three', bar: 3 },
{ foo: 'one', bar: 4 },
{ foo: 'one', bar: 5 },
{ foo: 'two', bar: 6 },
]);
var dim = data.dimension(function(d) { return d.foo; });
var group = dim.group();
// Equivalent to reductio().avg(function(d) { return d.bar; }), which sets the .sum() and .count() values.
var reducer = reductio()
.count(true)
.sum(function(d) { return d.bar; })
.avg(true);
// Now it should track count, sum, and avg.
reducer(group);
group.top(Infinity);
// [ { key: 'one', value: { count: 3, sum: 10, avg: 3.3333333 },
// { key: 'two', value: { count: 2, sum: 8, avg: 4 },
// { key: 'three', value: { count: 1, sum: 3, avg: 3 } ]
`
* Example
* Table of Contents
* Installation
* NPM
* Bower
* Download
* CDN
* Getting Help
* Accessor Functions
* Aggregations
* Standard aggregations
* reductio.count()
* reductio.sum(value)
* reductio.avg(boolean|value)
* reductio.min(boolean|value)
* reductio.max(boolean|value)
* reductio.median(boolean|value)
* reductio.sumOfSq(value)
* reductio.std(boolean|value)
* Histogram
* reductio.histogramBins(thresholdArray)
* reductio.histogramValue(value)
* reductio.value(propertyName)
* reductio.filter(filterFn)
* reductio.nest(keyAccessorArray)
* reductio.alias(mapping)
* reductio.aliasProp(mapping)
* reductio.valueList(accessor)
* reductio.dataList(boolean)
* reductio.custom(initial,add,remove)
* Exception aggregation
* reductio.exception(accessor)
* reductio.exceptionCount(boolean)
* reductio.exceptionSum(value)
* groupAll aggregations
* reductio.groupAll(groupingFunction)
* Chaining aggregations
* Post-processing
* group.post().cap(length)
* Utilities
* reductio().fromObject(parameters)
* reductio().toObject()
`shell`
npm install --save-dev reductio
`shell`
bower install --save-dev reductio
If something doesn't appear to be working or you're having trouble with implementing something, it will usually be best to ask a question on Stackoverflow and tag the question with the reductio tag. When you ask your question, it is best to put together a working example showing the problem you are having. A JSFiddle template is available that already includes the Reductio and Crossfilter libraries, as well as dc.js. Use that template to make an example showing what you are seeing. In your question, reference the example, explain what you are seeing, and explain what you expect or want to see instead.
There is also a Reductio Gitter, which is a good place to check in for help if you have a quick question that doesn't fit into the Stackoverflow format, or if you are looking for a more conceptual discussion.
In most cases when an accessor function is required, Reductio supports the use of the property name to be accessed in the form or a string instead. When appropriate, Reductio will even cast the value of a property to a number for you, though be aware that this will convert nulls and undefined values into 0s.
For example, the following:
`javascript`
reducer = reductio().sum(function(d) { return +d.number; });
reducer(group);
Is equivalent to:
`javascript`
reducer = reductio().sum('number');
reducer(group);
Aggregations that support this syntax with casting to a numeric value: sum, avg, exception sum, histogram value, min, max, median
Aggregations that support this syntax without casting: nest, exception, value list, standard deviation, sum of squares
Aggregations are composable (so you can track more than one aggregation on a given group) and may depend on each other (the 'avg' aggregation requires that 'count' and 'sum' be specified).
`javascript`
var data = crossfilter([...]);
var dim = data.dimension(...);
var group = dim.group();
var reducer;
group.reduceCount()`.`javascript
reducer = reductio().count(true);
reducer(group);
`Stored under the 'count' property of groups. The value will be a count of every record that matches the group accessor.
reductio.sum(value)
Works the same was as Crossfilter's standard group.reduceSum()`.`javascript
reducer = reductio().sum(function(d) { return +d.number; });
reducer(group);
`Stored under the 'sum' property of groups. The value is a sum of
accessor(d)` for every record `d` that matches the group accessor. The accessor function must return a number.reductio.avg(boolean|value)
`javascript
reductio().avg(function(d) { return +d.number; })(group);
`Stored under the 'avg' property of groups. Boolean variation depends on count and sum aggregations being specified. If an accessor function is provided, that function will be used to create a sum aggregation on the group, and a count aggregation will be created as well. The value on the 'avg' property is equal to sum/count for the group.
reductio.min(boolean|value)
reductio.max(boolean|value)
reductio.median(boolean|value)
javascript
reductio().min(function(d) { return +d.number; })
.max(true)
.median(true)(group);
`Stored under the 'median', 'min', and 'max' property of groups.
Once you've defined one accessor function for min, max, or median (or if you have explicitly defined a
redectio.valueList(value)`) it will be used by the others. This avoids warning messages about overwriting the valueList.reductio.sumOfSq(value)
`javascript
reductio().sumOfSq(function(d) { return d.number; })(group);
`Stored under the 'sumOfSq' property of the group. Defined as the square of the value returned by the accessor function summed over all records in the group. This is used in the standard deviation aggregation, but can be used on its own as well.
reductio.std(boolean|value)
javascript
reductio().sumOfSq(function(d) { return d.number; })
.sum(function(d) { return d.number; })
.count(true)
.std(true)(group);
reductio()
.std(function(d) { return d.number; })(group);
`Stored under the 'std' property of the group. Defined as the sum-of-squares minus the average of the square of sums for all records in the group. In other words, for group 'g',
g.sumOfSq - g.sum*g.sum/g.count`.If
`sumOfSq`, `sum`, and `count` are already defined, takes a boolean. Otherwise pass in an accessor function directly.Histogram
`javascript
reductio().histogramBins([0,2,6,10])
.histogramValue(function(d) { return +d.number; })(group)
`Histogram of values within grouping, stored on the 'histogram' property of the group. Acts like d3.layout.histogram defined using bins(thresholds).
This grouping should be usable anywhere d3.layout.histogram can be used. May be useful for small-multiples charts, or for use with the dc.js stack mixin.
The property
group.histogram` is an array. Each element of the array is a sorted array of values returned by `histogramValue` that fall into that bin. Each element of the array also has properties, x, dx, and y, as defined in the d3.layout.histogram documentation.reductio.histogramBins(thresholdArray)
Defines the bin thresholds for the histogram. Will result in `thresholdArray.length - 1` bins.reductio.histogramValue(value)
Accessor for the value to be binned.reductio.value(propertyName)
`javascript
var reducer = reductio();
reducer.value("x").sum(xSumAccessor);
reducer.value("y").count(true).sum(ySumAccessor).avg(true);
reducer(group);
`Allows group structures such as
javascript
{
x: { sum: 5 }
y: { count: 3, sum: 12, avg: 4 }
}
`Used for tracking multiple aggregations on a single group. For example, sum of x and sum of y. Useful for visualizations like scatter-plots where individual marks represent multiple dimensions in the data.
propertyName` must be a valid Javascript object property name and must not conflict with any of the property names already used by Reductio (i.e. `count`, `sum`, `avg`, etc.).As many values as desired can be defined and any aggregation in this list can be defined on a value and they are calculated independently from aggregations defined on other values or on the base level of the group. Therefore, values are the way to deal with scenarios in which you want to calculated the same aggregation twice with difference logic (sum credits and sum debits) or where you are struggling with aggregations that are interdependent (average and sum are linked, so if you want to average a different value than you are summing, use a value).
Note that exception aggregations are supported on values, but groupAll aggregations are not.
A more comprehensive example:
`javascript
var reducer = reductio();
reducer.value("w").exception(function(d) { return d.bar; }).exceptionSum(function(d) { return d.num1; });
reducer.value("x").exception(function(d) { return d.bar; }).exceptionSum(function(d) { return d.num2; });
reducer.value("y").sum(function(d) { return d.num3; });
reducer.value("z").sum(function(d) { return d.num4; });
reducer(group);
`Will result in groups that look like
javascript
{ key: groupKey, value: {
w: { exceptionSum: 2 },
x: { exceptionSum: 3 },
y: { sum: 4 },
z: { sum: 2 }
}}
`reductio.filter(filterFn)
javascript
reductio().filter(filterFn)(group)
`Limit the values that can be added to / removed from groups. Works well with
value`
chains, and is also very useful if you need to aggregate sparsely-populated fields.javascript
var reducer = reductio();
reducer.value("evens")
.count(true)
.filter(function(d) { return d.num % 2 === 0; });
reducer.value("rare")
.filter(function(d) { return typeof d.rareVal !== 'undefined'; })
.sum(function(d) { return d.rareVal; });
reducer(group);
`For example:
javascript
// Given:
[
{ foo: 'one', num: 1 },
{ foo: 'two', num: 2 },
{ foo: 'three', num: 3, rareVal: 98 },
{ foo: 'one', num: 3, rareVal: 99 },
{ foo: 'one', num: 4, rareVal: 100 },
{ foo: 'two', num: 6 }
]// The groups will look like:
[
{key: "one", value: {evens: {count: 1}, rare: {sum: 199}}},
{key: "two", value: {evens: {count: 2}, rare: {sum: 0 }}},
{key: "three", value: {evens: {count: 0}, rare: {sum: 98 }}}
]
`
reductio.nest(keyAccessorArray)
javascript
reductio().nest([keyAccessor1, keyAccessor2])(group)
`Stored under the 'nest' property of the group.
Provides a result similar to
d3.nest().key(keyAccessor1).key(keyAccessor2)` when applied to the records in the group.Usually you'll want to use the group key as the first level of nesting, then use this to accomplish sub-group nesting.
Note that leaves will not be created when there is no record with that value in the branch. However, once a leaf is created it is not removed, so there is the possibility of leaves with empty 'values' arrays.
reductio.alias(mapping)
`javascript
reductio().count(true).alias({ newCount: function(g) { return g.count; } });
`Allows definition of an accessor function of any name on the group that returns a value from the group.
mapping` is an object where keys are the new properties that will be added to the group and values are the accessor functions that get the required values from the group.On the group, we can then call the following function to retrieve the new count value.
`javascript
group.top(1)[0].newCount();
`This approach to aliases is more efficient than the aliasProp approach below because it executes no logic at the time off aggregation.
reductio.aliasProp(mapping)
javascript
reductio().count(true)
.sum(function(d) { return +d.num; })
.aliasProp({
newCount: function(g) { return g.count; },
average: function(g) { return g.sum / g.count; },
description: function (g, v) { return v.desc; }
});
`Allows definition of an accessor function of any name on the group that returns a value from the group.
mapping` is an object where keys are the new properties that will be added to the group and values are the values returned by the accessor function.Accessors also have access to the record, so you can use this function to do things like assigning additional descriptive information to the group property based on the records seen.
On the group, we can then call the following to retrieve the count value.
`javascript
group.top(1)[0].newCount;
group.top(1)[0].average;
group.top(1)[0].description;
`It is very important that the functions in the _mapping_ don't modify the group directly. The functions are run after all aggregations are calculated and the same function is run for adding and removing records. Because the accessor functions are run on the group every time a record is added or removed, this is less efficient than the function-based approach in reductio.alias above.
reductio.valueList(accessor)
javascript
var reducer = reductio()
.sum(function (d) { return d.bar; })
.valueList(function (d) { return d.bar; });
`Maintains a
property on the group containing an array of values returned by accessor for every record added to the group. This property is used internally by other aggregations like min, max, and median, so watch for warning messages on the console.reductio.dataList(boolean)
`javascript
var reducer = reductio()
.sum(function (d) { return d.bar; })
.dataList(true);
`Maintains a
property on the group containing an array of records included in the group. This is similar to valueList used with the identity function as the accessor, but is slightly more efficient.reductio.custom(initial,add,remove)
`javascript
var reducer = reductio()
.custom({
add: function(p, v) {
p.myProp += v.foo.length;
return p;
},
remove: function(p, v) {
p.myProp -= v.foo.length;
return p;
},
initial: function(p) {
p.myProp = 0;
return p;
}
});
`Applies specified custom reducer to every record of every group.
This property allows to plug any custom extension in Reductio's API.
Exception aggregation
We also support exception aggregation. For our purposes, this means only aggregating once for each unique value that the exception accessor returns. So:javascript
var data = crossfilter([
{ foo: 'one', bar: 'A', num: 1 },
{ foo: 'two', bar: 'B', num: 2 },
{ foo: 'three', bar: 'A', num: 3 },
{ foo: 'one', bar: 'B', num: 2 },
{ foo: 'one', bar: 'A', num: 1 },
{ foo: 'two', bar: 'B', num: 2 },
]);var dim = data.dimension(function(d) { return d.foo; });
var group = dim.group();
var reducer = reductio()
.exception(function(d) { return d.bar; })
.exceptionCount(true)
.exceptionSum(function(d) { return d.num; });
reducer(group);
group.top(Infinity);
// [ { key: 'one', value: { exceptionCount: 2, exceptionSum: 3 }, // 'bar' dimension has 2 values: 'A' and 'B'.
// { key: 'two', value: { exceptionCount: 1, exceptionSum: 2 }, // 'bar' dimension has 1 value: 'B'.
// { key: 'three', value: { exceptionCount: 1 , exceptionSum: 3} ] // 'bar' dimension has 1 value: 'A'.
`Right now we support exceptionCount and exceptionSum, but it might also make sense to support other types of dependent aggregation. These types of aggregations are meant to help with a situation where you want to use Crossfilter on a flattened one-to-many or many-to-many relational model, which can result in redundant values.
reductio.exception(accessor)
The exception accessor defines the value by which to restrict the calculation of the exception aggregation. In each group, only the first record with each unique value returned by this accessor will be considered for aggregation.
reductio.exceptionCount(boolean)
A count subject to exception calculation.
reductio.exceptionSum(value)
A sum subject to exception calculation. Make sure that for each value within a group that the exception accessor returns, the exceptionSum accessor returns an identical value, or results will be unpredictable because the record added for each exception value will not necessarily be the same record that is removed.
groupAll aggregations
Sometimes it is necessary to include one record in multiple groups. This is common in OLAP scenarios, classification, and tracking moving averages, to give a few examples. Say we have a data set like
javascript
[
{ foo: 'one', num: 1 },
{ foo: 'two', num: 2 },
{ foo: 'three', num: 2 },
{ foo: 'one', num: 3 },
{ foo: 'one', num: 4 },
{ foo: 'two', num: 5 },
]
`We want to track a moving count of the last 2 values on the
num` property. So our group with a key `2` should count up all records with a `num` of `2` or `1`. Normally this must be done using the Crossfilter dimension.groupAll method. With reductio we can use all the standard reductio reducers in this type of scenario by specifying some additional groupAll information and called the reducer on the output of `dimension.groupAll` instead of the output of `dimension.group`.The object returned by
`dimension.groupAll` in Crossfilter does not have the standard `all`, `top`, or `order` methods. As a convenience, the reducer function produced by Reductio adds an `all` method to this object if it does not already exist.reductio.groupAll(groupingFunction)
Takes a single argument: a function that takes a record from the data set (e.g.
`{ foo: 'three', num: 2 }`) and returns an array of keys of the groups that the record should be included in (e.g. `[2,3]`). This is a very simple example, but the same thing could be done for dates, with a function for a 5-day moving average returning an array of 5 dates.`javascript
data.dimension(function(d) { return d.num; });
filterDim = data.dimension(function(d) { return d.foo; });
groupAll = dim.groupAll();reducer = reductio()
.groupAll(function(record) {
if(record.num === 5) {
return [5];
} else {
return [record.num, record.num+1];
}
})
.count(true);
reducer(groupAll);
`Chaining aggregations
As seen above, aggregations can be chained on a given instance of reductio. For example:javascript
reductio().count(true)
.sum(function(d) { return d.bar; })
.avg(true)(group);
`Post-processing
Reductio adds a function to your group. Calling this function returns an object on which you can make settings to allow you to post-process your data in useful ways.group.post().cap(length)
Cap the output of your group so that it never exceeds length elements.The last element will be an aggregation of the rest of the elements. It might be wise to set your groups
order method on the Crossfilter group before using this.`javascript
reductio()
.count(true)
.sum('foo')
.avg(true)(group);
group.post().cap(4)().length // 4 or less
`Utilities
reductio().fromObject(parameters)
A utility that will allow you to assign directly to the inner object from which reductio creates its groupings.
Basic use:
js
reductio()
.fromObject({
sum: function(d){
return d.foo;
}
})(group);
``