compression of key-value data

if your data looks like this:

``js
var data = {
bedfordshire: 'England',
aberdeenshire: 'Scotland',
buckinghamshire: 'England',
argyllshire: 'Scotland',
bambridgeshire: 'England',
cheshire: 'England',
ayrshire: 'Scotland',
banffshire: 'Scotland'
}
`

you can compress it like this:

`js
import { pack } from 'efrt'
var str = pack(data)
//'England:b0che1;ambridge0edford0uckingham0;shire|Scotland:a0banff1;berdeen0rgyll0yr0;shire'
`

then \_very!\_ quickly flip it back into:

`js
import { unpack } from 'efrt'
var obj = unpack(str)
obj['bedfordshire'] //'England'
`

Yep,

efrt packs category-type data into a _very compressed prefix trie_ format, so that redundancies in the data are shared, and nothing is repeated.

By doing this clever-stuff ahead-of-time, efrt lets you ship _much more_ data to the client-side, without hassle or overhead.

The whole library is 8kb, the unpack half is barely 2kb.

it is based on:

- 😍 tamper by the NYTimes
- 💝 lookups by Mike Koss,
- 💓 bits.js by Steve Hanov

Benchmarks!


Demo!



Basically,


- get a js object into very compact form
- reduce filesize/bandwidth a bunch
- ensure the unpacking time is negligible
- keep word-lookups on critical-path

`js
import { pack, unpack } from 'efrt' // const {pack, unpack} = require('efrt')

var foods = {
strawberry: 'fruit',
blueberry: 'fruit',
blackberry: 'fruit',
tomato: ['fruit', 'vegetable'],
cucumber: 'vegetable',
pepper: 'vegetable'
}
var str = pack(foods)
//'{"fruit":"bl0straw1tomato;ack0ue0;berry","vegetable":"cucumb0pepp0tomato;er"}'

var obj = unpack(str)
console.log(obj.tomato)
//['fruit', 'vegetable']
`

---


or, an Array:


if you pass it an array of strings, it just creates an object with true values:

`js
const data = [
'january',
'february',
'april',
'june',
'july',
'august',
'september',
'october',
'november',
'december'
]
const packd = pack(data)
// true¦a6dec4febr3j1ma0nov4octo5sept4;rch,y;an1u0;ly,ne;uary;em0;ber;pril,ugust
const sameArray = Object.keys(unpack(packd))
// same thing !
`

Reserved characters

the keys of the object are normalized. Spaces/unicode are good, but numbers, case-sensitivity, and _some punctuation_ (semicolon, comma, exclamation-mark) are not (yet) supported.

`js
specialChars = new RegExp('[0-9A-Z,;!:|¦]')
`

_efrt_ is built-for, and used heavily in compromise, to expand the amount of data it can ship onto the client-side.
If you find another use for efrt, please drop us a line🎈

Performance

_efrt_ is tuned to be very quick to unzip. It is O(1) to lookup. Packing-up the data is the slowest part, which is usually fine:

`js
var compressed = pack(skateboarders) //1k words (on a macbook)
var trie = unpack(compressed)
// unpacking-step: 5.1ms

trie.hasOwnProperty('tony hawk')
// cached-lookup: 0.02ms
`

Size

efrt will pack filesize down as much as possible, depending upon the redundancy of the prefixes/suffixes in the words, and the size of the list.

- list of countries - 1.5k -> 0.8k _(46% compressed)_
- all adverbs in wordnet - 58k -> 24k _(58% compressed)_
- all adjectives in wordnet - 265k -> 99k _(62% compressed)_
- all nouns in wordnet - 1,775k -> 692k _(61% compressed)_

but there are some things to consider:

- bigger files compress further (see 🎈 birthday problem)
- using efrt will reduce gains from gzip compression, which most webservers quietly use
- english is more suffix-redundant than prefix-redundant, so non-english words may benefit from other styles

Assuming your data has a low _category-to-data ratio_, you will hit-breakeven with at about 250 keys. If your data is in the thousands, you can very be confident about saving your users some considerable bandwidth.

Use

IE9+

`html


`

if you're doing the second step in the client, you can load just the CJS unpack-half of the library(~3k):

`js
const unpack = require('efrt/unpack') // node/cjs
`

`html


``

Thanks to John Resig for his fun trie-compression post on his blog, and Wiktor Jakubczyc for his performance analysis work

MIT