hyperbeedeebee

A MongoDB-like database built on top of Hyperbee with support for indexing

WIP: There may be breaking changes in the indexing before the v1.0.0 release, don't use this for anything you don't mind migrating in the future.

Based on this design

Usage

``
npm i --save hyperbeedeebee
`

`JavaScript
const Hyperbee = require('hyperbee')
// This module handles networking and storage of hypercores for you
const SDK = require('hyper-sdk')
const {DB} = require('hyperbeedeebee')

const {Hypercore} = await SDK()

// Initialize a hypercore for loading data
const core = new Hypercore('example')
// Initialize the Hyperbee you want to use for storing data and indexes
const bee = new Hyperbee(core)

// Create a new DB
const db = new DB(bee)

// Open up a collection of documents and insert a new document
const doc = await db.collection('example').insert({
hello: 'World!'
})

// doc._id gets set to an ObjectId if you don't specify it
console.log(doc)

// Iterate through data as it's loaded (streaming)
// Usually faster and more memory / CPU efficient
for await (let doc of db.collection('example').find({
clout: {
$gt: 9000
},
})) {
console.log(doc)
}

// Create an index for properties in documents
// This drastically speeds up queries and is necessary for sorting by fields
await db.collection('example').createIndex('createdAt')

// Get all results in an array
// Can skip some results and limit total for pagination
const killbots = await db.collection('example')
.find({type: 'killbot'})
.sort('createdAt', -1)
.skip(30)
.limit(100)

// Get a single document that matches the query
const eggbert = await db.collection('example').findOne({name: 'Eggbert'})
`

Data Types

HyperbeeDeeBee uses MongoDB's BSON data types for encoding data.
You can import the bson library bundled with HyperbeeDeeBee using the following code:

`JavaScript
const { BSON } = require('hyperbeedeebee')
`

From there you can access any of the following data types:

`JavaScript
Binary,
Code,
DBRef,
Decimal128,
Double,
Int32,
Long,
UUID,
Map,
MaxKey,
MinKey,
ObjectId,
BSONRegExp,
BSONSymbol,
Timestamp
`


Important Differences From MongoDB

- There is a single writer for a hyperbee and multiple readers
- The indexing means that readers only need to download small subsets of the full dataset (if you index intelligently)
- No way to do "projections" so keep in mind you're always downloading the full document to disk
- Subset of find() API is implemented, no Map Reduce API, no $or/$and since it's difficult to optimize
- You can only sort by indexed fields, otherwise there's no difference from loading all the data and sorting in memory
- Fully open source under AGPL-3.0 and with mostly MIT dependencies.

Indexing considerations:

Indexes are super important to make your applications snappy and to reduce the overall CPU/Bandwidth/Storage usage of queries.

- If you do a search by fields that aren't indexed, you'll end up downloading the full collection (this is potentially really slow)
- The order of fields in the index matters, they're used to create an ordered key based on the values
- If you want to sort by a field, make sure it's the first field in an index
- You can have indexed fields before the sorted field if they are only used for $eq operations, this is due to the database's ability to turn them into a prefix to speed up the search.
- If an index cannot be found to satisfy a sort the query will fail.
- If you're using $gt/$lt/$gte/$lte in your query, they will perform best if the same considerations as the sort are applied.
- If the fields in the index can be used to rule out a document as matching, then you can avoid loading more documents and doing fewer overall comparisons on data.
- If your field is a unicode string which has 0x00 bytes in it, then the sorting might break due to the way BSON serializes unicode strings. Proceed with caution!

API

$3

Initialize a new DB instance using a hyperbee.
Note that it's up to you to figure out how to replicate the hyperbee (for added flexibility).
You may want to look into using hyper-sdk since it works out of the box in both Node.js and web browsers.

$3

Get a reference to a Collection of documents within this hyperbee.
This is where you will store documents as well as perform queries on them.

$3

Close the hyperbee and clean up any file descriptors it opened.

$3

Manually creates a Collection from a hyperbee and the collection name.

$3

The name of this collection in the hyperbeedeebee.

$3

Inserts a document into the collection.
Documents can contain any BSON data and most JS data types are automatically translated to their coresponding BSON types (e.g. Array and Date).
If the document doesn't have a _id field, one will be generated for it.
The _id field must be an ObjectID.
If you want to update a document, do another insert over the same _id to overwrite the old document.

$3

Create an index for a set of fields.
This will go over all the documents in the collection and if they have the apropriate fields, will add them to the index.
Indexing fields is important if you want to sort based on a query, or want to take advantage of sparsely querying the dataset to speed things up.

$3

$3

$3

$3

Search for a document that matches a given query.
If you specify a _id you can find the document without needing to perform an actual search.

$3

You can also search through all documents for a particular query using a cursor.
Cursors are like a "query builder" where you can specify additional properties like sorting and skipping.

You can get all matching documents in a cursor with await cursor, or you can use for await(const doc of cursor) to asynchronously iterate through the documents one at a time.
Using the AsyncIterator feature of cursors is preferred so that you can speed up your searches and avoid loading too much data into memory.

Note that the cursor will attempt to use any indexes that are in your query (or the sort) to speed up performance.

$3

You can treat the cursor as a promise to resolve the set of all documents within it.
Note that every time you await the cursor, you're fetching the documents from the database since it isn't a "real" promise.

$3

You can iterate through documents that match your query one at a time by using the cursor as an AsyncIterable.
Note that every time you use it as an async iterable, you are performing a new search.
This method is important if you're expecting a very large set of results or want to display things to users as data becomes available.

$3

You can skip a number of results for pagination (useful with cursor.limit)

$3

You can limit how many documents you wish you fetch from the database.

$3

You can sort the documents by a field if there is an index that uses that field available.
The direction specifies whether the values should be incrementing (1), or decrementing (-1).
If you want to sort by a timestamp with the latest first, use -1.

$3

Hint at which database index the search should use.

$3

Count the number of documents that match this query. Note that this operation _does_ download the documents from peers.

$3

Update one or more documents in a collection that match a particular query (same query format as .find()).
You can specify that you want to update all documents that match using multi: true.
You can have the DB insert a document if it doesn't exist by specifying upsert:true.
You can specify a hint for which DB to use when searching for documents.
The update can either be a plain JavaScript object that maps which properties should be set, or it can be an Update object with properties that are documented below.
Note that order is not guaranteed if you specify several update operations that use the same key.
The update can also be an Array of Update objects in which case the operations will be applied in that order.

The return value contains fields for nMatched (number of documents that got matched in the search),
nModified (number of documents that got modified), and nUpserted (number of documents that got upserted if upsert: true was set in the options.

E.g.

`JavaScript
const {nModified} = await collection.update({
birthday: today
}, {
$inc: {age: 1}
}, {
multi: true
})

const {nUpserted} = await collection.update({
some_impossible_search: Infinity
}, {
hello: 'World!'
}, {
upsert: true
})
`

$3

Find fields that are equal to a specific value.

E.g.

`JavaScript
const docs = await collection.find({
name: 'Bob'
})

// Equivalent to
const docs = await collection.find({
name: {
$eq: 'Bob'
}
})
`

$3

You can query by values that are greather than or less than a given value.

E.g.

`JavaScript
const docs = await collection.find({
createdAt: {
$lte: new Date(),
$gt: new Date(2012, 01, 01)
}
}).sort('createdAt', -1)
`

$3

Check if a field is equal to one of a set of values in an array.
If the field is an array, it checks that the array contains a subset of the query array.

E.g.

`JavaScript
const docs = await collection.find({
tags: {
$in: ['cool', 'cats', 'spaghetti']
}
})
`

$3

Check if a field (which is an array) contains all the values within the query array.

E.g.

`JavaScript
const docs = await collection.find({
permissions: {
$all: ['read', 'write', 'create']
}
})
`

$3

Check if a field exists in a document.
Note that it's impossible to use indexes for $exists: false at the moment.

`JavaScript
const docs = await collection.find({
secret: {
$exists: false
}
})
`

$3

You can set a field in a document by specifying it.

Note that nested fields with . are not yet supported, and fields with $ at the start may conflict with other query parameters.

Effectively an alias for update.$set[field]

`JavaScript
// add the field hello to all documents in the collection
await collection.update({}, {
hello: 'world',
goodbye: 'space'
}, {multi:true})
`

$3

You can set a field in the document to a specific value using $set.

`JavaScript
// add the field hello to all documents in the collection
await collection.update({}, {
$set: {
hello: 'world',
something: 'else'
}
}, {multi:true})
`

$3

You can delete a field from a document using $unset.

The value of the query can be anything.

`JavaScript
await collection.update({}, {
$unset: {
honor: ''
}
}, {multi:true})
`

$3

You can rename fields in a document using $rename

Effectively it deletes the existing field and sets the newName field to the value of the old field.

`JavaScript
await collection.update({}, {
$rename: {
oldFieldName: 'newFieldName'
}
}, {multi:true})
`

$3

You can use $inc to specify fields that should be incremented.

The number is the amount to increment by.

You can set number to a negative number to decrement fields.

If the field is not set in the document, the field will be set to number.

`JavaScript
await collection.update({}, {
$inc: {
points: 1000
}
}, {multi:true})
`

$3

You can use $mult to specify fields that should be multiplied.

The number is the amount to multiply by.

If the field is not set in the document, the field will be set to number.

`JavaScript
await collection.update({}, {
$mult: {
hp: 2,
mp: 0.5
}
}, {multi:true})
`

$3

You can append to the end of an array using $push

`JavaScript
// Add adorable to all objects with tags containing cute
await collection.update({tags: 'cute'}, {
$push: {
tags: 'adorable'
}
}, {multi:true})
`

$3

You can append a value to an array if that array doesn't already contain the value.

Useful for avoiding duplication.

If the field is not in the document, it will be set to an array with the value.

`JavaScript
// Add adorable to all objects with tags containing cute
// Avoids adding it to thing that are already adorable
await collection.update({tags: 'cute'}, {
$addToSet: {
tags: 'adorable'
}
}, {multi:true})
`

You can use $each in the value to add a set of values.

`JavaScript
await collection.update({tags: 'cute'}, {
$addToSet: {
tags: {
$each: ['adorable', 'fluffy']
}
}
}, {multi:true})
`

$3

You can remove an element from the end or start of an array using $pop.

The direction must be either 1 or -1 where 1 removes from the end, and -1 removes from the start.

`JavaScript
await collection.update({}, {
$pop: {
fromTheBack: 1,
fromTheFront: -1
}
}, {multi: true})
`

$3

You can remove all elements that match a given query using $pull.

The query should match the queries used for field fields in .find().

`JavaScript
// Find everyone that is cool, and remove uncool and boring from their qualities array.
await collection.update({
isCool: true
}, {
$pull: {
qualities: {
$in: ['uncool', 'boring']
}
}
})
`

TODO:

- [x] Sketch up API
- [x] Insert (with BSON encoding)
- [x] Find all docs
- [x] Find by _id
- [x] Find by field eq (no index)
- [x] Find by array field includes
- [x] Find by number field $gt/$gte/$lt/$lte
- [x] Numbers
- [x] Dates
- [x] Find using $in operator
- [x] Find using $all operator
- [x] Find using $exists operator
- [x] Index fields
- [x] Sort by index (with find)
- [x] Indexed find by field $eq
- [x] Flatten array for indexes
- [x] Get field values from index key without getting the doc
- [x] Find on fields that aren't indexed
- [x] Indexed find for $exists
- [x] Indexed find by number field
- [x] Indexed find for $in
- [x] Indexed find for $all
- [x] Hint API (specify index to use)
- [ ] Delete documents (clean up indexed values for them)
- [ ] Test if iterators clean up properly
- [ ] More efficient support for $gt/$gte/$lt/$lte indexes
- [ ] More efficient support for $all indexes
- [ ] More efficient support for $in` indexes
- [ ] Detect when data isn't available from peers and emit an error of some sort instead of waiting indefinately.