UltraDb

Probably the fastest node database in the world, blazing simple, uses nodejs n-api interface, designed to work in node clusters. It is also very low level database, so inappropriate handling might lead to data corruption. Read documentation carefully before you use it.

Works on 64 bit linux platforms. Other platforms not tested.

What for is this another database?

Some examples of usage:
* logging data
* storing sessions, large numger of documents
* building custom database data structures (trees, chain lists, ...)
* microservices
* services
* storing training data of ai, ann, ... storing trained results...
* building more advanced databases with ultradb as their backend
* everywhere where you need document ID without additional querying for it (you receive document ID always as a result of adding document to database)

It:
* is small - arround 300kB
* is written in C for node js intentionally - it is not a wrapper, or port of another database
* is multicore processors friendly - scales up with node processes, multiple node instances/forks can work simultaneously on same database file
* is pretty simple and straight forward - not intended to hide any kind of abstraction from you
* supports text, integers, floats and data buffers
* supports cursors that allows to iterate through documents starting from newest and allows paging
* is VERY FAST, try it on SSD drive, compare with other databases, tell me the results - I'm sure you'll be pleased when you'll compare it with other databases for node js
* is Binary JSON (BSON) and other binary formats friendly

If performance and/or costs of your infrastructure is for you a key factor you should consider use of Ultra DB. It is simple to use library, but requires little more attention from developer and more extensive testing, since there is not much control over what you do with database (for example there is no guarantee that database will trigger an error when trying to update document with non existing ID within a range of IDs in database, more over it might corrupt database). But in return you get a database, that runs within node process (no extra process, no extra overheads for TCP/IP stack, no waisteless copying data, no parsing queries, no switching to additional db tasks, no callbacks and promisses overheads ... and so on). With very little effort you can make pretty advanced services serving many thousands requests a second on a single cheap VPS machine.

Help highly appreciated! :-)

Feel free to drop a line of code, write tests, prepare documentation, tutorials, guidelines, come up with ideas and so on!

Important notes

It is unsafe to allow access to documents by their id from any kind of public api (unless all documents are fixed size and you control modulo of document ID). In case of public API additional step of document id validation should be taken (combining document ID and random key saved in the document as public document ID, accessing via another fixed size documents lookup database, hashing id or hasing id combined with kind of checksum ... and so on). Current API profides set of document ID checksum functions (8 - 32 bit) and unique key for checksums for each database file - this will be helpfull to prevent accessing non existing documents.

If document id's are used internally only and you have full control of how are they used, then you don't have to worry about above.

If you need full ACID database you have to design data and recovery mechanisms yourself. But if you only add new records and read them, you're safe, you might consider this database kinda of ACID in that case.

If you're going to use db in nodejs cluster make sure you use transactions where it needs to be synchronized across processes.

Since transactions are blocking other processes that want to use particular database file/files it is a good idea to split data to databases for different document types each.

Newly added documents are hidden till their content is filled into a database.

64 bit integers currently are actually limited to 53 bits. This is due javascript limitation for numbers, not database itself.

Before you start, check examples located in "tests" directory on github repo:
https://github.com/DanielMazurkiewicz/ultradb/tree/master/tests

Database file structure

...


Document ID points indirectly to address of most significant byte of document length. Most significant byte contains additional 3 bit information:
 * most significant bit if set to 1 means that document is hidden
 * second and third most significant bit encodes document length field size:
    - 00 : 1 byte
    - 01 : 2 bytes
    - 10 : 4 bytes
    - 11 : 8 bytes
Since Document ID is indirect address it is technically possible to remove phisically documents from the beginning of database while keeping same Document IDs.

Usage
Load database library

`const UltraDB = require('ultradb');`

`Open database`

`const testDb = UltraDB('./myDatabase.udb');

This will open database.

`const test1Db = UltraDB('./myDatabase.udb, 2048');

This will open database with specified page size, default is 16384. Page size has an impact on performance, the bigger the better performance. The best if it is multiplication of expected average document size.

`Close database`

`testDb.close();`

Database doesn't require to be closed so in most of cases you can skip this, however, if you're going to open dynamically a lot of databases it is good practice to close the ones you're not using any more.

`Add document (utf8 string with zero at its end)`

`const documentId = testDb.addUtf8z("My frirst ultradb document");`

This will add new document to database, and if successfull, it will return its id;

`Get document (utf8 string with zero at its end)`

`const documentContent = testDb.getUtf8z(documentId);`

This will retrieve document from database.

If document is hidden it will return "null"

If document doesn't exist it will return "undefined"

Note that "documentId" parameter is examined only briefly, so providing appropriate value is on your responsibility. Providing inappropriate value might lead to strange behaviour.

Note that it is your responsibility to use appropriate method to read appropriate data type, database doesn't encode document type.

`Default documents data format`

`const testDb = UltraDB('./dbTestCursor.udb').default('Utf8z');`

Adds shorten names for methods containing given default string. For above example:`const documentId = testDb.add('some document');`will be equivalent to`const documentId = testDb.addUtf8z('some document');`

Method returns current database instance

`Checking if database contains documents`

`testDb.isEmpty();`Returns true if database is empty

`Getting lattest added document identifier`

`const documentId = testDb.last();`It returns "undefined" if there is no documents in database. This method skips all hidden documents. If you need to get latest document, including hidden ones use:`const documentId = testDb.lastOfAll();`

`Getting previously added document identifier`

`const previousDocumentId = testDb.previous(documentId);`It returns "undefined" if there is no previous document to given one This method skips all hidden documents. If you need to get previous document, including hidden ones use:`const previousDocumentId = testDb.previousOfAll(documentId);`

Example: Iterate through all documents`for (let id = testDb.last(); id != undefined; id = testDb.previous(id)) { console.log(testDb.get(id)); }`

Note that "documentId" parameter is examined only briefly, so providing appropriate value is on your responsibility. Providing inappropriate value might lead to strange behaviour.

`Rolling database`

`testDb.roll();`

Writes new documents from begining of database file (with a guarantee of assigning non existing yet new document id's for new documents). It replaces oldest added documents in database.

`Hiding document`

`testDb.hide(documentId);`

Note that "documentId" parameter is examined only briefly, so providing appropriate value is on your responsibility. Providing inappropriate value might lead to strange behaviour or database corruption.

`Unhiding document`

`testDb.unhide(documentId);`

`Database access synchronization`

`testDb.start();

//code that needs to be run exclusively on database so other processes have to wait

testDb.stop();`

Method "start" informs that transaction has started, and method "stop" informs that transaction has ended.

Note that simple adding new documents to database and reading them is process safe, so there is no need to use transactions. Same if your node application is single process accessing particular database.

`$3`

`const testDb1 = UltraDb('./testdb1.udb'); const testDb2 = UltraDb('./testdb2.udb'); const testDb3 = UltraDb('./testdb3.udb'); const multiTransaction123 = UltraDB.prepareMultiTransaction(testDb1, testDb2, testDb3);

// ...

UltraDB.start(multiTransaction123);

//code that needs to be run exclusively on given set of databases

UltraDB.stop(multiTransaction123);`

Using this methods prevents cross transaction lock while transaction on multiple databases.

`Full API list`

(methods with skipped description simply have same functionality as previously described, but on different size of data)

`Document`


$3

creates empty hidden document and returns documentInfo reference
$3

creates empty document and returns documentInfo reference
$3

returns documentInfo reference for given documentId
$3

returns documentInfo reference for given documentId for both, visible and hidden documents
$3

returns id of document based on given documentInfo reference

Document ID obsfucation

$3

creates key which is a Buffer type containing instructions for encrypting/decrypting documentID, bits defines expected maximum size of documentID (6-64 bits), rounds defines ciphering complexity - the higher number the harder to crack (0 gives the weakest ciphering), randomData - at least 64 bytes of random buffer data has to be provided in order to be able to generate key - the more rounds the more data should be provided (approximatly each round should have additional 24 bytes of random data)
$3

encrypts given documentId with given key, returns Number which is a ciphered representation of documentId
$3

decrypts given encryptedDocumentID with given key, returns Number which is decrypted documentID

Database info and management

$3

returns true if database is empty
$3

rolls database, new documents will replace oldest documents in database
$3

allows to set top level document if someone need a document with some kind of settings, or keeping any kind of structures where entry document is required
$3

retrieves id of current root document
Document ID operations - 32 bit

$3

validates 32 bit checksum of document ID, provided indirectly by documentInfo, at given position
$3

sets 32 bit checksum for document provided by documentInfo reference, checksum is set at given position, but it is recommended for checksum not to be last element in the document.
Document ID operations - 16 bit

$3

$3
Document ID operations - 8 bit

$3

$3
Data operations - buffer in Buffer format

$3

adds new document with given buffer, returns id of created document, if length is -1 adds entire buffer, otherwise adds begining number of bytes from buffer and if length is bigger than buffer length, rest of space is left empty
$3

adds new document with given buffer, document is fixed length, remaining space after buffer is empty, writes only as many bytes as provided in "length" property (if -1 writes all of them, if bigger then length of buffer, then writes as much as available and rest remains empty), returns id of created document
$3

returns to buffer "length" number of bytes of document with given id, if length is -1 then reads all of document content
$3

sets/replaces buffer of document with given id, length points how many bytes to write (if -1 writes all of them, if bigger then length of buffer, then writes as much as available and rest remains unchanged)
$3

creates document with buffer and leaves begining of document empty, returns newly created document id, length points how many bytes to write (if -1 writes all of them, if bigger then length of buffer, then writes as much as available and rest remains empty)
$3

creates document with buffer placed in the middle of document, document is fixed size, returns newly created document id, length points how many bytes to write (if -1 writes all of them, if bigger then length of buffer, then writes as much as available and rest remains empty)
$3

returns buffer of document with given id at given position, if length is -1 then reads all content from given offset position
$3

sets/replaces buffer placed in the middle of document, length points how many bytes to write (if -1 writes all of them, if bigger then length of buffer, then writes as much as available and rest remains unchanged)
$3

returns buffer located at given position in document based on given documentInfo (see "Document API"), if length is -1 then reads all content from given offset position
$3

sets/replaces buffer located at given position in document based on given documentInfo, length points how many bytes to write (if -1 writes all of them, if bigger then length of buffer, then writes as much as available and rest remains unchanged)
Data operations - text in Utf8z format

$3

adds new document with given text, returns id of created document
$3

adds new document with given text, checks if it fits given maximum length fot text in utf8 format, document is fixed length, remaining space after text is empty. returns id of created document
$3

returns text of document with given id
$3

sets/replaces text of document with given id
$3

creates document with text and leaves begining of document empty, returns newly created document id
$3

creates document with text placed in the middle of document, document is fixed size, returns newly created document id
$3

returns text of document with given id at given position
$3

sets/replaces text placed in the middle of document
$3

returns text located at given position in document based on given documentInfo (see "Document API")
$3

sets/replaces text located at given position in document based on given documentInfo (see "Document API"), validates if it fits maximum length
Data operations - 64 bit float numbers

$3

adds new document with given number, returns id of created document
$3

adds new document with given number, document is fixed length, remaining space after number is empty. returns id of created document
$3

returns number of document with given id
$3

sets/replaces number of document with given id
$3

creates document with number and leaves begining of document empty, returns newly created document id
$3

creates document with number placed in the middle of document, document is fixed size, returns newly created document id
$3

returns number of document with given id at given position
$3

sets/replaces number placed in the middle of document
$3

returns number located at given position in document based on given documentInfo (see "Document API")
$3

sets/replaces number located at given position in document based on given documentInfo (see "Document API")
Data operations - 32 bit float numbers

$3

$3

$3

$3

$3

$3

$3

$3

$3

$3
Data operations - 64 bit unsigned integer numbers

$3

$3

$3

$3

$3

$3

$3

$3

$3

$3
Data operations - 48 bit unsigned integer numbers

$3

$3

$3

$3

$3

$3

$3

$3

$3

$3
Data operations - 32 bit unsigned integer numbers

$3

$3

$3

$3

$3

$3

$3

$3

$3

$3
Data operations - 24 bit unsigned integer numbers

$3

$3

$3

$3

$3

$3

$3

$3

$3

$3
Data operations - 16 bit unsigned integer numbers

$3

$3

$3

$3

$3

$3

$3

$3

$3

$3
Data operations - 8 bit unsigned integer numbers

$3

$3

$3

$3

$3

$3

$3

$3

$3

$3

Data operations - 64 bit signed integer numbers

$3

$3

$3

$3

$3

$3

$3

$3

$3

$3
Data operations - 48 bit signed integer numbers

$3

$3

$3

$3

$3

$3

$3

$3

$3

$3
Data operations - 32 bit signed integer numbers

$3

$3

$3

$3

$3

$3

$3

$3

$3

$3
Data operations - 24 bit signed integer numbers

$3

$3

$3

$3

$3

$3

$3

$3

$3

$3
Data operations - 16 bit signed integer numbers

$3

$3

$3

$3

$3

$3

$3

$3

$3

$3
Data operations - 8 bit signed integer numbers

$3

$3

$3

$3

$3

$3

$3

$3

$3

$3

Android termux prerequisites:

`pkg install python2 pkg install make pkg install clang``

Currently not working on Android due to Androids lack of support for sys/shm. :(

UltraDb

Works on 64 bit linux platforms. Other platforms not tested.

What for is this another database?

Help highly appreciated! :-)

Feel free to drop a line of code, write tests, prepare documentation, tutorials, guidelines, come up with ideas and so on!

Important notes

If document id's are used internally only and you have full control of how are they used, then you don't have to worry about above.

If you're going to use db in nodejs cluster make sure you use transactions where it needs to be synchronized across processes.

Since transactions are blocking other processes that want to use particular database file/files it is a good idea to split data to databases for different document types each.

Newly added documents are hidden till their content is filled into a database.

64 bit integers currently are actually limited to 53 bits. This is due javascript limitation for numbers, not database itself.

Before you start, check examples located in "tests" directory on github repo:
https://github.com/DanielMazurkiewicz/ultradb/tree/master/tests

Database file structure

...


Document ID points indirectly to address of most significant byte of document length. Most significant byte contains additional 3 bit information:
 * most significant bit if set to 1 means that document is hidden
 * second and third most significant bit encodes document length field size:
    - 00 : 1 byte
    - 01 : 2 bytes
    - 10 : 4 bytes
    - 11 : 8 bytes
Since Document ID is indirect address it is technically possible to remove phisically documents from the beginning of database while keeping same Document IDs.

Usage
Load database library

`const UltraDB = require('ultradb');`

`Open database`

`const testDb = UltraDB('./myDatabase.udb');

This will open database.

`const test1Db = UltraDB('./myDatabase.udb, 2048');

`Close database`

`testDb.close();`

`Add document (utf8 string with zero at its end)`

`const documentId = testDb.addUtf8z("My frirst ultradb document");`

This will add new document to database, and if successfull, it will return its id;

`Get document (utf8 string with zero at its end)`

`const documentContent = testDb.getUtf8z(documentId);`

This will retrieve document from database.

If document is hidden it will return "null"

If document doesn't exist it will return "undefined"

Note that "documentId" parameter is examined only briefly, so providing appropriate value is on your responsibility. Providing inappropriate value might lead to strange behaviour.

Note that it is your responsibility to use appropriate method to read appropriate data type, database doesn't encode document type.

`Default documents data format`

`const testDb = UltraDB('./dbTestCursor.udb').default('Utf8z');`

Method returns current database instance

`Checking if database contains documents`

`testDb.isEmpty();`Returns true if database is empty

`Getting lattest added document identifier`

`Getting previously added document identifier`

Example: Iterate through all documents`for (let id = testDb.last(); id != undefined; id = testDb.previous(id)) { console.log(testDb.get(id)); }`

Note that "documentId" parameter is examined only briefly, so providing appropriate value is on your responsibility. Providing inappropriate value might lead to strange behaviour.

`Rolling database`

`testDb.roll();`

Writes new documents from begining of database file (with a guarantee of assigning non existing yet new document id's for new documents). It replaces oldest added documents in database.

`Hiding document`

`testDb.hide(documentId);`

`Unhiding document`

`testDb.unhide(documentId);`

`Database access synchronization`

`testDb.start();

//code that needs to be run exclusively on database so other processes have to wait

testDb.stop();`

Method "start" informs that transaction has started, and method "stop" informs that transaction has ended.

`$3`

// ...

UltraDB.start(multiTransaction123);

//code that needs to be run exclusively on given set of databases

UltraDB.stop(multiTransaction123);`

Using this methods prevents cross transaction lock while transaction on multiple databases.

`Full API list`

(methods with skipped description simply have same functionality as previously described, but on different size of data)

`Document`


$3

creates empty hidden document and returns documentInfo reference
$3

creates empty document and returns documentInfo reference
$3

returns documentInfo reference for given documentId
$3

returns documentInfo reference for given documentId for both, visible and hidden documents
$3

returns id of document based on given documentInfo reference

Document ID obsfucation

$3

creates key which is a Buffer type containing instructions for encrypting/decrypting documentID, bits defines expected maximum size of documentID (6-64 bits), rounds defines ciphering complexity - the higher number the harder to crack (0 gives the weakest ciphering), randomData - at least 64 bytes of random buffer data has to be provided in order to be able to generate key - the more rounds the more data should be provided (approximatly each round should have additional 24 bytes of random data)
$3

encrypts given documentId with given key, returns Number which is a ciphered representation of documentId
$3

decrypts given encryptedDocumentID with given key, returns Number which is decrypted documentID

Database info and management

$3

returns true if database is empty
$3

rolls database, new documents will replace oldest documents in database
$3

allows to set top level document if someone need a document with some kind of settings, or keeping any kind of structures where entry document is required
$3

retrieves id of current root document
Document ID operations - 32 bit

$3

validates 32 bit checksum of document ID, provided indirectly by documentInfo, at given position
$3

sets 32 bit checksum for document provided by documentInfo reference, checksum is set at given position, but it is recommended for checksum not to be last element in the document.
Document ID operations - 16 bit

$3

$3
Document ID operations - 8 bit

$3

$3
Data operations - buffer in Buffer format

$3

adds new document with given buffer, returns id of created document, if length is -1 adds entire buffer, otherwise adds begining number of bytes from buffer and if length is bigger than buffer length, rest of space is left empty
$3

adds new document with given buffer, document is fixed length, remaining space after buffer is empty, writes only as many bytes as provided in "length" property (if -1 writes all of them, if bigger then length of buffer, then writes as much as available and rest remains empty), returns id of created document
$3

returns to buffer "length" number of bytes of document with given id, if length is -1 then reads all of document content
$3

sets/replaces buffer of document with given id, length points how many bytes to write (if -1 writes all of them, if bigger then length of buffer, then writes as much as available and rest remains unchanged)
$3

creates document with buffer and leaves begining of document empty, returns newly created document id, length points how many bytes to write (if -1 writes all of them, if bigger then length of buffer, then writes as much as available and rest remains empty)
$3

creates document with buffer placed in the middle of document, document is fixed size, returns newly created document id, length points how many bytes to write (if -1 writes all of them, if bigger then length of buffer, then writes as much as available and rest remains empty)
$3

returns buffer of document with given id at given position, if length is -1 then reads all content from given offset position
$3

sets/replaces buffer placed in the middle of document, length points how many bytes to write (if -1 writes all of them, if bigger then length of buffer, then writes as much as available and rest remains unchanged)
$3

returns buffer located at given position in document based on given documentInfo (see "Document API"), if length is -1 then reads all content from given offset position
$3

sets/replaces buffer located at given position in document based on given documentInfo, length points how many bytes to write (if -1 writes all of them, if bigger then length of buffer, then writes as much as available and rest remains unchanged)
Data operations - text in Utf8z format

$3

adds new document with given text, returns id of created document
$3

adds new document with given text, checks if it fits given maximum length fot text in utf8 format, document is fixed length, remaining space after text is empty. returns id of created document
$3

returns text of document with given id
$3

sets/replaces text of document with given id
$3

creates document with text and leaves begining of document empty, returns newly created document id
$3

creates document with text placed in the middle of document, document is fixed size, returns newly created document id
$3

returns text of document with given id at given position
$3

sets/replaces text placed in the middle of document
$3

returns text located at given position in document based on given documentInfo (see "Document API")
$3

sets/replaces text located at given position in document based on given documentInfo (see "Document API"), validates if it fits maximum length
Data operations - 64 bit float numbers

$3

adds new document with given number, returns id of created document
$3

adds new document with given number, document is fixed length, remaining space after number is empty. returns id of created document
$3

returns number of document with given id
$3

sets/replaces number of document with given id
$3

creates document with number and leaves begining of document empty, returns newly created document id
$3

creates document with number placed in the middle of document, document is fixed size, returns newly created document id
$3

returns number of document with given id at given position
$3

sets/replaces number placed in the middle of document
$3

returns number located at given position in document based on given documentInfo (see "Document API")
$3

sets/replaces number located at given position in document based on given documentInfo (see "Document API")
Data operations - 32 bit float numbers

$3

$3

$3

$3

$3

$3

$3

$3

$3

$3
Data operations - 64 bit unsigned integer numbers

$3

$3

$3

$3

$3

$3

$3

$3

$3

$3
Data operations - 48 bit unsigned integer numbers

$3

$3

$3

$3

$3

$3

$3

$3

$3

$3
Data operations - 32 bit unsigned integer numbers

$3

$3

$3

$3

$3

$3

$3

$3

$3

$3
Data operations - 24 bit unsigned integer numbers

$3

$3

$3

$3

$3

$3

$3

$3

$3

$3
Data operations - 16 bit unsigned integer numbers

$3

$3

$3

$3

$3

$3

$3

$3

$3

$3
Data operations - 8 bit unsigned integer numbers

$3

$3

$3

$3

$3

$3

$3

$3

$3

$3

Data operations - 64 bit signed integer numbers

$3

$3

$3

$3

$3

$3

$3

$3

$3

$3
Data operations - 48 bit signed integer numbers

$3

$3

$3

$3

$3

$3

$3

$3

$3

$3
Data operations - 32 bit signed integer numbers

$3

$3

$3

$3

$3

$3

$3

$3

$3

$3
Data operations - 24 bit signed integer numbers

$3

$3

$3

$3

$3

$3

$3

$3

$3

$3
Data operations - 16 bit signed integer numbers

$3

$3

$3

$3

$3

$3

$3

$3

$3

$3
Data operations - 8 bit signed integer numbers

$3

$3

$3

$3

$3

$3

$3

$3

$3

$3

Android termux prerequisites:

`pkg install python2 pkg install make pkg install clang``

Currently not working on Android due to Androids lack of support for sys/shm. :(