cuttlefish

A simple lowlevel synchronizing library for Joyent Manta.

USAGE

``javascript
var cuttlefish = require('cuttlefish');

// Minimal options:
var fishy = cuttlefish({
// The path on Manta where your stuff will be uploaded
path: '/isaacs/stor/path/on/manta/to/stuff',

// create your own manta client. See the Manta SDK.
client: myMantaClient,

// Pass a getter function to fetch the file stream.
request: function(filename, cb) {
// load up the stream, call cb(er, stream)
},

// the list of file objects to sync up to manta
files: {
// the object can contain optionally size, type, and/or md5
'filename.txt': {},

// paths with / will create the necessary dirs
'dir/a.txt': {,
size: 1234,
'md5': 'KwAEL3SBx7BWxLQQ0o8zzw==',
}
}
});

// With more optional options filled in
var fishy = cuttlefish({
// The path on Manta where your stuff will be uploaded
path: '/isaacs/stor/path/on/manta/to/stuff',

// create your own manta client. See the Manta SDK.
client: myMantaClient,

// Pass a getter function to fetch the file stream.
request: function(filename, cb) {
// load up the stream, call cb(er, stream)
},

// the list of file objects to sync up to manta
files: {
'filename.txt': {
// All fields for each file are optional
// will use md5 if provided, then size,
// then assume that all files must be written
size: 1234, // or content-length or length

type: 'text/plain', // or content_type, content-type, or type

// Multiple ways to specify md5, all work
// md5 is not necessary
'md5': 'KwAEL3SBx7BWxLQQ0o8zzw==',

// hex or base64 or buffer, totally ok
'md5': '2b00042f7481c7b056c4b410d28f33cf',
'md5': new Buffer('2b00042f7481c7b056c4b410d28f33cf', 'hex'),

// also supports this couchdb style 'digest' field
digest: 'md5-KwAEL3SBx7BWxLQQ0o8zzw==',

// optional headers for this one file when it gets mput'ed
// Note that content-type and content-md5 are not really
// necessary here.
headers: {
'access-control-allow-origin': '*',
'access-control-allow-methods': 'GET',
'x-fry-is': 'the greetest'
},
},
'sub/folder/file.txt': {
// if filename has slashes, then dirs will be made as needed
// if type is omitted, then manta will infer from extension, or
// use application/octet-stream by default
},
...
},

// optional headers that get sent along with EVERY put reqest to manta
headers: {
'access-control-allow-origin': '*',
'access-control-allow-methods': 'GET'
},

// Optionally delete files that are not in the list
// Default: delete=false
delete: true

// Optionally ONLY delete missing files, but don't send anything
// implies "delete"
onlyDelete: true,

// don't do ANY remote write operations, but act as if we did,
// emitting the same events etc. Default = false
dryRun: true
})

fishy.on('file', function(status, file, data) {
if (status === 'error')
console.error('%s failed: %s', file.name, data.stack)
else if (status === 'match')
console.error('%s already there', file.name)
else
console.error('%s ok!', file.name)
})

fishy.on('complete', function(error, data) {
if (error)
console.error('it didnt went well. first error was %s', error.stack)
else
console.log('ok! %d files uploaded', Object.keys(data).length)
})
`

Options

* client {MantaClient object} Required client for accessing Manta
* files {Object} The {:,...} hash. See below for
the fields that can be specified on each File object.
* path {String} The path on Manta where the stuf gets synced to
* request {Function} Function that gets a stream to send, if
appropriate
* concurrency {Number} The max number of tasks to be doing at any
one time. Default = 50.
* timeout {Number} Optional max amount of time to wait for any remote
task to complete, in ms. Default = Infinity
* headers {Object} Optional headers to send with every PUT
operation. Does not check for or overwrite headers on pre-existing
remote objects.
* delete {Boolean} Set to true to delete remote files that are not
found in the files hash. Default = false
* onlyDelete {Boolean} Set to true to only delete remote files
that are not found in the files hash, but do not send any new
files. Implies delete. Default = false
* dryRun {Boolean} Don't actually put or delete any files, but act
as if it would, performing the same length and MD5 comparisons etc.
* getMd5 {Function} Optionally provide a getter function to look up
md5 values only when necessary. This is handy if you have large
files, and don't want to look up md5 checksums unless necessary
because the file lengths match.
* timingDebug {Boolean} Optionally dump a bunch of timing info to
stderr. Defaults to false, obviously.

File Objects

Cuttlefish's file objects have the following fields. When you specify
one of the aliases, it'll be changed to the canonical name.

* md5 The md5 checksum of the file. Can be in Base64, Hex, or
Buffer format, or come with a md5- prefix. Aliases:
content-md5, computed-md5, digest
* size The length of the file in bytes. Aliases: length,
content-length, content_length, contentLength
* type The type of the file. Aliases: content-type,
contentType, content_type, mime-type, mime_type, mimeType
* headers Additional headers to pass to the Manta PUT operation.
Does not check against headers for pre-existing files.
* skip A boolean to say "do not send this file, even if the remote
does not have it, or the md5 doesn't match". This is useful if you
want to only remove certain missing files, but not all. A skipped
file will be emitted as a match if the remote has a file by that
name, without checking length or md5. If the remote does not have a
copy of the file, then it is still not sent, but is not emitted as a
match.
* name The key in the files hash. When file objects are cast to a
string, their name field is returned.
* mkdirs Boolean true, but only because the file is passed as an
argument to a Manta PUT operation.
* started Boolean false before the file is processed, true once
it starts.
* error Error object or null depending on whether the file
encountered an error.
* status Starts as null, but eventually changes to one of
'sent', 'match', or 'error'

Remote Objects

Remote objects will be either of the sort returned by Manta's ftw
operation, or returned by Manta's info operation if an md5 checksum
is provided and the ftw data does not contain it. Additionally,
they will have the following fields:

* status One of 'sent', 'match', 'error', or 'deleted'
* _path The full path of the remote object in Manta
* _remote The path relative to Cuttlefish's directory (corresponding
to the local file.name property)

Events

The cuttlefish object is an event emitter that emits the following
events.

$3

* error {Error object}

Emitted when there is a problem. This means something bad has
happened, which is probably unrecoverable. The error object may
have a file or task object attached with additional information.

$3

* results {Object} Collection of result information

Emitted when the sync operation is finished. The results object
contains as much information as cuttlefish has about all the remote
objects it saw, as well as the status of each remote object
('delete', 'sent', or 'match').

$3

* file {Object} An object representing the file that was processed
* status {String} Either the string 'sent' or 'match'
* remote {Object} An object representing the remote file

This is emitted whenever a local file is processed, to tell you that
either it was sent, or it was skipped because it matches the remote
file.

$3

* task {Object} The task being performed

This is emitted whenever a new async task is scheduled.

$3

* path {String} The remote path that is deleted (relative to
cuttlefish's root path)
* remote {Object} The remote object info

Emitted whenever a remote file or directory is deleted.

$3

* file {Object} The local file being sent
* result {Object} The results of the send operation

Emitted when a file is sent.

$3

* file {Obect} The local file that matches
* remote {Object} The remote data that it matches against

$3

* path {String} The remote path being walked

This is emitted when we're about to process the list of remote files.

It will usually be emitted. The only time it wouldn't be emitted is
if there's an error instead, or if the remote path doesn't exist (so
there's nothing to walk), or in the trivial case where we're not
sending any files and not deleting extra files and folders.

$3

* entry {Object} Remote object info

Emitted for each remote entry encountered in the ftw process.

$3

* remote {Object} The remote object info
* result {Object} The results of the unlink operation

Emitted when a remote object is unlinked.

$3

* remote {Object} The remote object info
* result {Object} The results of the rmr operation

Emitted when a remote directory is removed.

$3

* file {Object} The local file being queried for
* result {Object} The results of the info operation

Emitted when cuttlefish has to look up the detailed info about a
remote object. Currently, this is only done when it is necessary to
compare the md5 value.

FAQ

These questions may or may not be frequently asked, but I predict that
you might ask them, so here they are.

$3

No. You feed the cuttlefish a bunch of things that you want it to
sync. It just figures out what to sync, and then tells you when its
all done. Probably what you want is
manta-sync.

$3

Another way to approach this solution would have been to have a more
stream-like fishy.addFile(file) method, instead of requiring that
you provide all the file stat info up front.

However, that approach requires that an extra call be made for each
file to get the remote info, and a ftw at the end to clean up extra
files that need to be deleted. And, in order to handle deleting files
at all, you have to keep the names around anyway, which would
eventually hit a memory limit (albeit a much higher one).

That approach would require about twice as many HTTP calls, and an ftw
walk. If you have a small to medium number of files (ie, a million or
less), many of which are already present on Manta, and are typically
setting delete: true`, then cuttlefish's approach is much more
efficient.

A similarly efficient approach would be to require that you provide
another getter function to provide the file stat information, and then
a second getter to provide the file stream if needed, so that nothing
is stored in memory, and everything flows through, synchronizing
elegantly.

That's a much fancier lib with a more elaborate API, which should be
called thaumoctopus. If you have this use case, you should go write
it.

$3

The Joyent Manta service has a venerable tradition of naming things
after sea creatures. The cuttlefish is a little thing with tentacles
that stays on the bottom of the sea, and mirrors whatever it's placed
against. So it is a natural fit for a lowlevel synchronizing utility.