A promise besed paralel tool for uploading nodejs streams to s3.
npm install s3-upload-streamss3-upload-streams provides a thin wrapper on top of aws-sdk that allows multiple concurrent uploads of nodejs.stream.Readable streams to S3.
* a standard way for manipulating data streams
* an improved control flow using promises
* multiple concurrent uploads
* no prior knowledge of data size required
The uploader accepts a readable block stream as a parameter that enables the utlization of the full power of the streaming interface such as on-the-fly compression and encryption of data using the standard crypto and zlib modules. Flow and object mode are not supported, however, piping to a PassThrough stream provides a nice workaround. For example:
```
objectStream.pipe(passThroughStream)
npm i --save s3-upload-streams
let Uploader = require('s3-upload-streams');
let someFilePath = 'foo.txt';
let s3Uploader = new Uploader(s3, bucket, partSize, maxConcurentUploads);
let stream = fs.createReadStream(someFilePath);
let uploadIdPromise = s3Uploader.startUpload({ Key: 'Amazon S3 Object Key' }, stream, { orginalPath: someFilePath }); stream.on('end', () => {
uploadIdPromise
.then(uploadId => s3Uploader.completeUpload(uploadId))
.then((metadata) => {
console.log(
Uploaded ${metadata.additionalMetadata.orginalPath} to ${metadata.location});
currentUploads = currentUploads - 1;
tryNext();
})
.catch(err => console.log(err));
});
``See more example code here
s3 - An instance of an Amazon aws-sdk S3 object.
bucket - Name of the default S3 bucket for the specific S3 uploader. Can be overwritten by using the s3Params parameter of the startUpload method.
partSize - The size of the data chunk that will be read from the stream and will be passed to the Amazon uploader. Use this to optimize upload speed and memory usage.
concurrency - Maximum number of concurrent Amazon uploads.
readable - An instance of nodes Readable, Duplex or Transform stream. Be aware that the stream must be paused and not in object mode.
additionalMetadata - Metadata that will be passed back to the caller when the current upload is complete.
partialUploadParams - A key-value set of parameters for manipulation of partially uploaded objects not initiated by this s3Uploader. Available parameters are:
UploadId - UploadId of an already started multipartUpload
Parts - List of already uploaded parts of this object. The uploader will use this list when it completes the upload. See Advanced Usage section for more information.
location - URL of this object.
bucket - The S3 Bucket.
key - The key in the S3.
etag - Entity tag of the object.
additionalMetadata - The metadata passed in startUpload.
size - Size in bytes.
The following methods are designed in the case when a single S3Object is uploaded by multiple uploaders.
id - id of the current upload returned by startUpload method.
id - id of the current upload returned by startUpload method.