safely create multiple ReadStream or WriteStream objects from the same file descriptor
npm install better-fd-slicerSafe fs.ReadStream and fs.WriteStream using the same fd.
Let's say that you want to perform a parallel upload of a file to a remote
server. To do this, we want to create multiple read streams. The first thing
you might think of is to use the {start: 0, end: 0} API offs.createReadStream. This gives you two choices:
0. Use the same file descriptor for all fs.ReadStream objects.
0. Open the file multiple times, resulting in a separate file descriptor
for each read stream.
Neither of these are acceptable options. The first one is a severe bug,
because the API docs for fs.write state:
> Note that it is unsafe to use fs.write multiple times on the same file
> without waiting for the callback. For this scenario, fs.createWriteStream
> is strongly recommended.
fs.createWriteStream will solve the problem if you only create one of them
for the file descriptor, but it will exhibit this unsafety if you create
multiple write streams per file descriptor.
The second option suffers from a race condition. For each additional time the
file is opened after the first, it is possible that the file is modified. So
in our parallel uploading example, we might upload a corrupt file that never
existed on the client's computer.
This module solves this problem by providing createReadStream andcreateWriteStream that operate on a shared file descriptor and provides
the convenient stream API while still allowing slicing and dicing.
This module also gives you some additional power that the builtinfs.createWriteStream do not give you. These features are:
* Emitting a 'progress' event on write.
* Ability to set a maximum size and emit an error if this size is exceeded.
* Ability to create an FdSlicer instance from a Buffer. This enables you
to provide API for handling files as well as buffers using the same API.
``js
import {createFromFd} from 'better-fd-slicer';
import fs from 'node:fs';
fs.open("file.txt", 'r', function (err, fd) {
if (err) throw err;
const slicer = createFromFd(fd);
const firstPart = slicer.createReadStream({start: 0, end: 100});
const secondPart = slicer.createReadStream({start: 100});
const firstOut = fs.createWriteStream("first.txt");
const secondOut = fs.createWriteStream("second.txt");
firstPart.pipe(firstOut);
secondPart.pipe(secondOut);
});
`
You can also create from a buffer:
`js
import {createFromBuffer} from 'better-fd-slicer';
import fs from 'node:fs';
const slicer = createFromBuffer(someBuffer);
const firstPart = slicer.createReadStream({start: 0, end: 100});
const secondPart = slicer.createReadStream({start: 100});
const firstOut = fs.createWriteStream("first.txt");
const secondOut = fs.createWriteStream("second.txt");
firstPart.pipe(firstOut);
secondPart.pipe(secondOut);
`
`js
import {createFromFd} from 'better-fd-slicer';
fs.open("file.txt", 'r', function (err, fd) {
if (err) throw err;
const slicer = createFromFd(fd);
// ...
});
`
Make sure fd is a properly initialized file descriptor. If you want tocreateReadStream
use make sure you open it for reading and if you wantcreateWriteStream
to use make sure you open it for writing.
options is an optional object which can contain:
* autoClose - if set to true, the file descriptor will be automaticallyfalse
closed once the last stream that references it is closed. Defaults to
. ref() and unref() can be used to increase or decrease the
reference count, respectively.
`js`
import {createFromBuffer} from 'better-fd-slicer';
const slicer = createFromBuffer(someBuffer);
// ...
options is an optional object which can contain:
* maxChunkSize - A Number of bytes. see createReadStream().
If falsey, defaults to unlimited.
#### Properties
##### fd
The file descriptor passed in. undefined if created from a buffer.
#### Methods
##### createReadStream(options)
Available options:
* start - Number. The offset into the file to start reading from. Defaultsend
to 0.
* - Number. Exclusive upper bound offset into the file to stop readinghighWaterMark
from.
* - Number. The maximum number of bytes to store in theencoding
internal buffer before ceasing to read from the underlying resource.
Defaults to 16 KB.
* - String. If specified, then buffers will be decoded to stringsnull
using the specified encoding. Defaults to .
The ReadableStream that this returns has these additional methods:
* destroy(err) - stop streaming. err is optional and is the error thatnew Error("stream destroyed")
will be emitted in order to cause the streaming to stop. Defaults to
.
If maxChunkSize was specified (see createFromBuffer()), the read stream
will provide chunks of at most that size. Normally, the read stream provides
the entire range requested in a single chunk, but this can cause performance
problems in some circumstances.
See thejoshwolfe/yauzl#87.
##### createWriteStream(options)
Available options:
* start - Number. The offset into the file to start writing to. Defaults toend
0.
* - Number. Exclusive upper bound offset into the file. If this offseterr.code === 'ETOOBIG'
is reached, the write stream will emit an 'error' event and stop functioning.
In this situation, . Defaults to Infinity.highWaterMark
* - Number. Buffer level when write() starts returningdecodeStrings
false. Defaults to 16KB.
* - Boolean. Whether or not to decode strings into Buffers _write()
before passing them to. Defaults to true.
The WritableStream that this returns has these additional methods:
* destroy() - stop streaming
And these additional properties:
* bytesWritten - number of bytes written to the stream
And these additional events:
* 'progress' - emitted when bytesWritten changes.
##### read(buffer, offset, length, position, callback)
Equivalent to fs.read, but with concurrency protection.callback must be defined.
##### write(buffer, offset, length, position, callback)
Equivalent to fs.write, but with concurrency protection.callback must be defined.
##### ref()
Increase the autoClose reference count by 1.
##### unref()
Decrease the autoClose reference count by 1.
#### Events
##### 'error'
Emitted if fs.close returns an error when auto closing.
##### 'close'
Emitted when FdSlicer closes the file descriptor due to autoClose`. Never
emitted if created from a buffer.
This library is heavily based on an excellent fd-slicer library, and uses
some of it (modernized and improved) code.