Pack a File into Box for easy move/transfer between servers no matter of where it is.(local path, remote url, or cloud storage)
npm install @juzi/file-box



FileBox is a virtual container for packing a file data into it for future read, and easily transport between servers with the least payload, no mater than where it is (local path, remote url, or cloud storage).
Currently the FileBox supports almost all kinds of the data input/output methods/formats:
| File Type | Pack Method | Unpack Method | Description |
| :--- | :--- | :--- | :--- |
| Local File | fromFile() | toFile() | Local file in file system |
| Remote URL | fromUrl() | toUrl()(TBW) | Remote file in a HTTP/HTTPS URL |
| Buffer | fromBuffer() | toBuffer() | JavaScript Buffer |
| Stream | fromStream() | toStream() | JavaScript Stream |
| Base64 | fromBase64() | toBase64() | Base64 data |
| DataURL | fromDataURL() | toDataURL() | DataURL data |
| QRCode | fromQRCode() | toQRCode() | QR Code Image Decode/Encode |
| UUID | fromUuid() | toUuid() | UUID by loader/saver helper functions |
| JSON | fromJSON() | toJSON() | Serialize/Deserialize FileBox |
The following example demos:
1. Load local file
1. Save URL to local file
1. Convert buffer to stream
1. Pack from Base64 then Unpack to DataURL
``ts
import { FileBox } from 'file-box'
/**
* 0. Load local file
*/
const fileBox0 = FileBox.fromFile('/tmp/file.jpg')
/**
* 1. Save URL to File
*/
const fileBox1 = FileBox.fromUrl(
'https://huan.github.io/file-box/images/file-box-logo.jpg',
'logo.jpg',
)
fileBox1.toFile('/tmp/file-box-logo.jpg')
/**
* 2. Convert Buffer to Stream
*/
import fs from 'fs'
const fileBox2 = FileBox.fromBuffer(
Buffer.from('world'),
'hello.txt',
)
const writeStream = fs.createWriteStream('/tmp/hello.txt')
fileBox2.pipe(writeStream)
/**
* 3. Pack Base64, Unpack to DataURL
*/
const fileBox3 = FileBox.fromBase64('d29ybGQK', 'hello.txt')
fileBox3.toDataURL()
.then(console.log)
// Output: data:text/plain;base64,d29ybGQK
`
1. TypeError [ERR_UNESCAPED_CHARACTERS]: Request path contains unescaped characters #56
#### 1.1 fromFile(filePath: string, name?: string, md5?: string): FileBox
Alias: fromLocal()
About optional arguments:
name: filename, if not passed, will be parsed from file path, url, etc.
md5: file md5 code, only for checking purposes and will not be computed from file.
`ts`
const fileBox = FileBox.fromLocal('/tmp/test.txt')
#### 1.2 fromUrl(url: string, options?: {headers?: HTTP.OutgoingHttpHeaders, name?: string, size?: string, md5?: string}): FileBox
Alias: fromRemote()
`ts`
const fileBox = FileBox.fromUrl(
'https://huan.github.io/file-box/images/file-box-logo.jpg',
{name: 'logo.jpg'},
)
#### 1.3 fromStream(stream: Readable, name?: string, md5?: string): FileBox
Will be named 'stream.dat' if no name is provided.
`ts`
const fileBox = FileBox.fromStream(res, '/tmp/download.zip')
#### 1.4 fromBuffer(buffer: Buffer, name?: string, md5?: string): FileBox
Will be named 'buffer.dat' if no name is provided.
`ts`
const fileBox = FileBox.fromBuffer(buf, '/tmp/download.zip')
#### 1.5 FileBox.fromBase64(base64: string, name?: string, md5?: string): FileBox
Decoded a base64 encoded file data. Will be named 'base64.dat' if no name is provided.
`ts`
const fileBox = FileBox.fromBase64('d29ybGQK', 'hello.txt')
fileBox.toFile()
#### 1.6 FileBox.fromDataURL(dataUrl: string, name?: string, md5?: string): FileBox
Decoded a DataURL data. Will be named 'data-url.dat' if no name is provided.
`ts`
const fileBox = FileBox.fromDataURL('data:text/plain;base64,d29ybGQK', 'hello.txt')
fileBox.toFile()
#### 1.7 FileBox.fromJSON()
Restore a FileBox.toJSON() text string back to a FileBox instance.
`ts`
const restoredFileBox = FileBox.fromJSON(jsonText)
#### 1.8 FileBox.fromQRCode(qrCodeValue: string, md5?: string)
Get a FileBox instance that represent a QR Code value.
`ts`
const fileBox = FileBox.fromQRCode('https://github.com')
fileBox.toFile('qrcode.png')
#### 1.9 FileBox.fromUuid(uuid: string, options?: {name?: string, size?: string, md5?: string})
Load a FileBox from a UUID. Will be named ${uuid}.dat if no name is provided.
See: FileBox.setUuidLoader()
Save file to current work path(cwd) of the local file system with the default name.
if name specified with a full path, then will use the speficied file name instead.
`ts`
const fileBox = FileBox.fromRemote(
'https://huan.github.io/file-box/images/file-box-logo.jpg',
)
await fileBox.toFile('/tmp/logo.jpg')
#### 2.2 toStream(): Readable
Get the stream of file data.
`ts`
const fileBox = FileBox.fromRemote(
'https://huan.github.io/file-box/images/file-box-logo.jpg',
)
const readableStream = fileBox.toStream()
#### 2.3 pipe(destination: Writable): Promise
Pipe to a writable stream.
`ts`
const fileBox = FileBox.fromRemote(
'https://huan.github.io/file-box/images/file-box-logo.jpg',
)
const writableStream = fs.createWritable('/tmp/logo.jpg')
fileBox.pipe(writableStream)
#### 2.4 toBase64(): Promise
Get the base64 data of file.
`ts`
const fileBox = FileBox.fromRemote(
'https://huan.github.io/file-box/images/file-box-logo.jpg',
)
const base64Text = await fileBox.toBase64()
console.log(base64Text) // Output: the base64 encoded data of the file
#### 2.5 toJSON(): string
Get the JSON.stringify-ed text.
Not Implement Yet: Working In Progress...
`ts
const fileBox = FileBox.fromRemote(
'https://huan.github.io/file-box/images/file-box-logo.jpg',
)
const jsonText1 = fileBox.toJSON()
const jsonText2 = JSON.stringify(fileBox)
assert(jsonText1 === jsonText2)
console.log(jsonText1) // Output: the stringified data of the fileBox
const restoredFileBox = fileBox.fromJSON(jsonText1)
restoredFileBox.toFile('/tmp/file-box-logo.jpg')
`
#### 2.6 toDataURL(): Promise
Get the DataURL of the file.
`ts`
const fileBox = FileBox.fromFile('tests/fixtures/hello.txt')
const dataUrl = await fileBox.toDataURL()
console.log(dataUrl) // Output: data:text/plain;base64,d29ybGQK'
#### 2.7 toBuffer(): Promise
Get the Buffer of the file.
`ts`
const fileBox = FileBox.fromFile('tests/fixtures/hello.txt')
const buffer = await fileBox.toBuffer()
console.log(buffer.toString()) // Output: world
#### 2.8 toQRCode(): Promise
Decode the QR Code value from the file.
`tsQR Code decoded value is: "${qrCodeValue}"
const fileBox = FileBox.fromFile('qrcode.jpg')
const qrCodeValue = await fileBox.toQRCode()
console.log()`
// Output: QR Code decoded value is: "https://github.com"
#### 2.9 toUuid(): Promise
Save the FileBox to a UUID file and return the UUID.
See: FileBox.setUuidSaver()
#### 2.10 toJSON(): string
Encode a FileBox instance to JSON string so that we can transfer the FileBox on the wire.
`ts
const fileBox = FileBox.fromBase64('RmlsZUJveEJhc2U2NAo=')
const jsonText = JSON.stringify(fileBox)
// the above code equals to the following line of code:
// const jsonText = fileBox.toJSON()
// we will get the serialized data for this FileBox:
console.log(jsonText)
// Output: {"name":"qrcode.png","metadata":{},"boxType":1,"base64":"RmlsZUJveEJhc2U2NAo="}
// restore our fleBox:
// const newFileBox = FileBox.fromJSON(jsonText)
`
##### Limitation
Because we want to enable the JSON.stringify(fileBox), which will call fileBox.toJSON(), so the toJSON() can not be async, which means we can only support limited FileBoxType(s):
1. FileBoxType.Base64
1. FileBoxType.Url
1. FileBoxType.QRCode
For other types like FileBoxType.File, FileBoxType.Buffer, FileBoxType.Stream, etc, we need to transform them to FileBoxType.Base64 before we call toJSON:
`ts
const fileBoxLazy = FileBox.fromFile('./test.txt')
const base64 = await fileBoxLazy.toBase64()
const fileBox = FileBox.fromBase64(base64, 'test.txt')
// fileBox will be serializable because it do not need async operations
const jsonText = JSON.stringify(fileBox)
console.log(jsonText)
`
#### 3.1 name
File name of the file in the box
`ts`
const fileBox = FileBox.fromRemote(
'https://huan.github.io/file-box/images/file-box-logo.jpg',
)
console.log(fileBox.name) // Output: file-box-logo.jpg
#### 3.2 md5
File md5 of the file in the box. The value is set by user, not computed from file.
`ts`
const fileBox = FileBox.fromUrl(
'https://huan.github.io/file-box/images/file-box-logo.jpg',
{md5: 'computed-md5-string'}
)
console.log(fileBox.md5) // Output: computed-md5-string
#### 3.3 metadata: Metadata { [key: string]: any }
Metadata for the file in the box. This value can only be assigned once, and will be immutable afterwards, all following assign or modify actions on metadata will throw errors
`ts
const fileBox = FileBox.fromRemote(
'https://huan.github.io/file-box/images/file-box-logo.jpg',
)
fileBox.metadata = {
author : 'huan',
githubRepo : 'https://github.com/huan/file-box',
}
console.log(fileBox.metadata) // Output: { author: 'huan', githubRepo: 'https://github.com/huan/file-box' }
fileBox.metadata.author = 'Tank' // Will throw exception
`
#### 3.4 version(): string
Version of the FileBox
#### 3.5 toJSON(): string
Serialize FileBox metadata to JSON.
#### 3.6 ready(): Promise
Update the necessary internal data and make everything ready for use.
#### 3.7 syncRemoteName(): Promise
Sync the filename with the HTTP Response Header
HTTP Header Example:
> Content-Disposition: attachment; filename="filename.ext"
#### 3.8 type: FileBoxType
Return the type of the current FileBox instance.
The currently supported types are defined at file-box-type.ts as the following demonstrated:
`ts`
enum FileBoxType {
Unknown = 0,
Base64 = 1,
Url = 2,
QRCode = 3,
Buffer = 4,
File = 5,
Stream = 6,
Uuid = 7,
}
#### 3.9 FileBox.setUuidLoader(loader: UuidLoader): void
Required by static method FileBox.fromUuid()
`ts
class FileBoxUuid extends FileBox {}
const loader: UuidLoader = async (uuid: string) => {
const stream = new PassThrough()
stream.end('hello, world!')
return stream
})
FileBoxUuid.setUuidLoader(loader)
const fileBox = FileBoxUuid.fromUuid('12345678-1234-1234-1234-123456789012', 'test.txt')
await fileBox.toFile('test.txt')
`
The UuidLoader is a function that takes a UUID and return a readable stream.
`ts`
type UuidLoader = (this: FileBox, uuid: string) => Readable
#### 3.10 FileBox.setUuidSaver(saver: UuidSaver): void
Required by instance method fileBox.toUuid()
`ts
class FileBoxUuid extends FileBox {}
const saver: UuidSaver = async (stream: Readable) => {
// save the stream and get uuid
return '12345678-1234-1234-1234-123456789012'
})
FileBoxUuid.setUuidSaver(saver)
const fileBox = FileBoxUuid.fromFile('test.txt')
const uuid = await fileBox.toUuid()
`
The UuidSaver is a function that takes a readable stream and return a UUID promise.
`ts`
type UuidSaver = (this: FileBox, stream: Readable) => Promise
#### 3.11 size
The file box size in bytes. (-1 means unknown)
It is not the size of the target (boxed) file itself.
For example:
`ts`
const fileBox = FileBox.fromUrl('http://example.com/image.png')
console.log(fileBox.size)
// > 20 <- this is the length of the URL string
#### 3.12 remoteSize
The remote file size in bytes. (-1 or undefined means unknown)
For example:
`ts`
const fileBox = FileBox.fromUrl('http://example.com/image.png')
await fileBox.ready()
console.log(fileBox.remoteSize)
// > 102400 <- this is the size of the remote image.png
1. Present A File by Abstracting It's Meta Information that supports Reading & toJSON() API.
1. Follow DOM File/BLOB Interface
1. Present a file that could be: Local, Remote, Stream
1. Lazy load
1. Serializable
1. Can be Transfered from server to server, server to browser.
Environment variables can be used to control some behavior.
- FILEBOX_HTTP_REQUEST_TIMEOUT [default=10000] The timeout period for establishing a communication request with the server. For example, if the network is unavailable or the delay is too high, communication cannot be successfully established.FILEBOX_HTTP_RESPONSE_TIMEOUT
- [default=60000] Socket idle timeout when FileBox downloads data from URL. For example, when the network is temporarily interrupted, the request will be considered as failed after waiting for a specified time.FILEBOX_READY_RETRY
- [default=3] Maximum retry count when calling ready() method. If an error occurs while preparing the FileBox (e.g., fetching remote file metadata), it will retry up to this number of times before throwing an error.FILEBOX_HTTP_TIMEOUT
- ~~ [default=60000]~~ Deprecated! Please use FILEBOX_HTTP_RESPONSE_TIMEOUT.FILE_BOX_READY_RETRY
- ~~ [default=3]~~ Deprecated! Please use FILEBOX_READY_RETRY (with underscore naming for consistency).FILEBOX_NO_SLICE_DOWN
- ~~ [default=false]~~ Removed! Chunked download now automatically falls back to non-chunked mode when servers don't support range requests. No manual configuration needed.FILEBOX_HTTP_CHUNK_SIZE
- ~~ [default=524288]~~ Deprecated! Chunk size is no longer configurable. The implementation now uses open-ended range requests (bytes=start-) for automatic chunk management.
Node.js Documentation > URL Strings and URL Objects
`asciiart`
┌─────────────────────────────────────────────────────────────────────────────────────────────┐
│ href │
├──────────┬──┬─────────────────────┬─────────────────────┬───────────────────────────┬───────┤
│ protocol │ │ auth │ host │ path │ hash │
│ │ │ ├──────────────┬──────┼──────────┬────────────────┤ │
│ │ │ │ hostname │ port │ pathname │ search │ │
│ │ │ │ │ │ ├─┬──────────────┤ │
│ │ │ │ │ │ │ │ query │ │
" https: // user : pass @ sub.host.com : 8080 /p/a/t/h ? query=string #hash "
│ │ │ │ │ hostname │ port │ │ │ │
│ │ │ │ ├──────────────┴──────┤ │ │ │
│ protocol │ │ username │ password │ host │ │ │ │
├──────────┴──┼──────────┴──────────┼─────────────────────┤ │ │ │
│ origin │ │ origin │ pathname │ search │ hash │
├─────────────┴─────────────────────┴─────────────────────┴──────────┴────────────────┴───────┤
│ href │
└─────────────────────────────────────────────────────────────────────────────────────────────┘
Node.js Documentation > path.parse(path)
`asciiart`
┌─────────────────────┬────────────┐
│ dir │ base │
├──────┬ ├──────┬─────┤
│ root │ │ name │ ext │
" / home/user/dir / file .txt "
└──────┴──────────────┴──────┴─────┘
1. Environment variables FILEBOX_HTTP_TIMEOUT can be set by user. see Environments. (#80, by @binsee)
1. fileBox.md5 can be set by user. This filed is for the receiver of the filebox to check, and it is not computed from the file.
1. fileBox.size will be serialized to/from JSON, and present the Content-Length of the file. (-1 means unknown)mimeType
1. has been renamed to mediaType, and added to the FileBoxInterface
1. Suppert ES Module. (#54)
1. Add UUID boxType support: FileBox.fromUuid() and FileBox.toUuid()size
1. Add property to return the size of the file. (-1 means unknown)remoteSize
1. Add property to present the remote size of the file (if applicable, -1 means unknown)UniformResourceNameRegistry
1. Add class for providing a production-ready basic UUID management tool.FileBoxInterface
1. Add , FileBox.isInstance(), and FileBox.isInterface()
Breaking changes:
1. toJSON format renamed boxType to typetype()
1. has been changed to typeversion()
1. has been changed to version
1. Throw error when consume a stream twice to prevent data lost. (#50)
1. Add fileBox.type() to return the FileBoxType of a FileBox. (wechaty/wechaty#1918)Readable
1. Change to stream.Readable for better compatibility. (Jun 27, 2020)chunkerTransformStream
1. Add to toStream (#44)
Add support to JSON.stringify() (#25):
1. FileBox.fromJSON() - Static method for deserializationfileBox.toJSON()
1. - Instance method for serialization
1. Add support to QR Code: FileBox.fromQRCode() and FileBox.toQRCode()
1. Start using @chatie/tsconfig
1. Add two new factory methods: fromBase64(), fromDataURL()toBuffer()
1. Add , toBase64() and toDataURL() to get the Buffer and BASE64 encoded file datametadata
1. Add property to store additional information. (#3)
1. Add headers option for fromRemote()` method
Initial version.
* File API - W3C Working Draft, 26 October 2017
* MIME Sniffing - Living Standard — Last Updated 20 April 2018
* Using files from web applications
* Web technology for developers > Web APIs > File
* Web technology for developers > Web APIs > Blob
* Web technology for developers > Web APIs > FileReader
* A simple HTTP Request & Response Service.
* Hurl.it — Make HTTP Requests
This module is inspired by
Huan LI (李卓桓), Microsoft AI MVP, zixia@zixia.net

* Docs released under Creative Commons
* Code released under the Apache-2.0 License
* Code & Docs © 2018 Huan LI \