Node Backup (NPM, etc)

CLI for downloading public data (for now it is NPM packages meta or archives). Multithread.

Installation

npm install node-backup-loader -g

Usage

There is an important note. NPM has the only way to retrieve their packages, it is public
search, therefore we must do about 1-2k requests to obtain information about all their packages, BUT,
I found out it has much less packages in search, than actually can be found and installed
directly (using https://www.npmjs.com/package/)!

I found the repository called all-the-package-names
which updated NPM packages list every day, and it has all of them (even deleted or deprecated).

So, there are two algorythms implemented to download NPM packages:

1. Fast and reliable, not all packages, search-way. Just get meta from NPM database using search, and
download them. Bulk downloading (200 packages for each thread).

2. Slow, but you will receive all packages, package-api-way. You will request each packages one by
one (they are not available on search API), so you will send about 1mm request, and it will produce huge
load on NPM servers, your network connection, and, possibly, you hard drive (if you want to download archives).

$3

Meta data will be downloaded as JSON, one file for each package. At first, I implemented the way to download one
huge JSON, but it become too large (about 5-10 Gb), so it was not possible to debug it or even open it. So the
only way to download the data is to split it by files. Now all downloaded meta will be stored in the directory
you specify. Total count of files is about 1mm, so if you are using fat32 table partition, create an issue
then, you just will not be able to store all files in the same folder, restriction of fat32 systems is
about 65k files per folder.

Commands:
* node-backup --command meta --registry npm --threads (fast-download, not all packages)
* node-backup --command meta --registry npm --npm-force-package-api --threads (slow download, all packages)

Examples:
* node-backup --command meta --registry npm --threads 25 ../my-folder/to/store/packages/meta/files/
* node-backup --command meta --registry npm --npm-force-package-api --threads 50 ../my-folder/to/store/packages/meta/files/

$3

Useful when you want to download all packages from NPM.

Commands:
* node-backup --command copy --registry npm [--preserve-downloaded-files] --threads (fast-download, not all packages)
* node-backup --command copy --registry npm [--preserve-downloaded-files] --npm-force-package-api --threads (slow download, all packages)

Examples:
* node-backup --command copy --registry npm --threads 1 ../my-folder/to/store/packages/archives/
* node-backup --command copy --registry npm --npm-force-package-api --threads 10 ../my-folder/to/store/packages/archives/
* node-backup --command copy --registry npm --preserve-downloaded-files --threads 1 ../my-folder/to/store/packages/archives/
* node-backup --command copy --registry npm --preserve-downloaded-files --npm-force-package-api --threads 10 ../my-folder/to/store/packages/archives/

I would not recommend you to set huge threads number (like, 100-200). For my case at any threads number more than 10-20, I could
not get the downloading speed more than 5-15 packages per second due to npm pack command delay and my network connection speed.

The option --preserve-downloaded-files will force tool to not override already downloaded files in case if you
accidentally closed your terminal. This options use the module glob, but as is there is too much files, it may cause
performance issues, so it disabled by default.

Screenshots

How archives will be stored:


Downloading process:


Contribution

Look at the file src/modules/registry.ts and see what is registry. Every loaded data is abstracted from algorithm how it
used, so you can create any new registry (like, bower, or any other database, from news blog to archive.org) and add new
commands in this doc.

Be free to send pull requests.