A lightweight cloud storage system using discord as storage device written in nodejs
npm install @forscht/ddriveTurn Discord into a datastore that can manage and store your files.
##### DDrive A lightweight cloud storage system using discord as storage device written in nodejs. Supports an unlimited file size and unlimited storage, Implemented using node js streams with multi-part up & download.
https://user-images.githubusercontent.com/59018146/167635903-48cdace0-c383-4e7d-a037-4a32eaa4ab69.mp4
#### Current stable branch 4.x
This next major version release 4.0 is ddrive written from scratch. It comes with most requested features and several improvements.
- Now uses postgres to store files metadata. Why?
- Once you have huge amount of data stored on ddrive it makes ddrive significantly slow to start since ddrive have to fetch all the metadata from discord channel (For 3 TB of data it takes me 30+ minutes.)
- With postgres, deleting file is extremely faster because now ddrive don't have to delete files on discord channel and just need to remove from metadata only.
- With postgres now it's possible to move or rename files/folders which was impossible with older version.
- Added support for rename files/folders.
- Added support to move file/folder (Only via API, Not sure how to do it with frontend, PR welcomes.)
- Now uses webhooks instead of bot/user tokens to bypass the discord rate limit
- DDrive now uploads file chunks in parallel with limit. Which significantly increase the upload speed. I was able to upload file with 5GB of size in just 85 seconds.
- Public access mode - It is not now possible to provide users read-only access with just one config var
- Batch upload files - Now you can upload multiple files at once from panel. (DClone support has been removed from this version)
- Bug fix - download reset for few mobile devices
- Added support for optional encryption to files uploaded to discord
- DDrive now has proper rest API with OpenAPI 3.1 standards
- Added support for dark/light mode on panel
I spent several weeks finalizing this new version. Any support is highly appreciated - Buy me a coffee
cd .devcontainerdocker compose up -dconfig/env_sample to config/.env and make necessary changeswebhook.txt with \n seperated.npm run migration:upnode bin/ddrivehttp://localhost:3000 in your browser.npm install -g pm2pm2 start bin/ddrivepm2 list to check status of ddrivepm2 logs to check ddrive logsshell
config/.env
Required params
DATABASE_URL= # Database URL of postgres with valid postgres uriWEBHOOKS={url1},{url2} # Webhook urls seperated by ","
Optional params
PORT=3000 # HTTP Port where ddrive panel will start runningREQUEST_TIMEOUT=60000 # Time in ms after which ddrive will abort request to discord api server. Set it high if you have very slow internet
CHUNK_SIZE=7864320 # ChunkSize in bytes. You should probably never touch this and if you do don't set it to more than 8MB, with discord webhooks you can't upload file bigger than 8MB
SECRET=someverysecuresecret # If you set this every files on discord will be stored using strong encryption, but it will cause significantly high cpu usage, so don't use it unless you're storing important stuff
AUTH=admin:admin # Username password seperated by ":". If you set this panel will ask for username password before access
PUBLIC_ACCESS=READ_ONLY_FILE # If you want to give read only access to panel or file use this option. Check below for valid options.
# READ_ONLY_FILE - User will be only access download links of file and not panel
# READ_ONLY_PANEL - User will be able to browse the panel for files/directories but won't be able to upload/delete/rename any file/folder.
UPLOAD_CONCURRENCY=3 # ddrive will upload this many chunks in parallel to discord. If you have fast internet increasing it will significantly increase performance at cost of cpu/disk usage
`$3
`shell
docker run -rm -it -p 8080:8080 \
-e PORT=8080 \
-e WEBHOOKS={url1},{url2} \
-e DATABASE_URL={database url} \
--name ddrive forscht/ddrive
`
$3
$3
- Setup under 4 minutes in local/cloud server using neon.tech postgres - Youtube
API Usage
npm install @forscht/ddrive
`javascript
const { DFs, HttpServer } = require('@forscht/ddrive')const DFsConfig = {
chunkSize: 7864320,
webhooks: 'webhookURL1,webhookURL2',
secret: 'somerandomsecret',
maxConcurrency: 3, // UPLOAD_CONCURRENCY
restOpts: {
timeout: '60000',
},
}
const httpConfig = {
authOpts: {
auth: { user: 'admin', pass: 'admin' },
publicAccess: 'READ_ONLY_FILE', // or 'READ_ONLY_PANEL'
},
port: 8080,
}
const run = async () => {
// Create DFs Instance
const dfs = new DFs(DFsConfig)
// Create HTTP Server instance
const httpServer = HttpServer(dfs, httpConfig)
return httpServer.listen({ host: '0.0.0.0', port: httpConfig.port })
}
run().then()
`Migrate from v3 to v4
Migrating ddrive v3 to v4 is one way process once you migrate ddrive to v4 and add new files you can't migrate new files to v3 again but you can still use v3 with old files.1. Clone this project
2. Create few webhooks (1 webhook/text channel). Do not create webhook on old text channel where you have already stored v3 data.
3. Take pull of latest ddrive v3
4. Start ddrive v3 with option
--metadata=true. Ex - ddrive --channelId {id} --token {token} --metadata=true
5. Open localhost:{ddrive-port}/metadata in browser
6. Save JSON as old_data.json in cloned ddrive directory
7. Put valid DATABASE_URL in config/.env
8. Run node bin/migrate old_data.json
9. After few seconds once process is done you should see the message Migration is done`Feel free to create new issue if it's not working for you or need any help.