Schedule restic backups for multiple hosts and/or repos
npm install schedic 
schedic makes it easy to backup multiple hosts in your
network using the fantastic restic tool.
Everything is put into one YAML configuration file on your
backup server. There is no schedic configuration on the
hosts you like to backup. They require just an installation
of restic. The backup server executes restic remotely
using SSH and passes all required information in a secure
fashion.
restic is an ultra fast backup tool which performs super
efficient deduplication and encryption of your backup data.
It stores data locally or uploads to multiple different storage
backends.
- Installation
- From npm registry
- Building from source
- Automatic TypeScript transpilation
- Usage
- Options
- Commands
- run
- backup
- forget
- check
- update
- repo
- Backup configuration
- Cron configuration
- Example output of summaries
- Security considerations
- Storing restic password
- SSH access to all hosts
- CI pipeline
- License
- TODO / IDEAS
schedic requires Node.js (minimum v8.x). It's recommended
to use Yarn for installation, but npm is supported as well.
As well bash is required on the schedic host and on the
target hosts. That's no problem with Unix operating systems,
but additionals steps may be necessary to use schedic with
Windows and probably Mac (bash is default shell since
Mac OS 10.3).
schedic uses ssh (and assumes OpenSSH) to execute restic
commands on the target hosts. By default it sets -o 'BatchMode Yes'
to prevent interactive password prompts which causes the process to
hang or wait infinitely. If you don't use OpenSSH and require different
options to accomplish this you can specify them in thehost-defaults configuration section (see below);
``bashUsing Yarn
yarn global add schedic
Building from source
This will transpile the source from TypeScript to JavaScript
and then links the schedic script globally.
`bash
Using yarn
yarn && yarn build
yarn linkUsing npm (but without package locking)
npm install && npm run build
npm link
`$3
During development you should open a shell running
a nodemon watcher for automatic transpilation:
`bash
Using yarn
yarn watchUsing npm
npm run watch
`Usage
`bash
schedic command [options]
`Options
-
--config|c file.yml: specify configuration file, Default: ./schedic.yml, Environment: SCHEDIC_CONFIG
- --dry-run: just print commands, which would be executed
- --repo|r name: perform operations just for this repo, Default: all repos in configuration file
- --host|h name: perform operations just for this host, Default: all hosts in configuration file
- --path|p path: perform operation just for this backup path, Default: all paths in configuration file
- --sync-output: print log immediately, even if stdout is redirectedCommands
$3
Performs a combined execution of the
backup and forget
commands. Refer to their descriptions below.Example:
`bash
Perform backup, forget & prune of everything in ./schedic.yml
schedic run
`$3
Performs backups for all selected hosts and paths in the
configuration file.
Examples:
`bash
Backup all paths of all hosts listed in ./schedic.yml
schedic backupBackup all paths of host "homesrv"
schedic backup --host homesrvBackup /home of host "homesrv"
schedic backup --host homesrv --path /homeBackup /home of host "homesrv" to repo "mycloud" only
schedic backup --host homesrv --path /home --repo mycloud
`$3
Removes backups for all selected hosts and paths in the
configuration file according to the configured
forget
policies. Afterwards restic prune is executed on the
affected repos.Examples:
`bash
Remove all backups according to the configured "forget" policies
schedic forgetRemove all backups of host "homesrv" according to the "forget" policies
schedic forget --host homesrvRemove all "/home" backups of host "homesrv" according to the "forget" policies
schedic forget --host homesrv --path /home
`$3
This command checks if all selected hosts are accessable using
SSH and if the restic command is available (by executing
restic version).Examples:
`bash
Check all configured hosts
schedic checkCheck host "homesrv"
schedic check --host homesrv
`$3
Runs
restic self-update on all selected hosts.Examples:
`bash
Update restic on all configured hosts
schedic updateUpdate restic on host "homesrv"
schedic update --host homesrv
`$3
Prints shell code to set your environment to a specific restic repo.
Example:
`bash
Select repository "mycloud"
eval schedic repo mycloudList all snapshots in selected repo
restic snapshots
`Backup configuration
All backups for all hosts are configured in a single YAML
file. schedic validates the schema and prints specific
error messages, if the configuration file is invalid.
This is an annotated example configuration:
`yaml
Version of the configuration
version: "2.0"Repo definition
---------------
The following properties are known for a repository:
- name: an arbitrary name
- url: repository URL
- password-file: path to a file containing the repo password
- password: or the repo password itself (not recommended)
- restic: an object of additional restic options
use boolean values for switches, and numeric/string values
for options with parameters
- env: required environment variables (e.g. auth keys)
keys may be prefixed with "<<" to express that the
value is a filename where the env var is read from
(to keep secrets out of this file; eol removed).
env variables can be overridden by host-defaults and
host sections beyond.
repos:
- name: mycloud
url: "rest:https://mycloud.example.com"
password-file: "test/assets/.restic-mycloud.pw"
restic:
limit-upload: 900 - name: local
url: "/storage/backup/restic"
password: "super secret local restic password"
- name: s3
url: "s3:s3.amazonaws.com/myresticbucket"
password: "mys3awsresticpassword"
env:
AWS_ACCESS_KEY_ID: myawsaccesskey
<Default Host Options
--------------------
Specify default options for all backup hosts. Can be
overridden by each host.
- ssh: the ssh command to use (e.g. for a non-default port)
or false, if command should be executed locally;
the "BatchMode" option is important to prevent hanging
of the process in interactive password prompts
- arch: the CPU architecture (yet unused)
- restic: see "repos[].restic"
- env: see "repos[].env"
host-defaults:
ssh: "ssh -o 'BatchMode Yes'"
arch: linux_amd64
restic:
limit-upload: 500
Default Backup Options
----------------------
Specify default options for all backups. Can be overridden
by each backup.
- repos: list of repos to push this backup to
- exclude: list of paths resp. globs to exclude from the backup
- keep: your "forget" policy. All restic --keep-* options are supported
- restic: see "repos[].restic"
backup-defaults:
repos:
- local
exclude:
- .gvfs
- .thumbnails/
- .cache/
keep:
daily: 3
weekly: 4
monthly: 6
restic:
one-file-system: trueHost & Backup definition
------------------------
Specify all hosts and paths to be backed up.
- name: network hostname
- pre: a list of commands to execute prior backup
- ssh: the ssh command to use (e.g. for a non-default port)
or false, if command should be executed locally
- arch: the CPU architecture (yet unused)
- disabled: set to true, to disable a host (temporarily)
- restic: see "repos[].restic"
- env: see "repos[].env"
hosts:
- name: homesrv
ssh: false
pre:
- "cd /home/gitlab && ./mkbackup && mv $(ls -1atr /home/gitlab/data/gitlab/backups/* | tail -1) /home/backup/gitlab-latest.tar"
env:
PATH: /opt/restic/bin # Backup definition
# -----------------
# Specify all backup paths for this host. All settings here
# are merged with the "backup-defaults" specified above.
# - path: the local directory path
# - tags: single tag or list of tags
# - repos: list of repos to push this path to
# - exclude: list of paths resp. globs to exclude from the backup
# - restic: see "repos.restic"
backups:
- path: /home
tags: important
repos:
- mycloud
- local
- s3
exclude:
- /home/gitlab
- path: /etc
tags: etc
- path: /root
tags: root-home
- path: /var
tags: var
exclude:
- /var/cache
- /var/lib/docker
- /var/log
- path: /
tags: system
exclude:
- /root
- /var
- path: /mnt/music
tags: music
keep:
weekly: 2
- name: raspberry1
arch: linux_arm
backups:
- path: /etc
tags: etc
- path: /var
tags: var
exclude:
- /var/cache
- /var/log
- path: /
tags: system
exclude:
- /var
- name: raspberry2
arch: linux_arm
disabled: true
backups:
- path: /etc
tags: etc
- path: /var
tags: var
exclude:
- /var/cache
- /var/log
- path: /
tags: system
exclude:
- /var
`Cron configuration
This is an example cron configuration for regular operation:
`cron
Daily backup and removal
11 3 * root schedic --config /usr/local/etc/schedic.yml run 2>&1 | mail -s "Daily schedic report" mail@example.comWeekly restic self-update
1 3 0 root schedic --config /usr/local/etc/schedic.yml update 2>&1 | mail -s "Weekly schedic update report" mail@example.com
`Example output of summaries
schedic prints summaries of all executed tasks. If the output
is no terminal, all log messages are buffered and the summaries
are printed first. This is very useful when you send the output
with e-mail, so everything important is in front.
This is an example output of
schedic run --dry-run with
the shipped example schedic.yml (log messages are removed):`
> Backup summary
> OK: All tasks successfully completed
╔════════════╤═════════╤═════════╤═══════════════════════════╤════╤══════╤══════════╤═════════╗
║ Host │ Type │ Repo │ Details │ ID │ Size │ Duration │ Result ║
╟────────────┼─────────┼─────────┼───────────────────────────┼────┼──────┼──────────┼─────────╢
║ homesrv │ Command │ │ cd /home/gitlab && ./m... │ │ │ - │ Dry run ║
║ homesrv │ Backup │ mycloud │ /home │ - │ - │ - │ Dry run ║
║ homesrv │ Backup │ local │ /home │ - │ - │ - │ Dry run ║
║ homesrv │ Backup │ local │ /etc │ - │ - │ - │ Dry run ║
║ homesrv │ Backup │ local │ /root │ - │ - │ - │ Dry run ║
║ homesrv │ Backup │ local │ /var │ - │ - │ - │ Dry run ║
║ homesrv │ Backup │ local │ / │ - │ - │ - │ Dry run ║
║ homesrv │ Backup │ local │ /mnt/music │ - │ - │ - │ Dry run ║
╟────────────┼─────────┼─────────┼───────────────────────────┼────┼──────┼──────────┼─────────╢
║ raspberry1 │ Backup │ local │ /etc │ - │ - │ - │ Dry run ║
║ raspberry1 │ Backup │ local │ /var │ - │ - │ - │ Dry run ║
║ raspberry1 │ Backup │ local │ / │ - │ - │ - │ Dry run ║
╚════════════╧═════════╧═════════╧═══════════════════════════╧════╧══════╧══════════╧═════════╝> Forget summary
> OK: All tasks successfully completed
╔════════════╤═════════╤═══════════════════════════╤══════╤═════════╤═════════╗
║ Host │ Repo │ Path │ Kept │ Removed │ Result ║
╟────────────┼─────────┼───────────────────────────┼──────┼─────────┼─────────╢
║ homesrv │ mycloud │ /home │ - │ - │ Dry run ║
║ homesrv │ local │ /home │ - │ - │ Dry run ║
║ homesrv │ local │ /etc │ - │ - │ Dry run ║
║ homesrv │ local │ /root │ - │ - │ Dry run ║
║ homesrv │ local │ /var │ - │ - │ Dry run ║
║ homesrv │ local │ / │ - │ - │ Dry run ║
║ homesrv │ local │ /mnt/music │ - │ - │ Dry run ║
╟────────────┼─────────┼───────────────────────────┼──────┼─────────┼─────────╢
║ raspberry1 │ local │ /etc │ - │ - │ Dry run ║
║ raspberry1 │ local │ /var │ - │ - │ Dry run ║
║ raspberry1 │ local │ / │ - │ - │ Dry run ║
╚════════════╧═════════╧═══════════════════════════╧══════╧═════════╧═════════╝
> Prune summary
> OK: All tasks successfully completed
╔═════════╤══════════╤═════════╗
║ Repo │ Duration │ Result ║
╟─────────┼──────────┼─────────╢
║ mycloud │ - │ Dry run ║
║ local │ - │ Dry run ║
╚═════════╧══════════╧═════════╝
`Security considerations
A few words about security.
Storing restic password
restic requires a password to encrypt all backup data. You have
two options to store this passsord with schedic:
1. in a separate text file
2. directly in the schedic.yml file
It's recommended to use option 1) and important to mention, that
only the schedic server has to know the password. There is no
need to store this password on the hosts you back up. schedic
will pass the password through SSH using a pipe, so it's even
not visible in the processes environment.
You should use restricted permissions on the password file.
The advantage to not having the password in the schedic.yml is
that it's easier to not disclosure it by accident - e.g. by
showing someone else your schedic configuration or posting
questions to stackoverflow ;)
SSH access to all hosts
In terms of security it's not optimal that the schedic server
requires passwordless (key based) root access to all your hosts you
want to back up. A person who gains access to your schedic server
and the account you're running schedic with has also root access
to all your hosts.
So you really have to take care to protect your schedic server.
On the other hand you can still make use of schedic if this
open SSH configuration is no option for you. You could install
schedic on all hosts and run it locally (besides restic you have
to install anyway on every host). At least you have the benefit
of schedic's nice configuration file and reporting features.
CI pipeline
This project uses a simple GitLab CI pipeline to ensure
code quality (linting and unit testing) and that the code
is buildable. Deployment is a manual step I perform from
my workstation.
The pipeline can be simply executed locally just by launching
ci/local.sh from within the project directory. A temporary
Docker container is created to execute all steps (yarn install,
yarn lint, yarn cover). A few tests depend on the restic
binary, which is downloaded (and cached) automatically
from GitHub.Publishing a new release
`bash
add entry to CHANGELOG.md and
add all files to be commited along with release tag
git add CHANGELOG.md README.mdset new version (commits package.json and all added files)
yarn version [--patch]publish to npm registry
yarn publish
`License
Copyright (c) 2019 Jörn Reder
MIT License
See LICENSE file in this directory.
TODO / IDEAS
- ping log messages for long running commands
- configuration
- new key host.repos
- overrides backup-defaults.repos if present
- put some repo stats at top of mail report
- commands
-
schedic install -- install restic on target hosts
- schedic snapshots -- list snapshots for selected backups
- schedic setup` -- simple wizard to setup a basic schedic.yml