ci-pg-restore: A cached perf optimized restoration of a PostgreSQL database dump


usage: ci-pg-restore [-h] -d DBNAME --cache-name CACHE_NAME
                     [--deps-files "some/**/file1.sql\nfile2.sql\n..."]
                     [--deps-files-cmd "find . -name '*.sql'"]
                     [--cache-db-max-age 42, 42s, 30m, 8h, 3d, 1w]
                     [--restore-db-max-age 42, 42s, 30m, 8h, 3d, 1w]
                     ...
A cached perf optimized restoration of a PostgreSQL database dump.
To access the databases, the tool assumes that psql CLI is installed
in the system, and PGUSER/PGPASSWORD are correctly configured.
The tool uses the following observation: if we have a large rarely
changed *.sql file with a database dump, and we need to restore it
multiple times (in e.g. CI environment, when running integration
test matrix), it's way faster to restore it only once in some cache
"template" database and then run CREATE DATABASE WITH TEMPLATE to
clone that cache database multiple times.
While restoring, the tool behaves nicely in terms of concurrency: if
multiple processes of the tool are running at the same time, they
wait for each other.
The tool also takes care of cleaning up the cache databases and the
restored databases. This is typically useful in CI environment, when
the databases are not needed anymore after the tests are finished.
It also drops the databases having the name equals to the restored
database name following a "~" suffix ( means "any string"); this
allows the tests in CI to create extra databases if needed.
positional arguments:
  shell command         a shell command which will be run to restore
                        the database if its cache is invalid; the
                        database name will also be passed via
                        PGDATABASE environment variable; you may
                        call the regular psql here or use any other
                        database migration tool

options: -h, --help show this help message and exit -d, --dbname DBNAME the name of the database to restore; it will be pre-created empty prior to calling the restoration command --cache-name CACHE_NAME cache name; the tool will create an internal cached "template" database suffixed with the digest of that name plus any --deps-files or --deps-files-cmd involved --deps-files "some/**/file1.sql\nfile2.sql\n..." a newline separated list of file name wildcards involved in the restoration (you may use this arg multiple times; all files must exist); the tool will create an internal cached "template" database suffixed with the digest of those files content --deps-files-cmd "find . -name '*.sql'" a shell command which should return a newline separated list of all file names involved in the restoration; see --deps- files for details --cache-db-max-age 42, 42s, 30m, 8h, 3d, 1w if set, the cached "template" databases will be deleted if they are not used for that long --restore-db-max-age 42, 42s, 30m, 8h, 3d, 1w if set, the restored databases will be deleted if they are not used for that long``

usage: ci-pg-restore [-h] -d DBNAME --cache-name CACHE_NAME
[--deps-files "some/**/file1.sql\nfile2.sql\n..."]
[--deps-files-cmd "find . -name '*.sql'"]
[--cache-db-max-age 42, 42s, 30m, 8h, 3d, 1w]
[--restore-db-max-age 42, 42s, 30m, 8h, 3d, 1w]
...

A cached perf optimized restoration of a PostgreSQL database dump.

To access the databases, the tool assumes that psql CLI is installed
in the system, and PGUSER/PGPASSWORD are correctly configured.

The tool uses the following observation: if we have a large rarely
changed *.sql file with a database dump, and we need to restore it
multiple times (in e.g. CI environment, when running integration
test matrix), it's way faster to restore it only once in some cache
"template" database and then run CREATE DATABASE WITH TEMPLATE to
clone that cache database multiple times.

While restoring, the tool behaves nicely in terms of concurrency: if
multiple processes of the tool are running at the same time, they
wait for each other.

The tool also takes care of cleaning up the cache databases and the
restored databases. This is typically useful in CI environment, when
the databases are not needed anymore after the tests are finished.
It also drops the databases having the name equals to the restored
database name following a "~" suffix ( means "any string"); this
allows the tests in CI to create extra databases if needed.

positional arguments:
shell command a shell command which will be run to restore
the database if its cache is invalid; the
database name will also be passed via
PGDATABASE environment variable; you may
call the regular psql here or use any other
database migration tool

options:
  -h, --help            show this help message and exit
  -d, --dbname DBNAME   the name of the database to restore; it will
                        be pre-created empty prior to calling the
                        restoration command
  --cache-name CACHE_NAME
                        cache name; the tool will create an internal
                        cached "template" database suffixed with the
                        digest of that name plus any --deps-files or
                        --deps-files-cmd involved
  --deps-files "some/**/file1.sql\nfile2.sql\n..."
                        a newline separated list of file name
                        wildcards involved in the restoration (you
                        may use this arg multiple times; all files
                        must exist); the tool will create an
                        internal cached "template" database suffixed
                        with the digest of those files content
  --deps-files-cmd "find . -name '*.sql'"
                        a shell command which should return a
                        newline separated list of all file names
                        involved in the restoration; see --deps-
                        files for details
  --cache-db-max-age 42, 42s, 30m, 8h, 3d, 1w
                        if set, the cached "template" databases will
                        be deleted if they are not used for that
                        long
  --restore-db-max-age 42, 42s, 30m, 8h, 3d, 1w
                        if set, the restored databases will be
                        deleted if they are not used for that long