Backups

The one thing to remember

A backup set alone cannot bring Vaultigen back. You also need a few keys that live outside the database and outside the backup directory. That is deliberate: a stolen backup is useless without them, and a stolen live key does not unlock every old restore point. Keep the backups and the keys in different places, and practise a restore before you need one.

What you are protecting

Item	Location	Why it matters
Backup sets	storage/app/private/backups/	Encrypted database snapshot, encrypted files and a manifest.
Backup keyring	storage/app/private/backup.json	Decrypts backup sets.
Installation KEK keyring	storage/app/private/kek.json	Unwraps the vault keys inside the snapshot.
Installation identity	storage/app/private/installation.json	Installation ID, audit key, passkey secret.
APP_KEY	.env	Protects framework-stored authentication data.

On a Docker Compose installation, backup sets are in backups/ and the four key items are the files in data/keys/ (APP_KEY is data/keys/app.key). See Docker Compose below.

System → Operations → Keys lists each recovery input with its identifier and whether it is present. It never shows a value. See Key management.

Before the first backup

php artisan vaultigen:backup-key:generate --if-missing

Copy backup.json, kek.json, installation.json and APP_KEY to your escrow. A sensible minimum: three copies in three places (the live host, a sealed offline copy, and a safe or password manager a second person can open).

Then open System → Operations → Backup. Every readiness check must be green: backup encryption key, backup storage writable, database snapshot tool available, no other operation running.

Running a backup

In the browser: System → Operations → Run backup. The job runs in the queue. Success is reported only after every written file has been re-read and every checksum and the manifest MAC verified.

On the command line (no queue worker needed):

php artisan vaultigen:backup

The command exits non-zero on failure and prints the verified set’s absolute directory.

Schedule and local retention

Use the existing Laravel scheduler and a supervised durable queue worker:

VAULTIGEN_BACKUP_SCHEDULE_ENABLED=true
VAULTIGEN_BACKUP_SCHEDULE="30 2 * * *"
VAULTIGEN_BACKUP_TIMEZONE=UTC
VAULTIGEN_BACKUP_MAX_AGE_HOURS=48
VAULTIGEN_BACKUP_RETENTION_ENABLED=false
VAULTIGEN_BACKUP_RETENTION_DAYS=30

Scheduling defaults to disabled. Remove an old backup-specific cron before enabling it; keep the existing schedule:work or once-per-minute schedule:run running. Scheduled backups queue the same job as the UI without blocking the scheduler. A database, Redis, Beanstalkd, or SQS queue is required; sync/deferred/null queues are refused. Keep retry_after (default 3660 seconds for database queues) above every job timeout and supervisor shutdown grace longer than the longest job. SQS visibility must provide the same protection. Failed/skipped scheduled attempts appear in Operations and Activity; Operations also warns when a successful backup is missing or older than the configured 48-hour limit.

During a backup, write requests return 503, reads keep working, and cleanup waits. Normal completion and caught failures close their own maintenance window. A killed worker or host outage can leave it open: do not clear it by age alone. With Laravel down and web, worker, and scheduler stopped, run php artisan vaultigen:backup:recover --services-stopped. A host lock prevents clearing a live backup; the command also removes abandoned private staging. A worker timeout cleans up its own run.

Local deletion is separately opt-in: set VAULTIGEN_BACKUP_RETENTION_ENABLED=true and a positive retention in days. vaultigen:backups:prune runs at 05:00 in the configured timezone and removes only expired verified completed sets, preserving the newest verified set even when it is old. It skips pending, failed, and unverifiable sets and coordinates with backups and key rotation. It does not confirm off-host copying.

Storage locations

VAULTIGEN_BACKUP_ROOT controls every backup operation; the default remains storage/app/private/backups. Existing database paths backups/<run-id> resolve as <configured-root>/<run-id>. Before changing roots, stop backup/rotation work, move all retained run directories into the new root without an extra backups/ level, preserve private permissions, refresh configuration, and verify each set. VAULTIGEN_BACKUP_TEMPORARY_ROOT defaults to storage/app/private/backup-work; keep it private and on disk, with space for plaintext and encrypted snapshots. Capacity checks include VAULTIGEN_MIN_FREE_BYTES as reserved headroom, but estimates do not replace monitoring real free space.

Normal sets contain no .env, key files, resolved configuration caches, logs, or temporary plaintext. Vault images and site icons are in database rows; encrypted attachments and retained import sources are copied as objects. Temporary uploads and runtime caches are not recovery inputs.

Docker Compose

Everything above applies; only the commands and paths differ.

• Backup key: setup creates data/keys/backup.json. Save the whole data/keys/ folder off the server as described in Installation.
• Schedule: the template enables the nightly backup at 02:30 UTC. Change VAULTIGEN_BACKUP_SCHEDULE and VAULTIGEN_BACKUP_TIMEZONE in .env, or set VAULTIGEN_BACKUP_SCHEDULE_ENABLED=false, then run docker compose up -d --wait. The bundled worker and scheduler already satisfy the queue requirements above.
• On demand: docker compose exec web php artisan vaultigen:backup prints the set, for example /backups/<id>, which is backups/<id> next to compose.yaml. It exits non-zero on failure.
• Verify a set: docker compose exec web php artisan vaultigen:backup:verify /backups/<id>.
• Retention: set VAULTIGEN_BACKUP_RETENTION_ENABLED=true and VAULTIGEN_BACKUP_RETENTION_DAYS in .env. Without it, sets accumulate; watch the disk.
• Before every update the migrate service takes a verified backup (trigger migration) if migrations are pending.

Copy the sets off the server every night, well after the backup has finished. The folder belongs to the container user, so run the copy from root’s crontab (sudo crontab -e):

# Every night at 04:15, after the 02:30 backup.
15 4 * * * rsync -a /home/admin/vaultigen/backups/ backup-copy@backup-host:vaultigen-backups/

Prefer a destination that this server cannot delete or overwrite, for example a restic repository or object storage with object lock. A copy started while a backup is still running can contain an incomplete set; the next run completes it, and only sets that verify can be restored.

A backup set never contains data/keys/. Keep the key copy and the backup copies in different places.

Interrupted backup. If a killed worker or a host outage left a backup’s maintenance window open (writes keep returning 503), stop the application services and clear it with the stack’s own image:

docker compose stop web worker scheduler
docker compose run --rm migrate artisan down
docker compose run --rm migrate artisan vaultigen:backup:recover --services-stopped
docker compose run --rm migrate artisan up
docker compose up -d --wait

Get every set off the host

A backup next to the live database dies with that database. After each run, copy the new set to storage that the Vaultigen process cannot write to:

• an offline disk you rotate, or
• object storage with object lock, or with credentials the application does not hold.

Let something other than Vaultigen make the copy, for example a host account that can read the backup directory but cannot modify it.

Verify immediately before copying, from an operator-controlled account:

php artisan vaultigen:backup:verify /path/to/completed-set && \
  rsync -a /path/to/completed-set/ backup-copy@backup-host:/protected/vaultigen/set-id/

Keep copy credentials outside the application, and protect destination retention independently. Keep every old KEK and backup key needed by retained or off-host pre-rotation sets; retiring a key can make those independent copies unreadable.

When a backup fails

Failures appear on the Operations page and in Activity with a short code:

Code	Meaning	What to do
backup_key_missing	No backup keyring, or the active key is missing	Run vaultigen:backup-key:generate --if-missing or restore backup.json.
snapshot_unreadable / snapshot_unwritable	Snapshot tool or backup directory failed, often a full disk	Check free space and permissions.
object_missing	A referenced attachment or import file is gone	Find out why before you trust the set.
object_unreadable / object_unwritable	An object could not be copied	Check permissions on the attachment store and backup directory.
integrity_failed	The set did not verify after writing	The set is deleted; the next run starts clean.

A failed run never invalidates the previous good set.

Checklist

• A backup key exists and is escrowed separately from the live keyring.
• KEK, APP_KEY, installation identity and audit key are escrowed with a second person.
• Backups run on a schedule and complete.
• Every set is copied to storage the application cannot rewrite.
• At least one restore drill has succeeded and its notes are written down.
• After every key rotation, a fresh backup exists and old keys are still available for retained sets.