Backups & restore¶
Précis Finance MCP ships a backup subsystem driven by one declarative file,
instance/backup.yml: one scheduled run produces a bundle of configured managed
stores — a PostgreSQL dump, a ClickHouse backup when selected/managed, and a tarball of your
instance/ config — at a local volume or an S3-compatible bucket, with
retention pruning, a checksum manifest, and a restore/drill command. The
shipped tier is dump (nightly full backups by default), so your recovery
point equals the backup cadence; a pitr tier (continuous WAL archiving) is
reserved in the config and rejected until it ships.
What a bundle contains¶
| Store | Artifact | Mechanism |
|---|---|---|
Platform PostgreSQL (users, profiles, load_history, audit tables) |
pg/ |
pg_dump --format=custom |
ClickHouse (live, staging, semantic, plus planning when present) |
ch/ |
server-side BACKUP DATABASE … TO Disk('precis_backups') |
Your model (instance/ — catalogue, semantic SQL, integrations, scenarios) |
instance/ |
tarball, git SHA recorded |
| Manifest (checksums, sizes, package version, per-table row counts) | manifest/ |
written last — its presence means the bundle is complete |
Deliberately not in the bundle: secrets (your
secret manager is their durability boundary; backup config holds credential
references only, and validation rejects inline values). Your instance/
git remote remains the source of truth for the model — the bundled tarball
makes a restore self-contained and records exactly which model state was
live.
A store failure degrades the run to partial rather than aborting the
bundle, and any non-success outcome can POST to a webhook.
CH backups detect optional planning once and record the exact database list
in the existing manifest; row-count baselines and restore/drill operations use
the same list. The backup identity must be able to see and back up that database.
Inventory-query failure fails the CH store rather than silently omitting data.
Open deployments without planning still back up only the original three
databases. Legacy manifests without the list restore only those three and
leave existing planning untouched, including with --force; they cannot recover
planning data never captured. Take a new backup after upgrading for that
coverage. There is no new operator-maintained configuration file.
Configuring¶
Databricks and customer-managed warehouse recovery¶
PRECIS_ANALYTICAL_BACKEND=databricks makes warehouse recovery external.
The customer/operator owns recovery of all Databricks data, including Précis
scenario metadata and any application-owned tables stored there. Warehouse
durability alone does not establish a configured customer recovery process.
No Databricks backup/export or warehouse restore is performed by these tools.
Précis continues backing up PostgreSQL and instance configuration, plus files
when scope.files: managed. The existing backup.yml format is unchanged;
any legacy managed-CH scope is excluded automatically in Databricks mode.
The manifest records analytical_backend: databricks and a skipped
databricks store with external ownership. No CH client, disk XML or warehouse
credential is needed. backup init probes the platform backup destination and
reports the external warehouse boundary instead of rendering CH disk settings.
backup list --json includes the recorded backend. Success means the configured
platform stores succeeded; it does not mean warehouse data was backed up.
Use the Databricks overlay
for all Compose commands in this mode, including backup commands. Its optional
backup sidecar runs as UID 10001, matching the owner of private managed files;
the image initializes the shared /backups volume with write access. Existing
volumes retain their existing permissions. The CH sidecar's separate file-reader
limitation remains; this does not change ownership of CH backup directories.
Restore/drill accepts only successful artifacts actually present. It rejects
CH artifacts for a Databricks deployment or bundle before artifact fetch or
database operations, including with --force. Legacy manifests without the
backend field are treated as CH bundles. Explicit platform-only recovery from
an older bundle is supported with --stores postgres,instance,files when all
three have successful artifacts; select only the available stores. A database
drill can use --stores postgres --drill and explicitly reports Databricks
recovery as unexercised. Coordinate the recovered platform/config/files point
with the customer-managed warehouse recovery point; these commands do not
orchestrate consistency across them. Recover secrets through the secret manager.
Run the CLI inside the server container
(docker compose -f deploy/docker-compose.yml exec precis-mcp
precis-finance-mcp-admin backup …).
Prerequisites. The multi-user bundle deployed; for an S3 destination, the
s3 extra in the image (pip install ".[s3]" when building from source) and
two credential pairs — a writer that cannot delete objects and a
reader for restore — plus, for the ransomware posture, object-lock
(WORM) and a lifecycle policy on the bucket (the bucket policy is yours;
Précis Finance MCP verifies it). Bring-your-own ClickHouse needs the BACKUP grant
on the configured user (GRANT BACKUP ON *.* TO <chuser>); the bundled
service's default user has it implicitly.
- Write
instance/backup.yml. The bundled demo instance ships the local-volume default. An S3 variant:
mode: dump
destination:
type: s3
endpoint: https://s3.eu-central-1.amazonaws.com # omit for AWS; required for MinIO/R2
bucket: acme-precis-backups
prefix: prod
region: eu-central-1
credentials:
writer: BACKUP_WRITER # → BACKUP_WRITER_ACCESS_KEY_ID(_FILE) + _SECRET_ACCESS_KEY(_FILE)
reader: BACKUP_READER
schedule:
cron: "30 2 * * *"
retention:
postgres: { keep: 14, days: 90 }
clickhouse: { keep: 14, days: 90 }
instance: { keep: 30 }
scope:
postgres: managed
clickhouse: managed
files: external # back up artifact blobs separately; see Shared artifacts below
encryption:
kms_key_id: null # SSE-KMS key for process-side puts; null = bucket default
expect_worm: true
alert:
webhook_url: https://hooks.example.com/precis-backup
The file holds credential references, never values — safe to commit to your instance repository.
With files: external, this bundle excludes artifact blobs. Back up
USER_DATA_DIR separately and restore it together with the matching platform
database backup; see Shared artifacts for the sidecar's
file-permission limitation.
-
Set the credentials in
deploy/.env(or as_FILEsecrets):BACKUP_WRITER_ACCESS_KEY_ID,BACKUP_WRITER_SECRET_ACCESS_KEY,BACKUP_READER_ACCESS_KEY_ID,BACKUP_READER_SECRET_ACCESS_KEY. A local destination needs none. -
Validate:
precis-finance-mcp-admin backup validate(static checks). -
Render and check:
precis-finance-mcp-admin backup init. Writes the ClickHouse backup-disk config todeploy/secrets/precis_backup_disk.xml(for S3 it embeds the resolved writer credential and is written mode 0600, which is why it lives in the secrets area and is regenerated, never hand-edited; a local-destination render holds no secret), probes the destination for writability, and checks whether ClickHouse sees theprecis_backupsdisk — on first run that check warns, expected. -
Mount and restart ClickHouse: set
PRECIS_BACKUP_CH_CONFIG=./secrets/precis_backup_disk.xmlindeploy/.env,docker compose up -d clickhouse, re-runbackup init— the disk check now passes. S3 only: the 0600 render is owned by the CLI's uid, but the clickhouse service reads the mount as its own user — transfer ownership first (chown 101 deploy/secrets/precis_backup_disk.xmlfor the stock image), and re-do that after every credential rotation re-render. -
First manual run:
precis-finance-mcp-admin backup run, thenprecis-finance-mcp-admin backup listto confirm the bundle. -
Enable the schedule: append
backuptoCOMPOSE_PROFILESanddocker compose up -d— thebackup-schedulersidecar fires the same code path on the configured cron.
Credential rotation: update the secret values → re-run
backup init (re-renders the XML) → restart the clickhouse service.
Restore and drills¶
A backup that has not been restored is not a backup
Run a drill before trusting the chain with production data, and quarterly after.
precis-finance-mcp-admin backup restore --id <run_id> --drill
The drill restores into side databases (precis_platform_drill,
live_drill / staging_drill / semantic_drill), verifies per-table row
counts against the manifest, records the outcome in backup_history, and
drops the drill databases — it never touches live data. (Current limitation:
the drill verifies tables and checksums; restored semantic_drill views
still reference live.*, so metric-level verification means pointing an app
instance at the restored stores.)
The drill requires a successful PostgreSQL or ClickHouse artifact. Instance/files artifacts are fetched and checksum-checked where checksums exist, but reported as skipped because database drills do not exercise extraction. Selecting only instance/files for a drill is rejected.
A real restore:
precis-finance-mcp-admin backup restore --id <run_id> [--stores postgres,clickhouse,instance] [--force]
Checksums are verified before anything is touched; a non-empty target is
refused without --force. The instance artifact is always extracted
beside your live config, never over it — review and swap it in yourself.
Every explicitly requested store must have a successful, keyed artifact in
the bundle; absent, failed, or skipped stores reject the whole selection
before restoring anything. Empty selections are rejected too. Without
--stores, available successful stores are selected, even from a partial
bundle: inspect backup list to establish which stores were backed up.
After a full host loss, the order is: images + docker compose up →
backup init (ClickHouse disk) → restore postgres → restore clickhouse →
swap instance config in → restart → drill or a known-scenario metric
spot-check.
Verifying the chain¶
- [ ]
backup runexits 0;backup listshows the bundle with every storesuccess. - [ ] The destination holds
pg/,ch/,instance/,manifest/objects for the run id. - [ ]
backup restore --id <run> --drillexits 0 with all row counts matching. - [ ]
backup_history(platform PostgreSQL) shows the run and the drill, bothoutcome='success'. - [ ] The sidecar fires at the next cron tick (
docker compose logs backup-scheduler). - [ ] With
expect_worm: true: deleting a backup object with the writer credential fails, andbackup initverifies object-lock without warning. - [ ] Webhook configured: stop Postgres, run
backup run, confirm the POST arrives and the run reportspartial.
Disabling¶
Backups are additive: remove backup from COMPOSE_PROFILES and
docker compose up -d. The ClickHouse disk config can stay mounted (an
unused disk definition is inert). Existing bundles are untouched.
Related¶
- Environment variable reference
— the
BACKUP_*/PRECIS_BACKUP_*variables. - Security model — where the writer/reader split and the WORM chain sit in the wider posture.
- Observability — alerting;
backup_historyis the outcome record. - Troubleshooting
Shared artifacts¶
Artifact identities, upload reservations and retired-blob cleanup metadata live in the platform PostgreSQL database. Managed bytes live in USER_DATA_DIR, mounted read-only by the open backup sidecar. Back up and restore these together. Redis snapshots are disposable and excluded: rerun queries after losing or restoring the cache. Renew transfer capabilities for restored files under current grants; expired URLs are not durable recovery identifiers.
Current limitation: the shipped sidecar runs as ClickHouse UID 101,
while managed artifact blobs are private to application UID 10001
(mode 0600). The read-only mount does not grant access to these blobs;
scope.files: managed is not reliable in this sidecar configuration.
Managed-file coverage requires a reader identity and backup-volume layout
that also preserve ClickHouse checksum reads and retention deletion. Track
and validate that deployment work before relying on the bundle for recovery
of managed artifacts; do not make private blobs world-readable.