Aug 11 - Update photo storage using r2

This commit is contained in:
2026-08-11 11:30:57 -04:00
parent 3c35835505
commit c9984e7ae6
6 changed files with 999 additions and 1 deletions
+70 -1
View File
@@ -34,6 +34,7 @@
24. [Backup CLI](#24-backup-cli-controlbackuppy)
25. [Health Dashboard](#25-health-dashboard-health-on-panel)
26. [Coding Rules for AI Assistants](#26-coding-rules-for-ai-assistants)
29. [Photo Object Storage (R2)](#29-photo-object-storage-r2)
---
@@ -1937,4 +1938,72 @@ curl -sI -H "Host: ztest.jqc.app" http://127.0.0.1:8000/ | head -2
# 5. Wildcard TLS cert (DNS-01, certbot)
# 6. Add to /etc/jqc/control.env: MULTI_TENANT_ENABLED=true
# 7. sudo systemctl daemon-reload && sudo systemctl restart jqc
```
```
---
## 29. Photo Object Storage (R2)
**Goal:** photo **files** live in Cloudflare R2 (S3-compatible, $0 egress), not on
the server's local disk. The DB is not the bottleneck — rows are tiny, PDFs are
streamed via `BytesIO` and never written to disk. Only photos accumulate, and in
a multi-tenant deployment they accumulate from every tenant onto one volume.
**No schema change, ever.** The DB stores an unprefixed relative path
(`uploads/issue_photos/abc.jpg`) and continues to. That string is the storage
**key**; `app/utils/storage.py` maps it to a backend.
### Key mapping (the rule that matters)
| Layer | Value |
|---|---|
| DB (`issues.photo_path`, `result_photos[]`, `mobile_photo_paths[]`, `inspections.form_data`, `inspection_results.photo_path`) | `uploads/issue_photos/abc.jpg` |
| Local backend, on disk | `app/static/uploads/issue_photos/abc.jpg` |
| S3 backend, object key | `t<tenant_id>/uploads/issue_photos/abc.jpg` |
The `t<tenant_id>/` prefix is applied **only** inside `S3Backend._object_key()`,
from `g.tenant`. It never enters the DB, a template, or an API payload — so the
tenant DB stays portable and every caller stays tenant-agnostic. The **local**
backend deliberately does not prefix: prefixing would relocate every existing
file, and the local layout must remain byte-identical to what predates the seam.
Local mode therefore shares one uploads directory across tenants — an isolation
weakness inherited from before multi-tenancy, and the reason to move to `s3`.
`S3Backend.save()` **raises** when `MULTI_TENANT_ENABLED` is true and no tenant
is bound, rather than writing an unprefixed key that a second tenant could later
collide with. Reads are more forgiving: `read()` / `exists()` / `delete()` try
the prefixed key and then the bare key, so objects written before the prefix
existed stay reachable. Pinned by `tests/test_storage_backend.py`.
### Config (all env, per deployment)
`STORAGE_BACKEND=local|s3` (default `local`), plus `R2_ENDPOINT_URL`,
`R2_ACCESS_KEY_ID`, `R2_SECRET_ACCESS_KEY`, `R2_BUCKET`, `R2_PRESIGN_TTL`
(default 86400), `R2_MEDIA_FALLBACK` (default false). `boto3` is imported lazily,
so `local` deployments never touch it — but it **is** in `requirements.txt`,
because `STORAGE_BACKEND=s3` fails at first upload without it.
`STORAGE_BACKEND` is process-wide, not per-tenant: one bucket, one backend, all
tenants, isolated by prefix. A per-tenant backend would need the resolver to
carry a storage selector and `get_backend()` to cache per tenant instead of per
app — do not half-build it.
**CSP:** `set_security_headers` in `app/__init__.py` derives the R2 host from
`R2_ENDPOINT_URL` and appends it to `img-src` automatically. Presigned images are
blocked by the browser without this. A custom R2 domain must be added too.
### Operator scripts
- `scripts/audit_photos.py` — read-only. Per tenant, collects every key its DB
references and reconciles against disk. Records the baseline that must still
resolve after cutover, and flags any key claimed by more than one tenant.
- `scripts/migrate_photos_to_r2.py` — copy-only, idempotent, resumable,
MD5+size verified. Uploads each tenant's referenced files to `t<id>/…`.
Exit 0 only when everything verifies. Orphans (referenced by no tenant) are
**not** uploaded — no prefix could legitimately claim them.
Both establish file ownership from tenant DB references, because the shared
local directory carries none. Both need `CONTROL_DATABASE_URL` +
`CONTROL_FERNET_KEY`; neither writes to any database.
**Rollback is one env var:** `STORAGE_BACKEND=local` + restart. The sync never
deletes local files, so the old tree is intact indefinitely.