Aug 11 - Update photo storage using r2
This commit is contained in:
@@ -34,6 +34,7 @@
|
||||
24. [Backup CLI](#24-backup-cli-controlbackuppy)
|
||||
25. [Health Dashboard](#25-health-dashboard-health-on-panel)
|
||||
26. [Coding Rules for AI Assistants](#26-coding-rules-for-ai-assistants)
|
||||
29. [Photo Object Storage (R2)](#29-photo-object-storage-r2)
|
||||
|
||||
---
|
||||
|
||||
@@ -1937,4 +1938,72 @@ curl -sI -H "Host: ztest.jqc.app" http://127.0.0.1:8000/ | head -2
|
||||
# 5. Wildcard TLS cert (DNS-01, certbot)
|
||||
# 6. Add to /etc/jqc/control.env: MULTI_TENANT_ENABLED=true
|
||||
# 7. sudo systemctl daemon-reload && sudo systemctl restart jqc
|
||||
```
|
||||
```
|
||||
---
|
||||
|
||||
## 29. Photo Object Storage (R2)
|
||||
|
||||
**Goal:** photo **files** live in Cloudflare R2 (S3-compatible, $0 egress), not on
|
||||
the server's local disk. The DB is not the bottleneck — rows are tiny, PDFs are
|
||||
streamed via `BytesIO` and never written to disk. Only photos accumulate, and in
|
||||
a multi-tenant deployment they accumulate from every tenant onto one volume.
|
||||
|
||||
**No schema change, ever.** The DB stores an unprefixed relative path
|
||||
(`uploads/issue_photos/abc.jpg`) and continues to. That string is the storage
|
||||
**key**; `app/utils/storage.py` maps it to a backend.
|
||||
|
||||
### Key mapping (the rule that matters)
|
||||
|
||||
| Layer | Value |
|
||||
|---|---|
|
||||
| DB (`issues.photo_path`, `result_photos[]`, `mobile_photo_paths[]`, `inspections.form_data`, `inspection_results.photo_path`) | `uploads/issue_photos/abc.jpg` |
|
||||
| Local backend, on disk | `app/static/uploads/issue_photos/abc.jpg` |
|
||||
| S3 backend, object key | `t<tenant_id>/uploads/issue_photos/abc.jpg` |
|
||||
|
||||
The `t<tenant_id>/` prefix is applied **only** inside `S3Backend._object_key()`,
|
||||
from `g.tenant`. It never enters the DB, a template, or an API payload — so the
|
||||
tenant DB stays portable and every caller stays tenant-agnostic. The **local**
|
||||
backend deliberately does not prefix: prefixing would relocate every existing
|
||||
file, and the local layout must remain byte-identical to what predates the seam.
|
||||
Local mode therefore shares one uploads directory across tenants — an isolation
|
||||
weakness inherited from before multi-tenancy, and the reason to move to `s3`.
|
||||
|
||||
`S3Backend.save()` **raises** when `MULTI_TENANT_ENABLED` is true and no tenant
|
||||
is bound, rather than writing an unprefixed key that a second tenant could later
|
||||
collide with. Reads are more forgiving: `read()` / `exists()` / `delete()` try
|
||||
the prefixed key and then the bare key, so objects written before the prefix
|
||||
existed stay reachable. Pinned by `tests/test_storage_backend.py`.
|
||||
|
||||
### Config (all env, per deployment)
|
||||
|
||||
`STORAGE_BACKEND=local|s3` (default `local`), plus `R2_ENDPOINT_URL`,
|
||||
`R2_ACCESS_KEY_ID`, `R2_SECRET_ACCESS_KEY`, `R2_BUCKET`, `R2_PRESIGN_TTL`
|
||||
(default 86400), `R2_MEDIA_FALLBACK` (default false). `boto3` is imported lazily,
|
||||
so `local` deployments never touch it — but it **is** in `requirements.txt`,
|
||||
because `STORAGE_BACKEND=s3` fails at first upload without it.
|
||||
|
||||
`STORAGE_BACKEND` is process-wide, not per-tenant: one bucket, one backend, all
|
||||
tenants, isolated by prefix. A per-tenant backend would need the resolver to
|
||||
carry a storage selector and `get_backend()` to cache per tenant instead of per
|
||||
app — do not half-build it.
|
||||
|
||||
**CSP:** `set_security_headers` in `app/__init__.py` derives the R2 host from
|
||||
`R2_ENDPOINT_URL` and appends it to `img-src` automatically. Presigned images are
|
||||
blocked by the browser without this. A custom R2 domain must be added too.
|
||||
|
||||
### Operator scripts
|
||||
|
||||
- `scripts/audit_photos.py` — read-only. Per tenant, collects every key its DB
|
||||
references and reconciles against disk. Records the baseline that must still
|
||||
resolve after cutover, and flags any key claimed by more than one tenant.
|
||||
- `scripts/migrate_photos_to_r2.py` — copy-only, idempotent, resumable,
|
||||
MD5+size verified. Uploads each tenant's referenced files to `t<id>/…`.
|
||||
Exit 0 only when everything verifies. Orphans (referenced by no tenant) are
|
||||
**not** uploaded — no prefix could legitimately claim them.
|
||||
|
||||
Both establish file ownership from tenant DB references, because the shared
|
||||
local directory carries none. Both need `CONTROL_DATABASE_URL` +
|
||||
`CONTROL_FERNET_KEY`; neither writes to any database.
|
||||
|
||||
**Rollback is one env var:** `STORAGE_BACKEND=local` + restart. The sync never
|
||||
deletes local files, so the old tree is intact indefinitely.
|
||||
|
||||
Reference in New Issue
Block a user