Jul 14 - Preparation for using CDN
This commit is contained in:
@@ -29,6 +29,7 @@
|
||||
19. [Infrastructure](#19-infrastructure)
|
||||
20. [Known Constraints & Hard Rules](#20-known-constraints--hard-rules)
|
||||
21. [Change Philosophy](#21-change-philosophy)
|
||||
22. [Object Storage Migration (R2)](#22-object-storage-migration-r2)
|
||||
|
||||
---
|
||||
|
||||
@@ -603,7 +604,20 @@ The last eight styles (`SummaryTitle` through `TableCell`) were added for the fa
|
||||
|
||||
`stats.py` now returns `severity_breakdown` dict alongside the existing KPIs. Derived from the already-loaded `open_issues_all` list — zero extra DB queries.
|
||||
|
||||
### Issue API — `_issue_payload()` fields
|
||||
### Scheduled Inspections + Issue Handler Endpoints (July 2026)
|
||||
|
||||
| Endpoint | Auth | Description |
|
||||
|---|---|---|
|
||||
| `GET /api/v1/scheduled-inspections` | jwt_required | Active scheduled/recurring assignments (`app/api/scheduled.py`, new blueprint). **Inspector:** only rows where `inspector_id == self`. **admin/director/PM:** all active. Sorted by `next_due_date`. Returns per row: `id`, `facility_id`, `facility_name`, `template_id`, `template_name`, `inspector_id`, `frequency`, `frequency_label`, `next_due_date` (ISO date), `is_overdue`, `notes`, plus `total`/`limit`/`offset`. Powers the iPad "Scheduled" section on Dashboard + My Inspections. Read-only — the schedule lifecycle (fulfil/roll-forward) stays web-driven; the iPad "Start" just seeds the new-inspection flow. |
|
||||
| `PATCH /api/v1/issues/<id>/handler` | jwt_required | Set "Handled By" from the iPad (`update_issue_handler`). Body: `{ "handler_type": "internal"\|"facility"\|"vendor", ...optional detail keys }`. Detail keys (`facility_handler_name/contact/notes`, `vendor_name/contact/notes`) are updated only when present; empty string clears a field. **`log_action()` after commit.** |
|
||||
|
||||
**Handler permission divergence — deliberate (see rule 78).** The web issue form limits handler edits to admin/director/PM. This API endpoint additionally allows the assigned **inspector**, scoped by `get_inspector_scope()` (403 if the issue's facility isn't in their contracted set). The iPad is a field tool; inspectors set the handler from Issue Detail. Do not "align" the API back to the web restriction without explicit direction.
|
||||
|
||||
The new `scheduled` blueprint is registered in `app/api/__init__.py` and CSRF-exempted in `app/__init__.py` (`csrf.exempt(_api_scheduled_bp)` — parent-exempt does not cascade to child blueprints, per the CSRF pattern above).
|
||||
|
||||
No migration was needed for either feature: the `scheduled_inspections` table (phase36) and the issue handler columns (phase35) already existed; both additions are pure serialization + one new route.
|
||||
|
||||
|
||||
|
||||
```python
|
||||
{
|
||||
@@ -621,6 +635,11 @@ The last eight styles (`SummaryTitle` through `TableCell`) were added for the fa
|
||||
# Phase E additions:
|
||||
'area_name', # name of the Area the issue was flagged in (nullable)
|
||||
'assigned_to_name', # display_name of currently assigned User (nullable)
|
||||
# Handler ("Handled By", July 2026) additions:
|
||||
'handler_type', # 'internal' | 'facility' | 'vendor' (defaults 'internal')
|
||||
'handler_label', # human-readable label (Issue.handler_label property)
|
||||
'facility_handler_name', 'facility_handler_contact', 'facility_handler_notes', # nullable
|
||||
'vendor_name', 'vendor_contact', 'vendor_notes', # nullable
|
||||
}
|
||||
```
|
||||
|
||||
@@ -1279,6 +1298,8 @@ timeout = 30
|
||||
| 74 | **The `public` blueprint (`/f/*`) is login-free — keep it occupant-safe** | Pages are addressed by unguessable `public_token` (never facility id), 404 on inactive/unknown facilities, and expose only a quality rating, last-inspected date, and open-issue COUNT — **never** issue descriptions, inspector names, per-item scores, or any other facility's data. The `report` POST must stay CSRF-protected (Flask-WTF form), rate-limited, and honeypot-guarded; public-reported issues are created with `reported_by=NULL`, `severity='medium'`, and routed through `notify_by_matrix('issue_created', facility_id=...)`. Do not add fields that leak internal detail, and do not reuse `render_template('base.html')` here — the public page is a standalone template with no authenticated nav. |
|
||||
| 75 | **Email is stored lowercased; look it up case-insensitively** | User/customer email is normalized to `.strip().lower()` at every write site (`auth.py` profile/create/edit, `customers.py` invite/edit). Forgot-password lookup uses `db.func.lower(User.email) == input` so a mixed-case legacy row still matches — a plain `filter_by(email=...)` silently missed them and sent no reset (the failure was invisible because of the generic "if an account exists…" message). Keep both halves: normalize on write, case-insensitive on lookup. |
|
||||
| 76 | **Transactional email `From` must be an SMTP-authorized identity, per-domain branding via display name only** | Reset-password sends from `MAIL_DEFAULT_SENDER`; customer invite sends from `branded_sender()` = `(per-domain display name, authorized address)`. A per-host `noreply@<subdomain>` sender is accepted by the relay then dropped by SPF/DMARC. See rule 64 and §8 `mail_utils.py`. |
|
||||
| 77 | **`GET /api/v1/scheduled-inspections` is inspector-scoped by `inspector_id`, admin/director/PM see all** | New `app/api/scheduled.py` blueprint. Register in `app/api/__init__.py` AND `csrf.exempt(_api_scheduled_bp)` in `app/__init__.py` — the child-blueprint CSRF exemption never cascades from the parent. Read-only; do not add write/fulfil endpoints here (the schedule lifecycle stays in `routes/scheduled_inspections.py`). |
|
||||
| 78 | **`PATCH /api/v1/issues/<id>/handler` allows the inspector on purpose — do NOT align it to the web form's admin/director/PM restriction** | The iPad lets the assigned inspector set "Handled By" from the field, scoped via `get_inspector_scope()` (403 if the issue's facility isn't contracted). This is a deliberate divergence from the web form. `_issue_payload()` must keep returning all 8 handler fields (`handler_type`, `handler_label`, `facility_handler_*`, `vendor_*`) or the iPad's "Handled By" panel silently blanks — same failure mode as rule 40. |
|
||||
|
||||
---
|
||||
|
||||
@@ -1291,4 +1312,71 @@ timeout = 30
|
||||
5. **Migration existence checks** — all migrations safe to re-run
|
||||
6. **Full file contents for 1–3 file changes**; deployment map for larger changesets
|
||||
7. **Explicit deploy instructions** — migration steps separated from code steps
|
||||
8. **Root cause analysis** on errors — never apply temporary workarounds
|
||||
8. **Root cause analysis** on errors — never apply temporary workarounds
|
||||
---
|
||||
|
||||
## 22. Object Storage Migration (R2)
|
||||
|
||||
**Goal:** move photo **files** off the server's local disk (`app/static/uploads/`) to **Cloudflare R2** (S3-compatible, $0 egress) so storage expands without touching the server. The database is NOT the bottleneck — rows are tiny; PDFs are streamed (`BytesIO`), never written to disk. Only photos accumulate.
|
||||
|
||||
**No schema change.** The DB already stores a relative path string (`uploads/issue_photos/abc.jpg`). That exact string becomes the **object key** in R2, so rows never change and `media_url()` resolves the same key on either backend.
|
||||
|
||||
**Serving decision:** private bucket + **presigned GET URLs** (TTL 24h). This is *more* private than today (static files are served unauthenticated). `media_url()` is the single seam — switching to a public custom-domain CDN URL later is a one-function change.
|
||||
|
||||
**Complete photo-path inventory (the migration MUST cover all 5 — missing one loses those photos from the UI even if the file exists):**
|
||||
1. `Issue.photo_path` (string)
|
||||
2. `Issue.mobile_photo_paths[]` (JSON list — iPad evidence)
|
||||
3. `Issue.result_photos[]` (JSON list — web resolution photos)
|
||||
4. `Inspection.photo_path` (string)
|
||||
5. `Inspection.form_data{}` — any `uploads/...` string (image fields; walk nested)
|
||||
- Signatures are inline `data:` base64 in the DB, **not files** → correctly ignored.
|
||||
|
||||
**Zero-loss invariants (non-negotiable):**
|
||||
- Object key **==** the relative path string already in the DB. Never rewrite DB paths.
|
||||
- **Copy, never move.** The migration only uploads. Local `static/uploads/` is untouched through cutover and a 30-day safety window.
|
||||
- Per-file **MD5 + size verification** (local hash vs R2 `head_object`) — a file counts as migrated only when it matches.
|
||||
- Cutover is **gated**: only flip `STORAGE_BACKEND=s3` when `verified == referenced-present` (from the Phase 0 baseline) with zero mismatches.
|
||||
- S3 backend **falls back to the local file** if a key is absent (transition safety).
|
||||
- **Rollback = one env var** (`STORAGE_BACKEND=local` + restart). Local files never left disk.
|
||||
- Magic-byte validation stays in the **caller**, before `storage.save()`.
|
||||
- `media_url()` is the ONLY place a photo URL is built server-side — no bare `url_for('static', filename=<photo>)` anywhere after Phase 1.
|
||||
|
||||
### Phase 0 — Audit (READ-ONLY) — ✅ delivered
|
||||
`scripts/audit_photos.py`. Reads all 5 sources + disk, reconciles, writes a JSON baseline. Writes nothing to the DB or uploads tree.
|
||||
- [x] Script delivered (`scripts/audit_photos.py`).
|
||||
- [ ] Run on production: `python scripts/audit_photos.py --report /home/jqc/photo_audit_baseline.json`
|
||||
- [ ] Record the baseline number: **"N photos must still resolve after cutover."**
|
||||
- [ ] Investigate any `missing_on_disk` entries — these are PRE-EXISTING broken references, surfaced now so they can't be blamed on the move.
|
||||
|
||||
### Phase 1 — Storage abstraction + local backend (NO-OP, safe to deploy)
|
||||
- [ ] `app/utils/storage.py` — interface: `save(file_obj, subfolder) -> key`, `read(key) -> bytes`, `delete(key)`, `exists(key) -> bool`, `media_url(key) -> str`.
|
||||
- [ ] `LocalBackend` = byte-for-byte current behavior (`static/uploads` write; `media_url` = `url_for('static', filename=key)`).
|
||||
- [ ] Config: `STORAGE_BACKEND=local|s3` (default `local`).
|
||||
- [ ] Route `_save_photo()` in `inspections.py` + `issues.py`, and `api/photos.py`, through `storage.save()` (keep magic-byte check in the caller).
|
||||
- [ ] Templates: swap `url_for('static', filename=path)` → `media_url(path)` in `issues/view.html`, `inspections/view.html`, and any other photo render sites.
|
||||
- [ ] Deploy with `STORAGE_BACKEND=local` → identical behavior; regression check that web + iPad photos still load.
|
||||
|
||||
### Phase 2 — R2 backend (build, do NOT flip yet)
|
||||
- [ ] `pip install boto3`; add to `requirements.txt`.
|
||||
- [ ] `.env`: `STORAGE_BACKEND`, `R2_ENDPOINT_URL`, `R2_ACCESS_KEY_ID`, `R2_SECRET_ACCESS_KEY`, `R2_BUCKET`, `R2_PRESIGN_TTL=86400`.
|
||||
- [ ] `S3Backend`: boto3 client (`endpoint_url=R2_ENDPOINT_URL`, `region_name='auto'`). `save` → `put_object` (set `ContentType`); `media_url` → `generate_presigned_url('get_object', ..., ExpiresIn=TTL)`; `read` → `get_object`; `exists` → `head_object`.
|
||||
- [ ] Fallback: if `head_object` 404s, `S3Backend.read`/`media_url` fall back to the local static path.
|
||||
- [ ] Leave `STORAGE_BACKEND=local` in production for now.
|
||||
|
||||
### Phase 3 — Verified migration sync
|
||||
- [ ] `scripts/migrate_photos_to_r2.py` — idempotent, resumable, checksummed.
|
||||
- Upload **every** file under `static/uploads/` (orphans included — completeness over reference-only).
|
||||
- Per file: local MD5 → `put_object` → `head_object` → assert ETag/size match; append result (`ok`/`uploaded`/`mismatch`/`error`) to a JSON log.
|
||||
- Skip keys already present **and** verified (resumable).
|
||||
- [ ] Bulk run ahead of time (can run while `local` is still live — it only reads local + writes R2).
|
||||
- [ ] Re-run immediately before cutover to catch the delta (idempotent).
|
||||
- [ ] Gate check: `verified == referenced-present` (Phase 0 baseline) AND zero mismatches.
|
||||
|
||||
### Phase 4 — Cutover + iOS
|
||||
- [ ] Maintenance window: final incremental sync → set `STORAGE_BACKEND=s3` → `systemctl restart janitorial-qc`.
|
||||
- [ ] Smoke test: existing web issue/inspection photos load via presigned URLs; new upload from **web** and **iPad** lands in R2.
|
||||
- [ ] iOS: add absolute `photo_url` to issue/inspection payloads; iPad prefers `photo_url`, falls back to `ServerConfig.current + "/static/" + path` (backward-compatible). Record as an iOS CLAUDE.md rule + `APIAssignedIssue`/`APIIssueDetail` field.
|
||||
- [ ] Keep `static/uploads/` for 30 days as backup. Snapshot, then reclaim disk as a **separate, deliberate** step — never part of the migration.
|
||||
|
||||
### Rollback (any phase after cutover)
|
||||
- [ ] `STORAGE_BACKEND=local` → restart. Instant revert; local files were never touched.
|
||||
|
||||
Reference in New Issue
Block a user