diff --git a/CLAUDE.md b/CLAUDE.md index edb616c..b02d8c9 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -29,6 +29,7 @@ 19. [Infrastructure](#19-infrastructure) 20. [Known Constraints & Hard Rules](#20-known-constraints--hard-rules) 21. [Change Philosophy](#21-change-philosophy) +22. [Object Storage Migration (R2)](#22-object-storage-migration-r2) --- @@ -603,7 +604,20 @@ The last eight styles (`SummaryTitle` through `TableCell`) were added for the fa `stats.py` now returns `severity_breakdown` dict alongside the existing KPIs. Derived from the already-loaded `open_issues_all` list — zero extra DB queries. -### Issue API — `_issue_payload()` fields +### Scheduled Inspections + Issue Handler Endpoints (July 2026) + +| Endpoint | Auth | Description | +|---|---|---| +| `GET /api/v1/scheduled-inspections` | jwt_required | Active scheduled/recurring assignments (`app/api/scheduled.py`, new blueprint). **Inspector:** only rows where `inspector_id == self`. **admin/director/PM:** all active. Sorted by `next_due_date`. Returns per row: `id`, `facility_id`, `facility_name`, `template_id`, `template_name`, `inspector_id`, `frequency`, `frequency_label`, `next_due_date` (ISO date), `is_overdue`, `notes`, plus `total`/`limit`/`offset`. Powers the iPad "Scheduled" section on Dashboard + My Inspections. Read-only — the schedule lifecycle (fulfil/roll-forward) stays web-driven; the iPad "Start" just seeds the new-inspection flow. | +| `PATCH /api/v1/issues//handler` | jwt_required | Set "Handled By" from the iPad (`update_issue_handler`). Body: `{ "handler_type": "internal"\|"facility"\|"vendor", ...optional detail keys }`. Detail keys (`facility_handler_name/contact/notes`, `vendor_name/contact/notes`) are updated only when present; empty string clears a field. **`log_action()` after commit.** | + +**Handler permission divergence — deliberate (see rule 78).** The web issue form limits handler edits to admin/director/PM. This API endpoint additionally allows the assigned **inspector**, scoped by `get_inspector_scope()` (403 if the issue's facility isn't in their contracted set). The iPad is a field tool; inspectors set the handler from Issue Detail. Do not "align" the API back to the web restriction without explicit direction. + +The new `scheduled` blueprint is registered in `app/api/__init__.py` and CSRF-exempted in `app/__init__.py` (`csrf.exempt(_api_scheduled_bp)` — parent-exempt does not cascade to child blueprints, per the CSRF pattern above). + +No migration was needed for either feature: the `scheduled_inspections` table (phase36) and the issue handler columns (phase35) already existed; both additions are pure serialization + one new route. + + ```python { @@ -621,6 +635,11 @@ The last eight styles (`SummaryTitle` through `TableCell`) were added for the fa # Phase E additions: 'area_name', # name of the Area the issue was flagged in (nullable) 'assigned_to_name', # display_name of currently assigned User (nullable) + # Handler ("Handled By", July 2026) additions: + 'handler_type', # 'internal' | 'facility' | 'vendor' (defaults 'internal') + 'handler_label', # human-readable label (Issue.handler_label property) + 'facility_handler_name', 'facility_handler_contact', 'facility_handler_notes', # nullable + 'vendor_name', 'vendor_contact', 'vendor_notes', # nullable } ``` @@ -1279,6 +1298,8 @@ timeout = 30 | 74 | **The `public` blueprint (`/f/*`) is login-free — keep it occupant-safe** | Pages are addressed by unguessable `public_token` (never facility id), 404 on inactive/unknown facilities, and expose only a quality rating, last-inspected date, and open-issue COUNT — **never** issue descriptions, inspector names, per-item scores, or any other facility's data. The `report` POST must stay CSRF-protected (Flask-WTF form), rate-limited, and honeypot-guarded; public-reported issues are created with `reported_by=NULL`, `severity='medium'`, and routed through `notify_by_matrix('issue_created', facility_id=...)`. Do not add fields that leak internal detail, and do not reuse `render_template('base.html')` here — the public page is a standalone template with no authenticated nav. | | 75 | **Email is stored lowercased; look it up case-insensitively** | User/customer email is normalized to `.strip().lower()` at every write site (`auth.py` profile/create/edit, `customers.py` invite/edit). Forgot-password lookup uses `db.func.lower(User.email) == input` so a mixed-case legacy row still matches — a plain `filter_by(email=...)` silently missed them and sent no reset (the failure was invisible because of the generic "if an account exists…" message). Keep both halves: normalize on write, case-insensitive on lookup. | | 76 | **Transactional email `From` must be an SMTP-authorized identity, per-domain branding via display name only** | Reset-password sends from `MAIL_DEFAULT_SENDER`; customer invite sends from `branded_sender()` = `(per-domain display name, authorized address)`. A per-host `noreply@` sender is accepted by the relay then dropped by SPF/DMARC. See rule 64 and §8 `mail_utils.py`. | +| 77 | **`GET /api/v1/scheduled-inspections` is inspector-scoped by `inspector_id`, admin/director/PM see all** | New `app/api/scheduled.py` blueprint. Register in `app/api/__init__.py` AND `csrf.exempt(_api_scheduled_bp)` in `app/__init__.py` — the child-blueprint CSRF exemption never cascades from the parent. Read-only; do not add write/fulfil endpoints here (the schedule lifecycle stays in `routes/scheduled_inspections.py`). | +| 78 | **`PATCH /api/v1/issues//handler` allows the inspector on purpose — do NOT align it to the web form's admin/director/PM restriction** | The iPad lets the assigned inspector set "Handled By" from the field, scoped via `get_inspector_scope()` (403 if the issue's facility isn't contracted). This is a deliberate divergence from the web form. `_issue_payload()` must keep returning all 8 handler fields (`handler_type`, `handler_label`, `facility_handler_*`, `vendor_*`) or the iPad's "Handled By" panel silently blanks — same failure mode as rule 40. | --- @@ -1291,4 +1312,71 @@ timeout = 30 5. **Migration existence checks** — all migrations safe to re-run 6. **Full file contents for 1–3 file changes**; deployment map for larger changesets 7. **Explicit deploy instructions** — migration steps separated from code steps -8. **Root cause analysis** on errors — never apply temporary workarounds \ No newline at end of file +8. **Root cause analysis** on errors — never apply temporary workarounds +--- + +## 22. Object Storage Migration (R2) + +**Goal:** move photo **files** off the server's local disk (`app/static/uploads/`) to **Cloudflare R2** (S3-compatible, $0 egress) so storage expands without touching the server. The database is NOT the bottleneck — rows are tiny; PDFs are streamed (`BytesIO`), never written to disk. Only photos accumulate. + +**No schema change.** The DB already stores a relative path string (`uploads/issue_photos/abc.jpg`). That exact string becomes the **object key** in R2, so rows never change and `media_url()` resolves the same key on either backend. + +**Serving decision:** private bucket + **presigned GET URLs** (TTL 24h). This is *more* private than today (static files are served unauthenticated). `media_url()` is the single seam — switching to a public custom-domain CDN URL later is a one-function change. + +**Complete photo-path inventory (the migration MUST cover all 5 — missing one loses those photos from the UI even if the file exists):** +1. `Issue.photo_path` (string) +2. `Issue.mobile_photo_paths[]` (JSON list — iPad evidence) +3. `Issue.result_photos[]` (JSON list — web resolution photos) +4. `Inspection.photo_path` (string) +5. `Inspection.form_data{}` — any `uploads/...` string (image fields; walk nested) + - Signatures are inline `data:` base64 in the DB, **not files** → correctly ignored. + +**Zero-loss invariants (non-negotiable):** +- Object key **==** the relative path string already in the DB. Never rewrite DB paths. +- **Copy, never move.** The migration only uploads. Local `static/uploads/` is untouched through cutover and a 30-day safety window. +- Per-file **MD5 + size verification** (local hash vs R2 `head_object`) — a file counts as migrated only when it matches. +- Cutover is **gated**: only flip `STORAGE_BACKEND=s3` when `verified == referenced-present` (from the Phase 0 baseline) with zero mismatches. +- S3 backend **falls back to the local file** if a key is absent (transition safety). +- **Rollback = one env var** (`STORAGE_BACKEND=local` + restart). Local files never left disk. +- Magic-byte validation stays in the **caller**, before `storage.save()`. +- `media_url()` is the ONLY place a photo URL is built server-side — no bare `url_for('static', filename=)` anywhere after Phase 1. + +### Phase 0 — Audit (READ-ONLY) — ✅ delivered +`scripts/audit_photos.py`. Reads all 5 sources + disk, reconciles, writes a JSON baseline. Writes nothing to the DB or uploads tree. +- [x] Script delivered (`scripts/audit_photos.py`). +- [ ] Run on production: `python scripts/audit_photos.py --report /home/jqc/photo_audit_baseline.json` +- [ ] Record the baseline number: **"N photos must still resolve after cutover."** +- [ ] Investigate any `missing_on_disk` entries — these are PRE-EXISTING broken references, surfaced now so they can't be blamed on the move. + +### Phase 1 — Storage abstraction + local backend (NO-OP, safe to deploy) +- [ ] `app/utils/storage.py` — interface: `save(file_obj, subfolder) -> key`, `read(key) -> bytes`, `delete(key)`, `exists(key) -> bool`, `media_url(key) -> str`. +- [ ] `LocalBackend` = byte-for-byte current behavior (`static/uploads` write; `media_url` = `url_for('static', filename=key)`). +- [ ] Config: `STORAGE_BACKEND=local|s3` (default `local`). +- [ ] Route `_save_photo()` in `inspections.py` + `issues.py`, and `api/photos.py`, through `storage.save()` (keep magic-byte check in the caller). +- [ ] Templates: swap `url_for('static', filename=path)` → `media_url(path)` in `issues/view.html`, `inspections/view.html`, and any other photo render sites. +- [ ] Deploy with `STORAGE_BACKEND=local` → identical behavior; regression check that web + iPad photos still load. + +### Phase 2 — R2 backend (build, do NOT flip yet) +- [ ] `pip install boto3`; add to `requirements.txt`. +- [ ] `.env`: `STORAGE_BACKEND`, `R2_ENDPOINT_URL`, `R2_ACCESS_KEY_ID`, `R2_SECRET_ACCESS_KEY`, `R2_BUCKET`, `R2_PRESIGN_TTL=86400`. +- [ ] `S3Backend`: boto3 client (`endpoint_url=R2_ENDPOINT_URL`, `region_name='auto'`). `save` → `put_object` (set `ContentType`); `media_url` → `generate_presigned_url('get_object', ..., ExpiresIn=TTL)`; `read` → `get_object`; `exists` → `head_object`. +- [ ] Fallback: if `head_object` 404s, `S3Backend.read`/`media_url` fall back to the local static path. +- [ ] Leave `STORAGE_BACKEND=local` in production for now. + +### Phase 3 — Verified migration sync +- [ ] `scripts/migrate_photos_to_r2.py` — idempotent, resumable, checksummed. + - Upload **every** file under `static/uploads/` (orphans included — completeness over reference-only). + - Per file: local MD5 → `put_object` → `head_object` → assert ETag/size match; append result (`ok`/`uploaded`/`mismatch`/`error`) to a JSON log. + - Skip keys already present **and** verified (resumable). +- [ ] Bulk run ahead of time (can run while `local` is still live — it only reads local + writes R2). +- [ ] Re-run immediately before cutover to catch the delta (idempotent). +- [ ] Gate check: `verified == referenced-present` (Phase 0 baseline) AND zero mismatches. + +### Phase 4 — Cutover + iOS +- [ ] Maintenance window: final incremental sync → set `STORAGE_BACKEND=s3` → `systemctl restart janitorial-qc`. +- [ ] Smoke test: existing web issue/inspection photos load via presigned URLs; new upload from **web** and **iPad** lands in R2. +- [ ] iOS: add absolute `photo_url` to issue/inspection payloads; iPad prefers `photo_url`, falls back to `ServerConfig.current + "/static/" + path` (backward-compatible). Record as an iOS CLAUDE.md rule + `APIAssignedIssue`/`APIIssueDetail` field. +- [ ] Keep `static/uploads/` for 30 days as backup. Snapshot, then reclaim disk as a **separate, deliberate** step — never part of the migration. + +### Rollback (any phase after cutover) +- [ ] `STORAGE_BACKEND=local` → restart. Instant revert; local files were never touched. diff --git a/scripts/audit_photos.py b/scripts/audit_photos.py new file mode 100644 index 0000000..01231cb --- /dev/null +++ b/scripts/audit_photos.py @@ -0,0 +1,273 @@ +#!/usr/bin/env python3 +""" +scripts/audit_photos.py — Pre-migration photo audit (READ-ONLY) +================================================================= + +Phase 0 of the R2 object-storage migration. This script writes NOTHING to the +database and NOTHING to the uploads tree. It only reads, and emits a report. + +What it does +------------ +1. Collects every photo path REFERENCED by the database, across all 5 sources: + - Issue.photo_path (string) + - Issue.mobile_photo_paths[] (JSON list — iPad evidence) + - Issue.result_photos[] (JSON list — web resolution photos) + - Inspection.photo_path (string) + - Inspection.form_data{} (any 'uploads/...' string, incl. nested — image fields) + (Signatures are stored inline as 'data:' base64, NOT files — correctly ignored.) +2. Checks each referenced path against the file actually on disk. +3. Walks the uploads tree and finds files NOT referenced by any record (orphans). +4. Prints a summary and writes a detailed JSON report you can keep as the baseline. + +The number that matters: **distinct referenced photos that resolve on disk**. +That count must be identical after the migration. Any "missing on disk" here is a +PRE-EXISTING broken reference — surfaced now so it can never be blamed on the move. + +Usage +----- + cd /home/jqc/janitorial_qc + source venv/bin/activate # whatever your venv is + python scripts/audit_photos.py # report -> /tmp + python scripts/audit_photos.py --report /home/jqc/photo_audit.json + python scripts/audit_photos.py --list-orphans # also print orphan paths + +Exit code is always 0 (audit is informational). Nothing is modified. +""" + +import os +import sys +import json +import argparse +from collections import defaultdict +from datetime import datetime + +# Make the app package importable when run from the repo root. +sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))) + +from app import create_app, db # noqa: E402 +from app.models.issue import Issue # noqa: E402 +from app.models.inspection import Inspection # noqa: E402 + + +# ── Path helpers ────────────────────────────────────────────────────────────── + +def normalize(path): + """ + Return a canonical 'uploads/...' key, or None if this value is not a stored + photo file (base64 signatures, unsynced iPad sentinels, remote URLs, blanks). + """ + if not isinstance(path, str): + return None + p = path.strip() + if not p: + return None + # Not on-disk files: + if p.startswith('data:') or p.startswith('local://'): + return None + if p.startswith('http://') or p.startswith('https://'): + return None + # Tolerate a few stored variants and reduce to 'uploads/...' + for prefix in ('/static/', 'static/', '/'): + if p.startswith(prefix): + p = p[len(prefix):] + if not p.startswith('uploads/'): + return None + return p + + +def walk_strings(value): + """Yield every string found in an arbitrarily nested JSON value (dict/list/str).""" + if isinstance(value, str): + yield value + elif isinstance(value, dict): + for v in value.values(): + yield from walk_strings(v) + elif isinstance(value, (list, tuple)): + for v in value: + yield from walk_strings(v) + + +# ── Reference collection ────────────────────────────────────────────────────── + +def collect_referenced(): + """ + Return (referenced, by_source): + referenced : { normalized_path -> [ "issue:123:photo_path", ... ] } + by_source : { source_name -> count_of_refs } + """ + referenced = defaultdict(list) + by_source = defaultdict(int) + + def add(path, ref_label, source_name): + norm = normalize(path) + if norm is None: + return + referenced[norm].append(ref_label) + by_source[source_name] += 1 + + # --- Issues --- + for iss in Issue.query.yield_per(500): + add(iss.photo_path, f'issue:{iss.id}:photo_path', 'issue.photo_path') + for pth in (iss.mobile_photo_paths or []): + add(pth, f'issue:{iss.id}:mobile_photo_paths', 'issue.mobile_photo_paths') + for pth in (iss.result_photos or []): + add(pth, f'issue:{iss.id}:result_photos', 'issue.result_photos') + + # --- Inspections --- + for insp in Inspection.query.yield_per(500): + add(insp.photo_path, f'inspection:{insp.id}:photo_path', 'inspection.photo_path') + fd = insp.form_data or {} + for s in walk_strings(fd): + # Any 'uploads/...' string inside form_data is an uploaded image value. + add(s, f'inspection:{insp.id}:form_data', 'inspection.form_data(image)') + + return referenced, by_source + + +# ── Disk scan ───────────────────────────────────────────────────────────────── + +def scan_disk(uploads_root): + """Return { relative_uploads_key -> size_bytes } for every file under uploads/.""" + on_disk = {} + if not os.path.isdir(uploads_root): + return on_disk + base = os.path.dirname(uploads_root) # .../static (so keys start with 'uploads/') + for dirpath, _dirs, files in os.walk(uploads_root): + for name in files: + full = os.path.join(dirpath, name) + rel = os.path.relpath(full, base).replace(os.sep, '/') + try: + on_disk[rel] = os.path.getsize(full) + except OSError: + on_disk[rel] = -1 + return on_disk + + +# ── Main ────────────────────────────────────────────────────────────────────── + +def human(nbytes): + n = float(nbytes) + for unit in ('B', 'KB', 'MB', 'GB', 'TB'): + if abs(n) < 1024.0: + return f'{n:.1f} {unit}' + n /= 1024.0 + return f'{n:.1f} PB' + + +def main(): + parser = argparse.ArgumentParser(description='Read-only photo audit (R2 migration Phase 0).') + parser.add_argument('--report', default=None, + help='Path to write the JSON report (default: /tmp/jqc_photo_audit_.json)') + parser.add_argument('--list-orphans', action='store_true', + help='Also print orphan file paths to stdout') + args = parser.parse_args() + + app = create_app(os.getenv('FLASK_ENV') or 'production') + with app.app_context(): + static_folder = os.path.join(app.root_path, 'static') + uploads_root = os.path.join(static_folder, 'uploads') + + referenced, by_source = collect_referenced() + on_disk = scan_disk(uploads_root) + on_disk_keys = set(on_disk.keys()) + + referenced_keys = set(referenced.keys()) + present = referenced_keys & on_disk_keys + missing = referenced_keys - on_disk_keys # broken references (pre-existing) + orphans = on_disk_keys - referenced_keys # on disk, unreferenced + + present_bytes = sum(on_disk[k] for k in present if on_disk[k] > 0) + orphan_bytes = sum(on_disk[k] for k in orphans if on_disk[k] > 0) + total_bytes = sum(v for v in on_disk.values() if v > 0) + + # Per-subfolder disk breakdown + by_folder = defaultdict(lambda: [0, 0]) # folder -> [count, bytes] + for k, sz in on_disk.items(): + parts = k.split('/') + folder = parts[1] if len(parts) > 2 else '(root)' + by_folder[folder][0] += 1 + by_folder[folder][1] += sz if sz > 0 else 0 + + report = { + 'generated_at': datetime.utcnow().isoformat() + 'Z', + 'static_folder': static_folder, + 'uploads_root': uploads_root, + 'referenced': { + 'distinct_paths': len(referenced_keys), + 'by_source': dict(by_source), + 'present_on_disk': len(present), + 'missing_on_disk': len(missing), + }, + 'disk': { + 'total_files': len(on_disk), + 'total_bytes': total_bytes, + 'by_folder': {k: {'files': v[0], 'bytes': v[1]} for k, v in by_folder.items()}, + }, + 'orphans': { + 'count': len(orphans), + 'bytes': orphan_bytes, + }, + # Full detail for follow-up / verification after migration: + 'missing_on_disk': sorted( + [{'path': k, 'referenced_by': sorted(set(referenced[k]))} for k in missing], + key=lambda d: d['path'] + ), + 'orphan_files': sorted( + [{'path': k, 'bytes': on_disk[k]} for k in orphans], + key=lambda d: d['path'] + ), + } + + # ── stdout summary ── + print('=' * 68) + print(' JQC PHOTO AUDIT (read-only — nothing was modified)') + print('=' * 68) + print(f' uploads root : {uploads_root}') + print(f' generated : {report["generated_at"]}') + print('-' * 68) + print(' REFERENCED BY DATABASE') + for src, cnt in sorted(by_source.items()): + print(f' {src:<34} {cnt:>8} refs') + print(f' {"distinct photo files referenced":<34} {len(referenced_keys):>8}') + print('-' * 68) + print(' ON DISK') + for folder, (cnt, b) in sorted(by_folder.items()): + print(f' {folder:<24} {cnt:>8} files {human(b):>12}') + print(f' {"TOTAL":<24} {len(on_disk):>8} files {human(total_bytes):>12}') + print('-' * 68) + print(' RECONCILIATION') + print(f' referenced AND present on disk : {len(present):>8} <-- must be unchanged after migration') + print(f' referenced but MISSING on disk : {len(missing):>8} <-- pre-existing broken refs (investigate)') + print(f' on disk but UNREFERENCED : {len(orphans):>8} ({human(orphan_bytes)}, migrated anyway)') + print('=' * 68) + + if missing: + print(f'\n WARNING: {len(missing)} referenced photo(s) are already missing on disk.') + print(' These are NOT caused by migration. First few:') + for m in report['missing_on_disk'][:10]: + print(f' - {m["path"]} (by {", ".join(m["referenced_by"][:3])})') + if len(missing) > 10: + print(f' ... and {len(missing) - 10} more (see JSON report)') + + if args.list_orphans and orphans: + print(f'\n Orphan files ({len(orphans)}):') + for o in report['orphan_files']: + print(f' - {o["path"]} ({human(o["bytes"])})') + + # ── write JSON report ── + report_path = args.report or os.path.join( + '/tmp', f'jqc_photo_audit_{datetime.utcnow().strftime("%Y%m%d_%H%M%S")}.json' + ) + try: + with open(report_path, 'w') as fh: + json.dump(report, fh, indent=2) + print(f'\n Full JSON report written to: {report_path}') + except OSError as e: + print(f'\n Could not write report to {report_path}: {e}') + + print('\n BASELINE TO REMEMBER: ' + f'{len(present)} photos must still resolve after cutover.\n') + + +if __name__ == '__main__': + main()