Jul 14 - Preparation for using CDN
This commit is contained in:
@@ -29,6 +29,7 @@
|
|||||||
19. [Infrastructure](#19-infrastructure)
|
19. [Infrastructure](#19-infrastructure)
|
||||||
20. [Known Constraints & Hard Rules](#20-known-constraints--hard-rules)
|
20. [Known Constraints & Hard Rules](#20-known-constraints--hard-rules)
|
||||||
21. [Change Philosophy](#21-change-philosophy)
|
21. [Change Philosophy](#21-change-philosophy)
|
||||||
|
22. [Object Storage Migration (R2)](#22-object-storage-migration-r2)
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -603,7 +604,20 @@ The last eight styles (`SummaryTitle` through `TableCell`) were added for the fa
|
|||||||
|
|
||||||
`stats.py` now returns `severity_breakdown` dict alongside the existing KPIs. Derived from the already-loaded `open_issues_all` list — zero extra DB queries.
|
`stats.py` now returns `severity_breakdown` dict alongside the existing KPIs. Derived from the already-loaded `open_issues_all` list — zero extra DB queries.
|
||||||
|
|
||||||
### Issue API — `_issue_payload()` fields
|
### Scheduled Inspections + Issue Handler Endpoints (July 2026)
|
||||||
|
|
||||||
|
| Endpoint | Auth | Description |
|
||||||
|
|---|---|---|
|
||||||
|
| `GET /api/v1/scheduled-inspections` | jwt_required | Active scheduled/recurring assignments (`app/api/scheduled.py`, new blueprint). **Inspector:** only rows where `inspector_id == self`. **admin/director/PM:** all active. Sorted by `next_due_date`. Returns per row: `id`, `facility_id`, `facility_name`, `template_id`, `template_name`, `inspector_id`, `frequency`, `frequency_label`, `next_due_date` (ISO date), `is_overdue`, `notes`, plus `total`/`limit`/`offset`. Powers the iPad "Scheduled" section on Dashboard + My Inspections. Read-only — the schedule lifecycle (fulfil/roll-forward) stays web-driven; the iPad "Start" just seeds the new-inspection flow. |
|
||||||
|
| `PATCH /api/v1/issues/<id>/handler` | jwt_required | Set "Handled By" from the iPad (`update_issue_handler`). Body: `{ "handler_type": "internal"\|"facility"\|"vendor", ...optional detail keys }`. Detail keys (`facility_handler_name/contact/notes`, `vendor_name/contact/notes`) are updated only when present; empty string clears a field. **`log_action()` after commit.** |
|
||||||
|
|
||||||
|
**Handler permission divergence — deliberate (see rule 78).** The web issue form limits handler edits to admin/director/PM. This API endpoint additionally allows the assigned **inspector**, scoped by `get_inspector_scope()` (403 if the issue's facility isn't in their contracted set). The iPad is a field tool; inspectors set the handler from Issue Detail. Do not "align" the API back to the web restriction without explicit direction.
|
||||||
|
|
||||||
|
The new `scheduled` blueprint is registered in `app/api/__init__.py` and CSRF-exempted in `app/__init__.py` (`csrf.exempt(_api_scheduled_bp)` — parent-exempt does not cascade to child blueprints, per the CSRF pattern above).
|
||||||
|
|
||||||
|
No migration was needed for either feature: the `scheduled_inspections` table (phase36) and the issue handler columns (phase35) already existed; both additions are pure serialization + one new route.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
```python
|
```python
|
||||||
{
|
{
|
||||||
@@ -621,6 +635,11 @@ The last eight styles (`SummaryTitle` through `TableCell`) were added for the fa
|
|||||||
# Phase E additions:
|
# Phase E additions:
|
||||||
'area_name', # name of the Area the issue was flagged in (nullable)
|
'area_name', # name of the Area the issue was flagged in (nullable)
|
||||||
'assigned_to_name', # display_name of currently assigned User (nullable)
|
'assigned_to_name', # display_name of currently assigned User (nullable)
|
||||||
|
# Handler ("Handled By", July 2026) additions:
|
||||||
|
'handler_type', # 'internal' | 'facility' | 'vendor' (defaults 'internal')
|
||||||
|
'handler_label', # human-readable label (Issue.handler_label property)
|
||||||
|
'facility_handler_name', 'facility_handler_contact', 'facility_handler_notes', # nullable
|
||||||
|
'vendor_name', 'vendor_contact', 'vendor_notes', # nullable
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -1279,6 +1298,8 @@ timeout = 30
|
|||||||
| 74 | **The `public` blueprint (`/f/*`) is login-free — keep it occupant-safe** | Pages are addressed by unguessable `public_token` (never facility id), 404 on inactive/unknown facilities, and expose only a quality rating, last-inspected date, and open-issue COUNT — **never** issue descriptions, inspector names, per-item scores, or any other facility's data. The `report` POST must stay CSRF-protected (Flask-WTF form), rate-limited, and honeypot-guarded; public-reported issues are created with `reported_by=NULL`, `severity='medium'`, and routed through `notify_by_matrix('issue_created', facility_id=...)`. Do not add fields that leak internal detail, and do not reuse `render_template('base.html')` here — the public page is a standalone template with no authenticated nav. |
|
| 74 | **The `public` blueprint (`/f/*`) is login-free — keep it occupant-safe** | Pages are addressed by unguessable `public_token` (never facility id), 404 on inactive/unknown facilities, and expose only a quality rating, last-inspected date, and open-issue COUNT — **never** issue descriptions, inspector names, per-item scores, or any other facility's data. The `report` POST must stay CSRF-protected (Flask-WTF form), rate-limited, and honeypot-guarded; public-reported issues are created with `reported_by=NULL`, `severity='medium'`, and routed through `notify_by_matrix('issue_created', facility_id=...)`. Do not add fields that leak internal detail, and do not reuse `render_template('base.html')` here — the public page is a standalone template with no authenticated nav. |
|
||||||
| 75 | **Email is stored lowercased; look it up case-insensitively** | User/customer email is normalized to `.strip().lower()` at every write site (`auth.py` profile/create/edit, `customers.py` invite/edit). Forgot-password lookup uses `db.func.lower(User.email) == input` so a mixed-case legacy row still matches — a plain `filter_by(email=...)` silently missed them and sent no reset (the failure was invisible because of the generic "if an account exists…" message). Keep both halves: normalize on write, case-insensitive on lookup. |
|
| 75 | **Email is stored lowercased; look it up case-insensitively** | User/customer email is normalized to `.strip().lower()` at every write site (`auth.py` profile/create/edit, `customers.py` invite/edit). Forgot-password lookup uses `db.func.lower(User.email) == input` so a mixed-case legacy row still matches — a plain `filter_by(email=...)` silently missed them and sent no reset (the failure was invisible because of the generic "if an account exists…" message). Keep both halves: normalize on write, case-insensitive on lookup. |
|
||||||
| 76 | **Transactional email `From` must be an SMTP-authorized identity, per-domain branding via display name only** | Reset-password sends from `MAIL_DEFAULT_SENDER`; customer invite sends from `branded_sender()` = `(per-domain display name, authorized address)`. A per-host `noreply@<subdomain>` sender is accepted by the relay then dropped by SPF/DMARC. See rule 64 and §8 `mail_utils.py`. |
|
| 76 | **Transactional email `From` must be an SMTP-authorized identity, per-domain branding via display name only** | Reset-password sends from `MAIL_DEFAULT_SENDER`; customer invite sends from `branded_sender()` = `(per-domain display name, authorized address)`. A per-host `noreply@<subdomain>` sender is accepted by the relay then dropped by SPF/DMARC. See rule 64 and §8 `mail_utils.py`. |
|
||||||
|
| 77 | **`GET /api/v1/scheduled-inspections` is inspector-scoped by `inspector_id`, admin/director/PM see all** | New `app/api/scheduled.py` blueprint. Register in `app/api/__init__.py` AND `csrf.exempt(_api_scheduled_bp)` in `app/__init__.py` — the child-blueprint CSRF exemption never cascades from the parent. Read-only; do not add write/fulfil endpoints here (the schedule lifecycle stays in `routes/scheduled_inspections.py`). |
|
||||||
|
| 78 | **`PATCH /api/v1/issues/<id>/handler` allows the inspector on purpose — do NOT align it to the web form's admin/director/PM restriction** | The iPad lets the assigned inspector set "Handled By" from the field, scoped via `get_inspector_scope()` (403 if the issue's facility isn't contracted). This is a deliberate divergence from the web form. `_issue_payload()` must keep returning all 8 handler fields (`handler_type`, `handler_label`, `facility_handler_*`, `vendor_*`) or the iPad's "Handled By" panel silently blanks — same failure mode as rule 40. |
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -1292,3 +1313,70 @@ timeout = 30
|
|||||||
6. **Full file contents for 1–3 file changes**; deployment map for larger changesets
|
6. **Full file contents for 1–3 file changes**; deployment map for larger changesets
|
||||||
7. **Explicit deploy instructions** — migration steps separated from code steps
|
7. **Explicit deploy instructions** — migration steps separated from code steps
|
||||||
8. **Root cause analysis** on errors — never apply temporary workarounds
|
8. **Root cause analysis** on errors — never apply temporary workarounds
|
||||||
|
---
|
||||||
|
|
||||||
|
## 22. Object Storage Migration (R2)
|
||||||
|
|
||||||
|
**Goal:** move photo **files** off the server's local disk (`app/static/uploads/`) to **Cloudflare R2** (S3-compatible, $0 egress) so storage expands without touching the server. The database is NOT the bottleneck — rows are tiny; PDFs are streamed (`BytesIO`), never written to disk. Only photos accumulate.
|
||||||
|
|
||||||
|
**No schema change.** The DB already stores a relative path string (`uploads/issue_photos/abc.jpg`). That exact string becomes the **object key** in R2, so rows never change and `media_url()` resolves the same key on either backend.
|
||||||
|
|
||||||
|
**Serving decision:** private bucket + **presigned GET URLs** (TTL 24h). This is *more* private than today (static files are served unauthenticated). `media_url()` is the single seam — switching to a public custom-domain CDN URL later is a one-function change.
|
||||||
|
|
||||||
|
**Complete photo-path inventory (the migration MUST cover all 5 — missing one loses those photos from the UI even if the file exists):**
|
||||||
|
1. `Issue.photo_path` (string)
|
||||||
|
2. `Issue.mobile_photo_paths[]` (JSON list — iPad evidence)
|
||||||
|
3. `Issue.result_photos[]` (JSON list — web resolution photos)
|
||||||
|
4. `Inspection.photo_path` (string)
|
||||||
|
5. `Inspection.form_data{}` — any `uploads/...` string (image fields; walk nested)
|
||||||
|
- Signatures are inline `data:` base64 in the DB, **not files** → correctly ignored.
|
||||||
|
|
||||||
|
**Zero-loss invariants (non-negotiable):**
|
||||||
|
- Object key **==** the relative path string already in the DB. Never rewrite DB paths.
|
||||||
|
- **Copy, never move.** The migration only uploads. Local `static/uploads/` is untouched through cutover and a 30-day safety window.
|
||||||
|
- Per-file **MD5 + size verification** (local hash vs R2 `head_object`) — a file counts as migrated only when it matches.
|
||||||
|
- Cutover is **gated**: only flip `STORAGE_BACKEND=s3` when `verified == referenced-present` (from the Phase 0 baseline) with zero mismatches.
|
||||||
|
- S3 backend **falls back to the local file** if a key is absent (transition safety).
|
||||||
|
- **Rollback = one env var** (`STORAGE_BACKEND=local` + restart). Local files never left disk.
|
||||||
|
- Magic-byte validation stays in the **caller**, before `storage.save()`.
|
||||||
|
- `media_url()` is the ONLY place a photo URL is built server-side — no bare `url_for('static', filename=<photo>)` anywhere after Phase 1.
|
||||||
|
|
||||||
|
### Phase 0 — Audit (READ-ONLY) — ✅ delivered
|
||||||
|
`scripts/audit_photos.py`. Reads all 5 sources + disk, reconciles, writes a JSON baseline. Writes nothing to the DB or uploads tree.
|
||||||
|
- [x] Script delivered (`scripts/audit_photos.py`).
|
||||||
|
- [ ] Run on production: `python scripts/audit_photos.py --report /home/jqc/photo_audit_baseline.json`
|
||||||
|
- [ ] Record the baseline number: **"N photos must still resolve after cutover."**
|
||||||
|
- [ ] Investigate any `missing_on_disk` entries — these are PRE-EXISTING broken references, surfaced now so they can't be blamed on the move.
|
||||||
|
|
||||||
|
### Phase 1 — Storage abstraction + local backend (NO-OP, safe to deploy)
|
||||||
|
- [ ] `app/utils/storage.py` — interface: `save(file_obj, subfolder) -> key`, `read(key) -> bytes`, `delete(key)`, `exists(key) -> bool`, `media_url(key) -> str`.
|
||||||
|
- [ ] `LocalBackend` = byte-for-byte current behavior (`static/uploads` write; `media_url` = `url_for('static', filename=key)`).
|
||||||
|
- [ ] Config: `STORAGE_BACKEND=local|s3` (default `local`).
|
||||||
|
- [ ] Route `_save_photo()` in `inspections.py` + `issues.py`, and `api/photos.py`, through `storage.save()` (keep magic-byte check in the caller).
|
||||||
|
- [ ] Templates: swap `url_for('static', filename=path)` → `media_url(path)` in `issues/view.html`, `inspections/view.html`, and any other photo render sites.
|
||||||
|
- [ ] Deploy with `STORAGE_BACKEND=local` → identical behavior; regression check that web + iPad photos still load.
|
||||||
|
|
||||||
|
### Phase 2 — R2 backend (build, do NOT flip yet)
|
||||||
|
- [ ] `pip install boto3`; add to `requirements.txt`.
|
||||||
|
- [ ] `.env`: `STORAGE_BACKEND`, `R2_ENDPOINT_URL`, `R2_ACCESS_KEY_ID`, `R2_SECRET_ACCESS_KEY`, `R2_BUCKET`, `R2_PRESIGN_TTL=86400`.
|
||||||
|
- [ ] `S3Backend`: boto3 client (`endpoint_url=R2_ENDPOINT_URL`, `region_name='auto'`). `save` → `put_object` (set `ContentType`); `media_url` → `generate_presigned_url('get_object', ..., ExpiresIn=TTL)`; `read` → `get_object`; `exists` → `head_object`.
|
||||||
|
- [ ] Fallback: if `head_object` 404s, `S3Backend.read`/`media_url` fall back to the local static path.
|
||||||
|
- [ ] Leave `STORAGE_BACKEND=local` in production for now.
|
||||||
|
|
||||||
|
### Phase 3 — Verified migration sync
|
||||||
|
- [ ] `scripts/migrate_photos_to_r2.py` — idempotent, resumable, checksummed.
|
||||||
|
- Upload **every** file under `static/uploads/` (orphans included — completeness over reference-only).
|
||||||
|
- Per file: local MD5 → `put_object` → `head_object` → assert ETag/size match; append result (`ok`/`uploaded`/`mismatch`/`error`) to a JSON log.
|
||||||
|
- Skip keys already present **and** verified (resumable).
|
||||||
|
- [ ] Bulk run ahead of time (can run while `local` is still live — it only reads local + writes R2).
|
||||||
|
- [ ] Re-run immediately before cutover to catch the delta (idempotent).
|
||||||
|
- [ ] Gate check: `verified == referenced-present` (Phase 0 baseline) AND zero mismatches.
|
||||||
|
|
||||||
|
### Phase 4 — Cutover + iOS
|
||||||
|
- [ ] Maintenance window: final incremental sync → set `STORAGE_BACKEND=s3` → `systemctl restart janitorial-qc`.
|
||||||
|
- [ ] Smoke test: existing web issue/inspection photos load via presigned URLs; new upload from **web** and **iPad** lands in R2.
|
||||||
|
- [ ] iOS: add absolute `photo_url` to issue/inspection payloads; iPad prefers `photo_url`, falls back to `ServerConfig.current + "/static/" + path` (backward-compatible). Record as an iOS CLAUDE.md rule + `APIAssignedIssue`/`APIIssueDetail` field.
|
||||||
|
- [ ] Keep `static/uploads/` for 30 days as backup. Snapshot, then reclaim disk as a **separate, deliberate** step — never part of the migration.
|
||||||
|
|
||||||
|
### Rollback (any phase after cutover)
|
||||||
|
- [ ] `STORAGE_BACKEND=local` → restart. Instant revert; local files were never touched.
|
||||||
|
|||||||
@@ -0,0 +1,273 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
scripts/audit_photos.py — Pre-migration photo audit (READ-ONLY)
|
||||||
|
=================================================================
|
||||||
|
|
||||||
|
Phase 0 of the R2 object-storage migration. This script writes NOTHING to the
|
||||||
|
database and NOTHING to the uploads tree. It only reads, and emits a report.
|
||||||
|
|
||||||
|
What it does
|
||||||
|
------------
|
||||||
|
1. Collects every photo path REFERENCED by the database, across all 5 sources:
|
||||||
|
- Issue.photo_path (string)
|
||||||
|
- Issue.mobile_photo_paths[] (JSON list — iPad evidence)
|
||||||
|
- Issue.result_photos[] (JSON list — web resolution photos)
|
||||||
|
- Inspection.photo_path (string)
|
||||||
|
- Inspection.form_data{} (any 'uploads/...' string, incl. nested — image fields)
|
||||||
|
(Signatures are stored inline as 'data:' base64, NOT files — correctly ignored.)
|
||||||
|
2. Checks each referenced path against the file actually on disk.
|
||||||
|
3. Walks the uploads tree and finds files NOT referenced by any record (orphans).
|
||||||
|
4. Prints a summary and writes a detailed JSON report you can keep as the baseline.
|
||||||
|
|
||||||
|
The number that matters: **distinct referenced photos that resolve on disk**.
|
||||||
|
That count must be identical after the migration. Any "missing on disk" here is a
|
||||||
|
PRE-EXISTING broken reference — surfaced now so it can never be blamed on the move.
|
||||||
|
|
||||||
|
Usage
|
||||||
|
-----
|
||||||
|
cd /home/jqc/janitorial_qc
|
||||||
|
source venv/bin/activate # whatever your venv is
|
||||||
|
python scripts/audit_photos.py # report -> /tmp
|
||||||
|
python scripts/audit_photos.py --report /home/jqc/photo_audit.json
|
||||||
|
python scripts/audit_photos.py --list-orphans # also print orphan paths
|
||||||
|
|
||||||
|
Exit code is always 0 (audit is informational). Nothing is modified.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
import json
|
||||||
|
import argparse
|
||||||
|
from collections import defaultdict
|
||||||
|
from datetime import datetime
|
||||||
|
|
||||||
|
# Make the app package importable when run from the repo root.
|
||||||
|
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))
|
||||||
|
|
||||||
|
from app import create_app, db # noqa: E402
|
||||||
|
from app.models.issue import Issue # noqa: E402
|
||||||
|
from app.models.inspection import Inspection # noqa: E402
|
||||||
|
|
||||||
|
|
||||||
|
# ── Path helpers ──────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
def normalize(path):
|
||||||
|
"""
|
||||||
|
Return a canonical 'uploads/...' key, or None if this value is not a stored
|
||||||
|
photo file (base64 signatures, unsynced iPad sentinels, remote URLs, blanks).
|
||||||
|
"""
|
||||||
|
if not isinstance(path, str):
|
||||||
|
return None
|
||||||
|
p = path.strip()
|
||||||
|
if not p:
|
||||||
|
return None
|
||||||
|
# Not on-disk files:
|
||||||
|
if p.startswith('data:') or p.startswith('local://'):
|
||||||
|
return None
|
||||||
|
if p.startswith('http://') or p.startswith('https://'):
|
||||||
|
return None
|
||||||
|
# Tolerate a few stored variants and reduce to 'uploads/...'
|
||||||
|
for prefix in ('/static/', 'static/', '/'):
|
||||||
|
if p.startswith(prefix):
|
||||||
|
p = p[len(prefix):]
|
||||||
|
if not p.startswith('uploads/'):
|
||||||
|
return None
|
||||||
|
return p
|
||||||
|
|
||||||
|
|
||||||
|
def walk_strings(value):
|
||||||
|
"""Yield every string found in an arbitrarily nested JSON value (dict/list/str)."""
|
||||||
|
if isinstance(value, str):
|
||||||
|
yield value
|
||||||
|
elif isinstance(value, dict):
|
||||||
|
for v in value.values():
|
||||||
|
yield from walk_strings(v)
|
||||||
|
elif isinstance(value, (list, tuple)):
|
||||||
|
for v in value:
|
||||||
|
yield from walk_strings(v)
|
||||||
|
|
||||||
|
|
||||||
|
# ── Reference collection ──────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
def collect_referenced():
|
||||||
|
"""
|
||||||
|
Return (referenced, by_source):
|
||||||
|
referenced : { normalized_path -> [ "issue:123:photo_path", ... ] }
|
||||||
|
by_source : { source_name -> count_of_refs }
|
||||||
|
"""
|
||||||
|
referenced = defaultdict(list)
|
||||||
|
by_source = defaultdict(int)
|
||||||
|
|
||||||
|
def add(path, ref_label, source_name):
|
||||||
|
norm = normalize(path)
|
||||||
|
if norm is None:
|
||||||
|
return
|
||||||
|
referenced[norm].append(ref_label)
|
||||||
|
by_source[source_name] += 1
|
||||||
|
|
||||||
|
# --- Issues ---
|
||||||
|
for iss in Issue.query.yield_per(500):
|
||||||
|
add(iss.photo_path, f'issue:{iss.id}:photo_path', 'issue.photo_path')
|
||||||
|
for pth in (iss.mobile_photo_paths or []):
|
||||||
|
add(pth, f'issue:{iss.id}:mobile_photo_paths', 'issue.mobile_photo_paths')
|
||||||
|
for pth in (iss.result_photos or []):
|
||||||
|
add(pth, f'issue:{iss.id}:result_photos', 'issue.result_photos')
|
||||||
|
|
||||||
|
# --- Inspections ---
|
||||||
|
for insp in Inspection.query.yield_per(500):
|
||||||
|
add(insp.photo_path, f'inspection:{insp.id}:photo_path', 'inspection.photo_path')
|
||||||
|
fd = insp.form_data or {}
|
||||||
|
for s in walk_strings(fd):
|
||||||
|
# Any 'uploads/...' string inside form_data is an uploaded image value.
|
||||||
|
add(s, f'inspection:{insp.id}:form_data', 'inspection.form_data(image)')
|
||||||
|
|
||||||
|
return referenced, by_source
|
||||||
|
|
||||||
|
|
||||||
|
# ── Disk scan ─────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
def scan_disk(uploads_root):
|
||||||
|
"""Return { relative_uploads_key -> size_bytes } for every file under uploads/."""
|
||||||
|
on_disk = {}
|
||||||
|
if not os.path.isdir(uploads_root):
|
||||||
|
return on_disk
|
||||||
|
base = os.path.dirname(uploads_root) # .../static (so keys start with 'uploads/')
|
||||||
|
for dirpath, _dirs, files in os.walk(uploads_root):
|
||||||
|
for name in files:
|
||||||
|
full = os.path.join(dirpath, name)
|
||||||
|
rel = os.path.relpath(full, base).replace(os.sep, '/')
|
||||||
|
try:
|
||||||
|
on_disk[rel] = os.path.getsize(full)
|
||||||
|
except OSError:
|
||||||
|
on_disk[rel] = -1
|
||||||
|
return on_disk
|
||||||
|
|
||||||
|
|
||||||
|
# ── Main ──────────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
def human(nbytes):
|
||||||
|
n = float(nbytes)
|
||||||
|
for unit in ('B', 'KB', 'MB', 'GB', 'TB'):
|
||||||
|
if abs(n) < 1024.0:
|
||||||
|
return f'{n:.1f} {unit}'
|
||||||
|
n /= 1024.0
|
||||||
|
return f'{n:.1f} PB'
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
parser = argparse.ArgumentParser(description='Read-only photo audit (R2 migration Phase 0).')
|
||||||
|
parser.add_argument('--report', default=None,
|
||||||
|
help='Path to write the JSON report (default: /tmp/jqc_photo_audit_<ts>.json)')
|
||||||
|
parser.add_argument('--list-orphans', action='store_true',
|
||||||
|
help='Also print orphan file paths to stdout')
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
app = create_app(os.getenv('FLASK_ENV') or 'production')
|
||||||
|
with app.app_context():
|
||||||
|
static_folder = os.path.join(app.root_path, 'static')
|
||||||
|
uploads_root = os.path.join(static_folder, 'uploads')
|
||||||
|
|
||||||
|
referenced, by_source = collect_referenced()
|
||||||
|
on_disk = scan_disk(uploads_root)
|
||||||
|
on_disk_keys = set(on_disk.keys())
|
||||||
|
|
||||||
|
referenced_keys = set(referenced.keys())
|
||||||
|
present = referenced_keys & on_disk_keys
|
||||||
|
missing = referenced_keys - on_disk_keys # broken references (pre-existing)
|
||||||
|
orphans = on_disk_keys - referenced_keys # on disk, unreferenced
|
||||||
|
|
||||||
|
present_bytes = sum(on_disk[k] for k in present if on_disk[k] > 0)
|
||||||
|
orphan_bytes = sum(on_disk[k] for k in orphans if on_disk[k] > 0)
|
||||||
|
total_bytes = sum(v for v in on_disk.values() if v > 0)
|
||||||
|
|
||||||
|
# Per-subfolder disk breakdown
|
||||||
|
by_folder = defaultdict(lambda: [0, 0]) # folder -> [count, bytes]
|
||||||
|
for k, sz in on_disk.items():
|
||||||
|
parts = k.split('/')
|
||||||
|
folder = parts[1] if len(parts) > 2 else '(root)'
|
||||||
|
by_folder[folder][0] += 1
|
||||||
|
by_folder[folder][1] += sz if sz > 0 else 0
|
||||||
|
|
||||||
|
report = {
|
||||||
|
'generated_at': datetime.utcnow().isoformat() + 'Z',
|
||||||
|
'static_folder': static_folder,
|
||||||
|
'uploads_root': uploads_root,
|
||||||
|
'referenced': {
|
||||||
|
'distinct_paths': len(referenced_keys),
|
||||||
|
'by_source': dict(by_source),
|
||||||
|
'present_on_disk': len(present),
|
||||||
|
'missing_on_disk': len(missing),
|
||||||
|
},
|
||||||
|
'disk': {
|
||||||
|
'total_files': len(on_disk),
|
||||||
|
'total_bytes': total_bytes,
|
||||||
|
'by_folder': {k: {'files': v[0], 'bytes': v[1]} for k, v in by_folder.items()},
|
||||||
|
},
|
||||||
|
'orphans': {
|
||||||
|
'count': len(orphans),
|
||||||
|
'bytes': orphan_bytes,
|
||||||
|
},
|
||||||
|
# Full detail for follow-up / verification after migration:
|
||||||
|
'missing_on_disk': sorted(
|
||||||
|
[{'path': k, 'referenced_by': sorted(set(referenced[k]))} for k in missing],
|
||||||
|
key=lambda d: d['path']
|
||||||
|
),
|
||||||
|
'orphan_files': sorted(
|
||||||
|
[{'path': k, 'bytes': on_disk[k]} for k in orphans],
|
||||||
|
key=lambda d: d['path']
|
||||||
|
),
|
||||||
|
}
|
||||||
|
|
||||||
|
# ── stdout summary ──
|
||||||
|
print('=' * 68)
|
||||||
|
print(' JQC PHOTO AUDIT (read-only — nothing was modified)')
|
||||||
|
print('=' * 68)
|
||||||
|
print(f' uploads root : {uploads_root}')
|
||||||
|
print(f' generated : {report["generated_at"]}')
|
||||||
|
print('-' * 68)
|
||||||
|
print(' REFERENCED BY DATABASE')
|
||||||
|
for src, cnt in sorted(by_source.items()):
|
||||||
|
print(f' {src:<34} {cnt:>8} refs')
|
||||||
|
print(f' {"distinct photo files referenced":<34} {len(referenced_keys):>8}')
|
||||||
|
print('-' * 68)
|
||||||
|
print(' ON DISK')
|
||||||
|
for folder, (cnt, b) in sorted(by_folder.items()):
|
||||||
|
print(f' {folder:<24} {cnt:>8} files {human(b):>12}')
|
||||||
|
print(f' {"TOTAL":<24} {len(on_disk):>8} files {human(total_bytes):>12}')
|
||||||
|
print('-' * 68)
|
||||||
|
print(' RECONCILIATION')
|
||||||
|
print(f' referenced AND present on disk : {len(present):>8} <-- must be unchanged after migration')
|
||||||
|
print(f' referenced but MISSING on disk : {len(missing):>8} <-- pre-existing broken refs (investigate)')
|
||||||
|
print(f' on disk but UNREFERENCED : {len(orphans):>8} ({human(orphan_bytes)}, migrated anyway)')
|
||||||
|
print('=' * 68)
|
||||||
|
|
||||||
|
if missing:
|
||||||
|
print(f'\n WARNING: {len(missing)} referenced photo(s) are already missing on disk.')
|
||||||
|
print(' These are NOT caused by migration. First few:')
|
||||||
|
for m in report['missing_on_disk'][:10]:
|
||||||
|
print(f' - {m["path"]} (by {", ".join(m["referenced_by"][:3])})')
|
||||||
|
if len(missing) > 10:
|
||||||
|
print(f' ... and {len(missing) - 10} more (see JSON report)')
|
||||||
|
|
||||||
|
if args.list_orphans and orphans:
|
||||||
|
print(f'\n Orphan files ({len(orphans)}):')
|
||||||
|
for o in report['orphan_files']:
|
||||||
|
print(f' - {o["path"]} ({human(o["bytes"])})')
|
||||||
|
|
||||||
|
# ── write JSON report ──
|
||||||
|
report_path = args.report or os.path.join(
|
||||||
|
'/tmp', f'jqc_photo_audit_{datetime.utcnow().strftime("%Y%m%d_%H%M%S")}.json'
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
with open(report_path, 'w') as fh:
|
||||||
|
json.dump(report, fh, indent=2)
|
||||||
|
print(f'\n Full JSON report written to: {report_path}')
|
||||||
|
except OSError as e:
|
||||||
|
print(f'\n Could not write report to {report_path}: {e}')
|
||||||
|
|
||||||
|
print('\n BASELINE TO REMEMBER: '
|
||||||
|
f'{len(present)} photos must still resolve after cutover.\n')
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
main()
|
||||||
Reference in New Issue
Block a user