Aug 27 - Add MySQL optimize and security check
This commit is contained in:
@@ -31,6 +31,8 @@
|
||||
21. [Change Philosophy](#21-change-philosophy)
|
||||
22. [Object Storage Migration (R2)](#22-object-storage-migration-r2)
|
||||
23. [Photo Capture-Time / Geo Overlay](#23-photo-capture-time--geo-overlay)
|
||||
24. [Enrollment Form](#24-enrollment-form-enrollment)
|
||||
25. [Database Health Check](#25-database-health-check-scriptsdb_healthpy)
|
||||
|
||||
---
|
||||
|
||||
@@ -168,6 +170,8 @@ part of the tree — see §7. Device registration on the API side lives in
|
||||
| `ENROLLMENT_DIR` | Optional. Directory for enrollment-form JSON submissions. Defaults to `<instance_path>/enrollments` (git-ignored). Created at boot. |
|
||||
| `DEFAULT_UI_THEME` | Optional, default `modern` (phase50). The design shown when a user has no stored preference — i.e. new accounts and unauthenticated pages. A stored `users.ui_theme` always wins. Set `classic` to revert the default **without** touching anyone's saved choice. |
|
||||
| `COMMENTS_VISIBLE_TO_ALL` | Optional, default `true`. **TEMPORARY (Aug 2026).** When true, customers see *every* comment on an issue, not only those ticked "Share with customer". Set `false` to restore the phase22 staff-only filtering — `is_customer_visible` is still written on every comment, so the revert needs no data repair. |
|
||||
| `DB_POOL_RECYCLE` | Optional, default `1800` (seconds). Retires a pooled connection after this long. **Must stay below the server's `wait_timeout`** or MySQL closes the socket first and the next request gets `OperationalError 2006`. `scripts/db_health.py` cross-checks the two. |
|
||||
| `DB_POOL_SIZE` / `DB_MAX_OVERFLOW` | Optional, default `5` / `5`. Per-**worker** pool. Gunicorn runs `cpu*2+1` sync workers and each holds its own pool, so the ceiling is `workers x (size + overflow)` — the library defaults (5+10) put a 9-worker box at 135 against a `max_connections` of 151. A sync worker serves one request at a time and needs one connection in steady state; the overflow is headroom for the background email/notification threads. |
|
||||
| `PHOTO_STAMP_ENABLED` | Optional, default `true`. Burns a capture-time + geo overlay into photos uploaded via `POST /api/v1/photos/upload`. Set `false` to store raw uploads. |
|
||||
|
||||
### Email SSL Auto-Detection
|
||||
@@ -1974,3 +1978,47 @@ Login-free, so: CSRF-protected form, `@limiter.limit('5 per hour')` on POST only
|
||||
### Admin
|
||||
|
||||
`/enrollment/admin` (admin-only, linked from the **Admin** nav dropdown in both layouts). List → detail → office-use fields (Receive Date / Program By / Date email invitation) + status (new / in_progress / completed). `GET /admin/<id>.json` downloads the raw file; `GET /admin/export.csv` emits **one row per person, not per submission** — that is the unit of work when actually creating the accounts. Task cells a person's role cannot have export as `n/a`, distinct from an unticked `''`.
|
||||
---
|
||||
|
||||
## 25. Database Health Check (`scripts/db_health.py`)
|
||||
|
||||
A standalone operations tool for the MySQL side. It imports the app factory for
|
||||
config and nothing else — no request layer, no uploads tree — and the plain
|
||||
invocation is **strictly read-only** (INFORMATION_SCHEMA / SHOW / EXPLAIN only).
|
||||
|
||||
```bash
|
||||
python scripts/db_health.py # read-only report
|
||||
python scripts/db_health.py --json /tmp/db.json # + machine-readable
|
||||
python scripts/db_health.py --apply-indexes # create the missing indexes
|
||||
python scripts/db_health.py --analyze # refresh optimizer stats (safe)
|
||||
python scripts/db_health.py --optimize --yes # rebuild tables (LOCKS — window only)
|
||||
python scripts/db_health.py --emit-migration migrations/versions/phase54_perf_indexes.py --revision phase54_perf_indexes
|
||||
```
|
||||
|
||||
**It never DROPs anything.** Redundant indexes are reported with the SQL to run
|
||||
by hand, because "unused" is a judgement the tool should not make for you. It
|
||||
also refuses to offer a **FK-backed** index as a drop candidate — dropping one
|
||||
fails with errno 150, since MySQL needs it for the constraint.
|
||||
|
||||
Checks, in order: SQLAlchemy pool options (`pool_pre_ping` / `pool_recycle` —
|
||||
the two that decide whether an idle overnight produces `MySQL server has gone
|
||||
away`); server settings cross-checked against the app (`max_connections` vs the
|
||||
worst-case Gunicorn pool, `wait_timeout` vs `pool_recycle`,
|
||||
`innodb_buffer_pool_size` vs the live data size, slow-query log, STRICT mode);
|
||||
schema hygiene (non-InnoDB, non-utf8mb4, mixed collations — a collation mismatch
|
||||
on a join column silently disables the index); table footprint; missing indexes
|
||||
against a curated list; redundant indexes; unindexed foreign keys; and EXPLAIN
|
||||
over the dashboard, both list pages, the SLA cron and the mobile notification
|
||||
poll, flagging full scans / filesorts / temp tables.
|
||||
|
||||
`RECOMMENDED_INDEXES` in the script is the **single place** the index wish-list
|
||||
lives, and every entry names the query that justifies it. An index nothing runs
|
||||
is pure write-amplification, so keep speculative entries out — and when a new
|
||||
hot query lands, add its index there rather than to an ad-hoc migration, so the
|
||||
checker keeps agreeing with the schema.
|
||||
|
||||
`--emit-migration` writes a re-runnable Alembic migration (INFORMATION_SCHEMA
|
||||
guards per rule 16) rather than applying DDL out of band. It guesses
|
||||
`down_revision` from the versions directory — confirm against `flask db heads`
|
||||
before committing.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user