Jun 27 Update documents

2026-06-27 10:28:57 -04:00
parent 9f0f7d2f18
commit 7b764c76af
3 changed files with 466 additions and 61 deletions
@@ -2,7 +2,7 @@

 > **Audience:** AI assistants and developers working on this codebase.  
 > **Purpose:** Authoritative reference for architecture, conventions, gotchas, and decisions.  
-> **Last reviewed:** June 2026 (Phase 19 complete + mobile API gap-fill Phases A–E + customer UI refinements + Phase 22 comment visibility + Phase 23 support chat/tickets + inspector performance Excel export + inspection list filters + customer issue logging + AI chatbot + dashboard grouped sections + issues/inspections PDF export + date/ID filters + Reports expansion Phases R1–R4 + Phase 24 issue_created notify defaults + Phase 25 inspection GPS + Phase 26 issue vendor fields + Phase 27 facility score alerts)
+> **Last reviewed:** June 2026 (Phase 19 complete + mobile API gap-fill Phases A–E + customer UI refinements + Phase 22 comment visibility + Phase 23 support chat/tickets + inspector performance Excel export + inspection list filters + customer issue logging + AI chatbot + dashboard grouped sections + issues/inspections PDF export + date/ID filters + Reports expansion Phases R1–R4 + Phase 24 issue_created notify defaults + Phase 25 inspection GPS + Phase 26 issue vendor fields + Phase 27 facility score alerts + **MT-0 through MT-3 multi-tenant control plane**)

 ---

@@ -29,6 +29,7 @@
 19. [Infrastructure](#19-infrastructure)
 20. [Known Constraints & Hard Rules](#20-known-constraints--hard-rules)
 21. [Change Philosophy](#21-change-philosophy)
+22. [Multi-Tenant Architecture (MT-0 → MT-3)](#22-multi-tenant-architecture-mt-0--mt-3)

 ---

@@ -82,7 +83,14 @@ The application is actively deployed in production and maintained by a single de
 ```
 lt_janitorial_quality_control/
 ├── app/
-│   ├── __init__.py              # Application factory — limiter, csrf, db, mail, login_manager
+│   ├── __init__.py              # Application factory — limiter, csrf, db (RoutingSession), mail, login_manager, init_tenancy
+│   ├── tenancy/                 # MT-1 — tenant resolution + DB routing (inert unless MULTI_TENANT_ENABLED=true)
+│   │   ├── __init__.py          # public exports: RoutingSession, init_tenancy, TenantContext
+│   │   ├── context.py           # TenantContext frozen dataclass (g.tenant)
+│   │   ├── engine_cache.py      # per-tenant SQLAlchemy engine cache + invalidate()
+│   │   ├── middleware.py        # init_tenancy() — before_request Host→tenant resolver
+│   │   ├── resolver.py          # resolve_tenant(host) → TenantContext | None
+│   │   └── routing.py           # RoutingSession — routes db.session to g.tenant_engine
 │   ├── api/                     # Mobile REST API
 │   │   ├── __init__.py          # api_bp parent blueprint + register_api()
 │   │   ├── auth.py              # /api/v1/auth/* and /api/v1/devices/*
@@ -133,7 +141,24 @@ lt_janitorial_quality_control/
 │   └── utils/
 ├── migrations/
 │   └── versions/
-│       └── phase23_support_tickets.py   ← HEAD
+│       ├── 0003_add_user_active.py      ← squashed baseline (chain root, MT-2)
+│       └── phase32_device_token_columns ← HEAD
+├── migrations_tenant/           # MT-2 — standalone Alembic env for per-tenant upgrades
+│   ├── env.py                   # URL-driven, no Flask; reuses migrations/versions
+│   └── script.py.mako
+├── control/                     # MT-0 — control plane (tenant registry, plans, provisioning)
+│   ├── __init__.py
+│   ├── base.py                  # ControlBase, engine/session (CONTROL_DATABASE_URL)
+│   ├── cli.py                   # seed / create-superadmin / list-plans
+│   ├── crypto.py                # Fernet encrypt/decrypt (CONTROL_FERNET_KEY)
+│   ├── models.py                # Plan, PlanFeature, Tenant, TenantDomain,
+│   │                            #   Superadmin, ProvisioningJob, TenantAudit
+│   ├── provision.py             # MT-3 — create_tenant / register_tenant_zero / delete_tenant
+│   ├── seed.py                  # idempotent plan seeder
+│   ├── tenant_migrate.py        # MT-2 — upgrade_tenant / bootstrap_tenant / chain_head CLI
+│   ├── time_utils.py            # now_eastern() mirror (no app import)
+│   └── migrations/              # control Alembic chain (control{N}_…)
+│       └── versions/control0001_init.py ← HEAD
 └── ...
 ```

@@ -157,6 +182,15 @@ lt_janitorial_quality_control/
 | `REDIS_URL` | Optional. When set, Flask-Limiter uses Redis for shared rate-limit counters across Gunicorn workers. |
 | `GROQ_API_KEY` | Optional. When set, enables the AI chatbot at `/support/chat`. Absent → chat input disabled; customers see a "Submit to Support" fallback only. |
 | `GROQ_MODEL` | Optional. Groq model ID. Defaults to `llama-3.3-70b-versatile`. |
+| `MULTI_TENANT_ENABLED` | `false` by default. Set `true` to activate Host→tenant routing. Requires all control-plane vars below. |
+| `CONTROL_DATABASE_URL` | Control-plane MySQL URI, e.g. `mysql+pymysql://jqc_control:pw@127.0.0.1/jqc_control`. Required when `MULTI_TENANT_ENABLED=true`. |
+| `CONTROL_FERNET_KEY` | Fernet key for encrypting tenant DB passwords. Generate once; store in `/etc/jqc/control.env`. |
+| `PROVISION_DB_URL` | MySQL account that can `CREATE DATABASE` / `CREATE USER` / `GRANT`. e.g. `mysql+pymysql://jqc_provisioner:pw@127.0.0.1/`. |
+| `TENANT_BASE_DOMAIN` | Apex domain for subdomains, e.g. `jqc.app`. Used by provisioner to build `<slug>.jqc.app`. |
+| `MULTI_TENANT_EXEMPT_PATHS` | Comma-separated path prefixes that bypass the tenant gate (e.g. `/health`). `/static/` is always exempt. |
+| `TENANT_ENGINE_POOL_SIZE` | Per-tenant engine pool size (default 5). |
+| `TENANT_ENGINE_MAX_OVERFLOW` | Per-tenant pool max overflow (default 5). |
+| `TENANT_ENGINE_POOL_RECYCLE` | Pool recycle in seconds (default 1800). |

 ### Email SSL Auto-Detection

@@ -632,8 +666,13 @@ limiter = Limiter(

 ## 17. Alembic Migration Chain

+**Current HEAD:** `phase32_device_token_columns` (30 migrations total).
+
+**Chain root:** `0003_add_user_active` — a guarded squashed baseline (MT-2) that recreates the full 25-table schema with INFORMATION_SCHEMA guards. The original baseline migrations (0001/0002/0003) were lost; this file restores the chain root so Alembic can build the revision map. `down_revision = None`.
+
 ```
-phase1_projects_roles → phase6_features → phase7_mobile_api → phase8_notification_matrix
+0003_add_user_active (baseline, MT-2)
+  → phase1_projects_roles → phase6_features → phase7_mobile_api → phase8_notification_matrix
      → phase9_user_full_name → phase10_customer_password_setup → phase11_director_role
          → phase12_performance_indexes → phase_b_mobile_local_id
              → phase13_issue_facility → phase14_facility_created_at
@@ -649,7 +688,31 @@ phase1_projects_roles → phase6_features → phase7_mobile_api → phase8_notif
                                                      → phase24_notify_defaults
                                                          → phase25_inspection_gps
                                                              → phase26_issue_vendor
-                                                                → phase27_score_alerts   ← HEAD
+                                                                  → phase27_score_alerts
+                                                                      → ... → phase32_device_token_columns ← HEAD
+```
+
+### Fresh DB provisioning (multi-tenant)
+
+**Never use `flask db upgrade` on an empty database.** Fourteen of the phase migrations are not idempotent (no `INFORMATION_SCHEMA` guards) and will fail on a fresh DB that already has the baseline schema. Use the provisioner instead:
+
+```bash
+python -m control.tenant_migrate bootstrap --tenant <slug>
+# Runs: upgrade to 0003_add_user_active (builds full schema) → stamp head
+# Phase migrations are SKIPPED — baseline covers the full schema.
+```
+
+Ongoing incremental upgrades (phase33+, which MUST be guarded) use:
+
+```bash
+python -m control.tenant_migrate upgrade --tenant all
+```
+
+### Standard single-tenant migration deploy
+
+```bash
+flask db upgrade        # existing LT box — safe, already at head
+sudo systemctl restart jqc
 ```

 ### phase21_performance_indexes
@@ -945,6 +1008,47 @@ timeout = 30
 ### Nginx
 - `client_max_body_size 50M`
 - Passes `X-Forwarded-For`
+- **Multi-tenant wildcard block** (alongside the existing single-domain block):
+
+```nginx
+server {
+    listen 80;
+    server_name *.jqc.app jqc.app;
+
+    location / {
+        proxy_pass         http://127.0.0.1:8000;
+        proxy_set_header   Host              $host;   # resolver reads this
+        proxy_set_header   X-Real-IP         $remote_addr;
+        proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
+        proxy_set_header   X-Forwarded-Proto $scheme;
+    }
+}
+```
+
+**TLS for subdomains:** wildcard cert (`*.jqc.app`) via DNS-01 challenge (certbot + DNS plugin). HTTP-01 works for individual subdomains only and cannot auto-renew a wildcard. Custom-domain TLS via Caddy on-demand TLS (MT-6).
+
+### Control-plane environment file
+
+**All** control/provisioning env vars must live in one canonical file. Both the systemd unit and CLI sessions must source the same file — a key mismatch between them causes `cryptography.fernet.InvalidToken` at request time.
+
+```bash
+# /etc/jqc/control.env  (chmod 640, chown root:jqc)
+CONTROL_DATABASE_URL=mysql+pymysql://jqc_control:<pw>@127.0.0.1/jqc_control
+CONTROL_FERNET_KEY=<generated key — no quotes>
+PROVISION_DB_URL=mysql+pymysql://jqc_provisioner:<pw>@127.0.0.1/
+TENANT_BASE_DOMAIN=jqc.app
+MULTI_TENANT_ENABLED=true
+```
+
+```ini
+# systemd unit [Service]
+EnvironmentFile=/etc/jqc/control.env
+```
+
+```bash
+# CLI sessions
+set -a; . /etc/jqc/control.env; set +a
+```

 ### Recommended Cron Schedule
 ```bash
@@ -1026,6 +1130,12 @@ timeout = 30
 | 68 | **Support ticket customer replies revert status from `answered` → `open`** | When a customer posts a follow-up on an answered ticket, the route sets `ticket.status = 'open'` so admins see it in their open queue. Admin must manually close or re-answer. |
 | 69 | **Customer issue create: `assigned_to` field hidden, `facility_id` scoped to `get_customer_scope()`** | `issues.create()` detects `role == 'customer'`, scopes facilities to the customer's assigned set, sets `staff = []` for the assigned_to dropdown, and hides the field in `form.html`. `IssueForm.facility_id.choices` must still include all active facilities so POST validation passes. |
 | 70 | **`notify()` does NOT commit — caller must `db.session.commit()` after all `notify()` calls** | `notify()` adds a `Notification` row to the session but leaves the commit to the caller. The support helpers (`_notify_admins_new_ticket`, `_notify_customer_reply`, `_notify_admins_customer_reply`) each call `db.session.commit()` after the `notify()` loop. |
+| 71 | **`CONTROL_FERNET_KEY` must be identical between CLI sessions and the Gunicorn service** | Provisioning encrypts tenant creds; the app decrypts them at request time. A key mismatch causes `cryptography.fernet.InvalidToken`. Canonical source: `/etc/jqc/control.env` — sourced by both `EnvironmentFile=` in the systemd unit and `set -a; . /etc/jqc/control.env; set +a` in CLI sessions. |
+| 72 | **`MULTI_TENANT_ENABLED` must be flipped only after tenant-zero (LT) is registered** | The resolver gates all traffic by Host. If LT's domains aren't in `tenant_domains` when the flag goes `true`, LT's own traffic gets a 404 "Workspace not found" page. Register LT first via `register-tenant-zero`, confirm with `curl -H "Host: lts.jqc.app" http://127.0.0.1:8000/`, then flip the flag. |
+| 73 | **Tenant DB passwords use `_gen_password()` — not `secrets.token_urlsafe()`** | MySQL `validate_password` MEDIUM policy requires lower + upper + digit + special. `token_urlsafe` is alphanumeric-only and fails intermittently. `_gen_password()` guarantees all four character classes. |
+| 74 | **`bootstrap_tenant()` for fresh DBs; `upgrade_tenant()` for incremental** | Fourteen phase migrations are unguarded. Running the full chain on an empty DB (from the baseline) causes duplicate-column errors. `bootstrap_tenant()` runs the baseline to HEAD then stamps — phases skipped. `upgrade_tenant()` is for phase33+ incremental upgrades on already-provisioned DBs. |
+| 75 | **`delete_tenant()` + `_add_domains()` are retry-safe** | `_add_domains` uses `_upsert_domain()` (delete-then-insert) to handle orphan rows from failed partial runs. `delete_tenant()` also purges by derived domain string, not only by `tenant_id`, catching orphans whose parent tenant row was rolled back. |
+| 76 | **`register-tenant-zero` never bootstraps and never drops the DB** | `register_tenant_zero()` only reads the existing head, inserts control rows, and maps domains. `delete_tenant()` on tenant-zero must never use `--drop-db` — the guard checks `db_name == db_name_for(slug)` and refuses non-provisioner-named DBs (LT's DB name is `jqc_lt`, not `jqc_lts`). |

 ---

@@ -1039,3 +1149,102 @@ timeout = 30
 6. **Full file contents for 1–3 file changes**; deployment map for larger changesets
 7. **Explicit deploy instructions** — migration steps separated from code steps
 8. **Root cause analysis** on errors — never apply temporary workarounds
+
+---
+
+## 22. Multi-Tenant Architecture (MT-0 → MT-3)
+
+See `MULTI_TENANT_PLAN.md` for the full phased roadmap. This section summarises what is built and operational.
+
+### Model
+
+Shared codebase + **database-per-tenant**. One Flask/Gunicorn process serves all tenants. `before_request` resolves the `Host` header → tenant → per-tenant MySQL database. All existing `db.session` calls route transparently — zero changes to models or routes.
+
+### Control plane (`control/`)
+
+Self-contained package, own `ControlBase` + engine/session, own Alembic chain. No imports from `app/`.
+
+| Module | Purpose |
+|---|---|
+| `base.py` | `ControlBase`, `control_session()` context manager, engine from `CONTROL_DATABASE_URL` |
+| `models.py` | `Plan`, `PlanFeature`, `Tenant`, `TenantDomain`, `Superadmin`, `ProvisioningJob`, `TenantAudit` |
+| `crypto.py` | Fernet encrypt/decrypt for `tenants.db_password_enc` |
+| `seed.py` | Idempotent plan seeder (Free / Starter / Pro / Enterprise) |
+| `cli.py` | `seed`, `create-superadmin`, `list-plans` |
+| `tenant_migrate.py` | `bootstrap_tenant()`, `upgrade_tenant()`, `chain_head()`, `current_revision()` + CLI |
+| `provision.py` | `create_tenant()`, `register_tenant_zero()`, `delete_tenant()` + CLI |
+
+### Data-plane changes (`app/`)
+
+| File | Change |
+|---|---|
+| `app/__init__.py` | `db = SQLAlchemy(session_options={'class_': RoutingSession})` + `init_tenancy(app)` |
+| `config.py` | `MULTI_TENANT_ENABLED` (default `false`) + pool tunables |
+| `app/tenancy/` | New package — see §3 repo layout |
+
+### Plan tiers
+
+| Axis | Free | Starter | Pro | Enterprise |
+|---|---|---|---|---|
+| Max users | 3 | 15 | 50 | unlimited |
+| Max facilities | 2 | 10 | 50 | unlimited |
+| Inspections / month | 50 | 500 | 5 000 | unlimited |
+| Issues / month | 50 | 500 | 5 000 | unlimited |
+| Mobile API | ✗ | ✓ | ✓ | ✓ |
+| Scheduled reports | ✗ | ✗ | ✓ | ✓ |
+| Branding | ✗ | ✗ | ✓ | ✓ |
+| Custom domain | ✗ | ✗ | ✓ | ✓ |
+| Subdomain | ✓ | ✓ | ✓ | ✓ |
+
+Quota-exceed behaviour: **soft warn** — allow submit, flag for upgrade, never reject.
+
+### Provisioner MySQL account (required grants)
+
+```sql
+GRANT ALL PRIVILEGES ON *.* TO 'jqc_provisioner'@'localhost' WITH GRANT OPTION;
+GRANT CREATE USER ON *.* TO 'jqc_provisioner'@'localhost';
+FLUSH PRIVILEGES;
+```
+
+### CLI quick-reference
+
+```bash
+# Control schema + plans + first superadmin (run once)
+alembic -c control/migrations/alembic.ini upgrade head
+python -m control.cli seed
+python -m control.cli create-superadmin --username admin --email you@example.com
+
+# Adopt existing LT database as tenant-zero (run once, no data move)
+python -m control.provision register-tenant-zero \
+  --slug lts --name "LT Services" --plan enterprise \
+  --db-host 127.0.0.1 --db-name <LT_DB> --db-user <LT_USER> --db-password '<pw>' \
+  --custom-domain jqc.ltservicesinc.com --base-domain jqc.app
+
+# Provision a new tenant (creates MySQL DB + user + schema + first admin)
+python -m control.provision create-tenant \
+  --slug acme --name "Acme Corp" --plan pro --admin-email ops@acme.com
+
+# Delete / deregister a tenant
+python -m control.provision delete-tenant --slug ztest --drop-db --yes   # provisioned DB
+python -m control.provision delete-tenant --slug lts --yes               # adopted DB — no --drop-db
+
+# Migration status + upgrades
+python -m control.tenant_migrate heads
+python -m control.tenant_migrate current --tenant all
+python -m control.tenant_migrate upgrade --tenant all    # incremental (phase33+)
+python -m control.tenant_migrate bootstrap --tenant acme # fresh DB only
+```
+
+### Enable multi-tenancy (cutover sequence)
+
+```bash
+# 1. Register LT as tenant-zero (see above)
+# 2. Smoke-test routing (flag still off)
+curl -sI -H "Host: lts.jqc.app"  http://127.0.0.1:8000/ | head -2
+curl -sI -H "Host: ztest.jqc.app" http://127.0.0.1:8000/ | head -2
+# 3. Wildcard DNS: *.jqc.app A <SERVER_IP>
+# 4. Nginx wildcard server block (see §19)
+# 5. Wildcard TLS cert (DNS-01, certbot)
+# 6. Add to /etc/jqc/control.env: MULTI_TENANT_ENABLED=true
+# 7. sudo systemctl daemon-reload && sudo systemctl restart jqc
+```
@@ -1,6 +1,6 @@
 # JQC — Janitorial Quality Control System

-A production-grade web application for managing janitorial service contracts, facility inspections, issue tracking, and client reporting.
+A production-grade, multi-tenant web application for managing janitorial service contracts, facility inspections, issue tracking, and client reporting. Built as a shared-codebase SaaS with database-per-tenant isolation.

 ---

@@ -169,6 +169,7 @@ After=network.target mysql.service
 User=jqc
 WorkingDirectory=/home/jqc/lt_janitorial_quality_control
 EnvironmentFile=/home/jqc/.env
+EnvironmentFile=/etc/jqc/control.env    # control-plane + multi-tenant vars
 ExecStart=/home/jqc/venv/bin/gunicorn -c gunicorn_config.py wsgi:app
 Restart=on-failure

@@ -176,6 +177,8 @@ Restart=on-failure
 WantedBy=multi-user.target
 ```

+> `/etc/jqc/control.env` must be `chmod 640 / chown root:jqc`. It holds `CONTROL_DATABASE_URL`, `CONTROL_FERNET_KEY`, `PROVISION_DB_URL`, `TENANT_BASE_DOMAIN`, and `MULTI_TENANT_ENABLED`. The same file must be sourced in any shell session that runs provisioning commands (`set -a; . /etc/jqc/control.env; set +a`).
+
 ---

 ## Cron Jobs
@@ -260,7 +263,7 @@ Access tokens expire after 60 minutes. Refresh tokens are valid for 30 days and
 ## Database Migrations

 ```bash
-# Apply all pending migrations
+# Apply all pending migrations (existing / single-tenant DB)
 flask db upgrade

 # Create a new migration after model changes
@@ -270,6 +273,21 @@ flask db migrate -m "description of change"
 flask db downgrade
 ```

+### Multi-tenant migrations
+
+```bash
+# Bootstrap a brand-new tenant DB (baseline schema + stamp head)
+python -m control.tenant_migrate bootstrap --tenant <slug>
+
+# Incremental upgrade for all active tenants (phase33+ onwards)
+python -m control.tenant_migrate upgrade --tenant all
+
+# Check each tenant's current revision vs chain head
+python -m control.tenant_migrate current --tenant all
+```
+
+> **Never run `flask db upgrade` on an empty tenant database.** Fourteen historical phase migrations lack `INFORMATION_SCHEMA` guards and will fail against a DB that already has the baseline schema. Use `bootstrap` instead.
+
 ### MySQL Compatibility Notes

 - **ENUM changes** require three steps: expand → migrate data → contract. Never skip steps.
@@ -304,6 +322,64 @@ flask db downgrade
 | `REDIS_URL` | — | Redis connection URI for shared rate-limit storage; optional but recommended in production |
 | `GROQ_API_KEY` | — | Groq API key. When absent the AI chatbot is disabled; customers can still submit support tickets. |
 | `GROQ_MODEL` | `llama-3.3-70b-versatile` | Groq model ID override |
+| `MULTI_TENANT_ENABLED` | `false` | Set `true` to activate Host→tenant routing. Requires the four control-plane vars below. |
+| `CONTROL_DATABASE_URL` | — | Control-plane DB URI, e.g. `mysql+pymysql://jqc_control:pw@127.0.0.1/jqc_control` |
+| `CONTROL_FERNET_KEY` | — | Fernet key for encrypting tenant DB passwords. Generate once and store permanently. |
+| `PROVISION_DB_URL` | — | MySQL account with `CREATE DATABASE`/`CREATE USER`/`GRANT`. |
+| `TENANT_BASE_DOMAIN` | — | Subdomain apex, e.g. `jqc.app`. |
+
+---
+
+## Multi-Tenant Setup
+
+JQC runs as a shared-codebase SaaS with database-per-tenant isolation. Each tenant gets its own MySQL database and least-privilege MySQL user. The control plane (`control/`) manages the tenant registry separately from any tenant's data.
+
+### Prerequisites
+
+```sql
+-- Control database
+CREATE DATABASE jqc_control CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
+CREATE USER 'jqc_control'@'localhost' IDENTIFIED BY '<pw>';
+GRANT ALL PRIVILEGES ON jqc_control.* TO 'jqc_control'@'localhost'; FLUSH PRIVILEGES;
+
+-- Provisioner account (creates per-tenant DBs + users)
+CREATE USER 'jqc_provisioner'@'localhost' IDENTIFIED BY '<pw>';
+GRANT ALL PRIVILEGES ON *.* TO 'jqc_provisioner'@'localhost' WITH GRANT OPTION;
+GRANT CREATE USER ON *.* TO 'jqc_provisioner'@'localhost'; FLUSH PRIVILEGES;
+```
+
+### Bootstrap (run once)
+
+```bash
+# Set env (or add to /etc/jqc/control.env)
+export CONTROL_DATABASE_URL='mysql+pymysql://jqc_control:<pw>@127.0.0.1/jqc_control'
+export CONTROL_FERNET_KEY="$(python -c 'from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())')"
+export PROVISION_DB_URL='mysql+pymysql://jqc_provisioner:<pw>@127.0.0.1/'
+export TENANT_BASE_DOMAIN='jqc.app'
+
+# Apply control schema + seed plans + first superadmin
+alembic -c control/migrations/alembic.ini upgrade head
+python -m control.cli seed
+python -m control.cli create-superadmin --username admin --email you@example.com
+
+# Adopt existing LT database as tenant-zero (no data moved)
+python -m control.provision register-tenant-zero \
+  --slug lts --name "LT Services" --plan enterprise \
+  --db-host 127.0.0.1 --db-name <LT_DB> --db-user <LT_USER> --db-password '<pw>' \
+  --custom-domain jqc.ltservicesinc.com
+
+# Provision a new tenant
+python -m control.provision create-tenant \
+  --slug acme --name "Acme Corp" --plan pro --admin-email ops@acme.com
+```
+
+### Enable multi-tenancy
+
+1. Add `MULTI_TENANT_ENABLED=true` to `/etc/jqc/control.env`
+2. Configure wildcard DNS: `*.jqc.app A <SERVER_IP>`
+3. Add Nginx wildcard server block (see `CLAUDE.md §19`)
+4. Obtain wildcard TLS cert via DNS-01 challenge
+5. `sudo systemctl daemon-reload && sudo systemctl restart jqc`

 ---

@@ -1,84 +1,204 @@
-# Control Plane (MT-0)
+# Control Plane (MT-0 → MT-3)

 Tenant registry, plans, domains, provisioning state, and superadmin accounts
-for multi-tenant JQC. Self-contained and decoupled from `app/` — the existing
-single-tenant application is unaffected by this package.
+for multi-tenant JQC. Self-contained and decoupled from `app/` — imports
+nothing from the data-plane application.

-See `../MULTI_TENANT_PLAN.md` for the full architecture and roadmap.
+See `../MULTI_TENANT_PLAN.md` for the full phased roadmap.

 ## Layout

 ```
 control/
 ├── __init__.py        # package docs
-├── base.py            # ControlBase + engine/session (from CONTROL_DATABASE_URL)
-├── crypto.py          # Fernet encrypt/decrypt for tenant DB passwords
+├── base.py            # ControlBase + engine/session (CONTROL_DATABASE_URL)
+├── crypto.py          # Fernet encrypt/decrypt for tenant DB passwords (CONTROL_FERNET_KEY)
 ├── time_utils.py      # now_eastern() mirror (no app import)
 ├── models.py          # Plan, PlanFeature, Tenant, TenantDomain,
 │                      #   Superadmin, ProvisioningJob, TenantAudit
-├── seed.py            # idempotent baseline-plan seeder
+├── seed.py            # idempotent baseline-plan seeder (Free/Starter/Pro/Enterprise)
 ├── cli.py             # seed / create-superadmin / list-plans
+├── tenant_migrate.py  # MT-2 — upgrade_tenant / bootstrap_tenant / chain_head + CLI
+├── provision.py       # MT-3 — create_tenant / register_tenant_zero / delete_tenant + CLI
 └── migrations/        # standalone Alembic chain (control{N}_…)
-    └── versions/control0001_init.py   ← HEAD
+    └── versions/
+        └── control0001_init.py   ← HEAD (7 tables)
 ```

-## Environment variables (control plane only)
+## Environment variables
+
+All must live in `/etc/jqc/control.env` (see §3 below). Never hard-code.

 | Variable | Purpose |
 |---|---|
-| `CONTROL_DATABASE_URL` | e.g. `mysql+pymysql://jqc_control:pw@localhost/jqc_control` |
-| `CONTROL_FERNET_KEY` | Fernet key for encrypting tenant DB passwords |
+| `CONTROL_DATABASE_URL` | e.g. `mysql+pymysql://jqc_control:pw@127.0.0.1/jqc_control` |
+| `CONTROL_FERNET_KEY` | Fernet key for encrypting tenant DB passwords. Generate once, never rotate without re-encrypting all tenant rows. |
+| `PROVISION_DB_URL` | MySQL account with `CREATE DATABASE` / `CREATE USER` / `GRANT` rights. |
+| `TENANT_BASE_DOMAIN` | Apex for subdomains, e.g. `jqc.app`. Builds `<slug>.jqc.app`. |

-Generate a Fernet key:
+Generate a Fernet key (run once, store permanently):

 ```bash
 python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"
 ```

-## Deploy — MT-0 bootstrap (run once)
+## Canonical env file (required — prevents Fernet key mismatch)

-Migrations are independent of the tenant chain and the data plane. The existing
-app does **not** need to be touched or restarted for MT-0.
+The `CONTROL_FERNET_KEY` used at provisioning time must be identical to the key
+the running app uses to decrypt tenant creds. A mismatch causes
+`cryptography.fernet.InvalidToken` at request time. One file prevents this:

 ```bash
-# 1. Create the control database + its MySQL user (run as a MySQL admin)
-mysql -e "CREATE DATABASE jqc_control CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;"
-mysql -e "CREATE USER 'jqc_control'@'localhost' IDENTIFIED BY '<pw>';"
-mysql -e "GRANT ALL PRIVILEGES ON jqc_control.* TO 'jqc_control'@'localhost'; FLUSH PRIVILEGES;"
+# /etc/jqc/control.env   (chmod 640, chown root:jqc)
+CONTROL_DATABASE_URL=mysql+pymysql://jqc_control:<pw>@127.0.0.1/jqc_control
+CONTROL_FERNET_KEY=<key — no quotes, no trailing space>
+PROVISION_DB_URL=mysql+pymysql://jqc_provisioner:<pw>@127.0.0.1/
+TENANT_BASE_DOMAIN=jqc.app
+MULTI_TENANT_ENABLED=true
+```

-# 2. Export the env vars (add to .env or the systemd unit for the control panel later)
-export CONTROL_DATABASE_URL='mysql+pymysql://jqc_control:<pw>@localhost/jqc_control'
-export CONTROL_FERNET_KEY='<generated key>'
+```ini
+# systemd unit [Service]
+EnvironmentFile=/etc/jqc/control.env
+```

-# 3. Apply the control schema
+```bash
+# any CLI session that provisions tenants
+set -a; . /etc/jqc/control.env; set +a
+```
+
+## MySQL accounts required
+
+```sql
+-- Control DB user
+CREATE DATABASE jqc_control CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
+CREATE USER 'jqc_control'@'localhost' IDENTIFIED BY '<pw>';
+GRANT ALL PRIVILEGES ON jqc_control.* TO 'jqc_control'@'localhost'; FLUSH PRIVILEGES;
+
+-- Provisioner (creates per-tenant DBs + users)
+CREATE USER 'jqc_provisioner'@'localhost' IDENTIFIED BY '<pw>';
+GRANT ALL PRIVILEGES ON *.* TO 'jqc_provisioner'@'localhost' WITH GRANT OPTION;
+GRANT CREATE USER ON *.* TO 'jqc_provisioner'@'localhost'; FLUSH PRIVILEGES;
+```
+
+## Bootstrap — MT-0 (run once per environment)
+
+```bash
+set -a; . /etc/jqc/control.env; set +a
+
+# 1. Apply control schema (7 tables)
 alembic -c control/migrations/alembic.ini upgrade head

-# 4. Seed baseline plans (Free / Starter / Pro / Enterprise)
+# 2. Seed Free / Starter / Pro / Enterprise plans
 python -m control.cli seed

-# 5. Create the first superadmin
+# 3. First superadmin
 python -m control.cli create-superadmin --username admin --email you@example.com
+
+# Verify
+alembic -c control/migrations/alembic.ini current   # → control0001_init (head)
+python -m control.cli list-plans                    # → 4 plans
 ```

-Verify:
+## Tenant-zero — register existing LT in place (MT-3)
+
+No DB creation, no data movement, no schema changes to the live LT database.

 ```bash
-alembic -c control/migrations/alembic.ini current   # → control0001_init (head)
-python -m control.cli list-plans
+python -m control.provision register-tenant-zero \
+  --slug lts --name "LT Services" --plan enterprise \
+  --db-host 127.0.0.1 \
+  --db-name  <LT_DATABASE_NAME> \
+  --db-user  <LT_DB_USER> \
+  --db-password '<LT_DB_PASSWORD>' \
+  --custom-domain jqc.ltservicesinc.com \
+  --base-domain jqc.app
 ```

+Output includes `alembic_head` (read from the existing `alembic_version` table)
+and both domain mappings (`lts.jqc.app` + `jqc.ltservicesinc.com` as verified).
+
+## Provisioning a new tenant (MT-3)
+
+```bash
+python -m control.provision create-tenant \
+  --slug acme \
+  --name "Acme Corp" \
+  --plan pro \
+  --admin-email ops@acme.com \
+  --base-domain jqc.app
+```
+
+Steps executed automatically:
+1. `CREATE DATABASE jqc_acme` + `CREATE USER jqc_acme_u` + `GRANT` (scoped to that DB)
+2. Insert `tenants` row with Fernet-encrypted password
+3. `bootstrap_tenant()` → baseline schema + `stamp head` (phase migrations skipped)
+4. Insert first admin into tenant DB with a set-password token
+5. Register `acme.jqc.app` (verified) + optional custom domain (unverified)
+6. Set tenant `status = active`
+
+Output includes the **admin setup link** — send to the admin's email:
+`https://acme.jqc.app/customers/set-password/<token>`
+
+On failure, a best-effort rollback drops the DB/user and removes the tenant row
+so a retry starts clean.
+
+## Per-tenant migrations (MT-2)
+
+```bash
+# New fresh DB (empty → full schema + stamp head; phase migrations skipped)
+python -m control.tenant_migrate bootstrap --tenant acme
+
+# Incremental upgrade (phase33+ — must be INFORMATION_SCHEMA-guarded)
+python -m control.tenant_migrate upgrade --tenant all
+python -m control.tenant_migrate upgrade --tenant acme   # by slug
+python -m control.tenant_migrate upgrade --tenant 3      # by id
+
+# Status check
+python -m control.tenant_migrate current --tenant all
+python -m control.tenant_migrate heads
+```
+
+**Rule:** every new migration (phase33+) MUST use INFORMATION_SCHEMA existence
+checks (`_table_exists`, `_column_exists`, `_index_exists`) — the same pattern
+used by the squashed baseline. Unguarded migrations break idempotency and cannot
+be re-run safely across the fleet.
+
+## Delete / deregister a tenant
+
+```bash
+# Provisioned tenant — delete DB + control records
+python -m control.provision delete-tenant --slug acme --drop-db --yes
+
+# Adopted DB (tenant-zero) — remove control records only; NEVER --drop-db
+python -m control.provision delete-tenant --slug lts --yes
+```
+
+`--drop-db` is guarded: it only proceeds when the stored `db_name` matches the
+provisioner convention (`jqc_<slug>`). This prevents accidental drops of
+externally-named databases like LT's production DB.
+
 ## Rollback

 ```bash
-alembic -c control/migrations/alembic.ini downgrade base   # drops all control tables
+# Drop all 7 control tables (destructive)
+alembic -c control/migrations/alembic.ini downgrade base
 ```

-## Notes
+## Plan tier matrix

- The migration uses INFORMATION_SCHEMA existence checks (Rule 14) — safe to re-run.
- Plan seeding is idempotent (upsert by `code`) — re-running updates in place.
- Tenant DB passwords are stored Fernet-encrypted in `tenants.db_password_enc`;
-  `Tenant.db_uri` decrypts on demand. Provisioning that *creates* per-tenant
-  MySQL users/grants lands in MT-3.
- Control-panel write auditing (`tenant_audit`) is wired in MT-4; the bootstrap
-  CLI logs to stdout only.
+| Axis | Free | Starter | Pro | Enterprise |
+|---|---|---|---|---|
+| Max users | 3 | 15 | 50 | unlimited |
+| Max facilities | 2 | 10 | 50 | unlimited |
+| Inspections / month | 50 | 500 | 5 000 | unlimited |
+| Issues / month | 50 | 500 | 5 000 | unlimited |
+| Mobile API | ✗ | ✓ | ✓ | ✓ |
+| Scheduled reports | ✗ | ✗ | ✓ | ✓ |
+| Branding | ✗ | ✗ | ✓ | ✓ |
+| Custom domain | ✗ | ✗ | ✓ | ✓ |
+| Subdomain | ✓ | ✓ | ✓ | ✓ |
+
+Quota-exceed: **soft warn** — allow submit, flag for upgrade, never reject.
+Plans are seeded idempotently; re-running `python -m control.cli seed` updates
+in-place without creating duplicates.