JQC_multi_tenant/MULTI_TENANT_PLAN.md

# Multi-Tenant Plan — JQC

> **Audience:** AI assistants and developers extending JQC into a multi-tenant SaaS.
> **Companion to:** `CLAUDE.md` (single-tenant architecture reference).
> **Status:** MT-0 through MT-8 complete and deployed (MT-8 billing is flag-gated behind `BILLING_ENABLED`, default off). MT-9 (iOS multi-tenant) is fully pending — both the server-side discovery endpoints and the iOS client are unbuilt.

---

## 1. Decisions (locked)

| Decision | Choice |
|---|---|
| Isolation model | **Shared codebase + database-per-tenant** (one deploy, dynamic Host routing). NOT fork-per-tenant. |
| "Managed by tenant" | Tenant-admin self-serves branding / users / settings / in-plan feature toggles. No per-tenant code edits. |
| Plan-tier axes | (1) user quota, (2) facility quota, (3) inspections/month quota, (4) issues/month quota, (5) feature gates (mobile API, scheduled reports, …), (6) branding/white-label, (7) custom-domain vs subdomain-only. |
| Existing LT deploy | Migrated in as **tenant-zero** — register existing live DB in place, no data move (§8). |
| Quota-exceed behavior | **Soft warn** — allow submit, flag for upgrade. Never reject. |
| Tenant DB credentials | **Per-tenant MySQL user + password**. Provisioning creates the user/grants; creds encrypted at rest in control DB. |

Terminology: **control plane** = manages tenants (registry, plans, provisioning). **data plane** = serves tenant traffic (the existing Flask app, now tenant-aware).

---

## 2. Why this model

- **DB-per-tenant** → true data isolation; satisfies "separate database". Each tenant = own MySQL database, e.g. `jqc_<slug>`.
- **Shared code** → one codebase to patch. As a solo maintainer, N forks would mean N bug-fix deploys. Avoided.
- App already uses the **application-factory pattern** (`create_app`) and **relative `/api/v1` paths** — both ideal for adding Host-based tenant routing with minimal churn.
- Migrations already use `INFORMATION_SCHEMA` existence checks (CLAUDE.md Rule 14) → safe to re-run across every tenant DB unchanged.

---

## 3. Control-plane database (new, separate schema)

Lives in its own MySQL database (e.g. `jqc_control`), its own Alembic env (chain prefix `control{N}_…`), never routed by tenant middleware.

```
plans
  id, code (slug: free/starter/pro/enterprise), name, active,
  max_users            INT NULL,   -- NULL = unlimited
  max_facilities       INT NULL,
  max_inspections_month INT NULL,
  max_issues_month     INT NULL,
  allow_custom_domain  BOOL,
  allow_mobile_api     BOOL,
  allow_scheduled_reports BOOL,
  allow_branding       BOOL,        -- white-label
  price_cents          INT NULL,    -- future billing
  billing_period       VARCHAR(16) NULL

plan_features          -- optional EAV escape hatch for future boolean flags
  id, plan_id (FK), feature_key, enabled BOOL

tenants
  id, slug (subdomain label, UNIQUE), name, plan_id (FK),
  status ENUM(provisioning, active, suspended, deleted),
  db_host, db_port, db_name, db_user, db_password_enc,   -- creds encrypted at rest
  alembic_head VARCHAR(64),         -- last tenant-schema rev applied
  created_at, suspended_at, notes

tenant_domains
  id, tenant_id (FK), domain (UNIQUE), kind ENUM(subdomain, custom),
  is_primary BOOL, verified BOOL, verification_token,
  tls_status ENUM(pending, active, failed), created_at

superadmins                         -- cross-tenant accounts, control-plane only
  id, username, email, password_hash, active, created_at

provisioning_jobs
  id, tenant_id (FK), action ENUM(create_db, migrate, seed, suspend, delete),
  status ENUM(queued, running, ok, failed), log TEXT, created_at, finished_at

tenant_audit                        -- superadmin actions (separate from per-tenant log_action)
  id, superadmin_id, action, tenant_id, details, ip_address, created_at
```

Submission quotas (inspections/issues per month) are enforced by **live-counting current-period rows in the tenant DB** at submit time (cheap with an index on the date column). No separate counter table required; always accurate.

---

## 4. Tenant resolution + DB routing (the core mechanism)

**Resolution** (`before_request`): `host = request.host.split(':')[0]` → look up `tenant_domains.domain == host` where `verified AND tenant.status='active'` → load `tenant` + `plan` into `g`. Unknown/unverified/suspended host → branded error/landing page (no tenant DB touched).

**Routing without rewriting every `db.session` call** — use a routing Session so the entire existing codebase keeps using `db.session` unchanged:

```python
# app/tenancy/routing.py
from sqlalchemy.orm import Session
from flask import g

class RoutingSession(Session):
    def get_bind(self, mapper=None, clause=None, **kw):
        return getattr(g, 'tenant_engine', None) or _default_engine
```

> **MT-0 refinement (implemented):** the control plane uses its own
> `ControlBase` + engine + session (`control/base.py`), **not** `db.Model` with
> a `__control_plane__` flag. The `RoutingSession` (MT-1) therefore only handles tenant models.

```python
# app/__init__.py  (one-line change to the existing db init)
db = SQLAlchemy(session_options={'class_': RoutingSession})
```

- `_engine_cache`: `dict[tenant_id -> Engine]`, lazily built, small pools + `pool_recycle`.
- **Net code change to existing models/routes: zero.**

**MT-5 addition:** Plan limits and feature flags are loaded into `TenantContext` by the resolver in the same control-DB session — zero extra queries per request. `g.tenant.allow_mobile_api`, `g.tenant.max_users`, etc. are available everywhere.

---

## 5. Migrations

Two independent Alembic chains:

1. **Tenant schema** — existing chain (HEAD: **`phase33_tenant_settings`**). Runs per-tenant DB. New tenant features continue as `phase34_…` per existing naming.
2. **Control schema** — chain `control{N}_…`, runs once against `jqc_control`. HEAD: `control0004_dunning_tracking` (`control0001_init → control0002_billing → control0003_trial_reminder_sent → control0004_dunning_tracking`).

**CLI (always source env first):**
```bash
set -a; . /etc/jqc/control.env; set +a
python -m control.tenant_migrate upgrade --tenant all
```

Deploy ordering rule: run tenant migrations **before** shipping app code that references new columns.

---

## 6. Plan-tier matrix

| Axis | Free | Starter | Pro | Enterprise |
|---|---|---|---|---|
| Max users | 3 | 15 | 50 | unlimited |
| Max facilities | 2 | 10 | 50 | unlimited |
| Inspections / month | 50 | 500 | 5 000 | unlimited |
| Issues / month | 50 | 500 | 5 000 | unlimited |
| Mobile API (iPad app) | ✗ | ✓ | ✓ | ✓ |
| Scheduled reports | ✗ | ✗ | ✓ | ✓ |
| Branding / white-label | ✗ | ✗ | ✓ | ✓ |
| Custom domain | ✗ | ✗ | ✓ | ✓ |
| Subdomain (`*.jqc.app`) | ✓ | ✓ | ✓ | ✓ |

Enforcement: `@feature_required('mobile_api')` hard-blocks (403) disabled features. `@quota_soft_check('inspections')` sets `g.quota_warning` but **never rejects** — over-limit submits succeed, UI shows upgrade prompt. Both decorators are **inert** when `MULTI_TENANT_ENABLED=false`.

Feature + quota checks live in **both** web routes and `/api/v1` endpoints (iPad submits via API — web-only check is bypassable).

---

## 7. Phased roadmap

**MT-0 — Control-plane scaffold. ✅ DONE.**
Self-contained `control/` package: own `ControlBase` + engine + session, own Alembic chain (`control0001_init`), 7 models, Fernet-encrypted tenant creds, idempotent plan seeder, operator CLI. Zero imports into `app/`.

**MT-1 — Tenant resolution + routing. ✅ DONE.**
`app/tenancy/` package: `RoutingSession`, `resolve_tenant()`, `engine_cache`, `TenantContext`, `init_tenancy()` before_request hook + branded 404. Gated behind `MULTI_TENANT_ENABLED=false` — fully inert until flipped.

**MT-2 — Per-tenant migration runner. ✅ DONE.**
`migrations_tenant/env.py` + `control/tenant_migrate.py` (`upgrade_tenant`, `bootstrap_tenant`, `chain_head`, CLI). Guarded squashed baseline `0003_add_user_active` restores chain root. Fresh DBs use `bootstrap_tenant` (baseline → stamp head, skips unguarded phase migrations). Incremental upgrades use `upgrade_tenant` (phase33+).

> **⚠ Blocker found by MT-2 — RESOLVED.** Chain had no base; 14/30 phase migrations unguarded. Fixed by guarded squashed baseline. Fresh DB provisioning must use `bootstrap_tenant`, never raw `flask db upgrade`.

**MT-3 — Provisioning service. ✅ DONE.**
`control/provision.py`: `create_tenant()` (DB + user + schema + admin seed + domain), `register_tenant_zero()` (adopt LT DB in place), `delete_tenant()`. Fernet-encrypted creds, `provisioning_jobs` logged. CLI: `python -m control.provision`.

**MT-4 — Superadmin control panel. ✅ DONE.**
Standalone Flask app at `admin.jqc.app` (`control/panel/`). WSGI entry: `control/panel/wsgi_panel.py` → Gunicorn on port 8001 (`jqc-panel.service`). Separate Nginx server block — **must appear before the `*.jqc.app` wildcard block** or Nginx routes `admin.jqc.app` to port 8000 (main app).

Routes: tenant list/detail, plan change, suspend/resume, domain CRUD (add/verify/delete), migration status + upgrade trigger, provision new tenant, impersonation.

**Impersonation flow:**
1. Panel generates HMAC-SHA256 signed token (`PANEL_IMPERSONATE_KEY`, TTL 60 s).
2. Redirects to `https://<tenant-primary-domain>/auth/impersonate?token=<t>`.
3. Main app validates token, sets `session['impersonating_tenant_id']`.
4. Tenancy middleware reads this key and short-circuits Host resolution.
5. "End impersonation" banner clears key, redirects back to `admin.jqc.app`.

Required env vars (add to `/etc/jqc/control.env`):
```
PANEL_SECRET_KEY=<hex32>
PANEL_IMPERSONATE_KEY=<hex32>
```

**MT-5 — Plans + quota/feature gating. ✅ DONE.**
`app/tenancy/quota.py` — live counters (inspections/issues this month, total users/facilities) against tenant DB.
`app/tenancy/gates.py` — `@feature_required(key)` (hard 403) and `@quota_soft_check(axis)` (sets `g.quota_warning`, never rejects).
`app/tenancy/context.py` — `TenantContext` extended with 9 plan fields (all default to unlimited/True → single-tenant unchanged).
`app/tenancy/resolver.py` — loads `plan` in the same control session, populates `TenantContext` plan fields.
`app/templates/_quota_warning.html` — reusable upgrade-prompt banner partial.

Gated routes:

| Route | Gate |
|---|---|
| `inspections.start()` | `@quota_soft_check('inspections')` |
| `issues.create()` | `@quota_soft_check('issues')` |
| `auth.create_user()` | `@quota_soft_check('users')` |
| `facilities.create_facility()` | `@quota_soft_check('facilities')` |
| `scheduled_reports.index()` + `create()` | `@feature_required('scheduled_reports')` |
| `api.create_inspection()` | `@feature_required('mobile_api')` + `@quota_soft_check('inspections')` |
| `api.create_issue()` | `@feature_required('mobile_api')` + `@quota_soft_check('issues')` |

Decorator stack order: `@login_required` → `@role_required` → `@feature_required` → `@quota_soft_check`.

**MT-6 — Custom domain + TLS. ⚙ INFRASTRUCTURE ONLY — no Python deliverables.**
Wildcard cert `*.jqc.app` via DNS-01 challenge (certbot + DNS plugin). Custom-domain TLS via Caddy on-demand TLS. Domain-verification flow (TXT/CNAME) already in MT-7 self-service UI; superadmin marks `verified=True` in the control panel after DNS check. When Caddy is deployed, update Nginx to pass custom domains to Caddy rather than directly to port 8000.

**MT-7 — Tenant self-service. ✅ DONE.**
`app/models/tenant_settings.py` — `TenantSettings` model, one row per tenant DB, `get_or_default()` returns transient defaults when no row exists (zero migration burden for existing tenants).
`app/routes/tenant_settings.py` — blueprint at `/settings/`, `@admin_required`.
`app/templates/tenant_settings/` — `branding.html`, `plan.html`, `domains.html`.
Migration: `phase33_tenant_settings` (INFORMATION_SCHEMA guarded, safe to re-run).

Branding injection: `inject_tenant_branding()` context processor in `app/__init__.py` pushes `tenant_branding` into every template. `base.html` patches: navbar brand reads logo/name from `tenant_branding`; CSS vars `--bs-primary`, `--jqc-accent` injected via inline `<style>` from `tenant_branding.primary_color` / `accent_color`. Jinja2 filter `hex_to_rgb` registered for Bootstrap RGB var.

Self-service features:
- **Branding**: company name, logo upload (magic-byte validated), primary/accent colours, support email. Gated by `allow_branding` — shows warning but doesn't block form (soft, consistent with quota philosophy).
- **Plan & Usage**: read-only plan info + live quota progress bars (reads from control DB + tenant DB live counts). "Request upgrade" mailto link.
- **Domains**: lists all `TenantDomain` rows. Admin can request a custom domain (creates unverified row, shows TXT/CNAME DNS instructions). Delete unverified custom domains. Superadmin does final verification via control panel.

**MT-8 — Billing. ✅ DONE — flag-gated behind `BILLING_ENABLED` (default off).**
Stripe per-plan subscription, fully implemented in `app/billing/` (`routes.py`, `webhooks.py`, `emails.py`, `stripe_client.py`; blueprint registered + CSRF-exempted in `app/__init__.py`). Lifecycle: trial → active → past_due → suspended → cancelled. Billing gate (`_billing_gate()` in `app/tenancy/middleware.py`) enforces trial expiry + cancellation on every request. Dunning sequence (day 3/7/14) via cron `POST /notifications/dunning-reminders`, tracked by three `Tenant` columns (`past_due_since`, `dunning_stage`, `dunning_sent_at`; migration `control0004_dunning_tracking`). HTML + plain-text lifecycle emails, invoice history on `/settings/plan`, superadmin billing controls on the panel, and public self-service signup (`/signup`) with 14-day trial + welcome email. See CLAUDE.md §23 for the full reference.

**MT-9 — iOS multi-tenant. 🔲 PENDING (server + client both unbuilt).**
Planned server side: `GET /api/v1/discover?subdomain=acme` and `GET /api/v1/tenant` public endpoints — to be exempt from tenant middleware via `MULTI_TENANT_EXEMPT_PATHS`. **Neither endpoint exists in the code yet** — no `api_discovery` blueprint is registered. iOS side: also pending (web-first priority).

---

## 8. Tenant-zero (LT Services) migration

Register the **existing live LT database in place** — no dump/reload:

1. Insert `tenants` row (id 1), `db_*` pointing at the current LT DB + creds. Record its current alembic head.
2. Insert `tenant_domains`: primary subdomain `lts.jqc.app`; plus `jqc.ltservicesinc.com` (kind=custom, verified) so existing web users keep working.
3. Point wildcard DNS / Nginx at the (now tenant-aware) app. Resolver returns tenant 1 for both hosts.

Result: existing users notice nothing; LT is now "tenant 1".

---

## 9. Decisions resolved

1. **Quota-exceed** → soft warn (allow + flag upgrade). Never reject.
2. **Tenant DB credentials** → per-tenant MySQL user + password; creds encrypted at rest.
3. **Tenant-zero** → register existing live LT DB in place, no data move.
4. **Impersonation** → HMAC-signed token (60 s TTL), panel→tenant redirect, session key short-circuits Host resolution.
5. **Branding gate** → soft (shows warning, form still usable) — consistent with quota philosophy.

---

## 10. Nginx — block ordering (critical)

```nginx
# /etc/nginx/sites-available/jqc

# 1. PANEL — exact match, must be FIRST
server {
    listen 80;
    server_name admin.jqc.app;
    location / { proxy_pass http://127.0.0.1:8001; ... }
}

# 2. APEX — jqc.app serves the public marketing/landing page
#    (was a 301 redirect to lts.jqc.app; now proxied to the app, which the
#     tenant middleware serves the landing page for — see app/routes/landing.py)
server {
    listen 80;
    server_name jqc.app www.jqc.app;
    location / {
        proxy_pass         http://127.0.0.1:8000;
        proxy_set_header   Host              $host;   # resolver reads this
        proxy_set_header   X-Real-IP         $remote_addr;
        proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Proto $scheme;
    }
}

# 3. WILDCARD — all tenant subdomains
server {
    listen 80;
    server_name *.jqc.app;
    location / { proxy_pass http://127.0.0.1:8000; ... }
}
```

The apex block must pass `Host` through unchanged: the app's tenant middleware
compares `request.host` to `TENANT_BASE_DOMAIN` (and its `www.` variant) and
serves the landing page (`landing.index`) for `/`, bouncing any other apex path
back to `/`. `/signup`, `/welcome`, and `/static/` are tenant-exempt and served
directly. Tenant subdomains and `admin.jqc.app` are unaffected.

If `admin.jqc.app` is in the same block as `*.jqc.app`, Nginx routes it to port 8000 (main app), which returns "Workspace not found" because `admin.jqc.app` is not a registered tenant domain.

---

## 11. CLI quick-reference

```bash
# Source env first — CONTROL_DATABASE_URL not in interactive shell by default
set -a; . /etc/jqc/control.env; set +a

# Control schema + plans + first superadmin (run once)
alembic -c control/migrations/alembic.ini upgrade head
python -m control.cli seed
python -m control.cli create-superadmin --username admin --email you@example.com

# Adopt existing LT database as tenant-zero (run once, no data move)
python -m control.provision register-tenant-zero \
  --slug lts --name "LT Services" --plan enterprise \
  --db-host 127.0.0.1 --db-name <LT_DB> --db-user <LT_USER> --db-password '<pw>' \
  --custom-domain jqc.ltservicesinc.com --base-domain jqc.app

# Provision a new tenant
python -m control.provision create-tenant \
  --slug acme --name "Acme Corp" --plan pro --admin-email ops@acme.com

# Delete / deregister a tenant
python -m control.provision delete-tenant --slug ztest --drop-db --yes
python -m control.provision delete-tenant --slug lts --yes   # no --drop-db for adopted DB

# Migration status + upgrades
python -m control.tenant_migrate heads
python -m control.tenant_migrate current --tenant all
python -m control.tenant_migrate upgrade --tenant all    # incremental (phase33+)
python -m control.tenant_migrate bootstrap --tenant acme # fresh DB only

# Superadmin panel
sudo systemctl status jqc-panel
sudo systemctl restart jqc-panel
```