JQC_multi_tenant/MULTI_TENANT_PLAN.md

# Multi-Tenant Plan — JQC

> **Audience:** AI assistants and developers extending JQC into a multi-tenant SaaS.
> **Companion to:** `CLAUDE.md` (single-tenant architecture reference).
> **Status:** Planning. No code written yet. Existing single-tenant deploy stays live and untouched until cutover.

---

## 1. Decisions (locked)

| Decision | Choice |
|---|---|
| Isolation model | **Shared codebase + database-per-tenant** (one deploy, dynamic Host routing). NOT fork-per-tenant. |
| "Managed by tenant" | Tenant-admin self-serves branding / users / settings / in-plan feature toggles. No per-tenant code edits. |
| Plan-tier axes | (1) user quota, (2) facility quota, (3) inspections/month quota, (4) issues/month quota, (5) feature gates (mobile API, scheduled reports, …), (6) branding/white-label, (7) custom-domain vs subdomain-only. |
| Existing LT deploy | Migrated in as **tenant-zero** — register existing live DB in place, no data move (§8). |
| Quota-exceed behavior | **Soft warn** — allow submit, flag for upgrade. Never reject. |
| Tenant DB credentials | **Per-tenant MySQL user + password**. Provisioning creates the user/grants; creds encrypted at rest in control DB. |

Terminology: **control plane** = manages tenants (registry, plans, provisioning). **data plane** = serves tenant traffic (the existing Flask app, now tenant-aware).

---

## 2. Why this model

- **DB-per-tenant** → true data isolation; satisfies "separate database". Each tenant = own MySQL database, e.g. `jqc_<slug>`.
- **Shared code** → one codebase to patch. As a solo maintainer, N forks would mean N bug-fix deploys. Avoided.
- App already uses the **application-factory pattern** (`create_app`) and **relative `/api/v1` paths** — both ideal for adding Host-based tenant routing with minimal churn.
- Migrations already use `INFORMATION_SCHEMA` existence checks (CLAUDE.md Rule 14) → safe to re-run across every tenant DB unchanged.

---

## 3. Control-plane database (new, separate schema)

Lives in its own MySQL database (e.g. `jqc_control`), its own Alembic env (chain prefix `control{N}_…`), never routed by tenant middleware.

```
plans
  id, code (slug: free/starter/pro/enterprise), name, active,
  max_users            INT NULL,   -- NULL = unlimited
  max_facilities       INT NULL,
  max_inspections_month INT NULL,
  max_issues_month     INT NULL,
  allow_custom_domain  BOOL,
  allow_mobile_api     BOOL,
  allow_scheduled_reports BOOL,
  allow_branding       BOOL,        -- white-label
  price_cents          INT NULL,    -- future billing
  billing_period       VARCHAR(16) NULL

plan_features          -- optional EAV escape hatch for future boolean flags
  id, plan_id (FK), feature_key, enabled BOOL

tenants
  id, slug (subdomain label, UNIQUE), name, plan_id (FK),
  status ENUM(provisioning, active, suspended, deleted),
  db_host, db_port, db_name, db_user, db_password_enc,   -- creds encrypted at rest
  alembic_head VARCHAR(64),         -- last tenant-schema rev applied
  created_at, suspended_at, notes

tenant_domains
  id, tenant_id (FK), domain (UNIQUE), kind ENUM(subdomain, custom),
  is_primary BOOL, verified BOOL, verification_token,
  tls_status ENUM(pending, active, failed), created_at

superadmins                         -- cross-tenant accounts, control-plane only
  id, username, email, password_hash, active, created_at

provisioning_jobs
  id, tenant_id (FK), action ENUM(create_db, migrate, seed, suspend, delete),
  status ENUM(queued, running, ok, failed), log TEXT, created_at, finished_at

tenant_audit                        -- superadmin actions (separate from per-tenant log_action)
  id, superadmin_id, action, tenant_id, details, ip_address, created_at
```

Submission quotas (inspections/issues per month) are enforced by **live-counting current-period rows in the tenant DB** at submit time (cheap with an index on the date column). No separate counter table required; always accurate. A cached `usage_counters` table is an optional later optimization if submit-path counts ever show up in profiling.

---

## 4. Tenant resolution + DB routing (the core mechanism)

**Resolution** (`before_request`): `host = request.host.split(':')[0]` → look up `tenant_domains.domain == host` where `verified AND tenant.status='active'` → load `tenant` + `plan` into `g`. Unknown/unverified/suspended host → branded error/landing page (no tenant DB touched).

**Routing without rewriting every `db.session` call** — use a routing Session so the entire existing codebase keeps using `db.session` unchanged (preserves Change-Philosophy Rules 5/7):

```python
# app/tenancy/routing.py
from sqlalchemy.orm import Session
from flask import g

class RoutingSession(Session):
    def get_bind(self, mapper=None, clause=None, **kw):
        # Tenant models (app.db.Model) bind to the per-request tenant engine.
        # The control plane is a SEPARATE Base/engine/session (control/base.py),
        # so it never passes through this session — no special-casing needed here.
        return getattr(g, 'tenant_engine', None) or _default_engine
```

> **MT-0 refinement (implemented):** the control plane uses its own
> `ControlBase` + engine + session (`control/base.py`), **not** `db.Model` with
> a `__control_plane__` flag. Stronger isolation, zero metadata mixing with the
> tenant schema, and the control package imports nothing from `app/`. The
> `RoutingSession` (MT-1) therefore only handles tenant models.

```python
# app/__init__.py  (one-line change to the existing db init)
db = SQLAlchemy(session_options={'class_': RoutingSession})
```

```python
# before_request
g.tenant_engine = _engine_cache.get_or_create(tenant)   # create_engine(tenant.db_uri)
```

- `_engine_cache`: `dict[tenant_id -> Engine]`, lazily built. Use small pools + `pool_recycle`; switch to `NullPool` if tenant count grows large (connection-count = Σ tenant pools). Documented scaling lever.
- The **control plane is a separate Base/engine/session** (`control/base.py`); it never routes through `RoutingSession`. See §3 MT-0 refinement.
- **Net code change to existing models/routes: zero.** They keep importing `from app import db` and calling `db.session`.

---

## 5. Migrations

Two independent Alembic chains:

1. **Tenant schema** — the existing chain (current HEAD on disk: **`phase27_score_alerts`**; note `CLAUDE.md` text says phase23 — stale). Runs per-tenant DB. New tenant features continue this chain as `phase28_…`, `phase29_…` per existing naming + `down_revision` → current HEAD rule.
2. **Control schema** — new env, chain `control{N}_…`, runs once against `jqc_control`.

CLI added: `flask tenant db upgrade --tenant <id|all>` loops tenants, sets `sqlalchemy.url` to each tenant URI, runs `command.upgrade(cfg, 'head')`, writes resulting head back to `tenants.alembic_head`. All existing migrations are re-run-safe (Rule 14), so this is idempotent across the fleet.

Deploy ordering rule carries over (CLAUDE.md Rule 23): run tenant migrations **before** shipping app code that references new columns.

---

## 6. Plan-tier matrix (starting point — tune in §9)

| Axis | Free | Starter | Pro | Enterprise |
|---|---|---|---|---|
| Max users | 3 | 15 | 50 | unlimited |
| Max facilities | 2 | 10 | 50 | unlimited |
| Inspections / month | 50 | 500 | 5 000 | unlimited |
| Issues / month | 50 | 500 | 5 000 | unlimited |
| Mobile API (iPad app) | ✗ | ✓ | ✓ | ✓ |
| Scheduled reports | ✗ | ✗ | ✓ | ✓ |
| Branding / white-label | ✗ | ✗ | ✓ | ✓ |
| Custom domain | ✗ | ✗ | ✓ | ✓ |
| Subdomain (`*.jqc.app`) | ✓ | ✓ | ✓ | ✓ |

Enforcement: `@feature_required('mobile_api')` reads `g.tenant.plan` and **blocks** disabled features. Quota checks (`@quota_check('inspections')`) are **soft** — over-limit submits still succeed but the response/UI flags an upgrade prompt and the event is recorded; submits are never rejected. **Feature + quota checks must live in BOTH web routes and `/api/v1` endpoints** — the iPad submits via API, so a web-only check is bypassable (mirrors existing dual-enforcement Rules 39/56/57).

---

## 7. Phased roadmap

Each phase additive; existing tenant-zero traffic keeps working throughout.

**MT-0 — Control-plane scaffold. ✅ DONE.** Self-contained `control/` package at repo root: own `ControlBase` + engine + session, own Alembic chain (`control0001_init`), 7 models, Fernet-encrypted tenant creds, idempotent plan seeder, operator CLI. Zero imports into `app/` — existing app untouched.

**MT-1 — Tenant resolution + routing. ✅ DONE.** `app/tenancy/` package: `routing.py` (`RoutingSession` subclassing the Flask-SQLAlchemy session), `resolver.py` (Host→tenant via control plane, lazy import), `engine_cache.py` (per-tenant engines), `context.py` (`TenantContext`), `middleware.py` (`init_tenancy` before_request hook + branded unknown-host 404). Edits: `db` init in `app/__init__.py` (+`init_tenancy(app)` call) and `MULTI_TENANT_ENABLED` + pool flags in `config.py`. Gated behind `MULTI_TENANT_ENABLED` (default False) — fully inert until flipped.

**MT-2 — Per-tenant migration runner. ✅ DONE (with blocker found).** Standalone `migrations_tenant/env.py` (reads URL from config, reuses `migrations/versions` via `version_locations`, no Flask) + `control/tenant_migrate.py` (`upgrade_tenant()`, `current_revision()`, `chain_head()`, CLI: `python -m control.tenant_migrate upgrade|current|heads --tenant <id|slug|all>`). Records `tenants.alembic_head` + a `ProvisioningJob('migrate')` per run. Existing `migrations/env.py` untouched (normal `flask db` still works).

> **⚠ Blocker found by MT-2 — RESOLVED.** The live `migrations/versions` chain had **no base**: `phase1_projects_roles.down_revision = '0003_add_user_active'` pointed at a missing revision, and 14/30 phase migrations are not idempotent. **Fix:** a guarded squashed baseline `migrations/versions/0003_add_user_active.py` (`down_revision = None`) that recreates the full current schema (generated from the models, 25 tables, INFORMATION_SCHEMA-guarded) — restoring the chain root. Fresh tenants are built via `bootstrap_tenant()` = upgrade to the baseline **then `stamp head`**, so the non-idempotent phase migrations are never replayed. Existing DBs (LT) are at a later head, so the baseline is treated as applied ancestry and never runs. Ongoing migrations (phase33+, which MUST be guarded) apply incrementally to all tenants via `upgrade_tenant()`.
>
> **Operational note:** any *fresh* database (including a new dev DB) must use the bootstrap flow, not a naive `flask db upgrade` from empty, because the historical phase replay still hits the unguarded migrations. Cross-check the baseline against `mysqldump --no-data` of LT and test `bootstrap` on a scratch MySQL before going live.

**MT-3 — Provisioning service. ✅ DONE.** `control/provision.py`: `create_tenant()` (admin account from `PROVISION_DB_URL` creates per-tenant DB + least-privilege user + grant → `bootstrap_tenant` builds schema + stamps head → `seed_admin` inserts the first admin with a set-password token → registers subdomain + optional custom domain → status active; best-effort rollback drops the DB/user + tenant row on failure). `register_tenant_zero()` adopts an existing DB in place (no DB/user creation, no bootstrap; reads its current head, maps `lts.<base>` + LT's live domain as a verified custom domain — zero data movement). `delete_tenant()` removes control records, optional guarded DB drop (refuses non-provisioner-named DBs, protecting tenant-zero). CLI: `python -m control.provision create-tenant|register-tenant-zero|delete-tenant`. Admin setup link uses `/customers/set-password/<token>` (the route that sets `password_set=True`). Creds Fernet-encrypted; `provisioning_jobs`-logged.

**MT-4 — Superadmin control panel.** Tenant CRUD, plan assign, suspend/resume, domain mgmt, migration status, impersonation (scoped login into a tenant for support). At `admin.jqc.app`, separate blueprint, superadmin-gated.

**MT-5 — Plans + quota/feature gating.** `@feature_required`, `@quota_check`; new cross-tenant `superadmin` (control plane). Existing tenant `admin` = top role within a tenant (unchanged ENUM).

**MT-6 — Custom domain + TLS.** Wildcard cert for `*.jqc.app`; Caddy on-demand TLS in front for arbitrary custom domains. Domain-verification flow (TXT/CNAME) before activation.

**MT-7 — Tenant self-service.** Tenant-admin: branding (logo/name/colors), domain request, plan view, in-plan feature toggles. User mgmt auto tenant-scoped by routing.

**MT-8 — Billing (future).** Stripe per-plan; lifecycle (trial/suspend/dunning). Flag-gated.

**MT-9 — iOS multi-tenant.** Replace hardcoded `ServerOption` enum (`Utils/Constants.swift`) with onboarding: enter subdomain or work-email → discovery endpoint resolves base URL → store per-tenant in Keychain. All `/api/v1` calls unchanged (server resolves tenant by Host). Tenant-zero keeps `jqc.ltservicesinc.com` as a custom domain so existing builds keep working through rollout.

---

## 8. Tenant-zero (LT Services) migration

Register the **existing live LT database in place** — no dump/reload:

1. Insert `tenants` row (id 1), `db_*` pointing at the current LT DB + creds. Record its current alembic head.
2. Insert `tenant_domains`: primary subdomain `lts.jqc.app`; plus `jqc.ltservicesinc.com` (kind=custom, verified) so existing web users **and** existing iPad builds keep working with zero disruption.
3. Point wildcard DNS / Caddy at the (now tenant-aware) app. Resolver returns tenant 1 for both hosts.

Result: existing users notice nothing; LT is now just "tenant 1" inside the new model.

---

## 9. Decisions resolved — MT-0 unblocked

1. **Quota-exceed** → soft warn (allow + flag upgrade). Never reject. §6.
2. **Tenant DB credentials** → per-tenant MySQL user + password; provisioning creates user/grants scoped to the single tenant DB; creds encrypted at rest. §3, MT-3.
3. **Tenant-zero** → register existing live LT DB in place, no data move. §8.

All planning decisions locked. Ready to begin **MT-0** (control-plane scaffold) on request.

### MT-0 first deliverables (preview)
- `jqc_control` database + control Alembic env (`control0001_init`).
- Control models on a dedicated `ControlBase`: `Plan`, `PlanFeature`, `Tenant`, `TenantDomain`, `Superadmin`, `ProvisioningJob`, `TenantAudit`.
- Plan seeder (Free / Starter / Pro / Enterprise per §6 matrix).
- No data-plane change in MT-0 — existing app keeps running against the LT DB exactly as today.