Files

T

nngo 38ab66d021 Aug 7 - Update: documents, no code update

2026-08-07 17:13:08 -04:00

26 KiB

Raw Blame History

Multi-Tenant Plan — JQC

Audience: AI assistants and developers extending JQC into a multi-tenant SaaS. Companion to: CLAUDE.md (single-tenant architecture reference). Status: MT-0 through MT-8 complete and deployed (MT-8 billing is flag-gated behind BILLING_ENABLED, default off). MT-9 (iOS multi-tenant) is fully pending — both the server-side discovery endpoints and the iOS client are unbuilt.

Feature parity with the single-tenant tree (MT-10 → MT-17): complete. MT forked from ST before ST kept shipping, and that gap has now been closed phase by phase — see §12. The only deliberate divergence is ST's phase50_default_modern, which MT does not adopt (§12.3). Tenant migration head: phase52_user_ui_theme.

1. Decisions (locked)

Decision	Choice
Isolation model	Shared codebase + database-per-tenant (one deploy, dynamic Host routing). NOT fork-per-tenant.
"Managed by tenant"	Tenant-admin self-serves branding / users / settings / in-plan feature toggles. No per-tenant code edits.
Plan-tier axes	(1) user quota, (2) facility quota, (3) inspections/month quota, (4) issues/month quota, (5) feature gates (mobile API, scheduled reports, …), (6) branding/white-label, (7) custom-domain vs subdomain-only.
Existing LT deploy	Migrated in as tenant-zero — register existing live DB in place, no data move (§8).
Quota-exceed behavior	Soft warn — allow submit, flag for upgrade. Never reject.
Tenant DB credentials	Per-tenant MySQL user + password. Provisioning creates the user/grants; creds encrypted at rest in control DB.

Terminology: control plane = manages tenants (registry, plans, provisioning). data plane = serves tenant traffic (the existing Flask app, now tenant-aware).

2. Why this model

DB-per-tenant → true data isolation; satisfies "separate database". Each tenant = own MySQL database, e.g. jqc_<slug>.
Shared code → one codebase to patch. As a solo maintainer, N forks would mean N bug-fix deploys. Avoided.
App already uses the application-factory pattern (create_app) and relative /api/v1 paths — both ideal for adding Host-based tenant routing with minimal churn.
Migrations already use INFORMATION_SCHEMA existence checks (CLAUDE.md Rule 14) → safe to re-run across every tenant DB unchanged.

3. Control-plane database (new, separate schema)

Lives in its own MySQL database (e.g. jqc_control), its own Alembic env (chain prefix control{N}_…), never routed by tenant middleware.

plans
  id, code (slug: free/starter/pro/enterprise), name, active,
  max_users            INT NULL,   -- NULL = unlimited
  max_facilities       INT NULL,
  max_inspections_month INT NULL,
  max_issues_month     INT NULL,
  allow_custom_domain  BOOL,
  allow_mobile_api     BOOL,
  allow_scheduled_reports BOOL,
  allow_branding       BOOL,        -- white-label
  price_cents          INT NULL,    -- future billing
  billing_period       VARCHAR(16) NULL

plan_features          -- optional EAV escape hatch for future boolean flags
  id, plan_id (FK), feature_key, enabled BOOL

tenants
  id, slug (subdomain label, UNIQUE), name, plan_id (FK),
  status ENUM(provisioning, active, suspended, deleted),
  db_host, db_port, db_name, db_user, db_password_enc,   -- creds encrypted at rest
  alembic_head VARCHAR(64),         -- last tenant-schema rev applied
  created_at, suspended_at, notes

tenant_domains
  id, tenant_id (FK), domain (UNIQUE), kind ENUM(subdomain, custom),
  is_primary BOOL, verified BOOL, verification_token,
  tls_status ENUM(pending, active, failed), created_at

superadmins                         -- cross-tenant accounts, control-plane only
  id, username, email, password_hash, active, created_at,
  mfa_enabled, mfa_secret, mfa_recovery_codes   -- control0005: opt-in TOTP 2FA

provisioning_jobs
  id, tenant_id (FK), action ENUM(create_db, migrate, seed, suspend, delete),
  status ENUM(queued, running, ok, failed), log TEXT, created_at, finished_at

tenant_audit                        -- superadmin actions (separate from per-tenant log_action)
  id, superadmin_id, action, tenant_id, details, ip_address, created_at

Submission quotas (inspections/issues per month) are enforced by live-counting current-period rows in the tenant DB at submit time (cheap with an index on the date column). No separate counter table required; always accurate.

4. Tenant resolution + DB routing (the core mechanism)

Resolution (before_request): host = request.host.split(':')[0] → look up tenant_domains.domain == host where verified AND tenant.status='active' → load tenant + plan into g. Unknown/unverified/suspended host → branded error/landing page (no tenant DB touched).

Routing without rewriting every db.session call — use a routing Session so the entire existing codebase keeps using db.session unchanged:

# app/tenancy/routing.py
from sqlalchemy.orm import Session
from flask import g

class RoutingSession(Session):
    def get_bind(self, mapper=None, clause=None, **kw):
        return getattr(g, 'tenant_engine', None) or _default_engine

MT-0 refinement (implemented): the control plane uses its own ControlBase + engine + session (control/base.py), not db.Model with a __control_plane__ flag. The RoutingSession (MT-1) therefore only handles tenant models.

# app/__init__.py  (one-line change to the existing db init)
db = SQLAlchemy(session_options={'class_': RoutingSession})

_engine_cache: dict[tenant_id -> Engine], lazily built, small pools + pool_recycle.
Net code change to existing models/routes: zero.

MT-5 addition: Plan limits and feature flags are loaded into TenantContext by the resolver in the same control-DB session — zero extra queries per request. g.tenant.allow_mobile_api, g.tenant.max_users, etc. are available everywhere.

5. Migrations

Two independent Alembic chains:

Tenant schema — existing chain (HEAD: phase34_inspection_schedules). Runs per-tenant DB. New tenant features continue as phase35_… per existing naming.
Control schema — chain control{N}_…, runs once against jqc_control. HEAD: control0005_superadmin_mfa (control0001_init → control0002_billing → control0003_trial_reminder_sent → control0004_dunning_tracking → control0005_superadmin_mfa).

CLI (always source env first):

set -a; . /etc/jqc/control.env; set +a
python -m control.tenant_migrate upgrade --tenant all

Deploy ordering rule: run tenant migrations before shipping app code that references new columns.

6. Plan-tier matrix

Axis	Free	Starter	Pro	Enterprise
Max users	3	15	50	unlimited
Max facilities	2	10	50	unlimited
Inspections / month	50	500	5 000	unlimited
Issues / month	50	500	5 000	unlimited
Mobile API (iPad app)	✗	✓	✓	✓
Scheduled reports	✗	✗	✓	✓
Branding / white-label	✗	✗	✓	✓
Custom domain	✗	✗	✓	✓
Subdomain (`*.jqc.app`)	✓	✓	✓	✓

Enforcement: @feature_required('mobile_api') hard-blocks (403) disabled features. @quota_soft_check('inspections') sets g.quota_warning but never rejects — over-limit submits succeed, UI shows upgrade prompt. Both decorators are inert when MULTI_TENANT_ENABLED=false.

Feature + quota checks live in both web routes and /api/v1 endpoints (iPad submits via API — web-only check is bypassable).

7. Phased roadmap

MT-0 — Control-plane scaffold. ✅ DONE. Self-contained control/ package: own ControlBase + engine + session, own Alembic chain (control0001_init), 7 models, Fernet-encrypted tenant creds, idempotent plan seeder, operator CLI. Zero imports into app/.

MT-1 — Tenant resolution + routing. ✅ DONE. app/tenancy/ package: RoutingSession, resolve_tenant(), engine_cache, TenantContext, init_tenancy() before_request hook + branded 404. Gated behind MULTI_TENANT_ENABLED=false — fully inert until flipped.

MT-2 — Per-tenant migration runner. ✅ DONE. migrations_tenant/env.py + control/tenant_migrate.py (upgrade_tenant, bootstrap_tenant, chain_head, CLI). Guarded squashed baseline 0003_add_user_active restores chain root. Fresh DBs use bootstrap_tenant (baseline → stamp head, skips unguarded phase migrations). Incremental upgrades use upgrade_tenant (phase33+).

⚠ Blocker found by MT-2 — RESOLVED. Chain had no base; 14/30 phase migrations unguarded. Fixed by guarded squashed baseline. Fresh DB provisioning must use bootstrap_tenant, never raw flask db upgrade.

MT-3 — Provisioning service. ✅ DONE. control/provision.py: create_tenant() (DB + user + schema + admin seed + domain), register_tenant_zero() (adopt LT DB in place), delete_tenant(). Fernet-encrypted creds, provisioning_jobs logged. CLI: python -m control.provision.

MT-4 — Superadmin control panel. ✅ DONE. Standalone Flask app at admin.jqc.app (control/panel/). WSGI entry: control/panel/wsgi_panel.py → Gunicorn on port 8001 (jqc-panel.service). Separate Nginx server block — must appear before the *.jqc.app wildcard block or Nginx routes admin.jqc.app to port 8000 (main app).

Routes: tenant list/detail, plan change, suspend/resume, domain CRUD (add/verify/delete), migration status + upgrade trigger, provision new tenant, impersonation.

Impersonation flow:

Panel generates HMAC-SHA256 signed token (PANEL_IMPERSONATE_KEY, TTL 60 s).
Redirects to https://<tenant-primary-domain>/auth/impersonate?token=<t>.
Main app validates token, sets session['impersonating_tenant_id'].
Tenancy middleware reads this key and short-circuits Host resolution.
"End impersonation" banner clears key, redirects back to admin.jqc.app.

Required env vars (add to /etc/jqc/control.env):

PANEL_SECRET_KEY=<hex32>
PANEL_IMPERSONATE_KEY=<hex32>

MT-5 — Plans + quota/feature gating. ✅ DONE. app/tenancy/quota.py — live counters (inspections/issues this month, total users/facilities) against tenant DB. app/tenancy/gates.py — @feature_required(key) (hard 403) and @quota_soft_check(axis) (sets g.quota_warning, never rejects). app/tenancy/context.py — TenantContext extended with 9 plan fields (all default to unlimited/True → single-tenant unchanged). app/tenancy/resolver.py — loads plan in the same control session, populates TenantContext plan fields. app/templates/_quota_warning.html — reusable upgrade-prompt banner partial.

Gated routes:

Route	Gate
`inspections.start()`	`@quota_soft_check('inspections')`
`issues.create()`	`@quota_soft_check('issues')`
`auth.create_user()`	`@quota_soft_check('users')`
`facilities.create_facility()`	`@quota_soft_check('facilities')`
`scheduled_reports.index()` + `create()`	`@feature_required('scheduled_reports')`
`api.create_inspection()`	`@feature_required('mobile_api')` + `@quota_soft_check('inspections')`
`api.create_issue()`	`@feature_required('mobile_api')` + `@quota_soft_check('issues')`

Decorator stack order: @login_required → @role_required → @feature_required → @quota_soft_check.

MT-6 — Custom domain + TLS. ⚙ INFRASTRUCTURE ONLY — no Python deliverables. Wildcard cert *.jqc.app via DNS-01 challenge (certbot + DNS plugin). Custom-domain TLS via Caddy on-demand TLS. Domain-verification flow (TXT/CNAME) already in MT-7 self-service UI; superadmin marks verified=True in the control panel after DNS check. When Caddy is deployed, update Nginx to pass custom domains to Caddy rather than directly to port 8000.

MT-7 — Tenant self-service. ✅ DONE. app/models/tenant_settings.py — TenantSettings model, one row per tenant DB, get_or_default() returns transient defaults when no row exists (zero migration burden for existing tenants). app/routes/tenant_settings.py — blueprint at /settings/, @admin_required. app/templates/tenant_settings/ — branding.html, plan.html, domains.html. Migration: phase33_tenant_settings (INFORMATION_SCHEMA guarded, safe to re-run).

Branding injection: inject_tenant_branding() context processor in app/__init__.py pushes tenant_branding into every template. base.html patches: navbar brand reads logo/name from tenant_branding; CSS vars --bs-primary, --jqc-accent injected via inline <style> from tenant_branding.primary_color / accent_color. Jinja2 filter hex_to_rgb registered for Bootstrap RGB var.

Self-service features:

Branding: company name, logo upload (magic-byte validated), primary/accent colours, support email. Gated by allow_branding — shows warning but doesn't block form (soft, consistent with quota philosophy).
Plan & Usage: read-only plan info + live quota progress bars (reads from control DB + tenant DB live counts). "Request upgrade" mailto link.
Domains: lists all TenantDomain rows. Admin can request a custom domain (creates unverified row, shows TXT/CNAME DNS instructions). Delete unverified custom domains. Superadmin does final verification via control panel.

MT-8 — Billing. ✅ DONE — flag-gated behind BILLING_ENABLED (default off). Stripe per-plan subscription, fully implemented in app/billing/ (routes.py, webhooks.py, emails.py, stripe_client.py; blueprint registered + CSRF-exempted in app/__init__.py). Lifecycle: trial → active → past_due → suspended → cancelled. Billing gate (_billing_gate() in app/tenancy/middleware.py) enforces trial expiry + cancellation on every request. Dunning sequence (day 3/7/14) via cron POST /notifications/dunning-reminders, tracked by three Tenant columns (past_due_since, dunning_stage, dunning_sent_at; migration control0004_dunning_tracking). HTML + plain-text lifecycle emails, invoice history on /settings/plan, superadmin billing controls on the panel, and public self-service signup (/signup) with 14-day trial + welcome email. See CLAUDE.md §23 for the full reference.

MT-9 — iOS multi-tenant. 🔲 PENDING (server + client both unbuilt). Planned server side: GET /api/v1/discover?subdomain=acme and GET /api/v1/tenant public endpoints — to be exempt from tenant middleware via MULTI_TENANT_EXEMPT_PATHS. Neither endpoint exists in the code yet — no api_discovery blueprint is registered. iOS side: also pending (web-first priority).

8. Tenant-zero (LT Services) migration

Insert tenants row (id 1), db_* pointing at the current LT DB + creds. Record its current alembic head.
Insert tenant_domains: primary subdomain lts.jqc.app; plus jqc.ltservicesinc.com (kind=custom, verified) so existing web users keep working.
Point wildcard DNS / Nginx at the (now tenant-aware) app. Resolver returns tenant 1 for both hosts.

Result: existing users notice nothing; LT is now "tenant 1".

9. Decisions resolved

Quota-exceed → soft warn (allow + flag upgrade). Never reject.
Tenant DB credentials → per-tenant MySQL user + password; creds encrypted at rest.
Tenant-zero → register existing live LT DB in place, no data move.
Impersonation → HMAC-signed token (60 s TTL), panel→tenant redirect, session key short-circuits Host resolution.
Branding gate → soft (shows warning, form still usable) — consistent with quota philosophy.

10. Nginx — block ordering (critical)

# /etc/nginx/sites-available/jqc

# 1. PANEL — exact match, must be FIRST
server {
    listen 80;
    server_name admin.jqc.app;
    location / { proxy_pass http://127.0.0.1:8001; ... }
}

# 2. APEX — jqc.app serves the public marketing/landing page
#    (was a 301 redirect to lts.jqc.app; now proxied to the app, which the
#     tenant middleware serves the landing page for — see app/routes/landing.py)
server {
    listen 80;
    server_name jqc.app www.jqc.app;
    location / {
        proxy_pass         http://127.0.0.1:8000;
        proxy_set_header   Host              $host;   # resolver reads this
        proxy_set_header   X-Real-IP         $remote_addr;
        proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Proto $scheme;
    }
}

# 3. WILDCARD — all tenant subdomains
server {
    listen 80;
    server_name *.jqc.app;
    location / { proxy_pass http://127.0.0.1:8000; ... }
}

The apex block must pass Host through unchanged: the app's tenant middleware compares request.host to TENANT_BASE_DOMAIN (and its www. variant) and serves the landing page (landing.index) for /, bouncing any other apex path back to /. /signup, /welcome, and /static/ are tenant-exempt and served directly. Tenant subdomains and admin.jqc.app are unaffected.

If admin.jqc.app is in the same block as *.jqc.app, Nginx routes it to port 8000 (main app), which returns "Workspace not found" because admin.jqc.app is not a registered tenant domain.

11. CLI quick-reference

# Source env first — CONTROL_DATABASE_URL not in interactive shell by default
set -a; . /etc/jqc/control.env; set +a

# Control schema + plans + first superadmin (run once)
alembic -c control/migrations/alembic.ini upgrade head
python -m control.cli seed
python -m control.cli create-superadmin --username admin --email you@example.com

# Adopt existing LT database as tenant-zero (run once, no data move)
python -m control.provision register-tenant-zero \
  --slug lts --name "LT Services" --plan enterprise \
  --db-host 127.0.0.1 --db-name <LT_DB> --db-user <LT_USER> --db-password '<pw>' \
  --custom-domain jqc.ltservicesinc.com --base-domain jqc.app

# Provision a new tenant
python -m control.provision create-tenant \
  --slug acme --name "Acme Corp" --plan pro --admin-email ops@acme.com

# Delete / deregister a tenant
python -m control.provision delete-tenant --slug ztest --drop-db --yes
python -m control.provision delete-tenant --slug lts --yes   # no --drop-db for adopted DB

# Migration status + upgrades
python -m control.tenant_migrate heads
python -m control.tenant_migrate current --tenant all
python -m control.tenant_migrate upgrade --tenant all    # incremental (phase33+)
python -m control.tenant_migrate bootstrap --tenant acme # fresh DB only

# Superadmin panel
sudo systemctl status jqc-panel
sudo systemctl restart jqc-panel

12. Feature parity with the single-tenant tree (MT-10 → MT-17)

MT forked from the single-tenant codebase (LT_Janitorial_Quality_Control, "ST") before ST continued shipping features. This section records how that gap was closed. It is complete as of MT-17.

12.1 Why the two trees look more different than they are

A file-by-file comparison of the two trees overstates the gap. Several ST files have no MT counterpart by name while the feature is fully present under MT's own naming. These are not gaps and must not be "fixed":

ST	MT equivalent
`models/scheduled_inspection.py`	`models/inspection_schedule.py`
`routes/scheduled_inspections.py`	`routes/inspection_schedules.py`
`routes/public.py`	`routes/facility_qr.py`
`models/notification_recipient.py`	`models/project_recipient.py`
`ContractNotificationRecipient` / `get_event_types()`	`ProjectNotificationRecipient` / `get_events()`
`app/add_form_schema.py`	`scripts/add_form_schema.py`
`support/admin_conversation_detail.html`	`support/conversation_detail.html`

There is also one place where MT is ahead of ST: utils/forms.py strong_password() (length + complexity + common-password blocklist) versus ST's bare Length(min=6). Porting ST's version would be a downgrade.

12.2 Phase log

Migrations phase41 → phase52 in migrations/versions/ are the parity track: auditor role, area QR tokens, schedule plan fields, internal handler, frequency ENUM widening, recurrence, end date, parent inspection, follow-up attribution, schedule acknowledgement, and then:

MT-15 — External Inspector role. ✅ DONE (phase51_external_inspector) Adds external_inspector to the users.role ENUM: an inspector employed by the customer or a third party, with identical capabilities to inspector and scoped the same way through InspectorAssignment.

The ENUM widening and the code must ship together. MT had ~80 sites testing role == 'inspector' with a literal comparison; widening the ENUM alone would make every one of them evaluate False for the new role and fall through to the unscoped branch — get_inspector_scope() returns None, downstream queries drop their facility filter, and an inspector employed by one customer sees every other customer's contracts. User.INSPECTOR_ROLES (exposed as the is_inspector property) is now the single definition, and test_external_inspector_scope_is_not_unrestricted fails loudly if anyone reverts a membership test to a literal.

Also in this phase: external inspectors are invited, never given a password (password_set=False + emailed 72-hour token, reusing the customer invite mail), with a new auth.resend_invite route so a bounced invitation cannot brick an account permanently. Creating any other role with a blank password is now rejected — it previously stored the hash of the empty string.

MT-16 — Modern web portal design. ✅ DONE (phase52_user_ui_theme) Ports ST's phase48. base.html became a one-line dispatcher ({% extends jqc_layout %}); the old chrome moved verbatim to layouts/classic.html; layouts/modern.html is the sidebar shell. All existing page templates needed zero edits — Jinja resolves {% block %} overrides through the whole inheritance chain.

ThemedEnvironment.get_template() swaps x.html → modern/x.html for modern users. The swap is in get_template(), not the loader, on purpose: Jinja's template cache is keyed on the name get_template() receives, so a cached modern template can never be served to a classic user — and in MT, where one Gunicorn worker serves many tenants, a loader-level swap would leak across tenants too.

MT-specific adaptations that ST's files required: tenant branding (ST hardcodes its own company name), inspection_schedules for ST's scheduled_inspections, facilities.qr_print_all for ST's facility_qr_print_all, the billing banner, role_label for MT-15's new role, and a rewritten tenant-neutral About page. _quota_warning.html is deliberately not in the modern layout — it is a per-form include, not chrome, and would render twice on four pages.

MT-17 — Enrollment intake form. ✅ DONE (no migration) Ports ST's app/enrollment/ — a public, login-free intake form plus an admin-only inbox, kept deliberately outside the schema (flat JSON, no model, no migration, deletable package).

Tenant isolation was the change ST's version required. ST keeps every submission in one flat directory; in MT that directory is shared by every tenant on the host, so /enrollment/admin would list other organisations' submissions. Submissions are now filed under <ENROLLMENT_DIR>/t<tenant_id>/, mirroring storage.tenant_key_prefix(). When multi-tenancy is on and no tenant is bound, storage.enrollment_dir() raises TenantUnresolved rather than falling back to the root — a fallback would be a silent cross-tenant leak; an exception is loud and safe.

Branding was the second change: ST hardcodes its company name in four places and a personal Gmail address as the customer-facing "corrections" contact. Both now resolve from TenantSettings, with a test that greps the package so they cannot silently return.

12.3 Deliberate divergence: ST `phase50_default_modern` is NOT ported

ST's phase50 flips the ui_theme column default to modern and runs:

UPDATE users SET ui_theme = 'modern' WHERE ui_theme = 'classic';

That overwrites every saved preference. It was defensible for a single-tenant deployment deciding for its own staff after its own A/B test.

It is not portable to MT. The same statement runs against every tenant database, flipping the entire UI for tenants who never saw the test and never asked. MT therefore ships the phase48 semantics only: default classic, no backfill of any kind.

The effective default for accounts that never chose is config DEFAULT_UI_THEME (app/__init__.py::resolve_ui_theme), which reads the environment and itself defaults to classic. A stored users.ui_theme always wins. To put a tenant on the modern design, set DEFAULT_UI_THEME=modern in that tenant's process environment — a config change, reversible, with no preferences destroyed. test_new_user_defaults_to_classic pins this so a future port of phase50 has to be a deliberate act.

12.4 Open decisions

Seat quota. tenancy/quota.py::count_active_users() counts all active users regardless of role, so external inspectors consume a seat against max_users. Intentional (they are real accounts), but tenants near their cap will hit @quota_soft_check('users') when inviting third parties. Excluding them is a billing-policy decision, not a bug fix.
enrollment/schema.py::CORRECTIONS_EMAIL is now an empty last-resort default. Decide whether to drop the constant and its two config fallbacks in favour of requiring TenantSettings.support_email.

12.5 Deferred, with reasons

_handler_split dashboard cards. ST's classic dashboard shows handler breakdowns for opened today and unassigned as well as open issues. MT supplies handler_breakdown (open issues) and both MT dashboards render it; the other two would require converting .count() queries to .all() and fetching full rows for a cosmetic card, which regresses large tenants. If wanted, do it as a SQL GROUP BY handler_type rather than ST's Python-side count over fetched rows.
PDF / audit hardening.
MT-9 iOS client (see §7).

26 KiB Raw Blame History