nngo/JQC_multi_tenant
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Jun 27 Update documents

Browse Source
This commit is contained in:
nngo
2026-06-27 10:28:57 -04:00
parent 9f0f7d2f18
commit 7b764c76af
3 changed files with 466 additions and 61 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CLAUDE.md
+229 -20
View File
@@ -2,7 +2,7 @@
> **Audience:** AI assistants and developers working on this codebase. > **Audience:** AI assistants and developers working on this codebase.
> **Purpose:** Authoritative reference for architecture, conventions, gotchas, and decisions. > **Purpose:** Authoritative reference for architecture, conventions, gotchas, and decisions.
> **Last reviewed:** June 2026 (Phase 19 complete + mobile API gap-fill Phases A–E + customer UI refinements + Phase 22 comment visibility + Phase 23 support chat/tickets + inspector performance Excel export + inspection list filters + customer issue logging + AI chatbot + dashboard grouped sections + issues/inspections PDF export + date/ID filters + Reports expansion Phases R1–R4 + Phase 24 issue_created notify defaults + Phase 25 inspection GPS + Phase 26 issue vendor fields + Phase 27 facility score alerts) > **Last reviewed:** June 2026 (Phase 19 complete + mobile API gap-fill Phases A–E + customer UI refinements + Phase 22 comment visibility + Phase 23 support chat/tickets + inspector performance Excel export + inspection list filters + customer issue logging + AI chatbot + dashboard grouped sections + issues/inspections PDF export + date/ID filters + Reports expansion Phases R1–R4 + Phase 24 issue_created notify defaults + Phase 25 inspection GPS + Phase 26 issue vendor fields + Phase 27 facility score alerts + **MT-0 through MT-3 multi-tenant control plane**)
--- ---
@@ -29,6 +29,7 @@
19. [Infrastructure](#19-infrastructure) 19. [Infrastructure](#19-infrastructure)
20. [Known Constraints & Hard Rules](#20-known-constraints--hard-rules) 20. [Known Constraints & Hard Rules](#20-known-constraints--hard-rules)
21. [Change Philosophy](#21-change-philosophy) 21. [Change Philosophy](#21-change-philosophy)
22. [Multi-Tenant Architecture (MT-0 → MT-3)](#22-multi-tenant-architecture-mt-0--mt-3)
--- ---
@@ -82,7 +83,14 @@ The application is actively deployed in production and maintained by a single de
``` ```
lt_janitorial_quality_control/ lt_janitorial_quality_control/
├── app/ ├── app/
│ ├── __init__.py # Application factory — limiter, csrf, db, mail, login_manager │ ├── __init__.py # Application factory — limiter, csrf, db (RoutingSession), mail, login_manager, init_tenancy
│ ├── tenancy/ # MT-1 — tenant resolution + DB routing (inert unless MULTI_TENANT_ENABLED=true)
│ │ ├── __init__.py # public exports: RoutingSession, init_tenancy, TenantContext
│ │ ├── context.py # TenantContext frozen dataclass (g.tenant)
│ │ ├── engine_cache.py # per-tenant SQLAlchemy engine cache + invalidate()
│ │ ├── middleware.py # init_tenancy() — before_request Host→tenant resolver
│ │ ├── resolver.py # resolve_tenant(host) → TenantContext | None
│ │ └── routing.py # RoutingSession — routes db.session to g.tenant_engine
│ ├── api/ # Mobile REST API │ ├── api/ # Mobile REST API
│ │ ├── __init__.py # api_bp parent blueprint + register_api() │ │ ├── __init__.py # api_bp parent blueprint + register_api()
│ │ ├── auth.py # /api/v1/auth/* and /api/v1/devices/* │ │ ├── auth.py # /api/v1/auth/* and /api/v1/devices/*
@@ -133,7 +141,24 @@ lt_janitorial_quality_control/
│ └── utils/ │ └── utils/
├── migrations/ ├── migrations/
│ └── versions/ │ └── versions/
│ └── phase23_support_tickets.py ← HEAD │ ├── 0003_add_user_active.py ← squashed baseline (chain root, MT-2)
│ └── phase32_device_token_columns ← HEAD
├── migrations_tenant/ # MT-2 — standalone Alembic env for per-tenant upgrades
│ ├── env.py # URL-driven, no Flask; reuses migrations/versions
│ └── script.py.mako
├── control/ # MT-0 — control plane (tenant registry, plans, provisioning)
│ ├── __init__.py
│ ├── base.py # ControlBase, engine/session (CONTROL_DATABASE_URL)
│ ├── cli.py # seed / create-superadmin / list-plans
│ ├── crypto.py # Fernet encrypt/decrypt (CONTROL_FERNET_KEY)
│ ├── models.py # Plan, PlanFeature, Tenant, TenantDomain,
│ │ # Superadmin, ProvisioningJob, TenantAudit
│ ├── provision.py # MT-3 — create_tenant / register_tenant_zero / delete_tenant
│ ├── seed.py # idempotent plan seeder
│ ├── tenant_migrate.py # MT-2 — upgrade_tenant / bootstrap_tenant / chain_head CLI
│ ├── time_utils.py # now_eastern() mirror (no app import)
│ └── migrations/ # control Alembic chain (control{N}_…)
│ └── versions/control0001_init.py ← HEAD
└── ... └── ...
``` ```
@@ -157,6 +182,15 @@ lt_janitorial_quality_control/
| `REDIS_URL` | Optional. When set, Flask-Limiter uses Redis for shared rate-limit counters across Gunicorn workers. | | `REDIS_URL` | Optional. When set, Flask-Limiter uses Redis for shared rate-limit counters across Gunicorn workers. |
| `GROQ_API_KEY` | Optional. When set, enables the AI chatbot at `/support/chat`. Absent → chat input disabled; customers see a "Submit to Support" fallback only. | | `GROQ_API_KEY` | Optional. When set, enables the AI chatbot at `/support/chat`. Absent → chat input disabled; customers see a "Submit to Support" fallback only. |
| `GROQ_MODEL` | Optional. Groq model ID. Defaults to `llama-3.3-70b-versatile`. | | `GROQ_MODEL` | Optional. Groq model ID. Defaults to `llama-3.3-70b-versatile`. |
| `MULTI_TENANT_ENABLED` | `false` by default. Set `true` to activate Host→tenant routing. Requires all control-plane vars below. |
| `CONTROL_DATABASE_URL` | Control-plane MySQL URI, e.g. `mysql+pymysql://jqc_control:pw@127.0.0.1/jqc_control`. Required when `MULTI_TENANT_ENABLED=true`. |
| `CONTROL_FERNET_KEY` | Fernet key for encrypting tenant DB passwords. Generate once; store in `/etc/jqc/control.env`. |
| `PROVISION_DB_URL` | MySQL account that can `CREATE DATABASE` / `CREATE USER` / `GRANT`. e.g. `mysql+pymysql://jqc_provisioner:pw@127.0.0.1/`. |
| `TENANT_BASE_DOMAIN` | Apex domain for subdomains, e.g. `jqc.app`. Used by provisioner to build `<slug>.jqc.app`. |
| `MULTI_TENANT_EXEMPT_PATHS` | Comma-separated path prefixes that bypass the tenant gate (e.g. `/health`). `/static/` is always exempt. |
| `TENANT_ENGINE_POOL_SIZE` | Per-tenant engine pool size (default 5). |
| `TENANT_ENGINE_MAX_OVERFLOW` | Per-tenant pool max overflow (default 5). |
| `TENANT_ENGINE_POOL_RECYCLE` | Pool recycle in seconds (default 1800). |
### Email SSL Auto-Detection ### Email SSL Auto-Detection
@@ -632,24 +666,53 @@ limiter = Limiter(
## 17. Alembic Migration Chain ## 17. Alembic Migration Chain
**Current HEAD:** `phase32_device_token_columns` (30 migrations total).
**Chain root:** `0003_add_user_active` — a guarded squashed baseline (MT-2) that recreates the full 25-table schema with INFORMATION_SCHEMA guards. The original baseline migrations (0001/0002/0003) were lost; this file restores the chain root so Alembic can build the revision map. `down_revision = None`.
``` ```
phase1_projects_roles → phase6_features → phase7_mobile_api → phase8_notification_matrix 0003_add_user_active (baseline, MT-2)
→ phase9_user_full_name → phase10_customer_password_setup → phase11_director_role → phase1_projects_roles → phase6_features → phase7_mobile_api → phase8_notification_matrix
→ phase12_performance_indexes → phase_b_mobile_local_id → phase9_user_full_name → phase10_customer_password_setup → phase11_director_role
→ phase13_issue_facility → phase14_facility_created_at → phase12_performance_indexes → phase_b_mobile_local_id
→ phase15_audit_log_indexes → phase16_notifications_columns → phase13_issue_facility → phase14_facility_created_at
→ phase17_notification_event_type → phase15_audit_log_indexes → phase16_notifications_columns
→ phase18_issue_reported_by → phase17_notification_event_type
→ phase19_issue_mobile_photos → phase18_issue_reported_by
→ phase20_inspector_assignments → phase19_issue_mobile_photos
→ phase21_template_active → phase20_inspector_assignments
→ phase21_performance_indexes → phase21_template_active
→ phase22_comment_visibility → phase21_performance_indexes
→ phase23_support_tickets → phase22_comment_visibility
→ phase24_notify_defaults → phase23_support_tickets
→ phase25_inspection_gps → phase24_notify_defaults
→ phase26_issue_vendor → phase25_inspection_gps
→ phase27_score_alerts ← HEAD → phase26_issue_vendor
→ phase27_score_alerts
→ ... → phase32_device_token_columns ← HEAD
```
### Fresh DB provisioning (multi-tenant)
**Never use `flask db upgrade` on an empty database.** Fourteen of the phase migrations are not idempotent (no `INFORMATION_SCHEMA` guards) and will fail on a fresh DB that already has the baseline schema. Use the provisioner instead:
```bash
python -m control.tenant_migrate bootstrap --tenant <slug>
# Runs: upgrade to 0003_add_user_active (builds full schema) → stamp head
# Phase migrations are SKIPPED — baseline covers the full schema.
```
Ongoing incremental upgrades (phase33+, which MUST be guarded) use:
```bash
python -m control.tenant_migrate upgrade --tenant all
```
### Standard single-tenant migration deploy
```bash
flask db upgrade # existing LT box — safe, already at head
sudo systemctl restart jqc
``` ```
### phase21_performance_indexes ### phase21_performance_indexes
@@ -945,6 +1008,47 @@ timeout = 30
### Nginx ### Nginx
- `client_max_body_size 50M` - `client_max_body_size 50M`
- Passes `X-Forwarded-For` - Passes `X-Forwarded-For`
- **Multi-tenant wildcard block** (alongside the existing single-domain block):
```nginx
server {
listen 80;
server_name *.jqc.app jqc.app;
location / {
proxy_pass http://127.0.0.1:8000;
proxy_set_header Host $host; # resolver reads this
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
}
```
**TLS for subdomains:** wildcard cert (`*.jqc.app`) via DNS-01 challenge (certbot + DNS plugin). HTTP-01 works for individual subdomains only and cannot auto-renew a wildcard. Custom-domain TLS via Caddy on-demand TLS (MT-6).
### Control-plane environment file
**All** control/provisioning env vars must live in one canonical file. Both the systemd unit and CLI sessions must source the same file — a key mismatch between them causes `cryptography.fernet.InvalidToken` at request time.
```bash
# /etc/jqc/control.env (chmod 640, chown root:jqc)
CONTROL_DATABASE_URL=mysql+pymysql://jqc_control:<pw>@127.0.0.1/jqc_control
CONTROL_FERNET_KEY=<generated key — no quotes>
PROVISION_DB_URL=mysql+pymysql://jqc_provisioner:<pw>@127.0.0.1/
TENANT_BASE_DOMAIN=jqc.app
MULTI_TENANT_ENABLED=true
```
```ini
# systemd unit [Service]
EnvironmentFile=/etc/jqc/control.env
```
```bash
# CLI sessions
set -a; . /etc/jqc/control.env; set +a
```
### Recommended Cron Schedule ### Recommended Cron Schedule
```bash ```bash
@@ -1026,6 +1130,12 @@ timeout = 30
| 68 | **Support ticket customer replies revert status from `answered` → `open`** | When a customer posts a follow-up on an answered ticket, the route sets `ticket.status = 'open'` so admins see it in their open queue. Admin must manually close or re-answer. | | 68 | **Support ticket customer replies revert status from `answered` → `open`** | When a customer posts a follow-up on an answered ticket, the route sets `ticket.status = 'open'` so admins see it in their open queue. Admin must manually close or re-answer. |
| 69 | **Customer issue create: `assigned_to` field hidden, `facility_id` scoped to `get_customer_scope()`** | `issues.create()` detects `role == 'customer'`, scopes facilities to the customer's assigned set, sets `staff = []` for the assigned_to dropdown, and hides the field in `form.html`. `IssueForm.facility_id.choices` must still include all active facilities so POST validation passes. | | 69 | **Customer issue create: `assigned_to` field hidden, `facility_id` scoped to `get_customer_scope()`** | `issues.create()` detects `role == 'customer'`, scopes facilities to the customer's assigned set, sets `staff = []` for the assigned_to dropdown, and hides the field in `form.html`. `IssueForm.facility_id.choices` must still include all active facilities so POST validation passes. |
| 70 | **`notify()` does NOT commit — caller must `db.session.commit()` after all `notify()` calls** | `notify()` adds a `Notification` row to the session but leaves the commit to the caller. The support helpers (`_notify_admins_new_ticket`, `_notify_customer_reply`, `_notify_admins_customer_reply`) each call `db.session.commit()` after the `notify()` loop. | | 70 | **`notify()` does NOT commit — caller must `db.session.commit()` after all `notify()` calls** | `notify()` adds a `Notification` row to the session but leaves the commit to the caller. The support helpers (`_notify_admins_new_ticket`, `_notify_customer_reply`, `_notify_admins_customer_reply`) each call `db.session.commit()` after the `notify()` loop. |
| 71 | **`CONTROL_FERNET_KEY` must be identical between CLI sessions and the Gunicorn service** | Provisioning encrypts tenant creds; the app decrypts them at request time. A key mismatch causes `cryptography.fernet.InvalidToken`. Canonical source: `/etc/jqc/control.env` — sourced by both `EnvironmentFile=` in the systemd unit and `set -a; . /etc/jqc/control.env; set +a` in CLI sessions. |
| 72 | **`MULTI_TENANT_ENABLED` must be flipped only after tenant-zero (LT) is registered** | The resolver gates all traffic by Host. If LT's domains aren't in `tenant_domains` when the flag goes `true`, LT's own traffic gets a 404 "Workspace not found" page. Register LT first via `register-tenant-zero`, confirm with `curl -H "Host: lts.jqc.app" http://127.0.0.1:8000/`, then flip the flag. |
| 73 | **Tenant DB passwords use `_gen_password()` — not `secrets.token_urlsafe()`** | MySQL `validate_password` MEDIUM policy requires lower + upper + digit + special. `token_urlsafe` is alphanumeric-only and fails intermittently. `_gen_password()` guarantees all four character classes. |
| 74 | **`bootstrap_tenant()` for fresh DBs; `upgrade_tenant()` for incremental** | Fourteen phase migrations are unguarded. Running the full chain on an empty DB (from the baseline) causes duplicate-column errors. `bootstrap_tenant()` runs the baseline to HEAD then stamps — phases skipped. `upgrade_tenant()` is for phase33+ incremental upgrades on already-provisioned DBs. |
| 75 | **`delete_tenant()` + `_add_domains()` are retry-safe** | `_add_domains` uses `_upsert_domain()` (delete-then-insert) to handle orphan rows from failed partial runs. `delete_tenant()` also purges by derived domain string, not only by `tenant_id`, catching orphans whose parent tenant row was rolled back. |
| 76 | **`register-tenant-zero` never bootstraps and never drops the DB** | `register_tenant_zero()` only reads the existing head, inserts control rows, and maps domains. `delete_tenant()` on tenant-zero must never use `--drop-db` — the guard checks `db_name == db_name_for(slug)` and refuses non-provisioner-named DBs (LT's DB name is `jqc_lt`, not `jqc_lts`). |
--- ---
@@ -1039,3 +1149,102 @@ timeout = 30
6. **Full file contents for 1–3 file changes**; deployment map for larger changesets 6. **Full file contents for 1–3 file changes**; deployment map for larger changesets
7. **Explicit deploy instructions** — migration steps separated from code steps 7. **Explicit deploy instructions** — migration steps separated from code steps
8. **Root cause analysis** on errors — never apply temporary workarounds 8. **Root cause analysis** on errors — never apply temporary workarounds
---
## 22. Multi-Tenant Architecture (MT-0 → MT-3)
See `MULTI_TENANT_PLAN.md` for the full phased roadmap. This section summarises what is built and operational.
### Model
Shared codebase + **database-per-tenant**. One Flask/Gunicorn process serves all tenants. `before_request` resolves the `Host` header → tenant → per-tenant MySQL database. All existing `db.session` calls route transparently — zero changes to models or routes.
### Control plane (`control/`)
Self-contained package, own `ControlBase` + engine/session, own Alembic chain. No imports from `app/`.
| Module | Purpose |
|---|---|
| `base.py` | `ControlBase`, `control_session()` context manager, engine from `CONTROL_DATABASE_URL` |
| `models.py` | `Plan`, `PlanFeature`, `Tenant`, `TenantDomain`, `Superadmin`, `ProvisioningJob`, `TenantAudit` |
| `crypto.py` | Fernet encrypt/decrypt for `tenants.db_password_enc` |
| `seed.py` | Idempotent plan seeder (Free / Starter / Pro / Enterprise) |
| `cli.py` | `seed`, `create-superadmin`, `list-plans` |
| `tenant_migrate.py` | `bootstrap_tenant()`, `upgrade_tenant()`, `chain_head()`, `current_revision()` + CLI |
| `provision.py` | `create_tenant()`, `register_tenant_zero()`, `delete_tenant()` + CLI |
### Data-plane changes (`app/`)
| File | Change |
|---|---|
| `app/__init__.py` | `db = SQLAlchemy(session_options={'class_': RoutingSession})` + `init_tenancy(app)` |
| `config.py` | `MULTI_TENANT_ENABLED` (default `false`) + pool tunables |
| `app/tenancy/` | New package — see §3 repo layout |
### Plan tiers
| Axis | Free | Starter | Pro | Enterprise |
|---|---|---|---|---|
| Max users | 3 | 15 | 50 | unlimited |
| Max facilities | 2 | 10 | 50 | unlimited |
| Inspections / month | 50 | 500 | 5 000 | unlimited |
| Issues / month | 50 | 500 | 5 000 | unlimited |
| Mobile API | ✗ | ✓ | ✓ | ✓ |
| Scheduled reports | ✗ | ✗ | ✓ | ✓ |
| Branding | ✗ | ✗ | ✓ | ✓ |
| Custom domain | ✗ | ✗ | ✓ | ✓ |
| Subdomain | ✓ | ✓ | ✓ | ✓ |
Quota-exceed behaviour: **soft warn** — allow submit, flag for upgrade, never reject.
### Provisioner MySQL account (required grants)
```sql
GRANT ALL PRIVILEGES ON *.* TO 'jqc_provisioner'@'localhost' WITH GRANT OPTION;
GRANT CREATE USER ON *.* TO 'jqc_provisioner'@'localhost';
FLUSH PRIVILEGES;
```
### CLI quick-reference
```bash
# Control schema + plans + first superadmin (run once)
alembic -c control/migrations/alembic.ini upgrade head
python -m control.cli seed
python -m control.cli create-superadmin --username admin --email you@example.com
# Adopt existing LT database as tenant-zero (run once, no data move)
python -m control.provision register-tenant-zero \
--slug lts --name "LT Services" --plan enterprise \
--db-host 127.0.0.1 --db-name <LT_DB> --db-user <LT_USER> --db-password '<pw>' \
--custom-domain jqc.ltservicesinc.com --base-domain jqc.app
# Provision a new tenant (creates MySQL DB + user + schema + first admin)
python -m control.provision create-tenant \
--slug acme --name "Acme Corp" --plan pro --admin-email ops@acme.com
# Delete / deregister a tenant
python -m control.provision delete-tenant --slug ztest --drop-db --yes # provisioned DB
python -m control.provision delete-tenant --slug lts --yes # adopted DB — no --drop-db
# Migration status + upgrades
python -m control.tenant_migrate heads
python -m control.tenant_migrate current --tenant all
python -m control.tenant_migrate upgrade --tenant all # incremental (phase33+)
python -m control.tenant_migrate bootstrap --tenant acme # fresh DB only
```
### Enable multi-tenancy (cutover sequence)
```bash
# 1. Register LT as tenant-zero (see above)
# 2. Smoke-test routing (flag still off)
curl -sI -H "Host: lts.jqc.app" http://127.0.0.1:8000/ | head -2
curl -sI -H "Host: ztest.jqc.app" http://127.0.0.1:8000/ | head -2
# 3. Wildcard DNS: *.jqc.app A <SERVER_IP>
# 4. Nginx wildcard server block (see §19)
# 5. Wildcard TLS cert (DNS-01, certbot)
# 6. Add to /etc/jqc/control.env: MULTI_TENANT_ENABLED=true
# 7. sudo systemctl daemon-reload && sudo systemctl restart jqc
```
README.md
+78 -2
View File
@@ -1,6 +1,6 @@
# JQC — Janitorial Quality Control System # JQC — Janitorial Quality Control System
A production-grade web application for managing janitorial service contracts, facility inspections, issue tracking, and client reporting. A production-grade, multi-tenant web application for managing janitorial service contracts, facility inspections, issue tracking, and client reporting. Built as a shared-codebase SaaS with database-per-tenant isolation.
--- ---
@@ -169,6 +169,7 @@ After=network.target mysql.service
User=jqc User=jqc
WorkingDirectory=/home/jqc/lt_janitorial_quality_control WorkingDirectory=/home/jqc/lt_janitorial_quality_control
EnvironmentFile=/home/jqc/.env EnvironmentFile=/home/jqc/.env
EnvironmentFile=/etc/jqc/control.env # control-plane + multi-tenant vars
ExecStart=/home/jqc/venv/bin/gunicorn -c gunicorn_config.py wsgi:app ExecStart=/home/jqc/venv/bin/gunicorn -c gunicorn_config.py wsgi:app
Restart=on-failure Restart=on-failure
@@ -176,6 +177,8 @@ Restart=on-failure
WantedBy=multi-user.target WantedBy=multi-user.target
``` ```
> `/etc/jqc/control.env` must be `chmod 640 / chown root:jqc`. It holds `CONTROL_DATABASE_URL`, `CONTROL_FERNET_KEY`, `PROVISION_DB_URL`, `TENANT_BASE_DOMAIN`, and `MULTI_TENANT_ENABLED`. The same file must be sourced in any shell session that runs provisioning commands (`set -a; . /etc/jqc/control.env; set +a`).
--- ---
## Cron Jobs ## Cron Jobs
@@ -260,7 +263,7 @@ Access tokens expire after 60 minutes. Refresh tokens are valid for 30 days and
## Database Migrations ## Database Migrations
```bash ```bash
# Apply all pending migrations # Apply all pending migrations (existing / single-tenant DB)
flask db upgrade flask db upgrade
# Create a new migration after model changes # Create a new migration after model changes
@@ -270,6 +273,21 @@ flask db migrate -m "description of change"
flask db downgrade flask db downgrade
``` ```
### Multi-tenant migrations
```bash
# Bootstrap a brand-new tenant DB (baseline schema + stamp head)
python -m control.tenant_migrate bootstrap --tenant <slug>
# Incremental upgrade for all active tenants (phase33+ onwards)
python -m control.tenant_migrate upgrade --tenant all
# Check each tenant's current revision vs chain head
python -m control.tenant_migrate current --tenant all
```
> **Never run `flask db upgrade` on an empty tenant database.** Fourteen historical phase migrations lack `INFORMATION_SCHEMA` guards and will fail against a DB that already has the baseline schema. Use `bootstrap` instead.
### MySQL Compatibility Notes ### MySQL Compatibility Notes
- **ENUM changes** require three steps: expand → migrate data → contract. Never skip steps. - **ENUM changes** require three steps: expand → migrate data → contract. Never skip steps.
@@ -304,6 +322,64 @@ flask db downgrade
| `REDIS_URL` | — | Redis connection URI for shared rate-limit storage; optional but recommended in production | | `REDIS_URL` | — | Redis connection URI for shared rate-limit storage; optional but recommended in production |
| `GROQ_API_KEY` | — | Groq API key. When absent the AI chatbot is disabled; customers can still submit support tickets. | | `GROQ_API_KEY` | — | Groq API key. When absent the AI chatbot is disabled; customers can still submit support tickets. |
| `GROQ_MODEL` | `llama-3.3-70b-versatile` | Groq model ID override | | `GROQ_MODEL` | `llama-3.3-70b-versatile` | Groq model ID override |
| `MULTI_TENANT_ENABLED` | `false` | Set `true` to activate Host→tenant routing. Requires the four control-plane vars below. |
| `CONTROL_DATABASE_URL` | — | Control-plane DB URI, e.g. `mysql+pymysql://jqc_control:pw@127.0.0.1/jqc_control` |
| `CONTROL_FERNET_KEY` | — | Fernet key for encrypting tenant DB passwords. Generate once and store permanently. |
| `PROVISION_DB_URL` | — | MySQL account with `CREATE DATABASE`/`CREATE USER`/`GRANT`. |
| `TENANT_BASE_DOMAIN` | — | Subdomain apex, e.g. `jqc.app`. |
---
## Multi-Tenant Setup
JQC runs as a shared-codebase SaaS with database-per-tenant isolation. Each tenant gets its own MySQL database and least-privilege MySQL user. The control plane (`control/`) manages the tenant registry separately from any tenant's data.
### Prerequisites
```sql
-- Control database
CREATE DATABASE jqc_control CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
CREATE USER 'jqc_control'@'localhost' IDENTIFIED BY '<pw>';
GRANT ALL PRIVILEGES ON jqc_control.* TO 'jqc_control'@'localhost'; FLUSH PRIVILEGES;
-- Provisioner account (creates per-tenant DBs + users)
CREATE USER 'jqc_provisioner'@'localhost' IDENTIFIED BY '<pw>';
GRANT ALL PRIVILEGES ON *.* TO 'jqc_provisioner'@'localhost' WITH GRANT OPTION;
GRANT CREATE USER ON *.* TO 'jqc_provisioner'@'localhost'; FLUSH PRIVILEGES;
```
### Bootstrap (run once)
```bash
# Set env (or add to /etc/jqc/control.env)
export CONTROL_DATABASE_URL='mysql+pymysql://jqc_control:<pw>@127.0.0.1/jqc_control'
export CONTROL_FERNET_KEY="$(python -c 'from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())')"
export PROVISION_DB_URL='mysql+pymysql://jqc_provisioner:<pw>@127.0.0.1/'
export TENANT_BASE_DOMAIN='jqc.app'
# Apply control schema + seed plans + first superadmin
alembic -c control/migrations/alembic.ini upgrade head
python -m control.cli seed
python -m control.cli create-superadmin --username admin --email you@example.com
# Adopt existing LT database as tenant-zero (no data moved)
python -m control.provision register-tenant-zero \
--slug lts --name "LT Services" --plan enterprise \
--db-host 127.0.0.1 --db-name <LT_DB> --db-user <LT_USER> --db-password '<pw>' \
--custom-domain jqc.ltservicesinc.com
# Provision a new tenant
python -m control.provision create-tenant \
--slug acme --name "Acme Corp" --plan pro --admin-email ops@acme.com
```
### Enable multi-tenancy
1. Add `MULTI_TENANT_ENABLED=true` to `/etc/jqc/control.env`
2. Configure wildcard DNS: `*.jqc.app A <SERVER_IP>`
3. Add Nginx wildcard server block (see `CLAUDE.md §19`)
4. Obtain wildcard TLS cert via DNS-01 challenge
5. `sudo systemctl daemon-reload && sudo systemctl restart jqc`
--- ---
control/README.md
+158 -38
View File
@@ -1,84 +1,204 @@
# Control Plane (MT-0) # Control Plane (MT-0 → MT-3)
Tenant registry, plans, domains, provisioning state, and superadmin accounts Tenant registry, plans, domains, provisioning state, and superadmin accounts
for multi-tenant JQC. Self-contained and decoupled from `app/` — the existing for multi-tenant JQC. Self-contained and decoupled from `app/` — imports
single-tenant application is unaffected by this package. nothing from the data-plane application.
See `../MULTI_TENANT_PLAN.md` for the full architecture and roadmap. See `../MULTI_TENANT_PLAN.md` for the full phased roadmap.
## Layout ## Layout
``` ```
control/ control/
├── __init__.py # package docs ├── __init__.py # package docs
├── base.py # ControlBase + engine/session (from CONTROL_DATABASE_URL) ├── base.py # ControlBase + engine/session (CONTROL_DATABASE_URL)
├── crypto.py # Fernet encrypt/decrypt for tenant DB passwords ├── crypto.py # Fernet encrypt/decrypt for tenant DB passwords (CONTROL_FERNET_KEY)
├── time_utils.py # now_eastern() mirror (no app import) ├── time_utils.py # now_eastern() mirror (no app import)
├── models.py # Plan, PlanFeature, Tenant, TenantDomain, ├── models.py # Plan, PlanFeature, Tenant, TenantDomain,
│ # Superadmin, ProvisioningJob, TenantAudit │ # Superadmin, ProvisioningJob, TenantAudit
├── seed.py # idempotent baseline-plan seeder ├── seed.py # idempotent baseline-plan seeder (Free/Starter/Pro/Enterprise)
├── cli.py # seed / create-superadmin / list-plans ├── cli.py # seed / create-superadmin / list-plans
├── tenant_migrate.py # MT-2 — upgrade_tenant / bootstrap_tenant / chain_head + CLI
├── provision.py # MT-3 — create_tenant / register_tenant_zero / delete_tenant + CLI
└── migrations/ # standalone Alembic chain (control{N}_…) └── migrations/ # standalone Alembic chain (control{N}_…)
└── versions/control0001_init.py ← HEAD └── versions/
└── control0001_init.py ← HEAD (7 tables)
``` ```
## Environment variables (control plane only) ## Environment variables
All must live in `/etc/jqc/control.env` (see §3 below). Never hard-code.
| Variable | Purpose | | Variable | Purpose |
|---|---| |---|---|
| `CONTROL_DATABASE_URL` | e.g. `mysql+pymysql://jqc_control:pw@localhost/jqc_control` | | `CONTROL_DATABASE_URL` | e.g. `mysql+pymysql://jqc_control:pw@127.0.0.1/jqc_control` |
| `CONTROL_FERNET_KEY` | Fernet key for encrypting tenant DB passwords | | `CONTROL_FERNET_KEY` | Fernet key for encrypting tenant DB passwords. Generate once, never rotate without re-encrypting all tenant rows. |
| `PROVISION_DB_URL` | MySQL account with `CREATE DATABASE` / `CREATE USER` / `GRANT` rights. |
| `TENANT_BASE_DOMAIN` | Apex for subdomains, e.g. `jqc.app`. Builds `<slug>.jqc.app`. |
Generate a Fernet key: Generate a Fernet key (run once, store permanently):
```bash ```bash
python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())" python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"
``` ```
## Deploy — MT-0 bootstrap (run once) ## Canonical env file (required — prevents Fernet key mismatch)
Migrations are independent of the tenant chain and the data plane. The existing The `CONTROL_FERNET_KEY` used at provisioning time must be identical to the key
app does **not** need to be touched or restarted for MT-0. the running app uses to decrypt tenant creds. A mismatch causes
`cryptography.fernet.InvalidToken` at request time. One file prevents this:
```bash ```bash
# 1. Create the control database + its MySQL user (run as a MySQL admin) # /etc/jqc/control.env (chmod 640, chown root:jqc)
mysql -e "CREATE DATABASE jqc_control CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;" CONTROL_DATABASE_URL=mysql+pymysql://jqc_control:<pw>@127.0.0.1/jqc_control
mysql -e "CREATE USER 'jqc_control'@'localhost' IDENTIFIED BY '<pw>';" CONTROL_FERNET_KEY=<key — no quotes, no trailing space>
mysql -e "GRANT ALL PRIVILEGES ON jqc_control.* TO 'jqc_control'@'localhost'; FLUSH PRIVILEGES;" PROVISION_DB_URL=mysql+pymysql://jqc_provisioner:<pw>@127.0.0.1/
TENANT_BASE_DOMAIN=jqc.app
MULTI_TENANT_ENABLED=true
```
# 2. Export the env vars (add to .env or the systemd unit for the control panel later) ```ini
export CONTROL_DATABASE_URL='mysql+pymysql://jqc_control:<pw>@localhost/jqc_control' # systemd unit [Service]
export CONTROL_FERNET_KEY='<generated key>' EnvironmentFile=/etc/jqc/control.env
```
# 3. Apply the control schema ```bash
# any CLI session that provisions tenants
set -a; . /etc/jqc/control.env; set +a
```
## MySQL accounts required
```sql
-- Control DB user
CREATE DATABASE jqc_control CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
CREATE USER 'jqc_control'@'localhost' IDENTIFIED BY '<pw>';
GRANT ALL PRIVILEGES ON jqc_control.* TO 'jqc_control'@'localhost'; FLUSH PRIVILEGES;
-- Provisioner (creates per-tenant DBs + users)
CREATE USER 'jqc_provisioner'@'localhost' IDENTIFIED BY '<pw>';
GRANT ALL PRIVILEGES ON *.* TO 'jqc_provisioner'@'localhost' WITH GRANT OPTION;
GRANT CREATE USER ON *.* TO 'jqc_provisioner'@'localhost'; FLUSH PRIVILEGES;
```
## Bootstrap — MT-0 (run once per environment)
```bash
set -a; . /etc/jqc/control.env; set +a
# 1. Apply control schema (7 tables)
alembic -c control/migrations/alembic.ini upgrade head alembic -c control/migrations/alembic.ini upgrade head
# 4. Seed baseline plans (Free / Starter / Pro / Enterprise) # 2. Seed Free / Starter / Pro / Enterprise plans
python -m control.cli seed python -m control.cli seed
# 5. Create the first superadmin # 3. First superadmin
python -m control.cli create-superadmin --username admin --email you@example.com python -m control.cli create-superadmin --username admin --email you@example.com
# Verify
alembic -c control/migrations/alembic.ini current # → control0001_init (head)
python -m control.cli list-plans # → 4 plans
``` ```
Verify: ## Tenant-zero — register existing LT in place (MT-3)
No DB creation, no data movement, no schema changes to the live LT database.
```bash ```bash
alembic -c control/migrations/alembic.ini current # → control0001_init (head) python -m control.provision register-tenant-zero \
python -m control.cli list-plans --slug lts --name "LT Services" --plan enterprise \
--db-host 127.0.0.1 \
--db-name <LT_DATABASE_NAME> \
--db-user <LT_DB_USER> \
--db-password '<LT_DB_PASSWORD>' \
--custom-domain jqc.ltservicesinc.com \
--base-domain jqc.app
``` ```
Output includes `alembic_head` (read from the existing `alembic_version` table)
and both domain mappings (`lts.jqc.app` + `jqc.ltservicesinc.com` as verified).
## Provisioning a new tenant (MT-3)
```bash
python -m control.provision create-tenant \
--slug acme \
--name "Acme Corp" \
--plan pro \
--admin-email ops@acme.com \
--base-domain jqc.app
```
Steps executed automatically:
1. `CREATE DATABASE jqc_acme` + `CREATE USER jqc_acme_u` + `GRANT` (scoped to that DB)
2. Insert `tenants` row with Fernet-encrypted password
3. `bootstrap_tenant()` → baseline schema + `stamp head` (phase migrations skipped)
4. Insert first admin into tenant DB with a set-password token
5. Register `acme.jqc.app` (verified) + optional custom domain (unverified)
6. Set tenant `status = active`
Output includes the **admin setup link** — send to the admin's email:
`https://acme.jqc.app/customers/set-password/<token>`
On failure, a best-effort rollback drops the DB/user and removes the tenant row
so a retry starts clean.
## Per-tenant migrations (MT-2)
```bash
# New fresh DB (empty → full schema + stamp head; phase migrations skipped)
python -m control.tenant_migrate bootstrap --tenant acme
# Incremental upgrade (phase33+ — must be INFORMATION_SCHEMA-guarded)
python -m control.tenant_migrate upgrade --tenant all
python -m control.tenant_migrate upgrade --tenant acme # by slug
python -m control.tenant_migrate upgrade --tenant 3 # by id
# Status check
python -m control.tenant_migrate current --tenant all
python -m control.tenant_migrate heads
```
**Rule:** every new migration (phase33+) MUST use INFORMATION_SCHEMA existence
checks (`_table_exists`, `_column_exists`, `_index_exists`) — the same pattern
used by the squashed baseline. Unguarded migrations break idempotency and cannot
be re-run safely across the fleet.
## Delete / deregister a tenant
```bash
# Provisioned tenant — delete DB + control records
python -m control.provision delete-tenant --slug acme --drop-db --yes
# Adopted DB (tenant-zero) — remove control records only; NEVER --drop-db
python -m control.provision delete-tenant --slug lts --yes
```
`--drop-db` is guarded: it only proceeds when the stored `db_name` matches the
provisioner convention (`jqc_<slug>`). This prevents accidental drops of
externally-named databases like LT's production DB.
## Rollback ## Rollback
```bash ```bash
alembic -c control/migrations/alembic.ini downgrade base # drops all control tables # Drop all 7 control tables (destructive)
alembic -c control/migrations/alembic.ini downgrade base
``` ```
## Notes ## Plan tier matrix
- The migration uses INFORMATION_SCHEMA existence checks (Rule 14) — safe to re-run. | Axis | Free | Starter | Pro | Enterprise |
- Plan seeding is idempotent (upsert by `code`) — re-running updates in place. |---|---|---|---|---|
- Tenant DB passwords are stored Fernet-encrypted in `tenants.db_password_enc`; | Max users | 3 | 15 | 50 | unlimited |
`Tenant.db_uri` decrypts on demand. Provisioning that *creates* per-tenant | Max facilities | 2 | 10 | 50 | unlimited |
MySQL users/grants lands in MT-3. | Inspections / month | 50 | 500 | 5 000 | unlimited |
- Control-panel write auditing (`tenant_audit`) is wired in MT-4; the bootstrap | Issues / month | 50 | 500 | 5 000 | unlimited |
CLI logs to stdout only. | Mobile API | ✗ | ✓ | ✓ | ✓ |
| Scheduled reports | ✗ | ✗ | ✓ | ✓ |
| Branding | ✗ | ✗ | ✓ | ✓ |
| Custom domain | ✗ | ✗ | ✓ | ✓ |
| Subdomain | ✓ | ✓ | ✓ | ✓ |
Quota-exceed: **soft warn** — allow submit, flag for upgrade, never reject.
Plans are seeded idempotently; re-running `python -m control.cli seed` updates
in-place without creating duplicates.