diff --git a/CLAUDE.md b/CLAUDE.md index cefdd01..fe7f271 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -2,7 +2,7 @@ > **Audience:** AI assistants and developers working on this codebase. > **Purpose:** Authoritative reference for architecture, conventions, gotchas, and decisions. -> **Last reviewed:** June 2026 (Phase 19 complete + mobile API gap-fill Phases A–E + customer UI refinements + Phase 22 comment visibility + Phase 23 support chat/tickets + inspector performance Excel export + inspection list filters + customer issue logging + AI chatbot + dashboard grouped sections + issues/inspections PDF export + date/ID filters + Reports expansion Phases R1–R4 + Phase 24 issue_created notify defaults + Phase 25 inspection GPS + Phase 26 issue vendor fields + Phase 27 facility score alerts + **MT-0 through MT-7 complete; MT-8 flag-gated; MT-9 iOS pending**) +> **Last reviewed:** June 2026 (Phase 19 complete + mobile API gap-fill Phases A–E + customer UI refinements + Phase 22 comment visibility + Phase 23 support chat/tickets + inspector performance Excel export + inspection list filters + customer issue logging + AI chatbot + dashboard grouped sections + issues/inspections PDF export + date/ID filters + Reports expansion Phases R1–R4 + Phase 24 issue_created notify defaults + Phase 25 inspection GPS + Phase 26 issue vendor fields + Phase 27 facility score alerts + **MT-0 through MT-8 complete; self-service signup; trial enforcement; billing emails; invoice history; superadmin billing controls; per-tenant backup CLI; health dashboard; MT-9 iOS pending**) --- @@ -29,8 +29,11 @@ 19. [Infrastructure](#19-infrastructure) 20. [Known Constraints & Hard Rules](#20-known-constraints--hard-rules) 21. [Change Philosophy](#21-change-philosophy) -22. [Multi-Tenant Architecture (MT-0 → MT-7)](#22-multi-tenant-architecture-mt-0--mt-7) -23. [Coding Rules for AI Assistants](#23-coding-rules-for-ai-assistants) +22. [Multi-Tenant Architecture (MT-0 → MT-8)](#22-multi-tenant-architecture-mt-0--mt-8) +23. [Billing System (MT-8)](#23-billing-system-mt-8) +24. [Backup CLI](#24-backup-cli-controlbackuppy) +25. [Health Dashboard](#25-health-dashboard-health-on-panel) +26. [Coding Rules for AI Assistants](#26-coding-rules-for-ai-assistants) --- @@ -48,6 +51,8 @@ - **Support chat** — Groq AI chatbot for customers with preset FAQ chips; escalation to admin via ticketing system; customers can view and reply to their own tickets; admins manage tickets at `/support/admin/tickets` - **Mobile API** — JWT-authenticated REST layer for the iPad native app - **iPad native app** — SwiftUI + SwiftData offline-first inspection tool (Phase A + B + C complete) +- **Billing** — Stripe-backed subscription system with plan picker, Checkout, Customer Portal, dunning emails (HTML + plain text), invoice history, and trial-period enforcement +- **Self-service signup** — public `/signup` page provisions a new tenant immediately with a 14-day trial (no Stripe required at signup) The application is actively deployed in production and maintained by a single developer/administrator. @@ -115,11 +120,23 @@ lt_janitorial_quality_control/ │ │ └── ... │ ├── routes/ │ │ ├── support.py # /support/* — AI chat, ticket submit/list/detail (Phase 23) -│ │ ├── tenant_settings.py # MT-7: /settings/* — branding, plan view, domain mgmt (@admin_required) +│ │ ├── tenant_settings.py # MT-7: /settings/* — branding, plan view (+ invoice history), domain mgmt (@admin_required) +│ │ ├── signup.py # public /signup — self-service tenant signup (no auth, no tenant context) │ │ └── ... │ ├── templates/ │ │ ├── _quota_warning.html # MT-5: reusable quota exceeded banner partial │ │ ├── tenant_settings/ # MT-7: branding.html, plan.html, domains.html +│ │ ├── billing/ +│ │ │ ├── _billing_banner.html # past_due + trial_ending warning banners +│ │ │ ├── plan_picker.html # plan selection page (Starter/Pro/Enterprise cards) +│ │ │ └── email/ # HTML billing lifecycle emails +│ │ │ ├── base.html # shared branded email layout +│ │ │ ├── payment_failed.html +│ │ │ ├── trial_ending.html +│ │ │ └── subscription_cancelled.html +│ │ ├── signup/ +│ │ │ ├── index.html # self-service signup form (standalone, no base.html) +│ │ │ └── success.html # post-signup confirmation │ │ └── ... │ ├── static/ │ │ └── uploads/ # UPLOAD_FOLDER root @@ -163,6 +180,7 @@ lt_janitorial_quality_control/ │ ├── models.py # Plan, PlanFeature, Tenant, TenantDomain, │ │ # Superadmin, ProvisioningJob, TenantAudit │ ├── provision.py # MT-3 — create_tenant / register_tenant_zero / delete_tenant +│ ├── backup.py # per-tenant mysqldump backup CLI (python -m control.backup) │ ├── seed.py # idempotent plan seeder │ ├── tenant_migrate.py # MT-2 — upgrade_tenant / bootstrap_tenant / chain_head CLI │ ├── time_utils.py # now_eastern() mirror (no app import) @@ -171,9 +189,10 @@ lt_janitorial_quality_control/ │ │ ├── auth.py # /login, /logout (session-based, no Flask-Login) │ │ ├── decorators.py # @superadmin_required │ │ ├── impersonate.py # HMAC-SHA256 signed token generator/validator -│ │ ├── tenants.py # CRUD + plan/suspend/resume/domain/migrate/provision routes +│ │ ├── tenants.py # CRUD + plan/suspend/resume/domain/migrate/provision/billing routes +│ │ ├── health.py # GET /health/ — monitoring dashboard (tenant status, schema, trials) │ │ ├── wsgi_panel.py # WSGI entry point — Gunicorn on port 8001 -│ │ └── templates/panel/ # base.html, login.html, tenants_list.html, tenant_detail.html, provision.html +│ │ └── templates/panel/ # base.html, login.html, tenants_list.html, tenant_detail.html, provision.html, health.html │ └── migrations/ # control Alembic chain (control{N}_…) │ └── versions/control0001_init.py ← HEAD └── ... @@ -210,6 +229,13 @@ lt_janitorial_quality_control/ | `TENANT_ENGINE_POOL_RECYCLE` | Pool recycle in seconds (default 1800). | | `PANEL_SECRET_KEY` | Flask secret for the superadmin panel app (separate from `SECRET_KEY`). Generate: `python -c "import secrets; print(secrets.token_hex(32))"` | | `PANEL_IMPERSONATE_KEY` | HMAC key for impersonation tokens. Must be identical between panel and main app env. Generate same way. | +| `BILLING_ENABLED` | `false` by default. Set `true` to activate Stripe billing UI, gate, and webhooks. | +| `STRIPE_SECRET_KEY` | Stripe secret key (`sk_live_…` or `sk_test_…`). Required when `BILLING_ENABLED=true`. | +| `STRIPE_PUBLISHABLE_KEY` | Stripe publishable key (`pk_…`). Used in Checkout redirect. | +| `STRIPE_WEBHOOK_SECRET` | Stripe webhook signing secret (`whsec_…`). Validates webhook payloads. | +| `STRIPE_PRICE_STARTER` | Stripe Price ID for the Starter plan (`price_…`). Set via `python -m control.cli seed`. | +| `STRIPE_PRICE_PRO` | Stripe Price ID for the Pro plan. | +| `STRIPE_PRICE_ENTERPRISE` | Stripe Price ID for the Enterprise plan. | ### Email SSL Auto-Detection @@ -439,6 +465,8 @@ api_device_tokens: id, user_id, device_id, apns_token, device_name, app_version | `api_comments` | `/api/v1` | `GET /issues//comments`, `POST /issues//comments` (Phase D) | | `api_discovery` | `/api/v1` | `GET /discover?subdomain=` or `?email=`, `GET /tenant` — MT-9 server side (public, no auth, exempt from tenant middleware) | | `tenant_settings` | `/settings` | MT-7: `GET/POST /branding`, `GET /plan`, `GET /domains`, `POST /domains/request`, `POST /domains//delete` | +| `billing` | `/billing` | `GET /subscribe`, `POST /subscribe`, `GET /portal`, `POST /webhook`, `GET /suspended` | +| `signup` | `/signup` | `GET /`, `POST /` — public self-service signup (exempt from tenant middleware) | --- @@ -1174,7 +1202,7 @@ set -a; . /etc/jqc/control.env; set +a --- -## 22. Multi-Tenant Architecture (MT-0 → MT-7) +## 22. Multi-Tenant Architecture (MT-0 → MT-8) See `MULTI_TENANT_PLAN.md` for the full phased roadmap. This section summarises what is built and operational. @@ -1331,7 +1359,132 @@ curl -sI -H "Host: ztest.jqc.app" http://127.0.0.1:8000/ | head -2 --- -## 23. Coding Rules for AI Assistants +## 23. Billing System (MT-8) + +### Overview + +Stripe-backed subscription billing. Controlled by `BILLING_ENABLED` env var (default `false`). When enabled, a `_billing_gate()` runs in `init_tenancy()` before every request and enforces trial expiry and cancellation state. + +### Key files + +| File | Purpose | +|---|---| +| `app/billing/__init__.py` | Blueprint registration | +| `app/billing/routes.py` | `subscribe`, `portal`, `webhook`, `suspended` routes | +| `app/billing/webhooks.py` | Stripe event handlers (`payment_failed`, `subscription_updated`, `subscription_deleted`) | +| `app/billing/emails.py` | `send_billing_email()` — HTML + plain text dunning emails in background thread | +| `app/templates/billing/` | Plan picker, billing banner partial, HTML email templates | +| `control/seed.py` | Reads `STRIPE_PRICE_*` from env and writes to `Plan.stripe_price_id` | + +### Billing gate (`_billing_gate()` in `app/tenancy/middleware.py`) + +| `subscription_status` | Behaviour | +|---|---| +| `None` or `active` | Allow through | +| `trial` (not expired) | Allow; set `g.billing_warning = 'trial_ending'` if ≤ 3 days remain | +| `trial` (expired) | Redirect to `/billing/subscribe` (except `/billing/` and `/settings/`) | +| `past_due` | Allow; set `g.billing_warning = 'past_due'` | +| `cancelled` | Redirect to `/billing/suspended` | + +### Trial period + +- New tenants provisioned via `create_tenant()` get `subscription_status='trial'`, `trial_ends_at = now + 14 days` by default. +- Self-service signup (`/signup`) provisions a tenant immediately with a 14-day trial. +- Trial enforcement is in `_billing_gate()` — no Stripe required until they subscribe. + +### Billing emails (`app/billing/emails.py`) + +`send_billing_email(to_addr, event_type, context_dict)` sends multipart HTML + plain text. + +| `event_type` | Trigger | Required context keys | +|---|---|---| +| `payment_failed` | `invoice.payment_failed` Stripe webhook | `portal_url` | +| `trial_ending` | Cron (not yet wired — see §11 cron table) | `trial_ends_at`, `subscribe_url` | +| `subscription_cancelled` | `customer.subscription.deleted` Stripe webhook | `portal_url` | + +HTML templates live in `app/templates/billing/email/`. `base.html` provides the branded layout; each event type extends it. + +### Plan seeding + +```bash +# Add to .env: +STRIPE_PRICE_STARTER=price_... +STRIPE_PRICE_PRO=price_... +STRIPE_PRICE_ENTERPRISE=price_... + +# Then run: +python -m control.cli seed +python -m control.cli list-plans # verify price IDs +``` + +### Superadmin billing controls (panel `POST /tenants//billing`) + +| Action | Form field | Effect | +|---|---|---| +| `set_trial` | `trial_days` (int) | Sets `subscription_status=trial`, `trial_ends_at=now+N` | +| `set_status` | `subscription_status` | Overrides status directly | +| `apply_coupon` | `coupon_id` | Calls `stripe.Customer.modify(..., coupon=id)` | + +All three actions write a `TenantAudit` row. + +### Invoice history + +`GET /settings/plan` fetches last 10 Stripe invoices for the tenant's `stripe_customer_id` and passes them to `plan.html`. Shows date, amount, status badge, PDF link, hosted invoice link. + +### Self-service signup (`/signup`) + +Public route, exempt from tenant middleware. `SignupForm` validates: +- Company name, full name, email +- Subdomain: DNS label regex + uniqueness check against control DB +- Plan picker (Starter / Pro / Enterprise) +- Password + confirm + +On submit calls `create_tenant(admin_password=pw, trial_days=14)`. Shows `signup/success.html` with workspace URL and trial end date. + +--- + +## 24. Backup CLI (`control/backup.py`) + +Per-tenant MySQL backup using `mysqldump`. + +```bash +# Back up all tenants +python -m control.backup --tenant all --output-dir /var/backups/jqc + +# Back up one tenant +python -m control.backup --tenant acme --output-dir /var/backups/jqc + +# List tenants +python -m control.backup --list +``` + +Output: `_YYYYMMDD_HHMMSS.sql.gz` (gzip-compressed SQL dump). + +Flags passed to `mysqldump`: `--single-transaction`, `--routines`, `--triggers`, `--set-gtid-purged=OFF`. + +**Cron for nightly backup:** +```bash +0 2 * * * set -a; . /etc/jqc/control.env; set +a; \ + python -m control.backup --tenant all --output-dir /var/backups/jqc +``` + +--- + +## 25. Health Dashboard (`/health/` on panel) + +New nav item in the superadmin panel sidebar. Reads control DB only — no per-tenant DB queries. + +Shows: +- Summary cards: total tenants, active, suspended, schema behind +- Subscription breakdown (count per status) +- Trial alerts: expired trials, trials expiring within 3 days +- Full tenant table: slug, name, status, plan, subscription status, trial expiry countdown, Stripe presence, schema freshness, created date + +Row highlights: yellow = suspended, red = trial expired. + +--- + +## 26. Coding Rules for AI Assistants These rules apply to every change made to this codebase, without exception. diff --git a/SERVER_DUPLICATION.md b/SERVER_DUPLICATION.md new file mode 100644 index 0000000..3c04d88 --- /dev/null +++ b/SERVER_DUPLICATION.md @@ -0,0 +1,1398 @@ +# JQC Server Duplication Runbook + +**Purpose:** Step-by-step guide to duplicate the JQC production server to a new server. +**Audience:** Anyone who needs to migrate, clone, or recover the JQC production environment. +**Last updated:** June 2026 + +> **IMPORTANT:** Read the entire runbook before executing any steps. Steps 3–6 must be completed during a maintenance window or with the old server still serving traffic. DNS cutover (Step 10) is the moment of switchover — plan accordingly. + +--- + +## Table of Contents + +1. [Prerequisites](#1-prerequisites) +2. [Step 1 — Prepare the New Server](#step-1--prepare-the-new-server) +3. [Step 2 — Copy Application Code](#step-2--copy-application-code) +4. [Step 3 — Export Databases from Old Server](#step-3--export-databases-from-old-server) +5. [Step 4 — Restore Databases on New Server](#step-4--restore-databases-on-new-server) +6. [Step 5 — Copy Environment Files](#step-5--copy-environment-files) +7. [Step 6 — Copy Uploads](#step-6--copy-uploads) +8. [Step 7 — Configure systemd Services](#step-7--configure-systemd-services) +9. [Step 8 — Configure Nginx](#step-8--configure-nginx) +10. [Step 9 — TLS Certificates](#step-9--tls-certificates) +11. [Step 10 — DNS Cutover](#step-10--dns-cutover) +12. [Step 11 — Smoke Test](#step-11--smoke-test) +13. [Step 12 — Update Stripe Webhook](#step-12--update-stripe-webhook) +14. [Step 13 — Cron Jobs](#step-13--cron-jobs) +15. [Rollback Plan](#rollback-plan) +16. [Post-Cutover Checklist](#post-cutover-checklist) + +--- + +## 1. Prerequisites + +Gather everything on this list before you begin. Attempting the migration without these will require improvisation mid-procedure. + +### Access + +- [ ] SSH access to the **old server** (root or sudo) +- [ ] SSH access to the **new server** (root or sudo) +- [ ] SSH key pair — you will need to copy files between the two servers +- [ ] Access to the **DNS provider** control panel (to update A records at cutover) +- [ ] Access to the **Stripe Dashboard** (to re-register the webhook endpoint) +- [ ] Access to the **domain registrar** if DNS is managed there + +### New Server Specifications + +Minimum recommended specs (match or exceed old server): + +| Resource | Minimum | Recommended | +|---|---|---| +| CPU | 2 vCPUs | 4 vCPUs | +| RAM | 4 GB | 8 GB | +| Disk | 40 GB SSD | 80 GB SSD | +| OS | Ubuntu 22.04 LTS | Ubuntu 22.04 LTS | +| Network | 1 Gbps | 1 Gbps | + +### Information to collect from the old server before starting + +Run these on the **old server** and record the outputs: + +```bash +# Current MySQL version +mysql --version + +# Python version +python3 --version + +# List all tenant databases (you will need these names) +mysql -u root -p -e "SHOW DATABASES LIKE 'jqc_%';" + +# List all MySQL users (you will need to recreate these) +mysql -u root -p -e "SELECT user, host FROM mysql.user WHERE user LIKE 'jqc%';" + +# Current HEAD migration revision (record this) +cd /home/jqc/ +source /etc/jqc/control.env +source /etc/jqc/app.env +./venv/bin/flask db current + +# Record current public IP of old server +curl -s https://api.ipify.org + +# Record the number of tenant DBs to verify counts after restore +mysql -u root -p -e "SELECT COUNT(*) FROM jqc_control.tenants;" +``` + +> **Note:** Write down the MySQL root password if you do not have it stored. You will need it on the new server to create users and restore databases. + +--- + +## Step 1 — Prepare the New Server + +All commands in this section run on the **new server** as root (or with `sudo`). + +### 1.1 Update the OS and install base packages + +```bash +apt update && apt upgrade -y + +apt install -y \ + build-essential \ + curl \ + git \ + gnupg \ + htop \ + lsb-release \ + nginx \ + pkg-config \ + python3.11 \ + python3.11-dev \ + python3.11-venv \ + python3-pip \ + redis-server \ + rsync \ + software-properties-common \ + ufw \ + unzip \ + vim +``` + +### 1.2 Install MySQL 8.0 + +```bash +# Add the official MySQL APT repository +curl -fsSL https://dev.mysql.com/get/mysql-apt-config_0.8.29-1_all.deb -o /tmp/mysql-apt.deb +dpkg -i /tmp/mysql-apt.deb +# When prompted: select MySQL 8.0, then OK + +apt update +apt install -y mysql-server + +# Secure the installation +mysql_secure_installation +# Answer: set root password, remove anonymous users (Y), disallow remote root (Y), +# remove test database (Y), reload privilege tables (Y) + +# Start and enable MySQL +systemctl enable mysql +systemctl start mysql + +# Verify +mysql --version +``` + +### 1.3 Configure Redis + +```bash +# Redis is installed; configure it to start on boot +systemctl enable redis-server +systemctl start redis-server + +# Verify Redis is running +redis-cli ping +# Expected output: PONG +``` + +### 1.4 Install Certbot + +```bash +snap install --classic certbot +ln -sf /snap/bin/certbot /usr/bin/certbot + +# Install the DNS plugin for your DNS provider. +# Example for Cloudflare: +snap install certbot-dns-cloudflare + +# Example for Route53: +snap install certbot-dns-route53 + +# Verify +certbot --version +``` + +> **Note:** The specific certbot DNS plugin depends on your DNS provider. See https://certbot.eff.org/docs/using.html#dns-plugins for the full list. The wildcard cert in Step 9 requires DNS-01 challenge — HTTP-01 will not work for wildcards. + +### 1.5 Configure the firewall + +```bash +ufw default deny incoming +ufw default allow outgoing +ufw allow ssh +ufw allow 'Nginx Full' +ufw enable + +# Verify +ufw status +``` + +### 1.6 Create the `jqc` system user + +```bash +# Create system user with no login shell and a home directory +useradd --system --create-home --home-dir /home/jqc/ --shell /bin/bash jqc + +# Verify +id jqc +``` + +### 1.7 Create the directory structure + +```bash +# Create all required directories +mkdir -p /home/jqc/logs +mkdir -p /home/jqc/app/static/uploads/inspection_photos +mkdir -p /home/jqc/app/static/uploads/issue_photos +mkdir -p /home/jqc/app/static/uploads/issue_result_photos +mkdir -p /home/jqc/app/static/uploads/tenant_logos +mkdir -p /var/backups/jqc +mkdir -p /etc/jqc + +# Set ownership +chown -R jqc:jqc /home/jqc/ +chown -R jqc:jqc /var/backups/jqc + +# Set permissions on env directory (root owns it, readable by jqc) +chown root:jqc /etc/jqc +chmod 750 /etc/jqc +``` + +--- + +## Step 2 — Copy Application Code + +### Option A — Clone from Git (preferred) + +If the codebase is in a private Git repository: + +```bash +# Switch to jqc user +su - jqc + +# Clone the repository +cd /home/jqc/ +git clone https://github.com/YOUR_ORG/janitorial_qc_multi.git . + +# If the repo uses SSH keys, ensure jqc user has access +# Place the deploy key at /home/jqc/.ssh/id_rsa (chmod 600) + +# Verify the checkout +ls /home/jqc/app/ +``` + +### Option B — rsync from old server + +If Git is not available or the repo has uncommitted changes that need to be copied: + +```bash +# Run on the NEW server as root +# Replace OLD_SERVER_IP with the actual IP of the old server + +rsync -avz --exclude='venv/' \ + --exclude='__pycache__/' \ + --exclude='*.pyc' \ + --exclude='app/static/uploads/' \ + --exclude='logs/' \ + root@OLD_SERVER_IP:/home/jqc/ \ + /home/jqc/ + +chown -R jqc:jqc /home/jqc/ +``` + +### 2.2 Create and populate the Python virtual environment + +```bash +su - jqc +cd /home/jqc/ + +# Create the virtualenv using Python 3.11 +python3.11 -m venv venv + +# Activate and install dependencies +source venv/bin/activate +pip install --upgrade pip +pip install -r requirements.txt + +# Verify key packages installed +pip show flask gunicorn flask-sqlalchemy flask-migrate pymysql reportlab openpyxl +``` + +> **Note:** If `requirements.txt` is missing, run `pip freeze` on the old server (inside its venv) to generate one, then copy it over before running `pip install`. + +--- + +## Step 3 — Export Databases from Old Server + +Run all commands in this section on the **old server**. + +### 3.1 Source the control environment + +```bash +set -a; . /etc/jqc/control.env; set +a +``` + +### 3.2 Create the backup directory + +```bash +mkdir -p /var/backups/jqc/migration_$(date +%Y%m%d) +BACKUP_DIR=/var/backups/jqc/migration_$(date +%Y%m%d) +echo "Backup directory: $BACKUP_DIR" +``` + +### 3.3 Export the control plane database + +```bash +mysqldump \ + --single-transaction \ + --routines \ + --triggers \ + --set-gtid-purged=OFF \ + -u root -p \ + jqc_control \ + > $BACKUP_DIR/jqc_control.sql + +echo "Control DB dump size: $(du -sh $BACKUP_DIR/jqc_control.sql)" +``` + +### 3.4 Export all tenant databases + +Get the list of tenant databases from the control plane, then dump each one: + +```bash +# Get all tenant DB names from the control plane +cd /home/jqc/ +source venv/bin/activate + +python3 - <<'EOF' +import os, sys +sys.path.insert(0, '/home/jqc/') + +# Read directly from MySQL to avoid app startup dependency +import pymysql +url = os.environ['CONTROL_DATABASE_URL'] +# Parse: mysql+pymysql://user:pass@host/dbname +import re +m = re.match(r'mysql\+pymysql://([^:]+):([^@]+)@([^/]+)/(.+)', url) +user, password, host, db = m.groups() + +conn = pymysql.connect(host=host, user=user, password=password, database=db) +with conn.cursor() as cur: + cur.execute("SELECT slug, db_name FROM tenants WHERE active = 1") + rows = cur.fetchall() +conn.close() + +for slug, db_name in rows: + print(f"{slug}:{db_name}") +EOF +``` + +For each tenant database listed, run a dump. Replace the names with the actual output from above: + +```bash +# Dump each tenant DB — repeat this block for every tenant +# Example: TENANT_SLUG=lts TENANT_DB=jqc_lt + +for TENANT_DB in $(mysql -u root -p -N -e "SHOW DATABASES LIKE 'jqc_%';" | grep -v jqc_control); do + echo "Dumping $TENANT_DB..." + mysqldump \ + --single-transaction \ + --routines \ + --triggers \ + --set-gtid-purged=OFF \ + -u root -p \ + $TENANT_DB \ + > $BACKUP_DIR/${TENANT_DB}.sql + echo " Done: $(du -sh $BACKUP_DIR/${TENANT_DB}.sql)" +done + +echo "All dumps complete. Files in $BACKUP_DIR:" +ls -lh $BACKUP_DIR/ +``` + +### 3.5 Export MySQL user credentials + +You need to recreate the MySQL users on the new server. Record the current grants: + +```bash +mysql -u root -p -e " +SELECT CONCAT('SHOW GRANTS FOR ''', user, '''@''', host, ''';') +FROM mysql.user +WHERE user LIKE 'jqc%' +\G" | grep -v "^\*" | mysql -u root -p > $BACKUP_DIR/user_grants.sql 2>/dev/null + +# Also print to screen for manual inspection +mysql -u root -p -e "SELECT user, host, authentication_string FROM mysql.user WHERE user LIKE 'jqc%';" +``` + +> **Warning:** MySQL user passwords are hashed and cannot be extracted in plaintext from `mysql.user`. You have two options: +> 1. **Reset passwords on new server** — create new passwords for all MySQL users and update `/etc/jqc/app.env` and `/etc/jqc/control.env` accordingly. This is recommended. +> 2. **Copy the auth plugin + hash** — advanced; version-dependent. Only do this if both servers run the same MySQL version. + +### 3.6 Record the list of per-tenant MySQL users + +```bash +mysql -u root -p -e " +SELECT t.slug, t.db_name, t.db_user +FROM tenants t +ORDER BY t.slug;" jqc_control +``` + +Save this output — you will recreate each `db_user` in Step 4. + +### 3.7 Copy all dumps to the new server + +```bash +# Run on OLD server — push dumps to new server +NEW_SERVER_IP=YOUR_NEW_SERVER_IP + +rsync -avz --progress \ + $BACKUP_DIR/ \ + root@$NEW_SERVER_IP:/var/backups/jqc/migration_restore/ + +echo "Transfer complete." +``` + +--- + +## Step 4 — Restore Databases on New Server + +Run all commands in this section on the **new server**. + +### 4.1 Create MySQL users + +Log into MySQL on the new server as root: + +```bash +mysql -u root -p +``` + +Inside the MySQL shell, create all required users. Use **new, strong passwords** and update the environment files in Step 5 to match. + +```sql +-- ---------------------------------------------------------------- +-- Control plane user (owns jqc_control database) +-- ---------------------------------------------------------------- +CREATE USER 'jqc_control'@'127.0.0.1' IDENTIFIED BY 'CHOOSE_STRONG_PASSWORD_1'; +CREATE DATABASE IF NOT EXISTS jqc_control CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci; +GRANT ALL PRIVILEGES ON jqc_control.* TO 'jqc_control'@'127.0.0.1'; + +-- ---------------------------------------------------------------- +-- Provisioner user (creates new tenant DBs and users) +-- Must have WITH GRANT OPTION and CREATE USER +-- ---------------------------------------------------------------- +CREATE USER 'jqc_provisioner'@'127.0.0.1' IDENTIFIED BY 'CHOOSE_STRONG_PASSWORD_2'; +GRANT ALL PRIVILEGES ON *.* TO 'jqc_provisioner'@'127.0.0.1' WITH GRANT OPTION; +GRANT CREATE USER ON *.* TO 'jqc_provisioner'@'127.0.0.1'; + +-- ---------------------------------------------------------------- +-- Per-tenant users — repeat for EACH tenant from Step 3.6 +-- Replace db_name and db_user with values from the tenant list +-- ---------------------------------------------------------------- + +-- Example for tenant-zero (LT): +CREATE USER 'jqc_lts_user'@'127.0.0.1' IDENTIFIED BY 'CHOOSE_STRONG_PASSWORD_3'; +CREATE DATABASE IF NOT EXISTS jqc_lt CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci; +GRANT ALL PRIVILEGES ON jqc_lt.* TO 'jqc_lts_user'@'127.0.0.1'; + +-- Example for a second tenant (repeat as needed): +-- CREATE USER 'jqc_acme_user'@'127.0.0.1' IDENTIFIED BY 'CHOOSE_STRONG_PASSWORD_N'; +-- CREATE DATABASE IF NOT EXISTS jqc_acme CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci; +-- GRANT ALL PRIVILEGES ON jqc_acme.* TO 'jqc_acme_user'@'127.0.0.1'; + +FLUSH PRIVILEGES; +EXIT; +``` + +> **Note:** The per-tenant `db_user` and `db_name` values must match exactly what is stored (encrypted) in `jqc_control.tenants`. You are restoring both the data and the control plane DB, so the usernames must match the records in the dump. Use the output from Step 3.6 to get the exact usernames. + +### 4.2 Restore the control plane database + +```bash +RESTORE_DIR=/var/backups/jqc/migration_restore + +mysql -u root -p jqc_control < $RESTORE_DIR/jqc_control.sql + +echo "Control DB restored." +mysql -u root -p -e "SELECT COUNT(*) as tenant_count FROM jqc_control.tenants;" +``` + +### 4.3 Restore all tenant databases + +```bash +# Restore each tenant DB dump +for DUMP_FILE in $RESTORE_DIR/jqc_*.sql; do + # Skip the control DB — already restored + [[ "$DUMP_FILE" == *"jqc_control.sql" ]] && continue + + DB_NAME=$(basename "$DUMP_FILE" .sql) + echo "Restoring $DB_NAME from $DUMP_FILE..." + + # Ensure the database exists (may already be created above) + mysql -u root -p -e "CREATE DATABASE IF NOT EXISTS \`$DB_NAME\` CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;" + + mysql -u root -p $DB_NAME < $DUMP_FILE + echo " Done: $DB_NAME" +done +``` + +### 4.4 Verify table counts + +For each database, spot-check that table counts match the old server: + +```bash +# Check control DB +mysql -u root -p -e " +SELECT table_name, table_rows +FROM information_schema.tables +WHERE table_schema = 'jqc_control' +ORDER BY table_name;" 2>/dev/null + +# Check a tenant DB (replace jqc_lt with actual name) +mysql -u root -p -e " +SELECT table_name, table_rows +FROM information_schema.tables +WHERE table_schema = 'jqc_lt' +ORDER BY table_name;" 2>/dev/null +``` + +Compare these counts to the same query run on the old server. Row counts from `information_schema.tables` are estimates for InnoDB, so small discrepancies are normal — focus on table count and that key tables (`users`, `inspections`, `issues`, `tenants`) are not zero. + +### 4.5 Update the encrypted DB passwords in the control plane + +> **CRITICAL:** Because you created new MySQL passwords in Step 4.1, the `db_password_enc` column in `jqc_control.tenants` still holds the *old* encrypted passwords. The provisioner will fail to connect. You must re-encrypt the new passwords. + +```bash +cd /home/jqc/ +source /etc/jqc/control.env # must have CONTROL_FERNET_KEY set + +# For each tenant, re-encrypt the new password +python3 - <<'EOF' +import sys, os +sys.path.insert(0, '/home/jqc/') +from control.crypto import encrypt + +# Replace these with the actual slug → new_password mappings from Step 4.1 +tenant_passwords = { + 'lts': 'CHOOSE_STRONG_PASSWORD_3', + # 'acme': 'CHOOSE_STRONG_PASSWORD_N', +} + +for slug, pw in tenant_passwords.items(): + enc = encrypt(pw) + print(f"UPDATE tenants SET db_password_enc = '{enc}' WHERE slug = '{slug}';") +EOF +``` + +Copy the generated SQL statements and run them against `jqc_control`: + +```bash +mysql -u root -p jqc_control -e " +UPDATE tenants SET db_password_enc = 'ENCRYPTED_VALUE_FROM_ABOVE' WHERE slug = 'lts'; +-- repeat for each tenant +" +``` + +--- + +## Step 5 — Copy Environment Files + +### 5.1 Copy from old server to new server + +```bash +# Run on OLD server +NEW_SERVER_IP=YOUR_NEW_SERVER_IP + +scp /etc/jqc/app.env root@$NEW_SERVER_IP:/etc/jqc/app.env +scp /etc/jqc/control.env root@$NEW_SERVER_IP:/etc/jqc/control.env +``` + +### 5.2 Set correct permissions on new server + +```bash +# Run on NEW server +chmod 640 /etc/jqc/app.env +chmod 640 /etc/jqc/control.env +chown root:jqc /etc/jqc/app.env +chown root:jqc /etc/jqc/control.env +``` + +### 5.3 Update values that change with the new server + +Edit both files and update the following: + +```bash +vim /etc/jqc/app.env +``` + +Update in `/etc/jqc/app.env`: + +| Variable | What to change | +|---|---| +| `DATABASE_URL` | Update host from old server IP to `127.0.0.1` (if DB is local) or new DB host | +| `APP_BASE_URL` | Update if the domain changes (e.g. from old IP to new domain) | +| `REDIS_URL` | Update to `redis://127.0.0.1:6379/0` if Redis is local on new server | +| `DATABASE_URL` password | Update to the new MySQL password set in Step 4.1 | + +```bash +vim /etc/jqc/control.env +``` + +Update in `/etc/jqc/control.env`: + +| Variable | What to change | +|---|---| +| `CONTROL_DATABASE_URL` | Update host and password (use `jqc_control` user + new password from Step 4.1) | +| `PROVISION_DB_URL` | Update host and password (use `jqc_provisioner` user + new password from Step 4.1) | +| `MULTI_TENANT_ENABLED` | Leave as `true` if old server had it enabled | + +> **CRITICAL:** `CONTROL_FERNET_KEY` must be **identical** between the old and new servers. This key encrypts the tenant DB passwords stored in `jqc_control.tenants`. Copy it exactly — do not regenerate it. If you regenerate it, all tenant DB connections will fail with `InvalidToken` until you re-encrypt every password. + +### 5.4 Verify the environment files parse correctly + +```bash +# Source both files and check for syntax errors +set -a +. /etc/jqc/app.env +. /etc/jqc/control.env +set +a + +echo "DATABASE_URL is set: ${DATABASE_URL:+yes}" +echo "SECRET_KEY is set: ${SECRET_KEY:+yes}" +echo "CONTROL_FERNET_KEY is set: ${CONTROL_FERNET_KEY:+yes}" +echo "MULTI_TENANT_ENABLED: $MULTI_TENANT_ENABLED" +``` + +--- + +## Step 6 — Copy Uploads + +User-uploaded files (inspection photos, issue photos, tenant logos) must be copied from the old server. + +> **Note:** This can be a large transfer depending on how long the system has been running. Run it before the DNS cutover window and optionally run it again immediately before cutover to sync any files uploaded since the first rsync. + +```bash +# Run on NEW server (pulls from old server) +OLD_SERVER_IP=YOUR_OLD_SERVER_IP + +rsync -avz --progress \ + root@$OLD_SERVER_IP:/home/jqc/app/static/uploads/ \ + /home/jqc/app/static/uploads/ + +# Fix ownership after rsync +chown -R jqc:jqc /home/jqc/app/static/uploads/ + +# Verify counts match +echo "New server upload file count:" +find /home/jqc/app/static/uploads/ -type f | wc -l + +echo "Old server upload file count (run this on old server separately):" +echo " find /home/jqc/app/static/uploads/ -type f | wc -l" +``` + +--- + +## Step 7 — Configure systemd Services + +### 7.1 Create the main application service + +```bash +cat > /etc/systemd/system/jqc.service << 'EOF' +[Unit] +Description=JQC Web Application +After=network.target mysql.service redis.service + +[Service] +User=jqc +Group=jqc +WorkingDirectory=/home/jqc/ +EnvironmentFile=/etc/jqc/app.env +EnvironmentFile=/etc/jqc/control.env +ExecStart=/home/jqc/venv/bin/gunicorn wsgi:app \ + --bind 127.0.0.1:8000 \ + --workers 5 \ + --worker-class sync \ + --timeout 30 \ + --log-file /home/jqc/logs/gunicorn.log \ + --log-level info +ExecReload=/bin/kill -s HUP $MAINPID +Restart=always +RestartSec=5 + +[Install] +WantedBy=multi-user.target +EOF +``` + +### 7.2 Create the superadmin panel service + +```bash +cat > /etc/systemd/system/jqc-panel.service << 'EOF' +[Unit] +Description=JQC Superadmin Control Panel +After=network.target mysql.service + +[Service] +User=jqc +Group=jqc +WorkingDirectory=/home/jqc/ +EnvironmentFile=/etc/jqc/app.env +EnvironmentFile=/etc/jqc/control.env +ExecStart=/home/jqc/venv/bin/gunicorn wsgi_panel:panel_app \ + --bind 127.0.0.1:8001 \ + --workers 2 \ + --worker-class sync \ + --timeout 30 \ + --log-file /home/jqc/logs/gunicorn-panel.log \ + --log-level info +ExecReload=/bin/kill -s HUP $MAINPID +Restart=always +RestartSec=5 + +[Install] +WantedBy=multi-user.target +EOF +``` + +### 7.3 Enable and start the services + +```bash +# Reload systemd to read new unit files +systemctl daemon-reload + +# Enable both services to start on boot +systemctl enable jqc +systemctl enable jqc-panel + +# Start the services +systemctl start jqc +systemctl start jqc-panel + +# Verify both are running +systemctl status jqc --no-pager +systemctl status jqc-panel --no-pager + +# Check the application logs for startup errors +tail -50 /home/jqc/logs/gunicorn.log +tail -50 /home/jqc/logs/gunicorn-panel.log +``` + +### 7.4 Verify the applications respond on their ports + +```bash +# Main app should return a redirect or HTML (not a connection refused error) +curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:8000/ +# Expected: 302 (redirect to login) or 200 + +# Panel app +curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:8001/ +# Expected: 302 or 200 +``` + +If you get `000` (connection refused), check the logs: + +```bash +journalctl -u jqc -n 50 --no-pager +journalctl -u jqc-panel -n 50 --no-pager +``` + +--- + +## Step 8 — Configure Nginx + +> **CRITICAL:** The `admin.jqc.app` server block **must appear before** the `*.jqc.app` wildcard block. Nginx matches `server_name` in order — if the wildcard block comes first, it catches `admin.jqc.app` requests and proxies them to port 8000 (main app) instead of port 8001 (panel app). + +### 8.1 Write the Nginx configuration + +```bash +cat > /etc/nginx/sites-available/jqc << 'NGINXCONF' +# ---------------------------------------------------------------- +# Redirect HTTP to HTTPS (all hosts) +# ---------------------------------------------------------------- +server { + listen 80; + server_name *.jqc.app jqc.app admin.jqc.app; + + location /.well-known/acme-challenge/ { + root /var/www/html; + } + + location / { + return 301 https://$host$request_uri; + } +} + +# ---------------------------------------------------------------- +# Superadmin Control Panel — admin.jqc.app +# MUST BE BEFORE the wildcard *.jqc.app block +# ---------------------------------------------------------------- +server { + listen 443 ssl; + server_name admin.jqc.app; + + ssl_certificate /etc/letsencrypt/live/admin.jqc.app/fullchain.pem; + ssl_certificate_key /etc/letsencrypt/live/admin.jqc.app/privkey.pem; + include /etc/letsencrypt/options-ssl-nginx.conf; + ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; + + client_max_body_size 50M; + + access_log /var/log/nginx/panel-access.log; + error_log /var/log/nginx/panel-error.log; + + location / { + proxy_pass http://127.0.0.1:8001; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_read_timeout 60s; + } +} + +# ---------------------------------------------------------------- +# Main application — *.jqc.app wildcard (all tenants) +# MUST BE AFTER the admin.jqc.app exact block +# ---------------------------------------------------------------- +server { + listen 443 ssl; + server_name *.jqc.app jqc.app; + + ssl_certificate /etc/letsencrypt/live/jqc.app/fullchain.pem; + ssl_certificate_key /etc/letsencrypt/live/jqc.app/privkey.pem; + include /etc/letsencrypt/options-ssl-nginx.conf; + ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; + + client_max_body_size 50M; + + access_log /var/log/nginx/jqc-access.log; + error_log /var/log/nginx/jqc-error.log; + + # Serve static files directly without hitting Gunicorn + location /static/ { + alias /home/jqc/app/static/; + expires 30d; + add_header Cache-Control "public, immutable"; + } + + location / { + proxy_pass http://127.0.0.1:8000; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_read_timeout 60s; + } +} +NGINXCONF +``` + +### 8.2 Enable the site and test the configuration + +```bash +# Remove the default site if present +rm -f /etc/nginx/sites-enabled/default + +# Enable the JQC site +ln -sf /etc/nginx/sites-available/jqc /etc/nginx/sites-enabled/jqc + +# Test the configuration +nginx -t +# Expected: "configuration file ... syntax is ok" and "test is successful" +``` + +> **Note:** Do NOT reload Nginx yet if TLS certificates do not exist. The configuration references certificate files that must be created in Step 9 first. You can skip the reload here and do it after Step 9. + +--- + +## Step 9 — TLS Certificates + +### 9.1 Obtain the wildcard certificate for *.jqc.app + +Wildcard certificates require DNS-01 challenge. The exact command depends on your DNS provider. + +**Example for Cloudflare:** + +```bash +# Create Cloudflare credentials file +cat > /etc/letsencrypt/cloudflare.ini << 'EOF' +# Cloudflare API token with Zone:DNS:Edit permission +dns_cloudflare_api_token = YOUR_CLOUDFLARE_API_TOKEN +EOF +chmod 600 /etc/letsencrypt/cloudflare.ini + +# Request wildcard + apex cert +certbot certonly \ + --dns-cloudflare \ + --dns-cloudflare-credentials /etc/letsencrypt/cloudflare.ini \ + -d "jqc.app" \ + -d "*.jqc.app" \ + --email YOUR_EMAIL@example.com \ + --agree-tos \ + --non-interactive +``` + +**Example for Route53:** + +```bash +certbot certonly \ + --dns-route53 \ + -d "jqc.app" \ + -d "*.jqc.app" \ + --email YOUR_EMAIL@example.com \ + --agree-tos \ + --non-interactive +``` + +### 9.2 Obtain the certificate for admin.jqc.app + +```bash +# admin.jqc.app is an exact subdomain — still use DNS-01 for consistency +certbot certonly \ + --dns-cloudflare \ + --dns-cloudflare-credentials /etc/letsencrypt/cloudflare.ini \ + -d "admin.jqc.app" \ + --email YOUR_EMAIL@example.com \ + --agree-tos \ + --non-interactive +``` + +### 9.3 Verify certificates issued + +```bash +certbot certificates +# Should show two certificate entries: +# 1. jqc.app covering *.jqc.app and jqc.app +# 2. admin.jqc.app + +ls /etc/letsencrypt/live/ +``` + +### 9.4 Set up auto-renewal + +```bash +# Test renewal (dry run) +certbot renew --dry-run + +# Certbot installs a systemd timer automatically on Ubuntu 22.04 +# Verify it is active +systemctl status certbot.timer + +# Add a post-renewal hook to reload Nginx after cert renewal +cat > /etc/letsencrypt/renewal-hooks/post/reload-nginx.sh << 'EOF' +#!/bin/bash +nginx -t && systemctl reload nginx +EOF +chmod +x /etc/letsencrypt/renewal-hooks/post/reload-nginx.sh +``` + +### 9.5 Start Nginx with TLS + +```bash +# Now that certs exist, test and reload Nginx +nginx -t && systemctl reload nginx + +# Verify Nginx is running +systemctl status nginx --no-pager +``` + +--- + +## Step 10 — DNS Cutover + +> **IMPORTANT:** Plan this step for a low-traffic period. Reduce the TTL on your DNS records to 60–300 seconds **at least 24 hours before cutover** so the change propagates quickly. After cutover is confirmed working, restore TTL to normal (3600+). + +### 10.1 Pre-cutover: reduce TTL (do this 24 hours early) + +In your DNS provider's control panel, reduce the TTL on the following records to 60 seconds: +- `*.jqc.app` A record +- `jqc.app` A record +- `admin.jqc.app` A record + +### 10.2 Record the old server IP for rollback + +```bash +# On OLD server +curl -s https://api.ipify.org +# Save this IP — you will need it for rollback +``` + +### 10.3 Get the new server IP + +```bash +# On NEW server +NEW_SERVER_IP=$(curl -s https://api.ipify.org) +echo "New server IP: $NEW_SERVER_IP" +``` + +### 10.4 Final data sync before cutover + +Immediately before switching DNS (while the old server is still live), do a final rsync of uploads to capture anything uploaded since Step 6: + +```bash +# On NEW server — pull latest uploads from old server +OLD_SERVER_IP=YOUR_OLD_SERVER_IP + +rsync -avz --progress \ + root@$OLD_SERVER_IP:/home/jqc/app/static/uploads/ \ + /home/jqc/app/static/uploads/ + +chown -R jqc:jqc /home/jqc/app/static/uploads/ +``` + +Optionally do a final database sync as well if the cutover window is long: + +```bash +# On OLD server — dump and transfer latest data +mysqldump --single-transaction -u root -p jqc_control > /tmp/jqc_control_final.sql +scp /tmp/jqc_control_final.sql root@$NEW_SERVER_IP:/tmp/ + +# For each tenant DB: +mysqldump --single-transaction -u root -p jqc_lt > /tmp/jqc_lt_final.sql +scp /tmp/jqc_lt_final.sql root@$NEW_SERVER_IP:/tmp/ + +# On NEW server — restore the final dumps +mysql -u root -p jqc_control < /tmp/jqc_control_final.sql +mysql -u root -p jqc_lt < /tmp/jqc_lt_final.sql +# Repeat for each tenant DB +``` + +### 10.5 Switch DNS + +In your DNS provider's control panel, update the following A records to point to the new server IP: + +| Record | Old IP | New IP | +|---|---|---| +| `*.jqc.app` | OLD_IP | NEW_IP | +| `jqc.app` | OLD_IP | NEW_IP | +| `admin.jqc.app` | OLD_IP | NEW_IP | + +### 10.6 Monitor propagation + +```bash +# On your local machine — check when DNS has propagated +watch -n 5 "dig +short *.jqc.app @8.8.8.8" + +# Or use online tools: +# https://dnschecker.org/#A/lts.jqc.app +# https://www.whatsmydns.net/ +``` + +--- + +## Step 11 — Smoke Test + +After DNS propagates to your location, run this checklist. + +### 11.1 Basic HTTPS access + +```bash +# Test main app (replace lts.jqc.app with your tenant-zero subdomain) +curl -sI https://lts.jqc.app/ | head -5 +# Expected: HTTP/2 200 or 302 + +# Test admin panel +curl -sI https://admin.jqc.app/ | head -5 +# Expected: HTTP/2 200 or 302 + +# Verify TLS certificate +echo | openssl s_client -connect lts.jqc.app:443 -servername lts.jqc.app 2>/dev/null \ + | openssl x509 -noout -dates -subject +``` + +### 11.2 Web application checklist + +Open a browser and verify: + +- [ ] `https://lts.jqc.app/auth/login` loads the login page (no 502/504 error) +- [ ] Login with admin credentials succeeds +- [ ] Dashboard loads and shows data (inspections, issues counts) +- [ ] Navigate to Inspections list — data appears +- [ ] Navigate to Issues list — data appears +- [ ] Navigate to Reports — charts load +- [ ] An existing uploaded photo (inspection or issue) is visible (not broken img tag) +- [ ] Tenant branding/logo appears in the navbar (if configured) + +### 11.3 Multi-tenant routing + +```bash +# Test that the tenant resolver is working +# A second tenant subdomain should respond (or return 404 "Workspace not found" +# if that tenant does not exist — both are correct, 502 is not) + +curl -sI https://SECOND_TENANT.jqc.app/ | head -3 +``` + +### 11.4 Admin panel + +- [ ] `https://admin.jqc.app/login` loads +- [ ] Login with superadmin credentials succeeds +- [ ] Tenant list shows all tenants +- [ ] Impersonation: click "Impersonate" on a tenant → app loads for that tenant + +### 11.5 Mobile API + +```bash +# Test the mobile API auth endpoint +curl -s -X POST https://lts.jqc.app/api/v1/auth/login \ + -H "Content-Type: application/json" \ + -d '{"username": "INSPECTOR_USERNAME", "password": "INSPECTOR_PASSWORD"}' \ + | python3 -m json.tool | head -20 +# Expected: JSON with access_token and refresh_token (not an HTML error page) +``` + +### 11.6 Static file serving + +```bash +# Test that Nginx serves a static file directly (not via Gunicorn) +# Replace the path with an actual uploaded file path you know exists +curl -sI https://lts.jqc.app/static/uploads/inspection_photos/SOME_FILE.jpg | head -5 +# Expected: HTTP/2 200, Content-Type: image/jpeg +``` + +--- + +## Step 12 — Update Stripe Webhook + +The Stripe webhook endpoint was registered with the old server's URL. After DNS cutover, update it to use the new URL (which should be the same domain if DNS was simply re-pointed, or a new domain if applicable). + +### 12.1 When the domain stays the same (DNS re-pointed) + +If the domain is unchanged (e.g. still `lts.jqc.app`), the webhook URL is the same and **no Stripe changes are needed**. Verify the webhook is still receiving events: + +1. Log in to the [Stripe Dashboard](https://dashboard.stripe.com) +2. Navigate to **Developers → Webhooks** +3. Click on the webhook endpoint for `/billing/webhook` +4. Check recent deliveries — they should show `200 OK` responses after cutover + +### 12.2 When the domain changes + +If you are moving to a new domain: + +1. Log in to the [Stripe Dashboard](https://dashboard.stripe.com) +2. Navigate to **Developers → Webhooks** +3. Click **Add endpoint** +4. Enter: `https://NEW_DOMAIN.com/billing/webhook` +5. Select the same events as the old webhook (at minimum: `checkout.session.completed`, `customer.subscription.updated`, `customer.subscription.deleted`, `invoice.payment_failed`, `invoice.paid`) +6. Copy the new **Signing secret** (starts with `whsec_`) +7. Update `/etc/jqc/app.env` on the new server: + +```bash +vim /etc/jqc/app.env +# Update: STRIPE_WEBHOOK_SECRET=whsec_NEW_SECRET_FROM_STRIPE +``` + +8. Restart the main app to reload the environment: + +```bash +systemctl restart jqc +``` + +9. In Stripe, click **Send test webhook** → select `checkout.session.completed` → verify the new server returns `200 OK` + +10. After confirming the new endpoint works, delete the old endpoint from Stripe. + +--- + +## Step 13 — Cron Jobs + +Set up the cron schedule for the `jqc` user. These jobs run application maintenance tasks. + +```bash +# Edit the crontab for the jqc user +crontab -u jqc -e +``` + +Add the following entries. Replace `YOUR_DIGEST_SECRET` with the value of `DIGEST_SECRET` from `/etc/jqc/app.env`, and `YOUR_DOMAIN` with the primary tenant domain (e.g. `lts.jqc.app`): + +```cron +# JQC scheduled tasks +# Environment +MAILTO="" +SHELL=/bin/bash + +# ---------------------------------------------------------------- +# SLA breach/at-risk alerts — every 30 minutes +# ---------------------------------------------------------------- +*/30 * * * * curl -s -X POST https://YOUR_DOMAIN/notifications/check-sla \ + -d "token=YOUR_DIGEST_SECRET" >> /home/jqc/logs/cron.log 2>&1 + +# ---------------------------------------------------------------- +# Daily email digest — 7:00 AM Eastern +# ---------------------------------------------------------------- +0 7 * * * curl -s -X POST https://YOUR_DOMAIN/notifications/send-digest \ + -d "token=YOUR_DIGEST_SECRET&frequency=daily" >> /home/jqc/logs/cron.log 2>&1 + +# ---------------------------------------------------------------- +# Weekly email digest — 7:00 AM Eastern on Mondays +# ---------------------------------------------------------------- +0 7 * * 1 curl -s -X POST https://YOUR_DOMAIN/notifications/send-digest \ + -d "token=YOUR_DIGEST_SECRET&frequency=weekly" >> /home/jqc/logs/cron.log 2>&1 + +# ---------------------------------------------------------------- +# Facility score-drop alerts — 8:00 AM Eastern +# ---------------------------------------------------------------- +0 8 * * * curl -s -X POST https://YOUR_DOMAIN/notifications/check-score-trends \ + -d "token=YOUR_DIGEST_SECRET" >> /home/jqc/logs/cron.log 2>&1 + +# ---------------------------------------------------------------- +# Scheduled reports email delivery — 8:05 AM Eastern (offset from score alerts) +# ---------------------------------------------------------------- +5 8 * * * curl -s -X POST https://YOUR_DOMAIN/scheduled-reports/run \ + -d "secret=YOUR_DIGEST_SECRET" >> /home/jqc/logs/cron.log 2>&1 + +# ---------------------------------------------------------------- +# Expired API token cleanup — 3:00 AM Eastern +# ---------------------------------------------------------------- +0 3 * * * curl -s -X POST https://YOUR_DOMAIN/notifications/cleanup-tokens \ + -d "token=YOUR_DIGEST_SECRET" >> /home/jqc/logs/cron.log 2>&1 + +# ---------------------------------------------------------------- +# Nightly database backup — 2:00 AM Eastern +# ---------------------------------------------------------------- +0 2 * * * /home/jqc/scripts/backup.sh >> /home/jqc/logs/backup.log 2>&1 +``` + +### 13.1 Create the backup script + +```bash +cat > /home/jqc/scripts/backup.sh << 'BACKUPSCRIPT' +#!/bin/bash +# JQC nightly database backup + +set -a +. /etc/jqc/app.env +. /etc/jqc/control.env +set +a + +BACKUP_DIR=/var/backups/jqc/$(date +%Y%m%d) +mkdir -p "$BACKUP_DIR" + +# Back up control database +mysqldump --single-transaction \ + --routines --triggers --set-gtid-purged=OFF \ + -u root -p"$MYSQL_ROOT_PASSWORD" \ + jqc_control \ + | gzip > "$BACKUP_DIR/jqc_control.sql.gz" + +# Back up all tenant databases +for DB in $(mysql -u root -p"$MYSQL_ROOT_PASSWORD" -N \ + -e "SHOW DATABASES LIKE 'jqc_%';" 2>/dev/null | grep -v jqc_control); do + mysqldump --single-transaction \ + --routines --triggers --set-gtid-purged=OFF \ + -u root -p"$MYSQL_ROOT_PASSWORD" \ + "$DB" \ + | gzip > "$BACKUP_DIR/${DB}.sql.gz" +done + +# Remove backups older than 14 days +find /var/backups/jqc/ -maxdepth 1 -type d -mtime +14 -exec rm -rf {} + + +echo "[$(date)] Backup complete: $BACKUP_DIR" +ls -lh "$BACKUP_DIR/" +BACKUPSCRIPT + +mkdir -p /home/jqc/scripts +chmod +x /home/jqc/scripts/backup.sh +chown jqc:jqc /home/jqc/scripts/backup.sh +``` + +> **Note:** The backup script uses a `MYSQL_ROOT_PASSWORD` variable. Add this to `/etc/jqc/app.env` or pass it another way. Alternatively, create a `.my.cnf` file in the `jqc` user's home directory: +> +> ```bash +> cat > /home/jqc/.my.cnf << 'EOF' +> [client] +> user=root +> password=YOUR_MYSQL_ROOT_PASSWORD +> EOF +> chmod 600 /home/jqc/.my.cnf +> chown jqc:jqc /home/jqc/.my.cnf +> ``` +> +> Then remove the `-p"$MYSQL_ROOT_PASSWORD"` flags from the backup script and use `-u root` only. + +### 13.2 Verify the crontab + +```bash +crontab -u jqc -l +``` + +### 13.3 Test one cron job manually + +```bash +# Run the SLA check manually to confirm it works +su - jqc -c 'curl -s -X POST https://YOUR_DOMAIN/notifications/check-sla \ + -d "token=YOUR_DIGEST_SECRET"' +# Expected: JSON response like {"checked": 5, "notified": 0} — not an HTML error page +``` + +--- + +## Rollback Plan + +If the new server has problems after DNS cutover, revert traffic to the old server. + +### Immediate rollback (DNS revert) + +The old server should remain running and untouched until you are fully confident in the new server. + +```bash +# In your DNS provider's control panel, immediately update the A records back: +# *.jqc.app → OLD_SERVER_IP +# jqc.app → OLD_SERVER_IP +# admin.jqc.app → OLD_SERVER_IP +``` + +Because you reduced TTL to 60 seconds in Step 10.1, propagation will be fast (typically under 5 minutes for most resolvers, up to TTL seconds for others). + +### Before reverting, capture any new data written to the new server + +If the new server was live for any period and received real user activity (logins, new inspections, new issues), you should extract that data before rolling back: + +```bash +# On NEW server — dump any tenant DBs that received writes since cutover +mysqldump --single-transaction -u root -p jqc_lt \ + > /tmp/jqc_lt_rollback_$(date +%Y%m%d_%H%M).sql + +# Transfer to old server and import (carefully — manual merge required if old server also had writes) +scp /tmp/jqc_lt_rollback_*.sql root@OLD_SERVER_IP:/tmp/ +``` + +> **Warning:** If both servers received writes simultaneously (e.g. during a partial DNS propagation window where some users hit old and some hit new), reconciling the data is a manual process. This is the main reason to keep the maintenance window short and confirm full propagation before allowing the old server to accept new writes. + +### Keep old server running for 48 hours after cutover + +Do not decommission or wipe the old server until: +- DNS has fully propagated globally (use dnschecker.org to verify) +- You have run the full smoke test on the new server +- At least 48 hours have passed with no issues +- You have taken a final backup from the old server + +--- + +## Post-Cutover Checklist + +Work through this list after DNS propagation is confirmed. Check each item in the browser and in the application logs. + +### Application health + +- [ ] All tenant subdomains resolve to new server IP (`dig +short lts.jqc.app`) +- [ ] `admin.jqc.app` resolves to new server IP +- [ ] HTTPS works on all domains (no certificate warnings) +- [ ] `systemctl status jqc` shows `active (running)` +- [ ] `systemctl status jqc-panel` shows `active (running)` +- [ ] `systemctl status nginx` shows `active (running)` +- [ ] No errors in `/home/jqc/logs/gunicorn.log` (last 100 lines) +- [ ] No errors in `/var/log/nginx/jqc-error.log` + +### Authentication and roles + +- [ ] Admin login works +- [ ] Director login works +- [ ] Inspector login works +- [ ] Customer login works (verify scoped data, not all facilities) +- [ ] Superadmin panel login works + +### Core features + +- [ ] Create a new inspection (full submit, not just draft) +- [ ] Flag an issue from an inspection +- [ ] Upload a photo to an issue +- [ ] Add a comment to an issue +- [ ] Customer can see their facility data but not others +- [ ] PDF export works (inspection PDF, issues list PDF) +- [ ] Excel export works (inspector performance) + +### Multi-tenant + +- [ ] At least two tenant subdomains respond correctly +- [ ] Impersonation from admin panel works +- [ ] Tenant branding (logo, colours) appears correctly + +### Notifications and integrations + +- [ ] In-app notification bell shows notifications +- [ ] Test email notification delivery (e.g. assign an issue → check that assigned user receives email) +- [ ] Stripe: open Stripe dashboard → Webhooks → verify recent deliveries show 200 OK +- [ ] Groq AI chat: navigate to `/support/chat` as a customer → verify chat responds (if `GROQ_API_KEY` is set) + +### Infrastructure + +- [ ] Redis is running: `redis-cli ping` returns `PONG` +- [ ] MySQL is running: `systemctl status mysql` +- [ ] Certbot timer is active: `systemctl status certbot.timer` +- [ ] Cron jobs are scheduled: `crontab -u jqc -l` +- [ ] Log rotation is working: `/home/jqc/logs/` is not filling up unexpectedly +- [ ] First nightly backup completed (check `/var/backups/jqc/` the morning after cutover) +- [ ] Firewall is active: `ufw status` shows correct rules + +### Decommission old server (after 48 hours) + +- [ ] Final backup taken from old server +- [ ] Old server's Stripe webhook endpoint deleted +- [ ] Old server's cron jobs removed or disabled +- [ ] Old server's DNS records confirmed no longer pointing to old IP +- [ ] Old server shut down or deprovisioned + +--- + +*End of JQC Server Duplication Runbook*