Files

T

nngo 42790c499c Jun 29 - Update CLAUDE.md and create SERVER_DUPLICATION.md

2026-06-29 09:45:30 -04:00

42 KiB

Raw Blame History

JQC Server Duplication Runbook

Purpose: Step-by-step guide to duplicate the JQC production server to a new server.
Audience: Anyone who needs to migrate, clone, or recover the JQC production environment.
Last updated: June 2026

IMPORTANT: Read the entire runbook before executing any steps. Steps 3–6 must be completed during a maintenance window or with the old server still serving traffic. DNS cutover (Step 10) is the moment of switchover — plan accordingly.

Prerequisites
Step 1 — Prepare the New Server
Step 2 — Copy Application Code
Step 3 — Export Databases from Old Server
Step 4 — Restore Databases on New Server
Step 5 — Copy Environment Files
Step 6 — Copy Uploads
Step 7 — Configure systemd Services
Step 8 — Configure Nginx
Step 9 — TLS Certificates
Step 10 — DNS Cutover
Step 11 — Smoke Test
Step 12 — Update Stripe Webhook
Step 13 — Cron Jobs
Rollback Plan
Post-Cutover Checklist

1. Prerequisites

Gather everything on this list before you begin. Attempting the migration without these will require improvisation mid-procedure.

Access

SSH access to the old server (root or sudo)
SSH access to the new server (root or sudo)
SSH key pair — you will need to copy files between the two servers
Access to the DNS provider control panel (to update A records at cutover)
Access to the Stripe Dashboard (to re-register the webhook endpoint)
Access to the domain registrar if DNS is managed there

New Server Specifications

Minimum recommended specs (match or exceed old server):

Resource	Minimum	Recommended
CPU	2 vCPUs	4 vCPUs
RAM	4 GB	8 GB
Disk	40 GB SSD	80 GB SSD
OS	Ubuntu 22.04 LTS	Ubuntu 22.04 LTS
Network	1 Gbps	1 Gbps

Information to collect from the old server before starting

Run these on the old server and record the outputs:

# Current MySQL version
mysql --version

# Python version
python3 --version

# List all tenant databases (you will need these names)
mysql -u root -p -e "SHOW DATABASES LIKE 'jqc_%';"

# List all MySQL users (you will need to recreate these)
mysql -u root -p -e "SELECT user, host FROM mysql.user WHERE user LIKE 'jqc%';"

# Current HEAD migration revision (record this)
cd /home/jqc/
source /etc/jqc/control.env
source /etc/jqc/app.env
./venv/bin/flask db current

# Record current public IP of old server
curl -s https://api.ipify.org

# Record the number of tenant DBs to verify counts after restore
mysql -u root -p -e "SELECT COUNT(*) FROM jqc_control.tenants;"

Note: Write down the MySQL root password if you do not have it stored. You will need it on the new server to create users and restore databases.

Step 1 — Prepare the New Server

All commands in this section run on the new server as root (or with sudo).

1.1 Update the OS and install base packages

apt update && apt upgrade -y

apt install -y \
    build-essential \
    curl \
    git \
    gnupg \
    htop \
    lsb-release \
    nginx \
    pkg-config \
    python3.11 \
    python3.11-dev \
    python3.11-venv \
    python3-pip \
    redis-server \
    rsync \
    software-properties-common \
    ufw \
    unzip \
    vim

1.2 Install MySQL 8.0

# Add the official MySQL APT repository
curl -fsSL https://dev.mysql.com/get/mysql-apt-config_0.8.29-1_all.deb -o /tmp/mysql-apt.deb
dpkg -i /tmp/mysql-apt.deb
# When prompted: select MySQL 8.0, then OK

apt update
apt install -y mysql-server

# Secure the installation
mysql_secure_installation
# Answer: set root password, remove anonymous users (Y), disallow remote root (Y),
#         remove test database (Y), reload privilege tables (Y)

# Start and enable MySQL
systemctl enable mysql
systemctl start mysql

# Verify
mysql --version

1.3 Configure Redis

# Redis is installed; configure it to start on boot
systemctl enable redis-server
systemctl start redis-server

# Verify Redis is running
redis-cli ping
# Expected output: PONG

1.4 Install Certbot

snap install --classic certbot
ln -sf /snap/bin/certbot /usr/bin/certbot

# Install the DNS plugin for your DNS provider.
# Example for Cloudflare:
snap install certbot-dns-cloudflare

# Example for Route53:
snap install certbot-dns-route53

# Verify
certbot --version

Note: The specific certbot DNS plugin depends on your DNS provider. See https://certbot.eff.org/docs/using.html#dns-plugins for the full list. The wildcard cert in Step 9 requires DNS-01 challenge — HTTP-01 will not work for wildcards.

1.5 Configure the firewall

ufw default deny incoming
ufw default allow outgoing
ufw allow ssh
ufw allow 'Nginx Full'
ufw enable

# Verify
ufw status

1.6 Create the `jqc` system user

# Create system user with no login shell and a home directory
useradd --system --create-home --home-dir /home/jqc/ --shell /bin/bash jqc

# Verify
id jqc

1.7 Create the directory structure

# Create all required directories
mkdir -p /home/jqc/logs
mkdir -p /home/jqc/app/static/uploads/inspection_photos
mkdir -p /home/jqc/app/static/uploads/issue_photos
mkdir -p /home/jqc/app/static/uploads/issue_result_photos
mkdir -p /home/jqc/app/static/uploads/tenant_logos
mkdir -p /var/backups/jqc
mkdir -p /etc/jqc

# Set ownership
chown -R jqc:jqc /home/jqc/
chown -R jqc:jqc /var/backups/jqc

# Set permissions on env directory (root owns it, readable by jqc)
chown root:jqc /etc/jqc
chmod 750 /etc/jqc

Step 2 — Copy Application Code

Option A — Clone from Git (preferred)

If the codebase is in a private Git repository:

# Switch to jqc user
su - jqc

# Clone the repository
cd /home/jqc/
git clone https://github.com/YOUR_ORG/janitorial_qc_multi.git .

# If the repo uses SSH keys, ensure jqc user has access
# Place the deploy key at /home/jqc/.ssh/id_rsa (chmod 600)

# Verify the checkout
ls /home/jqc/app/

Option B — rsync from old server

If Git is not available or the repo has uncommitted changes that need to be copied:

# Run on the NEW server as root
# Replace OLD_SERVER_IP with the actual IP of the old server

rsync -avz --exclude='venv/' \
    --exclude='__pycache__/' \
    --exclude='*.pyc' \
    --exclude='app/static/uploads/' \
    --exclude='logs/' \
    root@OLD_SERVER_IP:/home/jqc/ \
    /home/jqc/

chown -R jqc:jqc /home/jqc/

2.2 Create and populate the Python virtual environment

su - jqc
cd /home/jqc/

# Create the virtualenv using Python 3.11
python3.11 -m venv venv

# Activate and install dependencies
source venv/bin/activate
pip install --upgrade pip
pip install -r requirements.txt

# Verify key packages installed
pip show flask gunicorn flask-sqlalchemy flask-migrate pymysql reportlab openpyxl

Note: If requirements.txt is missing, run pip freeze on the old server (inside its venv) to generate one, then copy it over before running pip install.

Step 3 — Export Databases from Old Server

Run all commands in this section on the old server.

3.1 Source the control environment

set -a; . /etc/jqc/control.env; set +a

3.2 Create the backup directory

mkdir -p /var/backups/jqc/migration_$(date +%Y%m%d)
BACKUP_DIR=/var/backups/jqc/migration_$(date +%Y%m%d)
echo "Backup directory: $BACKUP_DIR"

3.3 Export the control plane database

mysqldump \
    --single-transaction \
    --routines \
    --triggers \
    --set-gtid-purged=OFF \
    -u root -p \
    jqc_control \
    > $BACKUP_DIR/jqc_control.sql

echo "Control DB dump size: $(du -sh $BACKUP_DIR/jqc_control.sql)"

3.4 Export all tenant databases

Get the list of tenant databases from the control plane, then dump each one:

# Get all tenant DB names from the control plane
cd /home/jqc/
source venv/bin/activate

python3 - <<'EOF'
import os, sys
sys.path.insert(0, '/home/jqc/')

# Read directly from MySQL to avoid app startup dependency
import pymysql
url = os.environ['CONTROL_DATABASE_URL']
# Parse: mysql+pymysql://user:pass@host/dbname
import re
m = re.match(r'mysql\+pymysql://([^:]+):([^@]+)@([^/]+)/(.+)', url)
user, password, host, db = m.groups()

conn = pymysql.connect(host=host, user=user, password=password, database=db)
with conn.cursor() as cur:
    cur.execute("SELECT slug, db_name FROM tenants WHERE active = 1")
    rows = cur.fetchall()
conn.close()

for slug, db_name in rows:
    print(f"{slug}:{db_name}")
EOF

For each tenant database listed, run a dump. Replace the names with the actual output from above:

# Dump each tenant DB — repeat this block for every tenant
# Example: TENANT_SLUG=lts  TENANT_DB=jqc_lt

for TENANT_DB in $(mysql -u root -p -N -e "SHOW DATABASES LIKE 'jqc_%';" | grep -v jqc_control); do
    echo "Dumping $TENANT_DB..."
    mysqldump \
        --single-transaction \
        --routines \
        --triggers \
        --set-gtid-purged=OFF \
        -u root -p \
        $TENANT_DB \
        > $BACKUP_DIR/${TENANT_DB}.sql
    echo "  Done: $(du -sh $BACKUP_DIR/${TENANT_DB}.sql)"
done

echo "All dumps complete. Files in $BACKUP_DIR:"
ls -lh $BACKUP_DIR/

3.5 Export MySQL user credentials

You need to recreate the MySQL users on the new server. Record the current grants:

mysql -u root -p -e "
SELECT CONCAT('SHOW GRANTS FOR ''', user, '''@''', host, ''';')
FROM mysql.user
WHERE user LIKE 'jqc%'
\G" | grep -v "^\*" | mysql -u root -p > $BACKUP_DIR/user_grants.sql 2>/dev/null

# Also print to screen for manual inspection
mysql -u root -p -e "SELECT user, host, authentication_string FROM mysql.user WHERE user LIKE 'jqc%';"

Warning: MySQL user passwords are hashed and cannot be extracted in plaintext from mysql.user. You have two options:

Reset passwords on new server — create new passwords for all MySQL users and update /etc/jqc/app.env and /etc/jqc/control.env accordingly. This is recommended.

Copy the auth plugin + hash — advanced; version-dependent. Only do this if both servers run the same MySQL version.

3.6 Record the list of per-tenant MySQL users

mysql -u root -p -e "
SELECT t.slug, t.db_name, t.db_user
FROM tenants t
ORDER BY t.slug;" jqc_control

Save this output — you will recreate each db_user in Step 4.

3.7 Copy all dumps to the new server

# Run on OLD server — push dumps to new server
NEW_SERVER_IP=YOUR_NEW_SERVER_IP

rsync -avz --progress \
    $BACKUP_DIR/ \
    root@$NEW_SERVER_IP:/var/backups/jqc/migration_restore/

echo "Transfer complete."

Step 4 — Restore Databases on New Server

Run all commands in this section on the new server.

4.1 Create MySQL users

Log into MySQL on the new server as root:

mysql -u root -p

Inside the MySQL shell, create all required users. Use new, strong passwords and update the environment files in Step 5 to match.

-- ----------------------------------------------------------------
-- Control plane user (owns jqc_control database)
-- ----------------------------------------------------------------
CREATE USER 'jqc_control'@'127.0.0.1' IDENTIFIED BY 'CHOOSE_STRONG_PASSWORD_1';
CREATE DATABASE IF NOT EXISTS jqc_control CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
GRANT ALL PRIVILEGES ON jqc_control.* TO 'jqc_control'@'127.0.0.1';

-- ----------------------------------------------------------------
-- Provisioner user (creates new tenant DBs and users)
-- Must have WITH GRANT OPTION and CREATE USER
-- ----------------------------------------------------------------
CREATE USER 'jqc_provisioner'@'127.0.0.1' IDENTIFIED BY 'CHOOSE_STRONG_PASSWORD_2';
GRANT ALL PRIVILEGES ON *.* TO 'jqc_provisioner'@'127.0.0.1' WITH GRANT OPTION;
GRANT CREATE USER ON *.* TO 'jqc_provisioner'@'127.0.0.1';

-- ----------------------------------------------------------------
-- Per-tenant users — repeat for EACH tenant from Step 3.6
-- Replace db_name and db_user with values from the tenant list
-- ----------------------------------------------------------------

-- Example for tenant-zero (LT):
CREATE USER 'jqc_lts_user'@'127.0.0.1' IDENTIFIED BY 'CHOOSE_STRONG_PASSWORD_3';
CREATE DATABASE IF NOT EXISTS jqc_lt CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
GRANT ALL PRIVILEGES ON jqc_lt.* TO 'jqc_lts_user'@'127.0.0.1';

-- Example for a second tenant (repeat as needed):
-- CREATE USER 'jqc_acme_user'@'127.0.0.1' IDENTIFIED BY 'CHOOSE_STRONG_PASSWORD_N';
-- CREATE DATABASE IF NOT EXISTS jqc_acme CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
-- GRANT ALL PRIVILEGES ON jqc_acme.* TO 'jqc_acme_user'@'127.0.0.1';

FLUSH PRIVILEGES;
EXIT;

Note: The per-tenant db_user and db_name values must match exactly what is stored (encrypted) in jqc_control.tenants. You are restoring both the data and the control plane DB, so the usernames must match the records in the dump. Use the output from Step 3.6 to get the exact usernames.

4.2 Restore the control plane database

RESTORE_DIR=/var/backups/jqc/migration_restore

mysql -u root -p jqc_control < $RESTORE_DIR/jqc_control.sql

echo "Control DB restored."
mysql -u root -p -e "SELECT COUNT(*) as tenant_count FROM jqc_control.tenants;"

4.3 Restore all tenant databases

# Restore each tenant DB dump
for DUMP_FILE in $RESTORE_DIR/jqc_*.sql; do
    # Skip the control DB — already restored
    [[ "$DUMP_FILE" == *"jqc_control.sql" ]] && continue

    DB_NAME=$(basename "$DUMP_FILE" .sql)
    echo "Restoring $DB_NAME from $DUMP_FILE..."

    # Ensure the database exists (may already be created above)
    mysql -u root -p -e "CREATE DATABASE IF NOT EXISTS \`$DB_NAME\` CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;"

    mysql -u root -p $DB_NAME < $DUMP_FILE
    echo "  Done: $DB_NAME"
done

4.4 Verify table counts

For each database, spot-check that table counts match the old server:

# Check control DB
mysql -u root -p -e "
SELECT table_name, table_rows
FROM information_schema.tables
WHERE table_schema = 'jqc_control'
ORDER BY table_name;" 2>/dev/null

# Check a tenant DB (replace jqc_lt with actual name)
mysql -u root -p -e "
SELECT table_name, table_rows
FROM information_schema.tables
WHERE table_schema = 'jqc_lt'
ORDER BY table_name;" 2>/dev/null

Compare these counts to the same query run on the old server. Row counts from information_schema.tables are estimates for InnoDB, so small discrepancies are normal — focus on table count and that key tables (users, inspections, issues, tenants) are not zero.

4.5 Update the encrypted DB passwords in the control plane

CRITICAL: Because you created new MySQL passwords in Step 4.1, the db_password_enc column in jqc_control.tenants still holds the old encrypted passwords. The provisioner will fail to connect. You must re-encrypt the new passwords.

cd /home/jqc/
source /etc/jqc/control.env  # must have CONTROL_FERNET_KEY set

# For each tenant, re-encrypt the new password
python3 - <<'EOF'
import sys, os
sys.path.insert(0, '/home/jqc/')
from control.crypto import encrypt

# Replace these with the actual slug → new_password mappings from Step 4.1
tenant_passwords = {
    'lts': 'CHOOSE_STRONG_PASSWORD_3',
    # 'acme': 'CHOOSE_STRONG_PASSWORD_N',
}

for slug, pw in tenant_passwords.items():
    enc = encrypt(pw)
    print(f"UPDATE tenants SET db_password_enc = '{enc}' WHERE slug = '{slug}';")
EOF

Copy the generated SQL statements and run them against jqc_control:

mysql -u root -p jqc_control -e "
UPDATE tenants SET db_password_enc = 'ENCRYPTED_VALUE_FROM_ABOVE' WHERE slug = 'lts';
-- repeat for each tenant
"

Step 5 — Copy Environment Files

5.1 Copy from old server to new server

# Run on OLD server
NEW_SERVER_IP=YOUR_NEW_SERVER_IP

scp /etc/jqc/app.env root@$NEW_SERVER_IP:/etc/jqc/app.env
scp /etc/jqc/control.env root@$NEW_SERVER_IP:/etc/jqc/control.env

5.2 Set correct permissions on new server

# Run on NEW server
chmod 640 /etc/jqc/app.env
chmod 640 /etc/jqc/control.env
chown root:jqc /etc/jqc/app.env
chown root:jqc /etc/jqc/control.env

5.3 Update values that change with the new server

Edit both files and update the following:

vim /etc/jqc/app.env

Update in /etc/jqc/app.env:

Variable	What to change
`DATABASE_URL`	Update host from old server IP to `127.0.0.1` (if DB is local) or new DB host
`APP_BASE_URL`	Update if the domain changes (e.g. from old IP to new domain)
`REDIS_URL`	Update to `redis://127.0.0.1:6379/0` if Redis is local on new server
`DATABASE_URL` password	Update to the new MySQL password set in Step 4.1

vim /etc/jqc/control.env

Update in /etc/jqc/control.env:

Variable	What to change
`CONTROL_DATABASE_URL`	Update host and password (use `jqc_control` user + new password from Step 4.1)
`PROVISION_DB_URL`	Update host and password (use `jqc_provisioner` user + new password from Step 4.1)
`MULTI_TENANT_ENABLED`	Leave as `true` if old server had it enabled

CRITICAL: CONTROL_FERNET_KEY must be identical between the old and new servers. This key encrypts the tenant DB passwords stored in jqc_control.tenants. Copy it exactly — do not regenerate it. If you regenerate it, all tenant DB connections will fail with InvalidToken until you re-encrypt every password.

5.4 Verify the environment files parse correctly

# Source both files and check for syntax errors
set -a
. /etc/jqc/app.env
. /etc/jqc/control.env
set +a

echo "DATABASE_URL is set: ${DATABASE_URL:+yes}"
echo "SECRET_KEY is set: ${SECRET_KEY:+yes}"
echo "CONTROL_FERNET_KEY is set: ${CONTROL_FERNET_KEY:+yes}"
echo "MULTI_TENANT_ENABLED: $MULTI_TENANT_ENABLED"

Step 6 — Copy Uploads

User-uploaded files (inspection photos, issue photos, tenant logos) must be copied from the old server.

Note: This can be a large transfer depending on how long the system has been running. Run it before the DNS cutover window and optionally run it again immediately before cutover to sync any files uploaded since the first rsync.

# Run on NEW server (pulls from old server)
OLD_SERVER_IP=YOUR_OLD_SERVER_IP

rsync -avz --progress \
    root@$OLD_SERVER_IP:/home/jqc/app/static/uploads/ \
    /home/jqc/app/static/uploads/

# Fix ownership after rsync
chown -R jqc:jqc /home/jqc/app/static/uploads/

# Verify counts match
echo "New server upload file count:"
find /home/jqc/app/static/uploads/ -type f | wc -l

echo "Old server upload file count (run this on old server separately):"
echo "  find /home/jqc/app/static/uploads/ -type f | wc -l"

Step 7 — Configure systemd Services

7.1 Create the main application service

cat > /etc/systemd/system/jqc.service << 'EOF'
[Unit]
Description=JQC Web Application
After=network.target mysql.service redis.service

[Service]
User=jqc
Group=jqc
WorkingDirectory=/home/jqc/
EnvironmentFile=/etc/jqc/app.env
EnvironmentFile=/etc/jqc/control.env
ExecStart=/home/jqc/venv/bin/gunicorn wsgi:app \
    --bind 127.0.0.1:8000 \
    --workers 5 \
    --worker-class sync \
    --timeout 30 \
    --log-file /home/jqc/logs/gunicorn.log \
    --log-level info
ExecReload=/bin/kill -s HUP $MAINPID
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

7.2 Create the superadmin panel service

cat > /etc/systemd/system/jqc-panel.service << 'EOF'
[Unit]
Description=JQC Superadmin Control Panel
After=network.target mysql.service

[Service]
User=jqc
Group=jqc
WorkingDirectory=/home/jqc/
EnvironmentFile=/etc/jqc/app.env
EnvironmentFile=/etc/jqc/control.env
ExecStart=/home/jqc/venv/bin/gunicorn wsgi_panel:panel_app \
    --bind 127.0.0.1:8001 \
    --workers 2 \
    --worker-class sync \
    --timeout 30 \
    --log-file /home/jqc/logs/gunicorn-panel.log \
    --log-level info
ExecReload=/bin/kill -s HUP $MAINPID
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

7.3 Enable and start the services

# Reload systemd to read new unit files
systemctl daemon-reload

# Enable both services to start on boot
systemctl enable jqc
systemctl enable jqc-panel

# Start the services
systemctl start jqc
systemctl start jqc-panel

# Verify both are running
systemctl status jqc --no-pager
systemctl status jqc-panel --no-pager

# Check the application logs for startup errors
tail -50 /home/jqc/logs/gunicorn.log
tail -50 /home/jqc/logs/gunicorn-panel.log

7.4 Verify the applications respond on their ports

# Main app should return a redirect or HTML (not a connection refused error)
curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:8000/
# Expected: 302 (redirect to login) or 200

# Panel app
curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:8001/
# Expected: 302 or 200

If you get 000 (connection refused), check the logs:

journalctl -u jqc -n 50 --no-pager
journalctl -u jqc-panel -n 50 --no-pager

Step 8 — Configure Nginx

CRITICAL: The admin.jqc.app server block must appear before the *.jqc.app wildcard block. Nginx matches server_name in order — if the wildcard block comes first, it catches admin.jqc.app requests and proxies them to port 8000 (main app) instead of port 8001 (panel app).

8.1 Write the Nginx configuration

cat > /etc/nginx/sites-available/jqc << 'NGINXCONF'
# ----------------------------------------------------------------
# Redirect HTTP to HTTPS (all hosts)
# ----------------------------------------------------------------
server {
    listen 80;
    server_name *.jqc.app jqc.app admin.jqc.app;

    location /.well-known/acme-challenge/ {
        root /var/www/html;
    }

    location / {
        return 301 https://$host$request_uri;
    }
}

# ----------------------------------------------------------------
# Superadmin Control Panel — admin.jqc.app
# MUST BE BEFORE the wildcard *.jqc.app block
# ----------------------------------------------------------------
server {
    listen 443 ssl;
    server_name admin.jqc.app;

    ssl_certificate     /etc/letsencrypt/live/admin.jqc.app/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/admin.jqc.app/privkey.pem;
    include             /etc/letsencrypt/options-ssl-nginx.conf;
    ssl_dhparam         /etc/letsencrypt/ssl-dhparams.pem;

    client_max_body_size 50M;

    access_log /var/log/nginx/panel-access.log;
    error_log  /var/log/nginx/panel-error.log;

    location / {
        proxy_pass         http://127.0.0.1:8001;
        proxy_set_header   Host              $host;
        proxy_set_header   X-Real-IP         $remote_addr;
        proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Proto $scheme;
        proxy_read_timeout 60s;
    }
}

# ----------------------------------------------------------------
# Main application — *.jqc.app wildcard (all tenants)
# MUST BE AFTER the admin.jqc.app exact block
# ----------------------------------------------------------------
server {
    listen 443 ssl;
    server_name *.jqc.app jqc.app;

    ssl_certificate     /etc/letsencrypt/live/jqc.app/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/jqc.app/privkey.pem;
    include             /etc/letsencrypt/options-ssl-nginx.conf;
    ssl_dhparam         /etc/letsencrypt/ssl-dhparams.pem;

    client_max_body_size 50M;

    access_log /var/log/nginx/jqc-access.log;
    error_log  /var/log/nginx/jqc-error.log;

    # Serve static files directly without hitting Gunicorn
    location /static/ {
        alias /home/jqc/app/static/;
        expires 30d;
        add_header Cache-Control "public, immutable";
    }

    location / {
        proxy_pass         http://127.0.0.1:8000;
        proxy_set_header   Host              $host;
        proxy_set_header   X-Real-IP         $remote_addr;
        proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Proto $scheme;
        proxy_read_timeout 60s;
    }
}
NGINXCONF

8.2 Enable the site and test the configuration

# Remove the default site if present
rm -f /etc/nginx/sites-enabled/default

# Enable the JQC site
ln -sf /etc/nginx/sites-available/jqc /etc/nginx/sites-enabled/jqc

# Test the configuration
nginx -t
# Expected: "configuration file ... syntax is ok" and "test is successful"

Note: Do NOT reload Nginx yet if TLS certificates do not exist. The configuration references certificate files that must be created in Step 9 first. You can skip the reload here and do it after Step 9.

Step 9 — TLS Certificates

9.1 Obtain the wildcard certificate for *.jqc.app

Wildcard certificates require DNS-01 challenge. The exact command depends on your DNS provider.

Example for Cloudflare:

# Create Cloudflare credentials file
cat > /etc/letsencrypt/cloudflare.ini << 'EOF'
# Cloudflare API token with Zone:DNS:Edit permission
dns_cloudflare_api_token = YOUR_CLOUDFLARE_API_TOKEN
EOF
chmod 600 /etc/letsencrypt/cloudflare.ini

# Request wildcard + apex cert
certbot certonly \
    --dns-cloudflare \
    --dns-cloudflare-credentials /etc/letsencrypt/cloudflare.ini \
    -d "jqc.app" \
    -d "*.jqc.app" \
    --email YOUR_EMAIL@example.com \
    --agree-tos \
    --non-interactive

Example for Route53:

certbot certonly \
    --dns-route53 \
    -d "jqc.app" \
    -d "*.jqc.app" \
    --email YOUR_EMAIL@example.com \
    --agree-tos \
    --non-interactive

9.2 Obtain the certificate for admin.jqc.app

# admin.jqc.app is an exact subdomain — still use DNS-01 for consistency
certbot certonly \
    --dns-cloudflare \
    --dns-cloudflare-credentials /etc/letsencrypt/cloudflare.ini \
    -d "admin.jqc.app" \
    --email YOUR_EMAIL@example.com \
    --agree-tos \
    --non-interactive

9.3 Verify certificates issued

certbot certificates
# Should show two certificate entries:
# 1. jqc.app covering *.jqc.app and jqc.app
# 2. admin.jqc.app

ls /etc/letsencrypt/live/

9.4 Set up auto-renewal

# Test renewal (dry run)
certbot renew --dry-run

# Certbot installs a systemd timer automatically on Ubuntu 22.04
# Verify it is active
systemctl status certbot.timer

# Add a post-renewal hook to reload Nginx after cert renewal
cat > /etc/letsencrypt/renewal-hooks/post/reload-nginx.sh << 'EOF'
#!/bin/bash
nginx -t && systemctl reload nginx
EOF
chmod +x /etc/letsencrypt/renewal-hooks/post/reload-nginx.sh

9.5 Start Nginx with TLS

# Now that certs exist, test and reload Nginx
nginx -t && systemctl reload nginx

# Verify Nginx is running
systemctl status nginx --no-pager

Step 10 — DNS Cutover

IMPORTANT: Plan this step for a low-traffic period. Reduce the TTL on your DNS records to 60–300 seconds at least 24 hours before cutover so the change propagates quickly. After cutover is confirmed working, restore TTL to normal (3600+).

10.1 Pre-cutover: reduce TTL (do this 24 hours early)

In your DNS provider's control panel, reduce the TTL on the following records to 60 seconds:

*.jqc.app A record
jqc.app A record
admin.jqc.app A record

10.2 Record the old server IP for rollback

# On OLD server
curl -s https://api.ipify.org
# Save this IP — you will need it for rollback

10.3 Get the new server IP

# On NEW server
NEW_SERVER_IP=$(curl -s https://api.ipify.org)
echo "New server IP: $NEW_SERVER_IP"

10.4 Final data sync before cutover

Immediately before switching DNS (while the old server is still live), do a final rsync of uploads to capture anything uploaded since Step 6:

# On NEW server — pull latest uploads from old server
OLD_SERVER_IP=YOUR_OLD_SERVER_IP

rsync -avz --progress \
    root@$OLD_SERVER_IP:/home/jqc/app/static/uploads/ \
    /home/jqc/app/static/uploads/

chown -R jqc:jqc /home/jqc/app/static/uploads/

Optionally do a final database sync as well if the cutover window is long:

# On OLD server — dump and transfer latest data
mysqldump --single-transaction -u root -p jqc_control > /tmp/jqc_control_final.sql
scp /tmp/jqc_control_final.sql root@$NEW_SERVER_IP:/tmp/

# For each tenant DB:
mysqldump --single-transaction -u root -p jqc_lt > /tmp/jqc_lt_final.sql
scp /tmp/jqc_lt_final.sql root@$NEW_SERVER_IP:/tmp/

# On NEW server — restore the final dumps
mysql -u root -p jqc_control < /tmp/jqc_control_final.sql
mysql -u root -p jqc_lt < /tmp/jqc_lt_final.sql
# Repeat for each tenant DB

10.5 Switch DNS

In your DNS provider's control panel, update the following A records to point to the new server IP:

Record	Old IP	New IP
`*.jqc.app`	OLD_IP	NEW_IP
`jqc.app`	OLD_IP	NEW_IP
`admin.jqc.app`	OLD_IP	NEW_IP

10.6 Monitor propagation

# On your local machine — check when DNS has propagated
watch -n 5 "dig +short *.jqc.app @8.8.8.8"

# Or use online tools:
# https://dnschecker.org/#A/lts.jqc.app
# https://www.whatsmydns.net/

Step 11 — Smoke Test

After DNS propagates to your location, run this checklist.

11.1 Basic HTTPS access

# Test main app (replace lts.jqc.app with your tenant-zero subdomain)
curl -sI https://lts.jqc.app/ | head -5
# Expected: HTTP/2 200 or 302

# Test admin panel
curl -sI https://admin.jqc.app/ | head -5
# Expected: HTTP/2 200 or 302

# Verify TLS certificate
echo | openssl s_client -connect lts.jqc.app:443 -servername lts.jqc.app 2>/dev/null \
    | openssl x509 -noout -dates -subject

11.2 Web application checklist

Open a browser and verify:

https://lts.jqc.app/auth/login loads the login page (no 502/504 error)
Login with admin credentials succeeds
Dashboard loads and shows data (inspections, issues counts)
Navigate to Inspections list — data appears
Navigate to Issues list — data appears
Navigate to Reports — charts load
An existing uploaded photo (inspection or issue) is visible (not broken img tag)
Tenant branding/logo appears in the navbar (if configured)

11.3 Multi-tenant routing

# Test that the tenant resolver is working
# A second tenant subdomain should respond (or return 404 "Workspace not found"
# if that tenant does not exist — both are correct, 502 is not)

curl -sI https://SECOND_TENANT.jqc.app/ | head -3

11.4 Admin panel

https://admin.jqc.app/login loads
Login with superadmin credentials succeeds
Tenant list shows all tenants
Impersonation: click "Impersonate" on a tenant → app loads for that tenant

11.5 Mobile API

# Test the mobile API auth endpoint
curl -s -X POST https://lts.jqc.app/api/v1/auth/login \
    -H "Content-Type: application/json" \
    -d '{"username": "INSPECTOR_USERNAME", "password": "INSPECTOR_PASSWORD"}' \
    | python3 -m json.tool | head -20
# Expected: JSON with access_token and refresh_token (not an HTML error page)

11.6 Static file serving

# Test that Nginx serves a static file directly (not via Gunicorn)
# Replace the path with an actual uploaded file path you know exists
curl -sI https://lts.jqc.app/static/uploads/inspection_photos/SOME_FILE.jpg | head -5
# Expected: HTTP/2 200, Content-Type: image/jpeg

Step 12 — Update Stripe Webhook

The Stripe webhook endpoint was registered with the old server's URL. After DNS cutover, update it to use the new URL (which should be the same domain if DNS was simply re-pointed, or a new domain if applicable).

12.1 When the domain stays the same (DNS re-pointed)

If the domain is unchanged (e.g. still lts.jqc.app), the webhook URL is the same and no Stripe changes are needed. Verify the webhook is still receiving events:

Log in to the Stripe Dashboard
Navigate to Developers → Webhooks
Click on the webhook endpoint for /billing/webhook
Check recent deliveries — they should show 200 OK responses after cutover

12.2 When the domain changes

If you are moving to a new domain:

Log in to the Stripe Dashboard
Navigate to Developers → Webhooks
Click Add endpoint
Enter: https://NEW_DOMAIN.com/billing/webhook
Select the same events as the old webhook (at minimum: checkout.session.completed, customer.subscription.updated, customer.subscription.deleted, invoice.payment_failed, invoice.paid)
Copy the new Signing secret (starts with whsec_)
Update /etc/jqc/app.env on the new server:

vim /etc/jqc/app.env
# Update: STRIPE_WEBHOOK_SECRET=whsec_NEW_SECRET_FROM_STRIPE

Restart the main app to reload the environment:

systemctl restart jqc

In Stripe, click Send test webhook → select checkout.session.completed → verify the new server returns 200 OK
After confirming the new endpoint works, delete the old endpoint from Stripe.

Step 13 — Cron Jobs

Set up the cron schedule for the jqc user. These jobs run application maintenance tasks.

# Edit the crontab for the jqc user
crontab -u jqc -e

Add the following entries. Replace YOUR_DIGEST_SECRET with the value of DIGEST_SECRET from /etc/jqc/app.env, and YOUR_DOMAIN with the primary tenant domain (e.g. lts.jqc.app):

# JQC scheduled tasks
# Environment
MAILTO=""
SHELL=/bin/bash

# ----------------------------------------------------------------
# SLA breach/at-risk alerts — every 30 minutes
# ----------------------------------------------------------------
*/30 * * * * curl -s -X POST https://YOUR_DOMAIN/notifications/check-sla \
    -d "token=YOUR_DIGEST_SECRET" >> /home/jqc/logs/cron.log 2>&1

# ----------------------------------------------------------------
# Daily email digest — 7:00 AM Eastern
# ----------------------------------------------------------------
0 7 * * * curl -s -X POST https://YOUR_DOMAIN/notifications/send-digest \
    -d "token=YOUR_DIGEST_SECRET&frequency=daily" >> /home/jqc/logs/cron.log 2>&1

# ----------------------------------------------------------------
# Weekly email digest — 7:00 AM Eastern on Mondays
# ----------------------------------------------------------------
0 7 * * 1 curl -s -X POST https://YOUR_DOMAIN/notifications/send-digest \
    -d "token=YOUR_DIGEST_SECRET&frequency=weekly" >> /home/jqc/logs/cron.log 2>&1

# ----------------------------------------------------------------
# Facility score-drop alerts — 8:00 AM Eastern
# ----------------------------------------------------------------
0 8 * * * curl -s -X POST https://YOUR_DOMAIN/notifications/check-score-trends \
    -d "token=YOUR_DIGEST_SECRET" >> /home/jqc/logs/cron.log 2>&1

# ----------------------------------------------------------------
# Scheduled reports email delivery — 8:05 AM Eastern (offset from score alerts)
# ----------------------------------------------------------------
5 8 * * * curl -s -X POST https://YOUR_DOMAIN/scheduled-reports/run \
    -d "secret=YOUR_DIGEST_SECRET" >> /home/jqc/logs/cron.log 2>&1

# ----------------------------------------------------------------
# Expired API token cleanup — 3:00 AM Eastern
# ----------------------------------------------------------------
0 3 * * * curl -s -X POST https://YOUR_DOMAIN/notifications/cleanup-tokens \
    -d "token=YOUR_DIGEST_SECRET" >> /home/jqc/logs/cron.log 2>&1

# ----------------------------------------------------------------
# Nightly database backup — 2:00 AM Eastern
# ----------------------------------------------------------------
0 2 * * * /home/jqc/scripts/backup.sh >> /home/jqc/logs/backup.log 2>&1

13.1 Create the backup script

cat > /home/jqc/scripts/backup.sh << 'BACKUPSCRIPT'
#!/bin/bash
# JQC nightly database backup

set -a
. /etc/jqc/app.env
. /etc/jqc/control.env
set +a

BACKUP_DIR=/var/backups/jqc/$(date +%Y%m%d)
mkdir -p "$BACKUP_DIR"

# Back up control database
mysqldump --single-transaction \
    --routines --triggers --set-gtid-purged=OFF \
    -u root -p"$MYSQL_ROOT_PASSWORD" \
    jqc_control \
    | gzip > "$BACKUP_DIR/jqc_control.sql.gz"

# Back up all tenant databases
for DB in $(mysql -u root -p"$MYSQL_ROOT_PASSWORD" -N \
    -e "SHOW DATABASES LIKE 'jqc_%';" 2>/dev/null | grep -v jqc_control); do
    mysqldump --single-transaction \
        --routines --triggers --set-gtid-purged=OFF \
        -u root -p"$MYSQL_ROOT_PASSWORD" \
        "$DB" \
        | gzip > "$BACKUP_DIR/${DB}.sql.gz"
done

# Remove backups older than 14 days
find /var/backups/jqc/ -maxdepth 1 -type d -mtime +14 -exec rm -rf {} +

echo "[$(date)] Backup complete: $BACKUP_DIR"
ls -lh "$BACKUP_DIR/"
BACKUPSCRIPT

mkdir -p /home/jqc/scripts
chmod +x /home/jqc/scripts/backup.sh
chown jqc:jqc /home/jqc/scripts/backup.sh

Note: The backup script uses a MYSQL_ROOT_PASSWORD variable. Add this to /etc/jqc/app.env or pass it another way. Alternatively, create a .my.cnf file in the jqc user's home directory:
cat > /home/jqc/.my.cnf << 'EOF'
[client]
user=root
password=YOUR_MYSQL_ROOT_PASSWORD
EOF
chmod 600 /home/jqc/.my.cnf
chown jqc:jqc /home/jqc/.my.cnf
Then remove the -p"$MYSQL_ROOT_PASSWORD" flags from the backup script and use -u root only.

13.2 Verify the crontab

crontab -u jqc -l

13.3 Test one cron job manually

# Run the SLA check manually to confirm it works
su - jqc -c 'curl -s -X POST https://YOUR_DOMAIN/notifications/check-sla \
    -d "token=YOUR_DIGEST_SECRET"'
# Expected: JSON response like {"checked": 5, "notified": 0} — not an HTML error page

Rollback Plan

If the new server has problems after DNS cutover, revert traffic to the old server.

Immediate rollback (DNS revert)

The old server should remain running and untouched until you are fully confident in the new server.

# In your DNS provider's control panel, immediately update the A records back:
# *.jqc.app  → OLD_SERVER_IP
# jqc.app    → OLD_SERVER_IP
# admin.jqc.app → OLD_SERVER_IP

Because you reduced TTL to 60 seconds in Step 10.1, propagation will be fast (typically under 5 minutes for most resolvers, up to TTL seconds for others).

Before reverting, capture any new data written to the new server

If the new server was live for any period and received real user activity (logins, new inspections, new issues), you should extract that data before rolling back:

# On NEW server — dump any tenant DBs that received writes since cutover
mysqldump --single-transaction -u root -p jqc_lt \
    > /tmp/jqc_lt_rollback_$(date +%Y%m%d_%H%M).sql

# Transfer to old server and import (carefully — manual merge required if old server also had writes)
scp /tmp/jqc_lt_rollback_*.sql root@OLD_SERVER_IP:/tmp/

Warning: If both servers received writes simultaneously (e.g. during a partial DNS propagation window where some users hit old and some hit new), reconciling the data is a manual process. This is the main reason to keep the maintenance window short and confirm full propagation before allowing the old server to accept new writes.

Keep old server running for 48 hours after cutover

Do not decommission or wipe the old server until:

DNS has fully propagated globally (use dnschecker.org to verify)
You have run the full smoke test on the new server
At least 48 hours have passed with no issues
You have taken a final backup from the old server

Post-Cutover Checklist

Work through this list after DNS propagation is confirmed. Check each item in the browser and in the application logs.

Application health

All tenant subdomains resolve to new server IP (dig +short lts.jqc.app)
admin.jqc.app resolves to new server IP
HTTPS works on all domains (no certificate warnings)
systemctl status jqc shows active (running)
systemctl status jqc-panel shows active (running)
systemctl status nginx shows active (running)
No errors in /home/jqc/logs/gunicorn.log (last 100 lines)
No errors in /var/log/nginx/jqc-error.log

Authentication and roles

Admin login works
Director login works
Inspector login works
Customer login works (verify scoped data, not all facilities)
Superadmin panel login works

Core features

Create a new inspection (full submit, not just draft)
Flag an issue from an inspection
Upload a photo to an issue
Add a comment to an issue
Customer can see their facility data but not others
PDF export works (inspection PDF, issues list PDF)
Excel export works (inspector performance)

Multi-tenant

At least two tenant subdomains respond correctly
Impersonation from admin panel works
Tenant branding (logo, colours) appears correctly

Notifications and integrations

In-app notification bell shows notifications
Test email notification delivery (e.g. assign an issue → check that assigned user receives email)
Stripe: open Stripe dashboard → Webhooks → verify recent deliveries show 200 OK
Groq AI chat: navigate to /support/chat as a customer → verify chat responds (if GROQ_API_KEY is set)

Infrastructure

Redis is running: redis-cli ping returns PONG
MySQL is running: systemctl status mysql
Certbot timer is active: systemctl status certbot.timer
Cron jobs are scheduled: crontab -u jqc -l
Log rotation is working: /home/jqc/logs/ is not filling up unexpectedly
First nightly backup completed (check /var/backups/jqc/ the morning after cutover)
Firewall is active: ufw status shows correct rules

Decommission old server (after 48 hours)

Final backup taken from old server
Old server's Stripe webhook endpoint deleted
Old server's cron jobs removed or disabled
Old server's DNS records confirmed no longer pointing to old IP
Old server shut down or deprovisioned

End of JQC Server Duplication Runbook

42 KiB Raw Blame History Unescape Escape