JQC_multi_tenant/GOING_LIVE.md

# Going Live — JQC Billing & Email Checklist

Steps to complete before accepting real paying customers.  
Follow sections in order: Stripe first, then email, then cron, then verify.

---

## 1. Switch Stripe from Test → Live Mode

### 1.1 Get your live API keys

1. Log in to [dashboard.stripe.com](https://dashboard.stripe.com)
2. Click the toggle in the top-left from **Test mode** → **Live mode**
3. Go to **Developers → API keys**
4. Copy:
   - **Secret key** — starts with `sk_live_…`
   - **Publishable key** — starts with `pk_live_…`

### 1.2 Create live Products and Price IDs

> You must do this in **live mode** — test-mode prices are not valid for real payments.

1. Go to **Products → + Add product**
2. Create three products: **Starter**, **Pro**, **Enterprise**
3. For each, add a **recurring price** (monthly)
4. Copy each **Price ID** — starts with `price_live_…` (NOT `prod_…`)

### 1.3 Update `/etc/jqc/app.env`

```bash
sudo nano /etc/jqc/app.env
```

Replace the test values:

```env
STRIPE_SECRET_KEY=sk_live_XXXXXXXXXXXXXXXXXXXXXXXX
STRIPE_PUBLISHABLE_KEY=pk_live_XXXXXXXXXXXXXXXXXXXXXXXX
STRIPE_PRICE_STARTER=price_live_XXXXXXXXXXXXXXXXXXXXXXXX
STRIPE_PRICE_PRO=price_live_XXXXXXXXXXXXXXXXXXXXXXXX
STRIPE_PRICE_ENTERPRISE=price_live_XXXXXXXXXXXXXXXXXXXXXXXX
```

### 1.4 Seed the live Price IDs into the control DB

```bash
set -a; . /etc/jqc/app.env; . /etc/jqc/control.env; set +a
python -m control.cli seed
python -m control.cli list-plans   # verify all three show price_live_... IDs
```

### 1.5 Register the live Stripe webhook

1. In Stripe Dashboard (live mode) → **Developers → Webhooks → + Add endpoint**
2. **Endpoint URL:** `https://yourdomain.jqc.app/billing/webhook`
   *(use your tenant-zero domain or the primary custom domain)*
3. **Events to listen to** (select these exactly):
   - `invoice.payment_failed`
   - `customer.subscription.updated`
   - `customer.subscription.deleted`
   - `checkout.session.completed`
4. Click **Add endpoint**
5. Copy the **Signing secret** — starts with `whsec_…`

### 1.6 Update webhook secret

```bash
sudo nano /etc/jqc/app.env
```

```env
STRIPE_WEBHOOK_SECRET=whsec_XXXXXXXXXXXXXXXXXXXXXXXX
```

### 1.7 Restart the app

```bash
sudo systemctl daemon-reload
sudo systemctl restart jqc
```

### 1.8 Verify webhook delivery

1. In Stripe Dashboard → Webhooks → click your endpoint
2. Send a **test event**: `invoice.payment_failed`
3. Check your app logs: `tail -f /var/www/jqc/logs/jqc.log | grep BILLING`
4. You should see: `BILLING WEBHOOK | event=invoice.payment_failed`

---

## 2. Email Deliverability (SPF, DKIM, DMARC)

Without these DNS records, billing emails (trial ending, payment failed, etc.)
will land in spam or be rejected outright.

### 2.1 SPF Record

SPF tells receiving servers which servers are allowed to send email for your domain.

Add a **TXT record** on your sending domain (e.g. `jqc.app` or your custom domain):

| Name | Type | Value |
|---|---|---|
| `@` | TXT | `v=spf1 include:YOUR-SMTP-PROVIDER.com ~all` |

Replace `YOUR-SMTP-PROVIDER.com` with your actual SMTP provider's include directive:

| Provider | Include value |
|---|---|
| SendGrid | `include:sendgrid.net` |
| Mailgun | `include:mailgun.org` |
| Amazon SES | `include:amazonses.com` |
| Google Workspace | `include:_spf.google.com` |
| Your own server | `ip4:YOUR.SERVER.IP.ADDRESS` |

**Example for your own server:**
```
v=spf1 ip4:123.45.67.89 ~all
```

**Example for SendGrid:**
```
v=spf1 include:sendgrid.net ~all
```

> ⚠️ Only one SPF record per domain is allowed. If one already exists, append to it.

### 2.2 DKIM Record

DKIM cryptographically signs outgoing emails so recipients can verify they're authentic.

**If using SendGrid / Mailgun / Amazon SES:**
- Follow their dashboard instructions to generate a DKIM key pair
- They will give you TXT records to add (usually at `mail._domainkey.yourdomain.com`)
- Add those records exactly as provided

**If sending from your own server (Postfix + OpenDKIM):**

```bash
# Install OpenDKIM
sudo apt install opendkim opendkim-tools

# Generate key pair
sudo mkdir -p /etc/opendkim/keys/yourdomain.com
sudo opendkim-genkey -b 2048 -d yourdomain.com -D /etc/opendkim/keys/yourdomain.com -s mail -v

# View the TXT record to add to DNS
sudo cat /etc/opendkim/keys/yourdomain.com/mail.txt
```

Add the output as a TXT record at `mail._domainkey.yourdomain.com`.

### 2.3 DMARC Record

DMARC tells receiving servers what to do with emails that fail SPF/DKIM checks,
and where to send failure reports.

Add a **TXT record**:

| Name | Type | Value |
|---|---|---|
| `_dmarc` | TXT | `v=DMARC1; p=quarantine; pct=100; rua=mailto:dmarc@yourdomain.com` |

Start with `p=none` (monitoring only) for the first 2 weeks, then move to `p=quarantine`:

```
# Week 1-2: monitor only
v=DMARC1; p=none; rua=mailto:dmarc@yourdomain.com

# After confirming SPF and DKIM pass in reports:
v=DMARC1; p=quarantine; pct=100; rua=mailto:dmarc@yourdomain.com
```

### 2.4 Test email deliverability

Once DNS records propagate (up to 48 hours), test with:

1. **[mail-tester.com](https://mail-tester.com)** — free, shows your score out of 10 and exactly what's failing
2. **[MXToolbox](https://mxtoolbox.com/emailhealth)** — checks SPF, DKIM, DMARC, blacklists

Target score: **9/10 or higher** before sending billing emails to customers.

### 2.5 Verify from the app

```bash
# Send a test billing email from Flask shell
cd /var/www/jqc
source venv/bin/activate
flask shell

from app.billing.emails import send_billing_email
send_billing_email(
    'your@email.com',
    'trial_ending',
    {
        'trial_ends_at':  'July 15, 2026',
        'subscribe_url':  'https://demo.jqc.app/billing/subscribe',
        'days_left':      3,
        'tenant_name':    'Test Tenant',
    }
)
exit()
```

Check that it arrives in the inbox (not spam) and renders correctly.

---

## 3. Cron Jobs — Final Setup

After going live, make sure all cron jobs including the new trial reminder are configured.

Edit the crontab for the `jqc` user:

```bash
sudo crontab -u jqc -e
```

Add all six jobs (replace `YOUR_SECRET` and `yourdomain.jqc.app`):

```cron
# SLA breach alerts — every 30 minutes
*/30 * * * * curl -s -X POST https://yourdomain.jqc.app/notifications/check-sla \
    -d "token=YOUR_DIGEST_SECRET" >> /var/www/jqc/logs/cron.log 2>&1

# Daily email digest — 7 AM
0 7 * * * curl -s -X POST https://yourdomain.jqc.app/notifications/send-digest \
    -d "token=YOUR_DIGEST_SECRET&frequency=daily" >> /var/www/jqc/logs/cron.log 2>&1

# Weekly email digest — Monday 7 AM
0 7 * * 1 curl -s -X POST https://yourdomain.jqc.app/notifications/send-digest \
    -d "token=YOUR_DIGEST_SECRET&frequency=weekly" >> /var/www/jqc/logs/cron.log 2>&1

# Facility score trend alerts — 8 AM daily
0 8 * * * curl -s -X POST https://yourdomain.jqc.app/notifications/check-score-trends \
    -d "token=YOUR_DIGEST_SECRET" >> /var/www/jqc/logs/cron.log 2>&1

# Scheduled reports — 8 AM daily
0 8 * * * curl -s -X POST https://yourdomain.jqc.app/scheduled-reports/run \
    -d "secret=YOUR_DIGEST_SECRET" >> /var/www/jqc/logs/cron.log 2>&1

# Expired API token cleanup — 3 AM daily
0 3 * * * curl -s -X POST https://yourdomain.jqc.app/notifications/cleanup-tokens \
    -d "token=YOUR_DIGEST_SECRET" >> /var/www/jqc/logs/cron.log 2>&1

# Trial-ending reminder emails — 9 AM daily  ← NEW
0 9 * * * curl -s -X POST https://yourdomain.jqc.app/notifications/trial-reminders \
    -d "token=YOUR_DIGEST_SECRET" >> /var/www/jqc/logs/cron.log 2>&1

# Per-tenant DB backup — 2 AM daily
0 2 * * * set -a; . /etc/jqc/control.env; set +a; \
    python -m control.backup --tenant all \
    --output-dir /var/backups/jqc >> /var/www/jqc/logs/cron.log 2>&1
```

> **Note on `yourdomain.jqc.app`:** Use your tenant-zero subdomain (e.g. `lts.jqc.app`) or
> your custom domain (e.g. `jqc.ltservicesinc.com`). The trial-reminders endpoint is exempt
> from tenant middleware so any valid host works for it, but the others require a valid tenant host.

---

## 4. Final Pre-Launch Verification

Run through this checklist before announcing to customers:

### Stripe

- [ ] `python -m control.cli list-plans` shows `price_live_…` IDs (not `price_test_…`)
- [ ] Stripe Dashboard → Webhooks shows endpoint is **enabled** in **live mode**
- [ ] Made a real test payment through the subscribe flow using a live card
- [ ] Webhook delivery log shows `200 OK` responses
- [ ] `billing_email` is set on tenant-zero (check: `python -m control.cli list-plans`)

### Email

- [ ] mail-tester.com score ≥ 9/10
- [ ] SPF TXT record published and passes at mxtoolbox.com
- [ ] DKIM TXT record published and passes
- [ ] DMARC TXT record published (start with `p=none`)
- [ ] Test billing email arrives in Gmail inbox, not spam
- [ ] Test billing email HTML renders correctly (blue header, button, footer)

### Trial reminder cron

- [ ] Test manually: `curl -X POST https://yourdomain.jqc.app/notifications/trial-reminders -d "token=YOUR_SECRET"`
- [ ] Response: `{"ok": true, "sent": N, "skipped": N, "errors": 0}`
- [ ] Cron entry is in `crontab -u jqc -l`

### Backup

- [ ] `python -m control.backup --tenant all --output-dir /var/backups/jqc` runs without errors
- [ ] `/var/backups/jqc/` contains `.sql.gz` files for each tenant
- [ ] Backup cron entry is in the crontab

---

## 5. Post-Launch Monitoring

Watch these in the first week after going live:

```bash
# App errors
tail -f /var/www/jqc/logs/jqc.log | grep -E 'ERROR|WARNING|BILLING'

# Cron job results
tail -f /var/www/jqc/logs/cron.log

# Stripe events (Stripe Dashboard → Developers → Events)
# Look for: checkout.session.completed, customer.subscription.updated

# Trial reminders sent
grep 'TRIAL REMINDERS' /var/www/jqc/logs/jqc.log
```