WebChecker--Web-app-/CLAUDE.md

# CLAUDE.md — AI Developer Context

This file gives Claude (or any AI assistant) the context needed to continue development on this project without re-reading the entire codebase from scratch.

---

## What This Project Is

**Bid Checker Web** is a Flask web application that is a direct port of a Tkinter desktop application. Both apps share the **same MySQL database** — this is the single most important constraint. Any schema change, query, or encryption logic must remain compatible with what the desktop app writes and reads.

The app helps a small team at LT Services Inc. (Falls Church, VA) track government procurement websites across scheduled shifts, manage bid/opportunity follow-up, and run AI-powered solicitation document analysis.

---

## Architecture at a Glance

```
Browser → Nginx (reverse proxy) → Gunicorn (4 workers) → Flask app
                                                              │
                                                    MySQL (shared with desktop)
```

- **Entry point:** `wsgi.py` → `app.py::create_app()` — **only `wsgi.py` calls `create_app()`**; the bottom of `app.py` no longer has a module-level call (that caused double initialisation)
- **No ORM** — all DB access is raw SQL via `mysql-connector-python` in `models.py`
- **No frontend framework** — vanilla JS, no build step, no npm
- **Blueprints:** one file per feature area in `routes/`
- **Templates:** Jinja2; most extend `base.html` — **`login.html` is a standalone exception** (see CSRF section)
- **Static assets:** single `style.css` + `app.js` — no preprocessor

---

## Critical Constraints

### 1. Shared Database
Never rename columns, drop tables, or change column types without verifying the desktop app still works. Key schema facts:
- `activity_log` timestamp column is **`logged_at`** (DDL and queries both use `logged_at`; a migration in `config.py` renames `created_at` → `logged_at` for old web-only installs)
- `app_log` timestamp column is **`logged_at`**
- All other tables generally use `created_at`
- The desktop app writes `bid_tracker`, `bid_updates`, `ai_analysis_log`, `ai_criteria`, `app_settings`, `users`, `websites`, `website_credentials`, `shifts`, `shift_users`, `shift_websites`, `shift_checks`, `login_attempts`

### 2. Credential Encryption
`utils/crypto.py` must stay byte-for-byte compatible with the desktop's `utils/crypto.py`:
- `_APP_SECRET = b"WebsiteChecker-v1-CredentialKey"` — never change
- `_ITERATIONS = 100_000` — never change
- Salt stored as **base64** in `app_settings` under key `"crypto.salt"`
- Ciphertext has **`enc:`** prefix; values without this prefix are legacy plaintext and returned as-is
- Calling `reset_fernet()` forces key reload if the salt changes

### 3. No Inline JS with Dynamic Jinja Values
All button `onclick` handlers that need dynamic data (site ID, site name, note text) **must use `data-*` attributes** on the HTML element and read them in a delegated event listener. Direct `onclick="fn({{ value }})"` breaks when the value contains quotes, apostrophes, or backslashes.

Example of the correct pattern:
```html
<button class="js-check" data-id="{{ site.id }}" data-name="{{ site.name }}">Check</button>
```
```js
document.getElementById('list').addEventListener('click', function(e) {
  const btn = e.target.closest('.js-check');
  if (btn) openCheckModal(btn.dataset.id, btn.dataset.name);
});
```

### 4. Jinja Macros and `{% extends %}`
Jinja macros defined in the same file as `{% extends "base.html" %}` cannot be called with `{{ macro_name(...) }}` before the macro definition is reached. **Do not use macros in child templates.** Inline the HTML directly or use JS to build dynamic content.

### 5. CSS Specificity — Local `<style>` vs `style.css`
The global `static/css/style.css` is loaded in `base.html` and has equal or higher specificity than local `<style>` blocks in child templates. Always fix layout issues in `style.css` — do not rely on local `<style>` overrides.

### 6. CSRF Protection (Flask-WTF)
`CSRFProtect(app)` is initialised inside `create_app()` in `app.py`. All non-GET requests are validated automatically.

**How tokens reach the server:**

| Request type | Mechanism |
|---|---|
| Standard HTML form (in a template that extends `base.html`) | `app.js` IIFE injects a hidden `csrf_token` input into every POST form at page-load time, reading from `<meta name="csrf-token">` in `base.html` |
| Standalone template (e.g. `login.html`) | Must include `<input type="hidden" name="csrf_token" value="{{ csrf_token() }}">` **directly in the form** — these pages do not load `app.js` or `base.html` |
| `fetch()` / AJAX POST with `FormData` | Append `fd.append('csrf_token', getCsrfToken())` OR add header `'X-CSRFToken': getCsrfToken()` |
| `fetch()` / AJAX POST with JSON body | Add header `'X-CSRFToken': getCsrfToken()` (token must be a header; JSON body is not inspected) |
| JS-built form injected into DOM via `innerHTML` | Inline the token: `'<input type="hidden" name="csrf_token" value="' + getCsrfToken() + '">'` |

**`getCsrfToken()`** is defined in `app.js` and reads `document.querySelector('meta[name="csrf-token"]').content`.

**`SECRET_KEY` must be set in `.env`** — if it is absent, each Gunicorn worker generates its own random key, session cookies signed by one worker are rejected by another, and CSRF validation fails for any cross-worker request.

### 7. `SECRET_KEY` is Required
```
# .env
SECRET_KEY=<long-random-hex>   # generate: python -c "import secrets; print(secrets.token_hex(32))"
```
Without this, sessions and CSRF tokens are broken across the 4 Gunicorn workers. `app.py` logs a WARNING at startup if it is missing.

---

## Key Files Reference

| File | Purpose | Notes |
|------|---------|-------|
| `app.py` | Flask factory | Registers all 11 blueprints; `CSRFProtect(app)`; `hhmm` template filter; logs WARNING if `SECRET_KEY` not set |
| `wsgi.py` | Gunicorn entry point | Only place that calls `create_app()` — do not add a second call elsewhere |
| `config.py` | DB config, DDL, settings | Calls `load_dotenv()` at top — **must be before `DB_CONFIG` dict**; contains safe re-runnable migrations |
| `models.py` | All DB queries | ~1,650 lines; no ORM; every function opens/closes its own connection; `import datetime` at top |
| `utils/crypto.py` | Fernet encryption | Must match desktop exactly |
| `utils/decorators.py` | `@login_required`, `@admin_required` | Simple session checks |
| `utils/email.py` | SMTP email helper | `send_email(to, subject, body_text)` reads smtp settings from `app_settings`; used by bid reminders and password reset |
| `static/css/style.css` | Full design system | Light theme, DM Sans + DM Mono, CSS variables in `:root` |
| `static/js/app.js` | Global JS utilities | `openModal()`, `closeModal()`, `copyToClipboard()`, `timeAgo()`, mobile sidebar toggle, session timeout warning, `getCsrfToken()`, CSRF auto-inject IIFE |
| `templates/login.html` | Standalone login page | Does NOT extend `base.html`; has its own `<head>` and no `app.js`; CSRF token must be a direct hidden input |
| `templates/forgot_password.html` | Standalone forgot-password page | Same constraints as `login.html` — standalone, direct CSRF hidden input, no `app.js` |
| `templates/reset_password.html` | Standalone reset-password page | Same constraints as `login.html` — standalone, direct CSRF hidden input, no `app.js` |
| `templates/profile.html` | User profile editor | Extends `base.html`; users can update `full_name` and `email`; username/role are read-only |

---

## Session & Auth

- Session key: `session["user"]` — dict with `id`, `username`, `full_name`, `role`, `email`
- Role values: `"admin"` or `"user"`
- Session lifetime: 30 minutes (`app.permanent_session_lifetime`)
- Login rate limit: 5 attempts → 15-minute lockout (enforced in `models.check_login_allowed`)
- `@login_required` — redirects to `/login` if no session
- `@admin_required` — redirects to login or 403 if not admin
- **Session fixation prevention** — `session.clear()` is called in `auth.login` immediately before setting `session["user"]` on successful authentication
- **Keep-alive endpoint** — `GET /ping` (`routes/auth.py`, `@login_required`) touches `session.modified = True` and returns 204; called by the session-timeout warning in `app.js`
- **Password reset** — `POST /forgot-password` creates a 1-hour `password_reset_tokens` token and emails a link. `POST /reset-password/<token>` validates, sets the new password, and deletes the token. Expired tokens are purged on every successful login via `purge_expired_reset_tokens()`.
- **Profile editing** — `GET/POST /profile` lets any authenticated user update their `full_name` and `email`. Changes are immediately reflected in `session["user"]` so the sidebar name refreshes without re-login.

---

## Settings System

`app_settings` table stores key-value pairs for runtime configuration.

| `key_name` | `.env` fallback | Used by |
|---|---|---|
| `groq.api_key` | `GROQ_API_KEY` | AI Summary route |
| `groq.model` | `GROQ_MODEL` | AI Summary route |
| `email.smtp_host` | `SMTP_HOST` | Email reports |
| `email.smtp_port` | `SMTP_PORT` | Email reports |
| `email.smtp_user` | `SMTP_USER` | Email reports |
| `email.smtp_password` | `SMTP_PASSWORD` | Email reports |
| `email.smtp_from` | `SMTP_FROM` | Email reports |
| `crypto.salt` | *(generated)* | Fernet key derivation |

`get_setting(key, default)` reads DB first, then `.env` fallback, then `default`.  
`set_setting(key, value)` upserts into `app_settings`.  
On startup, `initialize_database()` seeds blank DB rows from `.env` using `ON DUPLICATE KEY UPDATE value = IF(value='', VALUES(value), value)`.

---

## AI Summary Feature

Route: `/ai-summary/` (`routes/ai_summary.py`)

- **Stage 1 prompt** (`_EXTRACTION_PROMPT`) — extracts 12 fields; treats all uploaded documents as one combined source (do NOT analyze per-file); driving distance from `2815 Hartland Road, Falls Church, VA 22043`
- **Stage 2 prompt** (`_CRITERIA_PROMPT_SUFFIX`) — appended only when active criteria exist; produces a machine-readable `RECOMMENDATION: PURSUE|PASS|UNCLEAR` line
- **File extraction** — `pypdf` for PDF, `python-docx` (imported as `docx`) for DOCX/DOC, `openpyxl` for XLSX, plain decode for TXT/CSV/MD
- **Magic-byte validation** — `_validate_magic()` rejects files whose bytes don't match their declared extension (disguised uploads)
- **API call** — direct `requests.post()` to `https://api.groq.com/openai/v1/chat/completions` — the `groq` Python SDK is **not** installed
- **Model** — `claude-sonnet-4-20250514` should NOT be used here; use `llama-3.3-70b-versatile` (Groq)
- **Max tokens** — 4,096; **temperature** — 0.2
- **Text limit** — combined document text is truncated to 14,000 characters before being sent; the JSON response includes `"truncated": true` when this occurs
- **`sort_order` input** — always wrap `int(request.form.get("sort_order", 0) or 0)` in `try/except (ValueError, TypeError)` — bad input must degrade gracefully to 0, not raise a 500
- **Output rendering** — `marked.js` (CDN) renders the AI summary as formatted markdown in both the live output and history modal
- **History detail** — `GET /ai-summary/history/<id>` returns `criteria_snapshot` (JSON string of `[{title, description}]`); the history modal parses and displays the criteria titles used
- **Delete history** — `POST /ai-summary/history/<id>/delete` enforces ownership (user can delete own; admin can delete any); logs the action
- **Verdict filter** — client-side pill buttons (All / PURSUE / PASS / UNCLEAR / No verdict) above the history table toggle row visibility

---

## Bid Tracker Feature

Route: `/bids/` (`routes/bid_tracker.py`)

Split-pane UI — left: filterable bid list, right: detail panel with update timeline.

Key AJAX endpoints (all return JSON, no page reload):
- `GET /bids/list/json?status=` — bid list filtered by status
- `GET /bids/<id>/json` — bid detail + updates + `can_edit` flag
- `POST /bids/<id>/updates/json` — post new update
- `POST /bids/updates/<id>/delete/json` — delete update

Ownership rule: users can only edit/delete their own bids. Admins can edit/delete any bid. Enforced via `can_edit = is_admin or bid.added_by == user.id`.

**`_ser(row)`** — module-level helper that converts a DB dict to JSON-serialisable form (dates/times → ISO strings). Do not redefine it inline inside individual functions.

All write operations (create, edit, delete bid; add, delete update — both form and JSON variants) call `log_action()`.

---

## User Dashboard

Route: `/dashboard/` (`routes/user_dashboard.py`)

- Sites are grouped by `check_type` (`daily` / `weekly` / other) and rendered as collapsible groups
- Each site card has two rows: `.sc-row1` (main flex row) and `.sc-row2` (note row, only if `user_note` is set)
- `.sc-actions` uses `margin-left:auto` to push buttons to the right edge of `.sc-row1`
- `.site-card` is `display:block` — **not flex** — so `.sc-row1` and `.sc-row2` stack vertically
- **Health dots** — `GET /dashboard/health/<id>` does a server-side HEAD request (6s timeout); returns `{status, ms}` JSON. Colors: green <3000ms, orange slow/timeout, red unreachable, gray error
- The `hhmm` Jinja filter (`app.py`) handles MySQL `TIME` columns returned as `datetime.timedelta`

---

## Common Patterns

### Adding a new admin page
1. Create `routes/admin_mypage.py` with a Blueprint at `/admin/mypage`
2. Register it in `app.py::create_app()`
3. Add a nav link in `templates/base.html` inside the admin nav section
4. Create `templates/admin/mypage.html` extending `base.html`
5. Add model functions to `models.py`
6. Add any new CSS classes to `static/css/style.css`

### Adding a new modal
```html
<div class="modal-overlay" id="modal-mymodal">
  <div class="modal">                          <!-- or modal-dialog for admin pages -->
    <div class="modal-header">
      <span class="modal-title">Title</span>
      <button class="modal-close" onclick="closeModal('modal-mymodal')">✕</button>
    </div>
    <div class="modal-body">…</div>
    <div class="modal-footer">
      <button class="btn btn-secondary" onclick="closeModal('modal-mymodal')">Cancel</button>
      <button class="btn btn-primary">Save</button>
    </div>
  </div>
</div>
```
Open with `openModal('modal-mymodal')` from `app.js`.

### Adding a new POST form (CSRF checklist)
- **Template extends `base.html`** → nothing extra needed; `app.js` IIFE injects the token automatically
- **Standalone template** (like `login.html`) → add `<input type="hidden" name="csrf_token" value="{{ csrf_token() }}">` inside the `<form>`
- **`fetch()` POST** → add `'X-CSRFToken': getCsrfToken()` to the `headers` object
- **JS-built form string** → concatenate `'<input type="hidden" name="csrf_token" value="' + getCsrfToken() + '">'`

### Serialising DB rows to JSON
MySQL connector returns `datetime`, `date`, `timedelta` objects which are not JSON-serialisable. Use the shared helper (already defined at module level in `bid_tracker.py`; replicate it where needed):
```python
def _ser(row):
    return {k: v.isoformat() if hasattr(v, 'isoformat') else v for k, v in row.items()}
```

### Adding a new model query with optional search/filter
Do **not** use f-strings to interpolate WHERE clauses into SQL. Build the query with string concatenation and always pass user values through the `%s` parameter list:
```python
sql = "SELECT ... FROM table"
params = []
if search:
    sql += " WHERE col LIKE %s"
    params.append(f"%{search}%")
sql += " ORDER BY id DESC LIMIT %s"
params.append(limit)
cur.execute(sql, params)
```

---

## Known Gotchas

| Gotcha | Detail |
|--------|--------|
| `activity_log.logged_at` | DDL now creates `logged_at`. A migration in `config.py` renames `created_at` → `logged_at` for old web-only installs. Queried as `al.logged_at AS created_at` in `get_activity_log()` |
| `TIME` columns → `timedelta` | MySQL returns `TIME` as `datetime.timedelta`. Use the `\|hhmm` filter in templates |
| `GROQ_API_KEY` in `.env` | Read directly via `os.environ.get("GROQ_API_KEY")` in the analyze route — not via `get_setting()` alone — because `get_setting()` depends on the DB row being populated |
| `div.modal` vs `.modal-overlay` | All modal backdrops must use `class="modal-overlay"`. Inner dialog boxes use `class="modal"` or `class="modal-dialog"`. Never use `div.modal` as a backdrop |
| Jinja macros in child templates | Cannot call `{{ macro_name() }}` before the `{% macro %}` definition is parsed. Inline the HTML instead |
| `INSERT IGNORE` for settings seed | Use `ON DUPLICATE KEY UPDATE value = IF(value='', VALUES(value), value)` — `INSERT IGNORE` silently skips rows that already exist, even with empty values |
| `bcrypt` not in original requirements | Added as `bcrypt==4.1.3`. Required by `models.py` for password hashing |
| `login.html` is standalone | Does NOT extend `base.html`. Has no `app.js`, no CSRF meta tag. Any form on this page needs `{{ csrf_token() }}` as a direct hidden input |
| `SECRET_KEY` must be fixed | `os.urandom(32)` fallback generates a different key per Gunicorn worker — sessions and CSRF break across workers. Always set `SECRET_KEY` in `.env` |
| `create_app()` called once | Only `wsgi.py` calls `create_app()`. Do not add a module-level call to `app.py` — it causes double initialisation (double `initialize_database()`, double blueprint registration) |
| `import datetime` in `models.py` | Already imported at the top of the file. Do not add inline `import datetime` inside functions |

---

## Development Workflow

```bash
# Activate venv
source /home/webchecker/venv/bin/activate

# Install / update dependencies (e.g. after adding Flask-WTF)
pip install -r requirements.txt

# After changing Python files — reload Gunicorn (zero-downtime)
sudo systemctl reload webchecker

# After changing templates or static files — no restart needed (served live)

# View live logs
sudo journalctl -u webchecker -f

# View Nginx errors
sudo tail -f /var/log/nginx/webchecker_error.log

# Run a quick DB query
mysql -u webchecker_user -p webchecker -e "SELECT key_name, value FROM app_settings;"
```

---

## File Placement on Server

```
/opt/webchecker/          ← project root (or /home/webchecker/app/)
/opt/webchecker/.env      ← secrets (chmod 600, owned by webchecker)
/opt/webchecker/venv/     ← Python virtual environment
/var/log/webchecker/      ← Gunicorn access.log + error.log
/run/webchecker/          ← Gunicorn UNIX socket (webchecker.sock)
/etc/systemd/system/webchecker.service
/etc/nginx/sites-available/webchecker
```

---

## Change Philosophy

1. **Surgical, additive patches** — smallest possible change to achieve the goal
2. **Preserve all routes, function names, variable names** unless explicitly directed otherwise
3. **Never remove existing functionality** unless explicitly directed
4. **Log all create/update/delete actions** via `log_action()`
5. **Migration existence checks** — all migrations safe to re-run
6. **Full file contents for 1–3 file changes**; deployment map for larger changesets
7. **Explicit deploy instructions** — migration steps separated from code steps
8. **Root cause analysis** on errors — never apply temporary workarounds

---

## Completed Improvements

All planned items have been implemented. The list below serves as a record and cross-reference.

### Security
- [x] File upload size cap (`MAX_CONTENT_LENGTH = 20 MB`); JSON error for `/ai-summary/`, flash+redirect elsewhere
- [x] HTTP security headers — `X-Frame-Options`, `X-Content-Type-Options`, `Referrer-Policy` on every response
- [x] Prevent last-admin demotion/deactivation — `update_user()` raises `ValueError` if zero active admins would remain
- [x] File MIME type validation — `_validate_magic()` checks magic bytes and UTF-8 decodability; no extra dependency
- [x] Session fixation prevention — `session.clear()` before setting session on login
- [x] Password reset tokens — single-use, 1-hour expiry, purged on every successful login

### UI / UX
- [x] Mobile-responsive sidebar — hamburger toggle (`sidebar-toggle` button in `base.html`), CSS `transform:translateX`, backdrop overlay; IIFE in `app.js`
- [x] Bid due-date urgency badges — "Overdue" (red) and "Due Soon" (yellow) client-side pills in bid list
- [x] Relative timestamps in Activity Log — `timeAgo()` in `app.js`; `[data-ts]` attribute on timestamp `<span>` elements in `logs.html`
- [x] Empty state for AI analysis history — robot emoji + "Start Analyzing" CTA when history is empty
- [x] Paginate bid list — `LIMIT`/`OFFSET` in `get_all_bids()`; "Load more" button in bid list; `_bidOffset` / `_PAGE_SIZE` state vars
- [x] AI history verdict filter — pill buttons (All / PURSUE / PASS / UNCLEAR / No verdict) filter rows client-side
- [x] Highlight today in shift calendar — JS IIFE maps `getDay()` → MySQL DAYOFWEEK → column index; adds `.cal-today` CSS class
- [x] Website list search + type filter — inline search input + dropdown filter client-side in `admin/websites.html`

### Functionality
- [x] Missed-shift alerting — `get_missed_shifts_today()` in `models.py`; warning card on admin dashboard
- [x] Bid deadline email reminders — `send_reminders` route; "📧 Remind" button (admin only) in bid toolbar
- [x] Server-side health checks — `/dashboard/health/<id>` HEAD request with 6s timeout; green/orange/red/gray dots
- [x] Password reset via email — `password_reset_tokens` table; `/forgot-password` + `/reset-password/<token>` routes; standalone templates; "Forgot password?" on login page
- [x] Shift calendar view — weekly grid tab; Mon–Sun columns; active shifts as rows; `day_map` server-rendered
- [x] Delete AI analysis history — `POST /ai-summary/history/<id>/delete`; ownership check; delete button in history modal
- [x] Criteria snapshot in history modal — `analysis_detail` returns `criteria_snapshot`; modal shows titles of criteria used
- [x] Purge expired password reset tokens — `purge_expired_reset_tokens()` called on every successful login (wrapped in try/except)
- [x] Test SMTP + Groq buttons — `POST /admin/settings/test-email` and `/test-groq`; inline AJAX result in settings page
- [x] Admin send password reset link — `POST /admin/users/<id>/send-reset`; generates token + sends email; "✉ Reset" button per user row
- [x] Export bids to CSV — `GET /bids/export.csv`; respects active status filter; "⬇ CSV" button in toolbar
- [x] Admin dashboard bid KPIs — Open Bids, Due This Week (red when >0), AI Analyses (30d) added to `get_admin_dashboard_stats()`
- [x] User profile page — `GET/POST /profile`; users edit own `full_name` and `email`; session updated immediately; "👤 My Profile" in sidebar
- [x] SMTP From Address field — `email.smtp_from` exposed in Admin → Settings form and saved alongside other email fields

- [x] Incomplete-shift reminder emails — manual `POST /admin/shifts/send-incomplete-reminders` + automated `GET /internal/cron/shift-reminders?token=<CRON_SECRET>`; `get_incomplete_shift_users_near_end()` + `record_shift_reminder()` in `models.py`; `shift_reminder_log` table deduplicates to one email per user/shift/day; `do_send_incomplete_reminders()` shared helper in `admin_shifts.py`; systemd timer fires every 5 min

### Pending
- No known pending items. Add new items here as they are identified.