Files
WebChecker--Web-app-/CLAUDE.md
T
2026-06-25 16:52:19 -04:00

373 lines
23 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# CLAUDE.md — AI Developer Context
This file gives Claude (or any AI assistant) the context needed to continue development on this project without re-reading the entire codebase from scratch.
---
## What This Project Is
**Website Checker Web** is a Flask web application that is a direct port of a Tkinter desktop application. Both apps share the **same MySQL database** — this is the single most important constraint. Any schema change, query, or encryption logic must remain compatible with what the desktop app writes and reads.
The app helps a small team at LT Services Inc. (Falls Church, VA) track government procurement websites across scheduled shifts, manage bid/opportunity follow-up, and run AI-powered solicitation document analysis.
---
## Architecture at a Glance
```
Browser → Nginx (reverse proxy) → Gunicorn (4 workers) → Flask app
MySQL (shared with desktop)
```
- **Entry point:** `wsgi.py``app.py::create_app()`**only `wsgi.py` calls `create_app()`**; the bottom of `app.py` no longer has a module-level call (that caused double initialisation)
- **No ORM** — all DB access is raw SQL via `mysql-connector-python` in `models.py`
- **No frontend framework** — vanilla JS, no build step, no npm
- **Blueprints:** one file per feature area in `routes/`
- **Templates:** Jinja2; most extend `base.html`**`login.html` is a standalone exception** (see CSRF section)
- **Static assets:** single `style.css` + `app.js` — no preprocessor
---
## Critical Constraints
### 1. Shared Database
Never rename columns, drop tables, or change column types without verifying the desktop app still works. Key schema facts:
- `activity_log` timestamp column is **`logged_at`** (DDL and queries both use `logged_at`; a migration in `config.py` renames `created_at``logged_at` for old web-only installs)
- `app_log` timestamp column is **`logged_at`**
- All other tables generally use `created_at`
- The desktop app writes `bid_tracker`, `bid_updates`, `ai_analysis_log`, `ai_criteria`, `app_settings`, `users`, `websites`, `website_credentials`, `shifts`, `shift_users`, `shift_websites`, `shift_checks`, `login_attempts`
### 2. Credential Encryption
`utils/crypto.py` must stay byte-for-byte compatible with the desktop's `utils/crypto.py`:
- `_APP_SECRET = b"WebsiteChecker-v1-CredentialKey"` — never change
- `_ITERATIONS = 100_000` — never change
- Salt stored as **base64** in `app_settings` under key `"crypto.salt"`
- Ciphertext has **`enc:`** prefix; values without this prefix are legacy plaintext and returned as-is
- Calling `reset_fernet()` forces key reload if the salt changes
### 3. No Inline JS with Dynamic Jinja Values
All button `onclick` handlers that need dynamic data (site ID, site name, note text) **must use `data-*` attributes** on the HTML element and read them in a delegated event listener. Direct `onclick="fn({{ value }})"` breaks when the value contains quotes, apostrophes, or backslashes.
Example of the correct pattern:
```html
<button class="js-check" data-id="{{ site.id }}" data-name="{{ site.name }}">Check</button>
```
```js
document.getElementById('list').addEventListener('click', function(e) {
const btn = e.target.closest('.js-check');
if (btn) openCheckModal(btn.dataset.id, btn.dataset.name);
});
```
### 4. Jinja Macros and `{% extends %}`
Jinja macros defined in the same file as `{% extends "base.html" %}` cannot be called with `{{ macro_name(...) }}` before the macro definition is reached. **Do not use macros in child templates.** Inline the HTML directly or use JS to build dynamic content.
### 5. CSS Specificity — Local `<style>` vs `style.css`
The global `static/css/style.css` is loaded in `base.html` and has equal or higher specificity than local `<style>` blocks in child templates. Always fix layout issues in `style.css` — do not rely on local `<style>` overrides.
### 6. CSRF Protection (Flask-WTF)
`CSRFProtect(app)` is initialised inside `create_app()` in `app.py`. All non-GET requests are validated automatically.
**How tokens reach the server:**
| Request type | Mechanism |
|---|---|
| Standard HTML form (in a template that extends `base.html`) | `app.js` IIFE injects a hidden `csrf_token` input into every POST form at page-load time, reading from `<meta name="csrf-token">` in `base.html` |
| Standalone template (e.g. `login.html`) | Must include `<input type="hidden" name="csrf_token" value="{{ csrf_token() }}">` **directly in the form** — these pages do not load `app.js` or `base.html` |
| `fetch()` / AJAX POST with `FormData` | Append `fd.append('csrf_token', getCsrfToken())` OR add header `'X-CSRFToken': getCsrfToken()` |
| `fetch()` / AJAX POST with JSON body | Add header `'X-CSRFToken': getCsrfToken()` (token must be a header; JSON body is not inspected) |
| JS-built form injected into DOM via `innerHTML` | Inline the token: `'<input type="hidden" name="csrf_token" value="' + getCsrfToken() + '">'` |
**`getCsrfToken()`** is defined in `app.js` and reads `document.querySelector('meta[name="csrf-token"]').content`.
**`SECRET_KEY` must be set in `.env`** — if it is absent, each Gunicorn worker generates its own random key, session cookies signed by one worker are rejected by another, and CSRF validation fails for any cross-worker request.
### 7. `SECRET_KEY` is Required
```
# .env
SECRET_KEY=<long-random-hex> # generate: python -c "import secrets; print(secrets.token_hex(32))"
```
Without this, sessions and CSRF tokens are broken across the 4 Gunicorn workers. `app.py` logs a WARNING at startup if it is missing.
---
## Key Files Reference
| File | Purpose | Notes |
|------|---------|-------|
| `app.py` | Flask factory | Registers all 11 blueprints; `CSRFProtect(app)`; `hhmm` template filter; logs WARNING if `SECRET_KEY` not set |
| `wsgi.py` | Gunicorn entry point | Only place that calls `create_app()` — do not add a second call elsewhere |
| `config.py` | DB config, DDL, settings | Calls `load_dotenv()` at top — **must be before `DB_CONFIG` dict**; contains safe re-runnable migrations |
| `models.py` | All DB queries | ~1,650 lines; no ORM; every function opens/closes its own connection; `import datetime` at top |
| `utils/crypto.py` | Fernet encryption | Must match desktop exactly |
| `utils/decorators.py` | `@login_required`, `@admin_required` | Simple session checks |
| `utils/email.py` | SMTP email helper | `send_email(to, subject, body_text)` reads smtp settings from `app_settings`; used by bid reminders and password reset |
| `static/css/style.css` | Full design system | Light theme, DM Sans + DM Mono, CSS variables in `:root` |
| `static/js/app.js` | Global JS utilities | `openModal()`, `closeModal()`, `copyToClipboard()`, `timeAgo()`, mobile sidebar toggle, session timeout warning, `getCsrfToken()`, CSRF auto-inject IIFE |
| `templates/login.html` | Standalone login page | Does NOT extend `base.html`; has its own `<head>` and no `app.js`; CSRF token must be a direct hidden input |
| `templates/forgot_password.html` | Standalone forgot-password page | Same constraints as `login.html` — standalone, direct CSRF hidden input, no `app.js` |
| `templates/reset_password.html` | Standalone reset-password page | Same constraints as `login.html` — standalone, direct CSRF hidden input, no `app.js` |
| `templates/profile.html` | User profile editor | Extends `base.html`; users can update `full_name` and `email`; username/role are read-only |
---
## Session & Auth
- Session key: `session["user"]` — dict with `id`, `username`, `full_name`, `role`, `email`
- Role values: `"admin"` or `"user"`
- Session lifetime: 30 minutes (`app.permanent_session_lifetime`)
- Login rate limit: 5 attempts → 15-minute lockout (enforced in `models.check_login_allowed`)
- `@login_required` — redirects to `/login` if no session
- `@admin_required` — redirects to login or 403 if not admin
- **Session fixation prevention** — `session.clear()` is called in `auth.login` immediately before setting `session["user"]` on successful authentication
- **Keep-alive endpoint** — `GET /ping` (`routes/auth.py`, `@login_required`) touches `session.modified = True` and returns 204; called by the session-timeout warning in `app.js`
- **Password reset** — `POST /forgot-password` creates a 1-hour `password_reset_tokens` token and emails a link. `POST /reset-password/<token>` validates, sets the new password, and deletes the token. Expired tokens are purged on every successful login via `purge_expired_reset_tokens()`.
- **Profile editing** — `GET/POST /profile` lets any authenticated user update their `full_name` and `email`. Changes are immediately reflected in `session["user"]` so the sidebar name refreshes without re-login.
---
## Settings System
`app_settings` table stores key-value pairs for runtime configuration.
| `key_name` | `.env` fallback | Used by |
|---|---|---|
| `groq.api_key` | `GROQ_API_KEY` | AI Summary route |
| `groq.model` | `GROQ_MODEL` | AI Summary route |
| `email.smtp_host` | `SMTP_HOST` | Email reports |
| `email.smtp_port` | `SMTP_PORT` | Email reports |
| `email.smtp_user` | `SMTP_USER` | Email reports |
| `email.smtp_password` | `SMTP_PASSWORD` | Email reports |
| `email.smtp_from` | `SMTP_FROM` | Email reports |
| `crypto.salt` | *(generated)* | Fernet key derivation |
`get_setting(key, default)` reads DB first, then `.env` fallback, then `default`.
`set_setting(key, value)` upserts into `app_settings`.
On startup, `initialize_database()` seeds blank DB rows from `.env` using `ON DUPLICATE KEY UPDATE value = IF(value='', VALUES(value), value)`.
---
## AI Summary Feature
Route: `/ai-summary/` (`routes/ai_summary.py`)
- **Stage 1 prompt** (`_EXTRACTION_PROMPT`) — extracts 12 fields; treats all uploaded documents as one combined source (do NOT analyze per-file); driving distance from `2815 Hartland Road, Falls Church, VA 22043`
- **Stage 2 prompt** (`_CRITERIA_PROMPT_SUFFIX`) — appended only when active criteria exist; produces a machine-readable `RECOMMENDATION: PURSUE|PASS|UNCLEAR` line
- **File extraction** — `pypdf` for PDF, `python-docx` (imported as `docx`) for DOCX/DOC, `openpyxl` for XLSX, plain decode for TXT/CSV/MD
- **Magic-byte validation** — `_validate_magic()` rejects files whose bytes don't match their declared extension (disguised uploads)
- **API call** — direct `requests.post()` to `https://api.groq.com/openai/v1/chat/completions` — the `groq` Python SDK is **not** installed
- **Model** — `claude-sonnet-4-20250514` should NOT be used here; use `llama-3.3-70b-versatile` (Groq)
- **Max tokens** — 4,096; **temperature** — 0.2
- **Text limit** — combined document text is truncated to 14,000 characters before being sent; the JSON response includes `"truncated": true` when this occurs
- **`sort_order` input** — always wrap `int(request.form.get("sort_order", 0) or 0)` in `try/except (ValueError, TypeError)` — bad input must degrade gracefully to 0, not raise a 500
- **Output rendering** — `marked.js` (CDN) renders the AI summary as formatted markdown in both the live output and history modal
- **History detail** — `GET /ai-summary/history/<id>` returns `criteria_snapshot` (JSON string of `[{title, description}]`); the history modal parses and displays the criteria titles used
- **Delete history** — `POST /ai-summary/history/<id>/delete` enforces ownership (user can delete own; admin can delete any); logs the action
- **Verdict filter** — client-side pill buttons (All / PURSUE / PASS / UNCLEAR / No verdict) above the history table toggle row visibility
---
## Bid Tracker Feature
Route: `/bids/` (`routes/bid_tracker.py`)
Split-pane UI — left: filterable bid list, right: detail panel with update timeline.
Key AJAX endpoints (all return JSON, no page reload):
- `GET /bids/list/json?status=` — bid list filtered by status
- `GET /bids/<id>/json` — bid detail + updates + `can_edit` flag
- `POST /bids/<id>/updates/json` — post new update
- `POST /bids/updates/<id>/delete/json` — delete update
Ownership rule: users can only edit/delete their own bids. Admins can edit/delete any bid. Enforced via `can_edit = is_admin or bid.added_by == user.id`.
**`_ser(row)`** — module-level helper that converts a DB dict to JSON-serialisable form (dates/times → ISO strings). Do not redefine it inline inside individual functions.
All write operations (create, edit, delete bid; add, delete update — both form and JSON variants) call `log_action()`.
---
## User Dashboard
Route: `/dashboard/` (`routes/user_dashboard.py`)
- Sites are grouped by `check_type` (`daily` / `weekly` / other) and rendered as collapsible groups
- Each site card has two rows: `.sc-row1` (main flex row) and `.sc-row2` (note row, only if `user_note` is set)
- `.sc-actions` uses `margin-left:auto` to push buttons to the right edge of `.sc-row1`
- `.site-card` is `display:block`**not flex** — so `.sc-row1` and `.sc-row2` stack vertically
- **Health dots** — `GET /dashboard/health/<id>` does a server-side HEAD request (6s timeout); returns `{status, ms}` JSON. Colors: green <3000ms, orange slow/timeout, red unreachable, gray error
- The `hhmm` Jinja filter (`app.py`) handles MySQL `TIME` columns returned as `datetime.timedelta`
---
## Common Patterns
### Adding a new admin page
1. Create `routes/admin_mypage.py` with a Blueprint at `/admin/mypage`
2. Register it in `app.py::create_app()`
3. Add a nav link in `templates/base.html` inside the admin nav section
4. Create `templates/admin/mypage.html` extending `base.html`
5. Add model functions to `models.py`
6. Add any new CSS classes to `static/css/style.css`
### Adding a new modal
```html
<div class="modal-overlay" id="modal-mymodal">
<div class="modal"> <!-- or modal-dialog for admin pages -->
<div class="modal-header">
<span class="modal-title">Title</span>
<button class="modal-close" onclick="closeModal('modal-mymodal')"></button>
</div>
<div class="modal-body"></div>
<div class="modal-footer">
<button class="btn btn-secondary" onclick="closeModal('modal-mymodal')">Cancel</button>
<button class="btn btn-primary">Save</button>
</div>
</div>
</div>
```
Open with `openModal('modal-mymodal')` from `app.js`.
### Adding a new POST form (CSRF checklist)
- **Template extends `base.html`** → nothing extra needed; `app.js` IIFE injects the token automatically
- **Standalone template** (like `login.html`) → add `<input type="hidden" name="csrf_token" value="{{ csrf_token() }}">` inside the `<form>`
- **`fetch()` POST** → add `'X-CSRFToken': getCsrfToken()` to the `headers` object
- **JS-built form string** → concatenate `'<input type="hidden" name="csrf_token" value="' + getCsrfToken() + '">'`
### Serialising DB rows to JSON
MySQL connector returns `datetime`, `date`, `timedelta` objects which are not JSON-serialisable. Use the shared helper (already defined at module level in `bid_tracker.py`; replicate it where needed):
```python
def _ser(row):
return {k: v.isoformat() if hasattr(v, 'isoformat') else v for k, v in row.items()}
```
### Adding a new model query with optional search/filter
Do **not** use f-strings to interpolate WHERE clauses into SQL. Build the query with string concatenation and always pass user values through the `%s` parameter list:
```python
sql = "SELECT ... FROM table"
params = []
if search:
sql += " WHERE col LIKE %s"
params.append(f"%{search}%")
sql += " ORDER BY id DESC LIMIT %s"
params.append(limit)
cur.execute(sql, params)
```
---
## Known Gotchas
| Gotcha | Detail |
|--------|--------|
| `activity_log.logged_at` | DDL now creates `logged_at`. A migration in `config.py` renames `created_at``logged_at` for old web-only installs. Queried as `al.logged_at AS created_at` in `get_activity_log()` |
| `TIME` columns → `timedelta` | MySQL returns `TIME` as `datetime.timedelta`. Use the `\|hhmm` filter in templates |
| `GROQ_API_KEY` in `.env` | Read directly via `os.environ.get("GROQ_API_KEY")` in the analyze route — not via `get_setting()` alone — because `get_setting()` depends on the DB row being populated |
| `div.modal` vs `.modal-overlay` | All modal backdrops must use `class="modal-overlay"`. Inner dialog boxes use `class="modal"` or `class="modal-dialog"`. Never use `div.modal` as a backdrop |
| Jinja macros in child templates | Cannot call `{{ macro_name() }}` before the `{% macro %}` definition is parsed. Inline the HTML instead |
| `INSERT IGNORE` for settings seed | Use `ON DUPLICATE KEY UPDATE value = IF(value='', VALUES(value), value)``INSERT IGNORE` silently skips rows that already exist, even with empty values |
| `bcrypt` not in original requirements | Added as `bcrypt==4.1.3`. Required by `models.py` for password hashing |
| `login.html` is standalone | Does NOT extend `base.html`. Has no `app.js`, no CSRF meta tag. Any form on this page needs `{{ csrf_token() }}` as a direct hidden input |
| `SECRET_KEY` must be fixed | `os.urandom(32)` fallback generates a different key per Gunicorn worker — sessions and CSRF break across workers. Always set `SECRET_KEY` in `.env` |
| `create_app()` called once | Only `wsgi.py` calls `create_app()`. Do not add a module-level call to `app.py` — it causes double initialisation (double `initialize_database()`, double blueprint registration) |
| `import datetime` in `models.py` | Already imported at the top of the file. Do not add inline `import datetime` inside functions |
---
## Development Workflow
```bash
# Activate venv
source /home/webchecker/venv/bin/activate
# Install / update dependencies (e.g. after adding Flask-WTF)
pip install -r requirements.txt
# After changing Python files — reload Gunicorn (zero-downtime)
sudo systemctl reload webchecker
# After changing templates or static files — no restart needed (served live)
# View live logs
sudo journalctl -u webchecker -f
# View Nginx errors
sudo tail -f /var/log/nginx/webchecker_error.log
# Run a quick DB query
mysql -u webchecker_user -p webchecker -e "SELECT key_name, value FROM app_settings;"
```
---
## File Placement on Server
```
/opt/webchecker/ ← project root (or /home/webchecker/app/)
/opt/webchecker/.env ← secrets (chmod 600, owned by webchecker)
/opt/webchecker/venv/ ← Python virtual environment
/var/log/webchecker/ ← Gunicorn access.log + error.log
/run/webchecker/ ← Gunicorn UNIX socket (webchecker.sock)
/etc/systemd/system/webchecker.service
/etc/nginx/sites-available/webchecker
```
---
## Change Philosophy
1. **Surgical, additive patches** — smallest possible change to achieve the goal
2. **Preserve all routes, function names, variable names** unless explicitly directed otherwise
3. **Never remove existing functionality** unless explicitly directed
4. **Log all create/update/delete actions** via `log_action()`
5. **Migration existence checks** — all migrations safe to re-run
6. **Full file contents for 13 file changes**; deployment map for larger changesets
7. **Explicit deploy instructions** — migration steps separated from code steps
8. **Root cause analysis** on errors — never apply temporary workarounds
---
## Completed Improvements
All planned items have been implemented. The list below serves as a record and cross-reference.
### Security
- [x] File upload size cap (`MAX_CONTENT_LENGTH = 20 MB`); JSON error for `/ai-summary/`, flash+redirect elsewhere
- [x] HTTP security headers — `X-Frame-Options`, `X-Content-Type-Options`, `Referrer-Policy` on every response
- [x] Prevent last-admin demotion/deactivation — `update_user()` raises `ValueError` if zero active admins would remain
- [x] File MIME type validation — `_validate_magic()` checks magic bytes and UTF-8 decodability; no extra dependency
- [x] Session fixation prevention — `session.clear()` before setting session on login
- [x] Password reset tokens — single-use, 1-hour expiry, purged on every successful login
### UI / UX
- [x] Mobile-responsive sidebar — hamburger toggle (`sidebar-toggle` button in `base.html`), CSS `transform:translateX`, backdrop overlay; IIFE in `app.js`
- [x] Bid due-date urgency badges — "Overdue" (red) and "Due Soon" (yellow) client-side pills in bid list
- [x] Relative timestamps in Activity Log — `timeAgo()` in `app.js`; `[data-ts]` attribute on timestamp `<span>` elements in `logs.html`
- [x] Empty state for AI analysis history — robot emoji + "Start Analyzing" CTA when history is empty
- [x] Paginate bid list — `LIMIT`/`OFFSET` in `get_all_bids()`; "Load more" button in bid list; `_bidOffset` / `_PAGE_SIZE` state vars
- [x] AI history verdict filter — pill buttons (All / PURSUE / PASS / UNCLEAR / No verdict) filter rows client-side
- [x] Highlight today in shift calendar — JS IIFE maps `getDay()` → MySQL DAYOFWEEK → column index; adds `.cal-today` CSS class
- [x] Website list search + type filter — inline search input + dropdown filter client-side in `admin/websites.html`
### Functionality
- [x] Missed-shift alerting — `get_missed_shifts_today()` in `models.py`; warning card on admin dashboard
- [x] Bid deadline email reminders — `send_reminders` route; "📧 Remind" button (admin only) in bid toolbar
- [x] Server-side health checks — `/dashboard/health/<id>` HEAD request with 6s timeout; green/orange/red/gray dots
- [x] Password reset via email — `password_reset_tokens` table; `/forgot-password` + `/reset-password/<token>` routes; standalone templates; "Forgot password?" on login page
- [x] Shift calendar view — weekly grid tab; MonSun columns; active shifts as rows; `day_map` server-rendered
- [x] Delete AI analysis history — `POST /ai-summary/history/<id>/delete`; ownership check; delete button in history modal
- [x] Criteria snapshot in history modal — `analysis_detail` returns `criteria_snapshot`; modal shows titles of criteria used
- [x] Purge expired password reset tokens — `purge_expired_reset_tokens()` called on every successful login (wrapped in try/except)
- [x] Test SMTP + Groq buttons — `POST /admin/settings/test-email` and `/test-groq`; inline AJAX result in settings page
- [x] Admin send password reset link — `POST /admin/users/<id>/send-reset`; generates token + sends email; "✉ Reset" button per user row
- [x] Export bids to CSV — `GET /bids/export.csv`; respects active status filter; "⬇ CSV" button in toolbar
- [x] Admin dashboard bid KPIs — Open Bids, Due This Week (red when >0), AI Analyses (30d) added to `get_admin_dashboard_stats()`
- [x] User profile page — `GET/POST /profile`; users edit own `full_name` and `email`; session updated immediately; "👤 My Profile" in sidebar
- [x] SMTP From Address field — `email.smtp_from` exposed in Admin → Settings form and saved alongside other email fields
- [x] Incomplete-shift reminder emails — manual `POST /admin/shifts/send-incomplete-reminders` + automated `GET /internal/cron/shift-reminders?token=<CRON_SECRET>`; `get_incomplete_shift_users_near_end()` + `record_shift_reminder()` in `models.py`; `shift_reminder_log` table deduplicates to one email per user/shift/day; `do_send_incomplete_reminders()` shared helper in `admin_shifts.py`; systemd timer fires every 5 min
### Pending
- No known pending items. Add new items here as they are identified.