345 lines
20 KiB
Markdown
345 lines
20 KiB
Markdown
# CLAUDE.md — AI Developer Context
|
||
|
||
This file gives Claude (or any AI assistant) the context needed to continue development on this project without re-reading the entire codebase from scratch.
|
||
|
||
---
|
||
|
||
## What This Project Is
|
||
|
||
**Website Checker Web** is a Flask web application that is a direct port of a Tkinter desktop application. Both apps share the **same MySQL database** — this is the single most important constraint. Any schema change, query, or encryption logic must remain compatible with what the desktop app writes and reads.
|
||
|
||
The app helps a small team at LT Services Inc. (Falls Church, VA) track government procurement websites across scheduled shifts, manage bid/opportunity follow-up, and run AI-powered solicitation document analysis.
|
||
|
||
---
|
||
|
||
## Architecture at a Glance
|
||
|
||
```
|
||
Browser → Nginx (reverse proxy) → Gunicorn (4 workers) → Flask app
|
||
│
|
||
MySQL (shared with desktop)
|
||
```
|
||
|
||
- **Entry point:** `wsgi.py` → `app.py::create_app()` — **only `wsgi.py` calls `create_app()`**; the bottom of `app.py` no longer has a module-level call (that caused double initialisation)
|
||
- **No ORM** — all DB access is raw SQL via `mysql-connector-python` in `models.py`
|
||
- **No frontend framework** — vanilla JS, no build step, no npm
|
||
- **Blueprints:** one file per feature area in `routes/`
|
||
- **Templates:** Jinja2; most extend `base.html` — **`login.html` is a standalone exception** (see CSRF section)
|
||
- **Static assets:** single `style.css` + `app.js` — no preprocessor
|
||
|
||
---
|
||
|
||
## Critical Constraints
|
||
|
||
### 1. Shared Database
|
||
Never rename columns, drop tables, or change column types without verifying the desktop app still works. Key schema facts:
|
||
- `activity_log` timestamp column is **`logged_at`** (DDL and queries both use `logged_at`; a migration in `config.py` renames `created_at` → `logged_at` for old web-only installs)
|
||
- `app_log` timestamp column is **`logged_at`**
|
||
- All other tables generally use `created_at`
|
||
- The desktop app writes `bid_tracker`, `bid_updates`, `ai_analysis_log`, `ai_criteria`, `app_settings`, `users`, `websites`, `website_credentials`, `shifts`, `shift_users`, `shift_websites`, `shift_checks`, `login_attempts`
|
||
|
||
### 2. Credential Encryption
|
||
`utils/crypto.py` must stay byte-for-byte compatible with the desktop's `utils/crypto.py`:
|
||
- `_APP_SECRET = b"WebsiteChecker-v1-CredentialKey"` — never change
|
||
- `_ITERATIONS = 100_000` — never change
|
||
- Salt stored as **base64** in `app_settings` under key `"crypto.salt"`
|
||
- Ciphertext has **`enc:`** prefix; values without this prefix are legacy plaintext and returned as-is
|
||
- Calling `reset_fernet()` forces key reload if the salt changes
|
||
|
||
### 3. No Inline JS with Dynamic Jinja Values
|
||
All button `onclick` handlers that need dynamic data (site ID, site name, note text) **must use `data-*` attributes** on the HTML element and read them in a delegated event listener. Direct `onclick="fn({{ value }})"` breaks when the value contains quotes, apostrophes, or backslashes.
|
||
|
||
Example of the correct pattern:
|
||
```html
|
||
<button class="js-check" data-id="{{ site.id }}" data-name="{{ site.name }}">Check</button>
|
||
```
|
||
```js
|
||
document.getElementById('list').addEventListener('click', function(e) {
|
||
const btn = e.target.closest('.js-check');
|
||
if (btn) openCheckModal(btn.dataset.id, btn.dataset.name);
|
||
});
|
||
```
|
||
|
||
### 4. Jinja Macros and `{% extends %}`
|
||
Jinja macros defined in the same file as `{% extends "base.html" %}` cannot be called with `{{ macro_name(...) }}` before the macro definition is reached. **Do not use macros in child templates.** Inline the HTML directly or use JS to build dynamic content.
|
||
|
||
### 5. CSS Specificity — Local `<style>` vs `style.css`
|
||
The global `static/css/style.css` is loaded in `base.html` and has equal or higher specificity than local `<style>` blocks in child templates. Always fix layout issues in `style.css` — do not rely on local `<style>` overrides.
|
||
|
||
### 6. CSRF Protection (Flask-WTF)
|
||
`CSRFProtect(app)` is initialised inside `create_app()` in `app.py`. All non-GET requests are validated automatically.
|
||
|
||
**How tokens reach the server:**
|
||
|
||
| Request type | Mechanism |
|
||
|---|---|
|
||
| Standard HTML form (in a template that extends `base.html`) | `app.js` IIFE injects a hidden `csrf_token` input into every POST form at page-load time, reading from `<meta name="csrf-token">` in `base.html` |
|
||
| Standalone template (e.g. `login.html`) | Must include `<input type="hidden" name="csrf_token" value="{{ csrf_token() }}">` **directly in the form** — these pages do not load `app.js` or `base.html` |
|
||
| `fetch()` / AJAX POST with `FormData` | Append `fd.append('csrf_token', getCsrfToken())` OR add header `'X-CSRFToken': getCsrfToken()` |
|
||
| `fetch()` / AJAX POST with JSON body | Add header `'X-CSRFToken': getCsrfToken()` (token must be a header; JSON body is not inspected) |
|
||
| JS-built form injected into DOM via `innerHTML` | Inline the token: `'<input type="hidden" name="csrf_token" value="' + getCsrfToken() + '">'` |
|
||
|
||
**`getCsrfToken()`** is defined in `app.js` and reads `document.querySelector('meta[name="csrf-token"]').content`.
|
||
|
||
**`SECRET_KEY` must be set in `.env`** — if it is absent, each Gunicorn worker generates its own random key, session cookies signed by one worker are rejected by another, and CSRF validation fails for any cross-worker request.
|
||
|
||
### 7. `SECRET_KEY` is Required
|
||
```
|
||
# .env
|
||
SECRET_KEY=<long-random-hex> # generate: python -c "import secrets; print(secrets.token_hex(32))"
|
||
```
|
||
Without this, sessions and CSRF tokens are broken across the 4 Gunicorn workers. `app.py` logs a WARNING at startup if it is missing.
|
||
|
||
---
|
||
|
||
## Key Files Reference
|
||
|
||
| File | Purpose | Notes |
|
||
|------|---------|-------|
|
||
| `app.py` | Flask factory | Registers all 11 blueprints; `CSRFProtect(app)`; `hhmm` template filter; logs WARNING if `SECRET_KEY` not set |
|
||
| `wsgi.py` | Gunicorn entry point | Only place that calls `create_app()` — do not add a second call elsewhere |
|
||
| `config.py` | DB config, DDL, settings | Calls `load_dotenv()` at top — **must be before `DB_CONFIG` dict**; contains safe re-runnable migrations |
|
||
| `models.py` | All DB queries | ~1,650 lines; no ORM; every function opens/closes its own connection; `import datetime` at top |
|
||
| `utils/crypto.py` | Fernet encryption | Must match desktop exactly |
|
||
| `utils/decorators.py` | `@login_required`, `@admin_required` | Simple session checks |
|
||
| `utils/email.py` | SMTP email helper | `send_email(to, subject, body_text)` reads smtp settings from `app_settings`; used by bid reminders and password reset |
|
||
| `static/css/style.css` | Full design system | Light theme, DM Sans + DM Mono, CSS variables in `:root` |
|
||
| `static/js/app.js` | Global JS utilities | `openModal()`, `closeModal()`, `copyToClipboard()`, `timeAgo()`, mobile sidebar toggle, session timeout warning, `getCsrfToken()`, CSRF auto-inject IIFE |
|
||
| `templates/login.html` | Standalone login page | Does NOT extend `base.html`; has its own `<head>` and no `app.js`; CSRF token must be a direct hidden input |
|
||
| `templates/forgot_password.html` | Standalone forgot-password page | Same constraints as `login.html` — standalone, direct CSRF hidden input, no `app.js` |
|
||
| `templates/reset_password.html` | Standalone reset-password page | Same constraints as `login.html` — standalone, direct CSRF hidden input, no `app.js` |
|
||
|
||
---
|
||
|
||
## Session & Auth
|
||
|
||
- Session key: `session["user"]` — dict with `id`, `username`, `full_name`, `role`, `email`
|
||
- Role values: `"admin"` or `"user"`
|
||
- Session lifetime: 30 minutes (`app.permanent_session_lifetime`)
|
||
- Login rate limit: 5 attempts → 15-minute lockout (enforced in `models.check_login_allowed`)
|
||
- `@login_required` — redirects to `/login` if no session
|
||
- `@admin_required` — redirects to login or 403 if not admin
|
||
- **Session fixation prevention** — `session.clear()` is called in `auth.login` immediately before setting `session["user"]` on successful authentication
|
||
- **Keep-alive endpoint** — `GET /ping` (`routes/auth.py`, `@login_required`) touches `session.modified = True` and returns 204; called by the session-timeout warning in `app.js`
|
||
|
||
---
|
||
|
||
## Settings System
|
||
|
||
`app_settings` table stores key-value pairs for runtime configuration.
|
||
|
||
| `key_name` | `.env` fallback | Used by |
|
||
|---|---|---|
|
||
| `groq.api_key` | `GROQ_API_KEY` | AI Summary route |
|
||
| `groq.model` | `GROQ_MODEL` | AI Summary route |
|
||
| `email.smtp_host` | `SMTP_HOST` | Email reports |
|
||
| `email.smtp_port` | `SMTP_PORT` | Email reports |
|
||
| `email.smtp_user` | `SMTP_USER` | Email reports |
|
||
| `email.smtp_password` | `SMTP_PASSWORD` | Email reports |
|
||
| `email.smtp_from` | `SMTP_FROM` | Email reports |
|
||
| `crypto.salt` | *(generated)* | Fernet key derivation |
|
||
|
||
`get_setting(key, default)` reads DB first, then `.env` fallback, then `default`.
|
||
`set_setting(key, value)` upserts into `app_settings`.
|
||
On startup, `initialize_database()` seeds blank DB rows from `.env` using `ON DUPLICATE KEY UPDATE value = IF(value='', VALUES(value), value)`.
|
||
|
||
---
|
||
|
||
## AI Summary Feature
|
||
|
||
Route: `/ai-summary/` (`routes/ai_summary.py`)
|
||
|
||
- **Stage 1 prompt** (`_EXTRACTION_PROMPT`) — extracts 12 fields per document including driving distance from `2815 Hartland Road, Falls Church, VA 22043` to the work site
|
||
- **Stage 2 prompt** (`_CRITERIA_PROMPT_SUFFIX`) — appended only when active criteria exist; produces a machine-readable `RECOMMENDATION: PURSUE|PASS|UNCLEAR` line
|
||
- **File extraction** — `pypdf` for PDF, `python-docx` (imported as `docx`) for DOCX/DOC, `openpyxl` for XLSX, plain decode for TXT/CSV/MD
|
||
- **API call** — direct `requests.post()` to `https://api.groq.com/openai/v1/chat/completions` — the `groq` Python SDK is **not** installed
|
||
- **Model** — `claude-sonnet-4-20250514` should NOT be used here; use `llama-3.3-70b-versatile` (Groq)
|
||
- **Max tokens** — 4,096; **temperature** — 0.2
|
||
- **Text limit** — combined document text is truncated to 14,000 characters before being sent; the JSON response includes `"truncated": true` when this occurs so the UI can warn the user
|
||
- **`sort_order` input** — always wrap `int(request.form.get("sort_order", 0) or 0)` in `try/except (ValueError, TypeError)` — bad input must degrade gracefully to 0, not raise a 500
|
||
|
||
---
|
||
|
||
## Bid Tracker Feature
|
||
|
||
Route: `/bids/` (`routes/bid_tracker.py`)
|
||
|
||
Split-pane UI — left: filterable bid list, right: detail panel with update timeline.
|
||
|
||
Key AJAX endpoints (all return JSON, no page reload):
|
||
- `GET /bids/list/json?status=` — bid list filtered by status
|
||
- `GET /bids/<id>/json` — bid detail + updates + `can_edit` flag
|
||
- `POST /bids/<id>/updates/json` — post new update
|
||
- `POST /bids/updates/<id>/delete/json` — delete update
|
||
|
||
Ownership rule: users can only edit/delete their own bids. Admins can edit/delete any bid. Enforced via `can_edit = is_admin or bid.added_by == user.id`.
|
||
|
||
**`_ser(row)`** — module-level helper that converts a DB dict to JSON-serialisable form (dates/times → ISO strings). Do not redefine it inline inside individual functions.
|
||
|
||
All write operations (create, edit, delete bid; add, delete update — both form and JSON variants) call `log_action()`.
|
||
|
||
---
|
||
|
||
## User Dashboard
|
||
|
||
Route: `/dashboard/` (`routes/user_dashboard.py`)
|
||
|
||
- Sites are grouped by `check_type` (`daily` / `weekly` / other) and rendered as collapsible groups
|
||
- Each site card has two rows: `.sc-row1` (main flex row) and `.sc-row2` (note row, only if `user_note` is set)
|
||
- `.sc-actions` uses `margin-left:auto` to push buttons to the right edge of `.sc-row1`
|
||
- `.site-card` is `display:block` — **not flex** — so `.sc-row1` and `.sc-row2` stack vertically
|
||
- Health dots probe site reachability using Google's favicon service (cross-origin safe)
|
||
- The `hhmm` Jinja filter (`app.py`) handles MySQL `TIME` columns returned as `datetime.timedelta`
|
||
|
||
---
|
||
|
||
## Common Patterns
|
||
|
||
### Adding a new admin page
|
||
1. Create `routes/admin_mypage.py` with a Blueprint at `/admin/mypage`
|
||
2. Register it in `app.py::create_app()`
|
||
3. Add a nav link in `templates/base.html` inside the admin nav section
|
||
4. Create `templates/admin/mypage.html` extending `base.html`
|
||
5. Add model functions to `models.py`
|
||
6. Add any new CSS classes to `static/css/style.css`
|
||
|
||
### Adding a new modal
|
||
```html
|
||
<div class="modal-overlay" id="modal-mymodal">
|
||
<div class="modal"> <!-- or modal-dialog for admin pages -->
|
||
<div class="modal-header">
|
||
<span class="modal-title">Title</span>
|
||
<button class="modal-close" onclick="closeModal('modal-mymodal')">✕</button>
|
||
</div>
|
||
<div class="modal-body">…</div>
|
||
<div class="modal-footer">
|
||
<button class="btn btn-secondary" onclick="closeModal('modal-mymodal')">Cancel</button>
|
||
<button class="btn btn-primary">Save</button>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
```
|
||
Open with `openModal('modal-mymodal')` from `app.js`.
|
||
|
||
### Adding a new POST form (CSRF checklist)
|
||
- **Template extends `base.html`** → nothing extra needed; `app.js` IIFE injects the token automatically
|
||
- **Standalone template** (like `login.html`) → add `<input type="hidden" name="csrf_token" value="{{ csrf_token() }}">` inside the `<form>`
|
||
- **`fetch()` POST** → add `'X-CSRFToken': getCsrfToken()` to the `headers` object
|
||
- **JS-built form string** → concatenate `'<input type="hidden" name="csrf_token" value="' + getCsrfToken() + '">'`
|
||
|
||
### Serialising DB rows to JSON
|
||
MySQL connector returns `datetime`, `date`, `timedelta` objects which are not JSON-serialisable. Use the shared helper (already defined at module level in `bid_tracker.py`; replicate it where needed):
|
||
```python
|
||
def _ser(row):
|
||
return {k: v.isoformat() if hasattr(v, 'isoformat') else v for k, v in row.items()}
|
||
```
|
||
|
||
### Adding a new model query with optional search/filter
|
||
Do **not** use f-strings to interpolate WHERE clauses into SQL. Build the query with string concatenation and always pass user values through the `%s` parameter list:
|
||
```python
|
||
sql = "SELECT ... FROM table"
|
||
params = []
|
||
if search:
|
||
sql += " WHERE col LIKE %s"
|
||
params.append(f"%{search}%")
|
||
sql += " ORDER BY id DESC LIMIT %s"
|
||
params.append(limit)
|
||
cur.execute(sql, params)
|
||
```
|
||
|
||
---
|
||
|
||
## Known Gotchas
|
||
|
||
| Gotcha | Detail |
|
||
|--------|--------|
|
||
| `activity_log.logged_at` | DDL now creates `logged_at`. A migration in `config.py` renames `created_at` → `logged_at` for old web-only installs. Queried as `al.logged_at AS created_at` in `get_activity_log()` |
|
||
| `TIME` columns → `timedelta` | MySQL returns `TIME` as `datetime.timedelta`. Use the `\|hhmm` filter in templates |
|
||
| `GROQ_API_KEY` in `.env` | Read directly via `os.environ.get("GROQ_API_KEY")` in the analyze route — not via `get_setting()` alone — because `get_setting()` depends on the DB row being populated |
|
||
| `div.modal` vs `.modal-overlay` | All modal backdrops must use `class="modal-overlay"`. Inner dialog boxes use `class="modal"` or `class="modal-dialog"`. Never use `div.modal` as a backdrop |
|
||
| Jinja macros in child templates | Cannot call `{{ macro_name() }}` before the `{% macro %}` definition is parsed. Inline the HTML instead |
|
||
| `INSERT IGNORE` for settings seed | Use `ON DUPLICATE KEY UPDATE value = IF(value='', VALUES(value), value)` — `INSERT IGNORE` silently skips rows that already exist, even with empty values |
|
||
| `bcrypt` not in original requirements | Added as `bcrypt==4.1.3`. Required by `models.py` for password hashing |
|
||
| `login.html` is standalone | Does NOT extend `base.html`. Has no `app.js`, no CSRF meta tag. Any form on this page needs `{{ csrf_token() }}` as a direct hidden input |
|
||
| `SECRET_KEY` must be fixed | `os.urandom(32)` fallback generates a different key per Gunicorn worker — sessions and CSRF break across workers. Always set `SECRET_KEY` in `.env` |
|
||
| `create_app()` called once | Only `wsgi.py` calls `create_app()`. Do not add a module-level call to `app.py` — it causes double initialisation (double `initialize_database()`, double blueprint registration) |
|
||
| `import datetime` in `models.py` | Already imported at the top of the file. Do not add inline `import datetime` inside functions |
|
||
|
||
---
|
||
|
||
## Development Workflow
|
||
|
||
```bash
|
||
# Activate venv
|
||
source /home/webchecker/venv/bin/activate
|
||
|
||
# Install / update dependencies (e.g. after adding Flask-WTF)
|
||
pip install -r requirements.txt
|
||
|
||
# After changing Python files — reload Gunicorn (zero-downtime)
|
||
sudo systemctl reload webchecker
|
||
|
||
# After changing templates or static files — no restart needed (served live)
|
||
|
||
# View live logs
|
||
sudo journalctl -u webchecker -f
|
||
|
||
# View Nginx errors
|
||
sudo tail -f /var/log/nginx/webchecker_error.log
|
||
|
||
# Run a quick DB query
|
||
mysql -u webchecker_user -p webchecker -e "SELECT key_name, value FROM app_settings;"
|
||
```
|
||
|
||
---
|
||
|
||
## File Placement on Server
|
||
|
||
```
|
||
/opt/webchecker/ ← project root (or /home/webchecker/app/)
|
||
/opt/webchecker/.env ← secrets (chmod 600, owned by webchecker)
|
||
/opt/webchecker/venv/ ← Python virtual environment
|
||
/var/log/webchecker/ ← Gunicorn access.log + error.log
|
||
/run/webchecker/ ← Gunicorn UNIX socket (webchecker.sock)
|
||
/etc/systemd/system/webchecker.service
|
||
/etc/nginx/sites-available/webchecker
|
||
```
|
||
|
||
---
|
||
|
||
## Change Philosophy
|
||
|
||
1. **Surgical, additive patches** — smallest possible change to achieve the goal
|
||
2. **Preserve all routes, function names, variable names** unless explicitly directed otherwise
|
||
3. **Never remove existing functionality** unless explicitly directed
|
||
4. **Log all create/update/delete actions** via `log_action()`
|
||
5. **Migration existence checks** — all migrations safe to re-run
|
||
6. **Full file contents for 1–3 file changes**; deployment map for larger changesets
|
||
7. **Explicit deploy instructions** — migration steps separated from code steps
|
||
8. **Root cause analysis** on errors — never apply temporary workarounds
|
||
|
||
---
|
||
|
||
## Pending Improvements (To-Do)
|
||
|
||
### Security
|
||
- [x] **File upload size cap** — `MAX_CONTENT_LENGTH = 20 MB` set in `create_app()`; `RequestEntityTooLarge` handler returns JSON for `/ai-summary/` paths and flash+redirect for form routes
|
||
- [x] **HTTP security headers** — `@app.after_request` in `app.py` sets `X-Frame-Options`, `X-Content-Type-Options`, `Referrer-Policy` on every response
|
||
- [x] **Prevent last-admin demotion/deactivation** — `update_user()` in `models.py` checks that demoting or deactivating an admin won't leave zero active admins; raises `ValueError` surfaced as a flash message
|
||
- [x] **File MIME type validation** — `_validate_magic()` in `ai_summary.py` checks magic bytes for PDF (`%PDF`), DOCX/XLSX (`PK\x03\x04`), DOC/XLS (OLE header), and UTF-8 decodability for text types; no extra dependency needed
|
||
|
||
### UI / UX
|
||
- [ ] **Mobile-responsive sidebar** — No media queries exist; the sidebar + main-content layout is unusable on phones; implement a hamburger toggle that collapses the sidebar on small screens
|
||
- [ ] **Bid due-date urgency badges** — Client-side only: highlight bids due within 7 days with a warning badge and bids past due with a red "Overdue" pill in the bid list
|
||
- [ ] **Relative timestamps in Activity Log** — Format raw `YYYY-MM-DD HH:MM:SS` timestamps as "2 hours ago" / "3 days ago" using a small JS formatter
|
||
- [ ] **Empty state for AI analysis history** — When no analyses exist, show a call-to-action ("Upload your first document ↑") instead of a blank panel
|
||
- [ ] **Paginate bid list** — `get_all_bids()` fetches all rows with no limit; add `LIMIT`/`OFFSET` to the model query and a "load more" button in the split-pane list
|
||
|
||
### Functionality
|
||
- [x] **Missed-shift alerting** — `get_missed_shifts_today()` in `models.py` queries shift+user pairs scheduled today with 0 checks; surfaced as a warning card on the admin dashboard
|
||
- [x] **Bid deadline email reminders** — `get_bids_due_soon()` + `get_admin_emails()` in `models.py`; `send_reminders` route in `bid_tracker.py`; "📧 Remind" button in bid tracker toolbar (admin only); uses `utils/email.py`
|
||
- [x] **Server-side health checks** — `/dashboard/health/<id>` in `user_dashboard.py` does a server-side HEAD request; user dashboard JS updated to call this instead of the Google favicon proxy
|
||
- [x] **Password reset via email** — `password_reset_tokens` table (migration-safe DDL in `config.py`); model functions in `models.py`; `/forgot-password` and `/reset-password/<token>` routes in `auth.py`; standalone templates `forgot_password.html` and `reset_password.html`; "Forgot password?" link on login page; uses `utils/email.py`
|
||
- [x] **"Copy password" button in Credentials modal** — Already implemented in the original code via `data-copy` attribute and delegated click handler in `user/dashboard.html`
|
||
- [x] **Shift calendar view** — Weekly grid tab added to admin shifts page; Mon–Sun columns, active shifts as rows; server-side rendered with Jinja2 using existing `day_map` data
|