Files

T

2026-05-24 17:49:44 -04:00

20 KiB

Raw Blame History

CLAUDE.md — AI Developer Context

This file gives Claude (or any AI assistant) the context needed to continue development on this project without re-reading the entire codebase from scratch.

What This Project Is

Website Checker Web is a Flask web application that is a direct port of a Tkinter desktop application. Both apps share the same MySQL database — this is the single most important constraint. Any schema change, query, or encryption logic must remain compatible with what the desktop app writes and reads.

The app helps a small team at LT Services Inc. (Falls Church, VA) track government procurement websites across scheduled shifts, manage bid/opportunity follow-up, and run AI-powered solicitation document analysis.

Architecture at a Glance

Browser → Nginx (reverse proxy) → Gunicorn (4 workers) → Flask app
                                                              │
                                                    MySQL (shared with desktop)

Entry point: wsgi.py → app.py::create_app() — only wsgi.py calls create_app(); the bottom of app.py no longer has a module-level call (that caused double initialisation)
No ORM — all DB access is raw SQL via mysql-connector-python in models.py
No frontend framework — vanilla JS, no build step, no npm
Blueprints: one file per feature area in routes/
Templates: Jinja2; most extend base.html — login.html is a standalone exception (see CSRF section)
Static assets: single style.css + app.js — no preprocessor

Critical Constraints

1. Shared Database

Never rename columns, drop tables, or change column types without verifying the desktop app still works. Key schema facts:

activity_log timestamp column is logged_at (DDL and queries both use logged_at; a migration in config.py renames created_at → logged_at for old web-only installs)
app_log timestamp column is logged_at
All other tables generally use created_at
The desktop app writes bid_tracker, bid_updates, ai_analysis_log, ai_criteria, app_settings, users, websites, website_credentials, shifts, shift_users, shift_websites, shift_checks, login_attempts

2. Credential Encryption

utils/crypto.py must stay byte-for-byte compatible with the desktop's utils/crypto.py:

_APP_SECRET = b"WebsiteChecker-v1-CredentialKey" — never change
_ITERATIONS = 100_000 — never change
Salt stored as base64 in app_settings under key "crypto.salt"
Ciphertext has enc: prefix; values without this prefix are legacy plaintext and returned as-is
Calling reset_fernet() forces key reload if the salt changes

3. No Inline JS with Dynamic Jinja Values

All button onclick handlers that need dynamic data (site ID, site name, note text) must use data-* attributes on the HTML element and read them in a delegated event listener. Direct onclick="fn({{ value }})" breaks when the value contains quotes, apostrophes, or backslashes.

Example of the correct pattern:

<button class="js-check" data-id="{{ site.id }}" data-name="{{ site.name }}">Check</button>

document.getElementById('list').addEventListener('click', function(e) {
  const btn = e.target.closest('.js-check');
  if (btn) openCheckModal(btn.dataset.id, btn.dataset.name);
});

4. Jinja Macros and `{% extends %}`

Jinja macros defined in the same file as {% extends "base.html" %} cannot be called with {{ macro_name(...) }} before the macro definition is reached. Do not use macros in child templates. Inline the HTML directly or use JS to build dynamic content.

5. CSS Specificity — Local `<style>` vs `style.css`

The global static/css/style.css is loaded in base.html and has equal or higher specificity than local <style> blocks in child templates. Always fix layout issues in style.css — do not rely on local <style> overrides.

6. CSRF Protection (Flask-WTF)

CSRFProtect(app) is initialised inside create_app() in app.py. All non-GET requests are validated automatically.

How tokens reach the server:

Request type	Mechanism
Standard HTML form (in a template that extends `base.html`)	`app.js` IIFE injects a hidden `csrf_token` input into every POST form at page-load time, reading from `<meta name="csrf-token">` in `base.html`
Standalone template (e.g. `login.html`)	Must include `<input type="hidden" name="csrf_token" value="{{ csrf_token() }}">` directly in the form — these pages do not load `app.js` or `base.html`
`fetch()` / AJAX POST with `FormData`	Append `fd.append('csrf_token', getCsrfToken())` OR add header `'X-CSRFToken': getCsrfToken()`
`fetch()` / AJAX POST with JSON body	Add header `'X-CSRFToken': getCsrfToken()` (token must be a header; JSON body is not inspected)
JS-built form injected into DOM via `innerHTML`	Inline the token: `'<input type="hidden" name="csrf_token" value="' + getCsrfToken() + '">'`

getCsrfToken() is defined in app.js and reads document.querySelector('meta[name="csrf-token"]').content.

SECRET_KEY must be set in .env — if it is absent, each Gunicorn worker generates its own random key, session cookies signed by one worker are rejected by another, and CSRF validation fails for any cross-worker request.

7. `SECRET_KEY` is Required

# .env
SECRET_KEY=<long-random-hex>   # generate: python -c "import secrets; print(secrets.token_hex(32))"

Without this, sessions and CSRF tokens are broken across the 4 Gunicorn workers. app.py logs a WARNING at startup if it is missing.

Key Files Reference

File	Purpose	Notes
`app.py`	Flask factory	Registers all 11 blueprints; `CSRFProtect(app)`; `hhmm` template filter; logs WARNING if `SECRET_KEY` not set
`wsgi.py`	Gunicorn entry point	Only place that calls `create_app()` — do not add a second call elsewhere
`config.py`	DB config, DDL, settings	Calls `load_dotenv()` at top — must be before `DB_CONFIG` dict; contains safe re-runnable migrations
`models.py`	All DB queries	~1,430 lines; no ORM; every function opens/closes its own connection; `import datetime` at top
`utils/crypto.py`	Fernet encryption	Must match desktop exactly
`utils/decorators.py`	`@login_required`, `@admin_required`	Simple session checks
`static/css/style.css`	Full design system	Light theme, DM Sans + DM Mono, CSS variables in `:root`
`static/js/app.js`	Global JS utilities	`openModal()`, `closeModal()`, `copyToClipboard()`, session timeout warning, `getCsrfToken()`, CSRF auto-inject IIFE
`templates/login.html`	Standalone login page	Does NOT extend `base.html`; has its own `<head>` and no `app.js`; CSRF token must be a direct hidden input

Session & Auth

Session key: session["user"] — dict with id, username, full_name, role, email
Role values: "admin" or "user"
Session lifetime: 30 minutes (app.permanent_session_lifetime)
Login rate limit: 5 attempts → 15-minute lockout (enforced in models.check_login_allowed)
@login_required — redirects to /login if no session
@admin_required — redirects to login or 403 if not admin
Session fixation prevention — session.clear() is called in auth.login immediately before setting session["user"] on successful authentication
Keep-alive endpoint — GET /ping (routes/auth.py, @login_required) touches session.modified = True and returns 204; called by the session-timeout warning in app.js

Settings System

app_settings table stores key-value pairs for runtime configuration.

`key_name`	`.env` fallback	Used by
`groq.api_key`	`GROQ_API_KEY`	AI Summary route
`groq.model`	`GROQ_MODEL`	AI Summary route
`email.smtp_host`	`SMTP_HOST`	Email reports
`email.smtp_port`	`SMTP_PORT`	Email reports
`email.smtp_user`	`SMTP_USER`	Email reports
`email.smtp_password`	`SMTP_PASSWORD`	Email reports
`email.smtp_from`	`SMTP_FROM`	Email reports
`crypto.salt`	(generated)	Fernet key derivation

get_setting(key, default) reads DB first, then .env fallback, then default.
set_setting(key, value) upserts into app_settings.
On startup, initialize_database() seeds blank DB rows from .env using ON DUPLICATE KEY UPDATE value = IF(value='', VALUES(value), value).

AI Summary Feature

Route: /ai-summary/ (routes/ai_summary.py)

Stage 1 prompt (_EXTRACTION_PROMPT) — extracts 12 fields per document including driving distance from 2815 Hartland Road, Falls Church, VA 22043 to the work site
Stage 2 prompt (_CRITERIA_PROMPT_SUFFIX) — appended only when active criteria exist; produces a machine-readable RECOMMENDATION: PURSUE|PASS|UNCLEAR line
File extraction — pypdf for PDF, python-docx (imported as docx) for DOCX/DOC, openpyxl for XLSX, plain decode for TXT/CSV/MD
API call — direct requests.post() to https://api.groq.com/openai/v1/chat/completions — the groq Python SDK is not installed
Model — claude-sonnet-4-20250514 should NOT be used here; use llama-3.3-70b-versatile (Groq)
Max tokens — 4,096; temperature — 0.2
Text limit — combined document text is truncated to 14,000 characters before being sent; the JSON response includes "truncated": true when this occurs so the UI can warn the user
sort_order input — always wrap int(request.form.get("sort_order", 0) or 0) in try/except (ValueError, TypeError) — bad input must degrade gracefully to 0, not raise a 500

Bid Tracker Feature

Route: /bids/ (routes/bid_tracker.py)

Split-pane UI — left: filterable bid list, right: detail panel with update timeline.

Key AJAX endpoints (all return JSON, no page reload):

GET /bids/list/json?status= — bid list filtered by status
GET /bids/<id>/json — bid detail + updates + can_edit flag
POST /bids/<id>/updates/json — post new update
POST /bids/updates/<id>/delete/json — delete update

Ownership rule: users can only edit/delete their own bids. Admins can edit/delete any bid. Enforced via can_edit = is_admin or bid.added_by == user.id.

_ser(row) — module-level helper that converts a DB dict to JSON-serialisable form (dates/times → ISO strings). Do not redefine it inline inside individual functions.

All write operations (create, edit, delete bid; add, delete update — both form and JSON variants) call log_action().

User Dashboard

Route: /dashboard/ (routes/user_dashboard.py)

Sites are grouped by check_type (daily / weekly / other) and rendered as collapsible groups
Each site card has two rows: .sc-row1 (main flex row) and .sc-row2 (note row, only if user_note is set)
.sc-actions uses margin-left:auto to push buttons to the right edge of .sc-row1
.site-card is display:block — not flex — so .sc-row1 and .sc-row2 stack vertically
Health dots probe site reachability using Google's favicon service (cross-origin safe)
The hhmm Jinja filter (app.py) handles MySQL TIME columns returned as datetime.timedelta

Common Patterns

Adding a new admin page

Create routes/admin_mypage.py with a Blueprint at /admin/mypage
Register it in app.py::create_app()
Add a nav link in templates/base.html inside the admin nav section
Create templates/admin/mypage.html extending base.html
Add model functions to models.py
Add any new CSS classes to static/css/style.css

<div class="modal-overlay" id="modal-mymodal">
  <div class="modal">                          <!-- or modal-dialog for admin pages -->
    <div class="modal-header">
      <span class="modal-title">Title</span>
      <button class="modal-close" onclick="closeModal('modal-mymodal')">✕</button>
    </div>
    <div class="modal-body">…</div>
    <div class="modal-footer">
      <button class="btn btn-secondary" onclick="closeModal('modal-mymodal')">Cancel</button>
      <button class="btn btn-primary">Save</button>
    </div>
  </div>
</div>

Open with openModal('modal-mymodal') from app.js.

Adding a new POST form (CSRF checklist)

Template extends base.html → nothing extra needed; app.js IIFE injects the token automatically
Standalone template (like login.html) → add <input type="hidden" name="csrf_token" value="{{ csrf_token() }}"> inside the <form>
fetch() POST → add 'X-CSRFToken': getCsrfToken() to the headers object
JS-built form string → concatenate '<input type="hidden" name="csrf_token" value="' + getCsrfToken() + '">'

Serialising DB rows to JSON

MySQL connector returns datetime, date, timedelta objects which are not JSON-serialisable. Use the shared helper (already defined at module level in bid_tracker.py; replicate it where needed):

def _ser(row):
    return {k: v.isoformat() if hasattr(v, 'isoformat') else v for k, v in row.items()}

Adding a new model query with optional search/filter

Do not use f-strings to interpolate WHERE clauses into SQL. Build the query with string concatenation and always pass user values through the %s parameter list:

sql = "SELECT ... FROM table"
params = []
if search:
    sql += " WHERE col LIKE %s"
    params.append(f"%{search}%")
sql += " ORDER BY id DESC LIMIT %s"
params.append(limit)
cur.execute(sql, params)

Known Gotchas

Gotcha	Detail
`activity_log.logged_at`	DDL now creates `logged_at`. A migration in `config.py` renames `created_at` → `logged_at` for old web-only installs. Queried as `al.logged_at AS created_at` in `get_activity_log()`
`TIME` columns → `timedelta`	MySQL returns `TIME` as `datetime.timedelta`. Use the `\|hhmm` filter in templates
`GROQ_API_KEY` in `.env`	Read directly via `os.environ.get("GROQ_API_KEY")` in the analyze route — not via `get_setting()` alone — because `get_setting()` depends on the DB row being populated
`div.modal` vs `.modal-overlay`	All modal backdrops must use `class="modal-overlay"`. Inner dialog boxes use `class="modal"` or `class="modal-dialog"`. Never use `div.modal` as a backdrop
Jinja macros in child templates	Cannot call `{{ macro_name() }}` before the `{% macro %}` definition is parsed. Inline the HTML instead
`INSERT IGNORE` for settings seed	Use `ON DUPLICATE KEY UPDATE value = IF(value='', VALUES(value), value)` — `INSERT IGNORE` silently skips rows that already exist, even with empty values
`bcrypt` not in original requirements	Added as `bcrypt==4.1.3`. Required by `models.py` for password hashing
`login.html` is standalone	Does NOT extend `base.html`. Has no `app.js`, no CSRF meta tag. Any form on this page needs `{{ csrf_token() }}` as a direct hidden input
`SECRET_KEY` must be fixed	`os.urandom(32)` fallback generates a different key per Gunicorn worker — sessions and CSRF break across workers. Always set `SECRET_KEY` in `.env`
`create_app()` called once	Only `wsgi.py` calls `create_app()`. Do not add a module-level call to `app.py` — it causes double initialisation (double `initialize_database()`, double blueprint registration)
`import datetime` in `models.py`	Already imported at the top of the file. Do not add inline `import datetime` inside functions

Development Workflow

# Activate venv
source /home/webchecker/venv/bin/activate

# Install / update dependencies (e.g. after adding Flask-WTF)
pip install -r requirements.txt

# After changing Python files — reload Gunicorn (zero-downtime)
sudo systemctl reload webchecker

# After changing templates or static files — no restart needed (served live)

# View live logs
sudo journalctl -u webchecker -f

# View Nginx errors
sudo tail -f /var/log/nginx/webchecker_error.log

# Run a quick DB query
mysql -u webchecker_user -p webchecker -e "SELECT key_name, value FROM app_settings;"

File Placement on Server

/opt/webchecker/          ← project root (or /home/webchecker/app/)
/opt/webchecker/.env      ← secrets (chmod 600, owned by webchecker)
/opt/webchecker/venv/     ← Python virtual environment
/var/log/webchecker/      ← Gunicorn access.log + error.log
/run/webchecker/          ← Gunicorn UNIX socket (webchecker.sock)
/etc/systemd/system/webchecker.service
/etc/nginx/sites-available/webchecker

Change Philosophy

Surgical, additive patches — smallest possible change to achieve the goal
Preserve all routes, function names, variable names unless explicitly directed otherwise
Never remove existing functionality unless explicitly directed
Log all create/update/delete actions via log_action()
Migration existence checks — all migrations safe to re-run
Full file contents for 1–3 file changes; deployment map for larger changesets
Explicit deploy instructions — migration steps separated from code steps
Root cause analysis on errors — never apply temporary workarounds

Pending Improvements (To-Do)

Security

File upload size cap — MAX_CONTENT_LENGTH = 20 MB set in create_app(); RequestEntityTooLarge handler returns JSON for /ai-summary/ paths and flash+redirect for form routes
HTTP security headers — @app.after_request in app.py sets X-Frame-Options, X-Content-Type-Options, Referrer-Policy on every response
Prevent last-admin demotion/deactivation — update_user() in models.py checks that demoting or deactivating an admin won't leave zero active admins; raises ValueError surfaced as a flash message
File MIME type validation — _validate_magic() in ai_summary.py checks magic bytes for PDF (%PDF), DOCX/XLSX (PK\x03\x04), DOC/XLS (OLE header), and UTF-8 decodability for text types; no extra dependency needed

UI / UX

Mobile-responsive sidebar — No media queries exist; the sidebar + main-content layout is unusable on phones; implement a hamburger toggle that collapses the sidebar on small screens
Bid due-date urgency badges — Client-side only: highlight bids due within 7 days with a warning badge and bids past due with a red "Overdue" pill in the bid list
Relative timestamps in Activity Log — Format raw YYYY-MM-DD HH:MM:SS timestamps as "2 hours ago" / "3 days ago" using a small JS formatter
Empty state for AI analysis history — When no analyses exist, show a call-to-action ("Upload your first document ↑") instead of a blank panel
Paginate bid list — get_all_bids() fetches all rows with no limit; add LIMIT/OFFSET to the model query and a "load more" button in the split-pane list

Functionality

Missed-shift alerting — Query or report that flags shifts where zero shift_checks records exist for a given date, surfaced on the admin dashboard or via email
Bid deadline email reminders — Use the existing email.smtp_* settings to send a daily digest of bids with due_date within the next 7 days
Server-side health checks — Replace the Google favicon proxy in the user dashboard with a /dashboard/health/<id> route that makes a server-side HEAD request (with short timeout) for a real reachability signal
Password reset via email — Time-limited token flow so users can self-service instead of requiring an admin edit; needs a password_reset_tokens table and SMTP integration
"Copy password" button in Credentials modal — Wire copyToClipboard() (already in app.js) to the password field in the user dashboard credentials modal
Shift calendar view — Weekly grid (Mon–Sun columns, shifts as rows) on the admin shifts page to make schedule gaps and overlaps visible at a glance

20 KiB Raw Blame History Unescape Escape