WebChecker--Web-app-/CLAUDE.md

CLAUDE.md — AI Developer Context

This file gives Claude (or any AI assistant) the context needed to continue development on this project without re-reading the entire codebase from scratch.

---

What This Project Is

Website Checker Web is a Flask web application that is a direct port of a Tkinter desktop application. Both apps share the same MySQL database — this is the single most important constraint. Any schema change, query, or encryption logic must remain compatible with what the desktop app writes and reads.

The app helps a small team at LT Services Inc. (Falls Church, VA) track government procurement websites across scheduled shifts, manage bid/opportunity follow-up, and run AI-powered solicitation document analysis.

---

Architecture at a Glance

Browser → Nginx (reverse proxy) → Gunicorn (4 workers) → Flask app
                                                              │
                                                    MySQL (shared with desktop)

• Entry point: wsgi.py → app.py::create_app() — only wsgi.py calls create_app(); the bottom of app.py no longer has a module-level call (that caused double initialisation)
• No ORM — all DB access is raw SQL via mysql-connector-python in models.py
• No frontend framework — vanilla JS, no build step, no npm
• Blueprints: one file per feature area in routes/
• Templates: Jinja2; most extend base.html — login.html is a standalone exception (see CSRF section)
• Static assets: single style.css + app.js — no preprocessor

---

Critical Constraints

1. Shared Database

Never rename columns, drop tables, or change column types without verifying the desktop app still works. Key schema facts:

• activity_log timestamp column is logged_at (DDL and queries both use logged_at; a migration in config.py renames created_at → logged_at for old web-only installs)
• app_log timestamp column is logged_at
• All other tables generally use created_at
• The desktop app writes bid_tracker, bid_updates, ai_analysis_log, ai_criteria, app_settings, users, websites, website_credentials, shifts, shift_users, shift_websites, shift_checks, login_attempts

2. Credential Encryption

utils/crypto.py must stay byte-for-byte compatible with the desktop's utils/crypto.py:

• _APP_SECRET = b"WebsiteChecker-v1-CredentialKey" — never change
• _ITERATIONS = 100_000 — never change
• Salt stored as base64 in app_settings under key "crypto.salt"
• Ciphertext has enc: prefix; values without this prefix are legacy plaintext and returned as-is
• Calling reset_fernet() forces key reload if the salt changes

3. No Inline JS with Dynamic Jinja Values

All button onclick handlers that need dynamic data (site ID, site name, note text) must use data-* attributes on the HTML element and read them in a delegated event listener. Direct onclick="fn({{ value }})" breaks when the value contains quotes, apostrophes, or backslashes.

Example of the correct pattern:

<button class="js-check" data-id="{{ site.id }}" data-name="{{ site.name }}">Check</button>

document.getElementById('list').addEventListener('click', function(e) {
  const btn = e.target.closest('.js-check');
  if (btn) openCheckModal(btn.dataset.id, btn.dataset.name);
});

4. Jinja Macros and {% extends %}

Jinja macros defined in the same file as {% extends "base.html" %} cannot be called with {{ macro_name(...) }} before the macro definition is reached. Do not use macros in child templates. Inline the HTML directly or use JS to build dynamic content.

5. CSS Specificity — Local <style> vs style.css

The global static/css/style.css is loaded in base.html and has equal or higher specificity than local <style> blocks in child templates. Always fix layout issues in style.css — do not rely on local <style> overrides.

6. CSRF Protection (Flask-WTF)

CSRFProtect(app) is initialised inside create_app() in app.py. All non-GET requests are validated automatically.

How tokens reach the server:

Request type	Mechanism
Standard HTML form (in a template that extends base.html)	app.js IIFE injects a hidden csrf_token input into every POST form at page-load time, reading from <meta name="csrf-token"> in base.html
Standalone template (e.g. login.html)	Must include <input type="hidden" name="csrf_token" value="{{ csrf_token() }}"> directly in the form — these pages do not load app.js or base.html
fetch() / AJAX POST with FormData	Append fd.append('csrf_token', getCsrfToken()) OR add header 'X-CSRFToken': getCsrfToken()
fetch() / AJAX POST with JSON body	Add header 'X-CSRFToken': getCsrfToken() (token must be a header; JSON body is not inspected)
JS-built form injected into DOM via innerHTML	Inline the token: '<input type="hidden" name="csrf_token" value="' + getCsrfToken() + '">'

getCsrfToken() is defined in app.js and reads document.querySelector('meta[name="csrf-token"]').content.

SECRET_KEY must be set in .env — if it is absent, each Gunicorn worker generates its own random key, session cookies signed by one worker are rejected by another, and CSRF validation fails for any cross-worker request.

7. SECRET_KEY is Required

# .env
SECRET_KEY=<long-random-hex>   # generate: python -c "import secrets; print(secrets.token_hex(32))"

Without this, sessions and CSRF tokens are broken across the 4 Gunicorn workers. app.py logs a WARNING at startup if it is missing.

---

Key Files Reference

File	Purpose	Notes
app.py	Flask factory	Registers all 11 blueprints; CSRFProtect(app); hhmm template filter; logs WARNING if SECRET_KEY not set
wsgi.py	Gunicorn entry point	Only place that calls create_app() — do not add a second call elsewhere
config.py	DB config, DDL, settings	Calls load_dotenv() at top — must be before DB_CONFIG dict; contains safe re-runnable migrations
models.py	All DB queries	~1,650 lines; no ORM; every function opens/closes its own connection; import datetime at top
utils/crypto.py	Fernet encryption	Must match desktop exactly
utils/decorators.py	@login_required, @admin_required	Simple session checks
utils/email.py	SMTP email helper	send_email(to, subject, body_text) reads smtp settings from app_settings; used by bid reminders and password reset
static/css/style.css	Full design system	Light theme, DM Sans + DM Mono, CSS variables in :root
static/js/app.js	Global JS utilities	openModal(), closeModal(), copyToClipboard(), timeAgo(), mobile sidebar toggle, session timeout warning, getCsrfToken(), CSRF auto-inject IIFE
templates/login.html	Standalone login page	Does NOT extend base.html; has its own <head> and no app.js; CSRF token must be a direct hidden input
templates/forgot_password.html	Standalone forgot-password page	Same constraints as login.html — standalone, direct CSRF hidden input, no app.js
templates/reset_password.html	Standalone reset-password page	Same constraints as login.html — standalone, direct CSRF hidden input, no app.js
templates/profile.html	User profile editor	Extends base.html; users can update full_name and email; username/role are read-only

---

Session & Auth

• Session key: session["user"] — dict with id, username, full_name, role, email
• Role values: "admin" or "user"
• Session lifetime: 30 minutes (app.permanent_session_lifetime)
• Login rate limit: 5 attempts → 15-minute lockout (enforced in models.check_login_allowed)
• @login_required — redirects to /login if no session
• @admin_required — redirects to login or 403 if not admin
• Session fixation prevention — session.clear() is called in auth.login immediately before setting session["user"] on successful authentication
• Keep-alive endpoint — GET /ping (routes/auth.py, @login_required) touches session.modified = True and returns 204; called by the session-timeout warning in app.js
• Password reset — POST /forgot-password creates a 1-hour password_reset_tokens token and emails a link. POST /reset-password/<token> validates, sets the new password, and deletes the token. Expired tokens are purged on every successful login via purge_expired_reset_tokens().
• Profile editing — GET/POST /profile lets any authenticated user update their full_name and email. Changes are immediately reflected in session["user"] so the sidebar name refreshes without re-login.

---

Settings System

app_settings table stores key-value pairs for runtime configuration.

key_name	.env fallback	Used by
groq.api_key	GROQ_API_KEY	AI Summary route
groq.model	GROQ_MODEL	AI Summary route
email.smtp_host	SMTP_HOST	Email reports
email.smtp_port	SMTP_PORT	Email reports
email.smtp_user	SMTP_USER	Email reports
email.smtp_password	SMTP_PASSWORD	Email reports
email.smtp_from	SMTP_FROM	Email reports
crypto.salt	(generated)	Fernet key derivation

get_setting(key, default) reads DB first, then .env fallback, then default.
set_setting(key, value) upserts into app_settings.
On startup, initialize_database() seeds blank DB rows from .env using ON DUPLICATE KEY UPDATE value = IF(value='', VALUES(value), value).

---

AI Summary Feature

Route: /ai-summary/ (routes/ai_summary.py)

• Stage 1 prompt (_EXTRACTION_PROMPT) — extracts 12 fields; treats all uploaded documents as one combined source (do NOT analyze per-file); driving distance from 2815 Hartland Road, Falls Church, VA 22043
• Stage 2 prompt (_CRITERIA_PROMPT_SUFFIX) — appended only when active criteria exist; produces a machine-readable RECOMMENDATION: PURSUE|PASS|UNCLEAR line
• File extraction — pypdf for PDF, python-docx (imported as docx) for DOCX/DOC, openpyxl for XLSX, plain decode for TXT/CSV/MD
• Magic-byte validation — _validate_magic() rejects files whose bytes don't match their declared extension (disguised uploads)
• API call — direct requests.post() to https://api.groq.com/openai/v1/chat/completions — the groq Python SDK is not installed
• Model — claude-sonnet-4-20250514 should NOT be used here; use llama-3.3-70b-versatile (Groq)
• Max tokens — 4,096; temperature — 0.2
• Text limit — combined document text is truncated to 14,000 characters before being sent; the JSON response includes "truncated": true when this occurs
• sort_order input — always wrap int(request.form.get("sort_order", 0) or 0) in try/except (ValueError, TypeError) — bad input must degrade gracefully to 0, not raise a 500
• Output rendering — marked.js (CDN) renders the AI summary as formatted markdown in both the live output and history modal
• History detail — GET /ai-summary/history/<id> returns criteria_snapshot (JSON string of [{title, description}]); the history modal parses and displays the criteria titles used
• Delete history — POST /ai-summary/history/<id>/delete enforces ownership (user can delete own; admin can delete any); logs the action
• Verdict filter — client-side pill buttons (All / PURSUE / PASS / UNCLEAR / No verdict) above the history table toggle row visibility

---

Bid Tracker Feature

Route: /bids/ (routes/bid_tracker.py)

Split-pane UI — left: filterable bid list, right: detail panel with update timeline.

Key AJAX endpoints (all return JSON, no page reload):

• GET /bids/list/json?status= — bid list filtered by status
• GET /bids/<id>/json — bid detail + updates + can_edit flag
• POST /bids/<id>/updates/json — post new update
• POST /bids/updates/<id>/delete/json — delete update

Ownership rule: users can only edit/delete their own bids. Admins can edit/delete any bid. Enforced via can_edit = is_admin or bid.added_by == user.id.

_ser(row) — module-level helper that converts a DB dict to JSON-serialisable form (dates/times → ISO strings). Do not redefine it inline inside individual functions.

All write operations (create, edit, delete bid; add, delete update — both form and JSON variants) call log_action().

---

User Dashboard

Route: /dashboard/ (routes/user_dashboard.py)

• Sites are grouped by check_type (daily / weekly / other) and rendered as collapsible groups
• Each site card has two rows: .sc-row1 (main flex row) and .sc-row2 (note row, only if user_note is set)
• .sc-actions uses margin-left:auto to push buttons to the right edge of .sc-row1
• .site-card is display:block — not flex — so .sc-row1 and .sc-row2 stack vertically
• Health dots — GET /dashboard/health/<id> does a server-side HEAD request (6s timeout); returns {status, ms} JSON. Colors: green <3000ms, orange slow/timeout, red unreachable, gray error
• The hhmm Jinja filter (app.py) handles MySQL TIME columns returned as datetime.timedelta

---

Common Patterns

Adding a new admin page

1. Create routes/admin_mypage.py with a Blueprint at /admin/mypage
2. Register it in app.py::create_app()
3. Add a nav link in templates/base.html inside the admin nav section
4. Create templates/admin/mypage.html extending base.html
5. Add model functions to models.py
6. Add any new CSS classes to static/css/style.css

Adding a new modal

<div class="modal-overlay" id="modal-mymodal">
  <div class="modal">                          <!-- or modal-dialog for admin pages -->
    <div class="modal-header">
      <span class="modal-title">Title</span>
      <button class="modal-close" onclick="closeModal('modal-mymodal')">✕</button>
    </div>
    <div class="modal-body">…</div>
    <div class="modal-footer">
      <button class="btn btn-secondary" onclick="closeModal('modal-mymodal')">Cancel</button>
      <button class="btn btn-primary">Save</button>
    </div>
  </div>
</div>

Open with openModal('modal-mymodal') from app.js.

Adding a new POST form (CSRF checklist)

• Template extends base.html → nothing extra needed; app.js IIFE injects the token automatically
• Standalone template (like login.html) → add <input type="hidden" name="csrf_token" value="{{ csrf_token() }}"> inside the <form>
• fetch() POST → add 'X-CSRFToken': getCsrfToken() to the headers object
• JS-built form string → concatenate '<input type="hidden" name="csrf_token" value="' + getCsrfToken() + '">'

Serialising DB rows to JSON

MySQL connector returns datetime, date, timedelta objects which are not JSON-serialisable. Use the shared helper (already defined at module level in bid_tracker.py; replicate it where needed):

def _ser(row):
    return {k: v.isoformat() if hasattr(v, 'isoformat') else v for k, v in row.items()}

Adding a new model query with optional search/filter

Do not use f-strings to interpolate WHERE clauses into SQL. Build the query with string concatenation and always pass user values through the %s parameter list:

sql = "SELECT ... FROM table"
params = []
if search:
    sql += " WHERE col LIKE %s"
    params.append(f"%{search}%")
sql += " ORDER BY id DESC LIMIT %s"
params.append(limit)
cur.execute(sql, params)

---

Known Gotchas

Gotcha	Detail
activity_log.logged_at	DDL now creates logged_at. A migration in config.py renames created_at → logged_at for old web-only installs. Queried as al.logged_at AS created_at in get_activity_log()
TIME columns → timedelta	MySQL returns TIME as datetime.timedelta. Use the |hhmm filter in templates
GROQ_API_KEY in .env	Read directly via os.environ.get("GROQ_API_KEY") in the analyze route — not via get_setting() alone — because get_setting() depends on the DB row being populated
div.modal vs .modal-overlay	All modal backdrops must use class="modal-overlay". Inner dialog boxes use class="modal" or class="modal-dialog". Never use div.modal as a backdrop
Jinja macros in child templates	Cannot call {{ macro_name() }} before the {% macro %} definition is parsed. Inline the HTML instead
INSERT IGNORE for settings seed	Use ON DUPLICATE KEY UPDATE value = IF(value='', VALUES(value), value) — INSERT IGNORE silently skips rows that already exist, even with empty values
bcrypt not in original requirements	Added as bcrypt==4.1.3. Required by models.py for password hashing
login.html is standalone	Does NOT extend base.html. Has no app.js, no CSRF meta tag. Any form on this page needs {{ csrf_token() }} as a direct hidden input
SECRET_KEY must be fixed	os.urandom(32) fallback generates a different key per Gunicorn worker — sessions and CSRF break across workers. Always set SECRET_KEY in .env
create_app() called once	Only wsgi.py calls create_app(). Do not add a module-level call to app.py — it causes double initialisation (double initialize_database(), double blueprint registration)
import datetime in models.py	Already imported at the top of the file. Do not add inline import datetime inside functions

---

Development Workflow

# Activate venv
source /home/webchecker/venv/bin/activate

# Install / update dependencies (e.g. after adding Flask-WTF)
pip install -r requirements.txt

# After changing Python files — reload Gunicorn (zero-downtime)
sudo systemctl reload webchecker

# After changing templates or static files — no restart needed (served live)

# View live logs
sudo journalctl -u webchecker -f

# View Nginx errors
sudo tail -f /var/log/nginx/webchecker_error.log

# Run a quick DB query
mysql -u webchecker_user -p webchecker -e "SELECT key_name, value FROM app_settings;"

---

File Placement on Server

/opt/webchecker/          ← project root (or /home/webchecker/app/)
/opt/webchecker/.env      ← secrets (chmod 600, owned by webchecker)
/opt/webchecker/venv/     ← Python virtual environment
/var/log/webchecker/      ← Gunicorn access.log + error.log
/run/webchecker/          ← Gunicorn UNIX socket (webchecker.sock)
/etc/systemd/system/webchecker.service
/etc/nginx/sites-available/webchecker

---

Change Philosophy

1. Surgical, additive patches — smallest possible change to achieve the goal
2. Preserve all routes, function names, variable names unless explicitly directed otherwise
3. Never remove existing functionality unless explicitly directed
4. Log all create/update/delete actions via log_action()
5. Migration existence checks — all migrations safe to re-run
6. Full file contents for 1–3 file changes; deployment map for larger changesets
7. Explicit deploy instructions — migration steps separated from code steps
8. Root cause analysis on errors — never apply temporary workarounds

---

Completed Improvements

All planned items have been implemented. The list below serves as a record and cross-reference.

Security

• File upload size cap (MAX_CONTENT_LENGTH = 20 MB); JSON error for /ai-summary/, flash+redirect elsewhere
• HTTP security headers — X-Frame-Options, X-Content-Type-Options, Referrer-Policy on every response
• Prevent last-admin demotion/deactivation — update_user() raises ValueError if zero active admins would remain
• File MIME type validation — _validate_magic() checks magic bytes and UTF-8 decodability; no extra dependency
• Session fixation prevention — session.clear() before setting session on login
• Password reset tokens — single-use, 1-hour expiry, purged on every successful login

UI / UX

• Mobile-responsive sidebar — hamburger toggle (sidebar-toggle button in base.html), CSS transform:translateX, backdrop overlay; IIFE in app.js
• Bid due-date urgency badges — "Overdue" (red) and "Due Soon" (yellow) client-side pills in bid list
• Relative timestamps in Activity Log — timeAgo() in app.js; [data-ts] attribute on timestamp <span> elements in logs.html
• Empty state for AI analysis history — robot emoji + "Start Analyzing" CTA when history is empty
• Paginate bid list — LIMIT/OFFSET in get_all_bids(); "Load more" button in bid list; _bidOffset / _PAGE_SIZE state vars
• AI history verdict filter — pill buttons (All / PURSUE / PASS / UNCLEAR / No verdict) filter rows client-side
• Highlight today in shift calendar — JS IIFE maps getDay() → MySQL DAYOFWEEK → column index; adds .cal-today CSS class
• Website list search + type filter — inline search input + dropdown filter client-side in admin/websites.html

Functionality

• Missed-shift alerting — get_missed_shifts_today() in models.py; warning card on admin dashboard

• Bid deadline email reminders — send_reminders route; "📧 Remind" button (admin only) in bid toolbar

• Server-side health checks — /dashboard/health/<id> HEAD request with 6s timeout; green/orange/red/gray dots

• Password reset via email — password_reset_tokens table; /forgot-password + /reset-password/<token> routes; standalone templates; "Forgot password?" on login page

• Shift calendar view — weekly grid tab; Mon–Sun columns; active shifts as rows; day_map server-rendered

• Delete AI analysis history — POST /ai-summary/history/<id>/delete; ownership check; delete button in history modal

• Criteria snapshot in history modal — analysis_detail returns criteria_snapshot; modal shows titles of criteria used

• Purge expired password reset tokens — purge_expired_reset_tokens() called on every successful login (wrapped in try/except)

• Test SMTP + Groq buttons — POST /admin/settings/test-email and /test-groq; inline AJAX result in settings page

• Admin send password reset link — POST /admin/users/<id>/send-reset; generates token + sends email; "✉ Reset" button per user row

• Export bids to CSV — GET /bids/export.csv; respects active status filter; "⬇ CSV" button in toolbar

• Admin dashboard bid KPIs — Open Bids, Due This Week (red when >0), AI Analyses (30d) added to get_admin_dashboard_stats()

• User profile page — GET/POST /profile; users edit own full_name and email; session updated immediately; "👤 My Profile" in sidebar

• SMTP From Address field — email.smtp_from exposed in Admin → Settings form and saved alongside other email fields

• Incomplete-shift reminder emails — manual POST /admin/shifts/send-incomplete-reminders + automated GET /internal/cron/shift-reminders?token=<CRON_SECRET>; get_incomplete_shift_users_near_end() + record_shift_reminder() in models.py; shift_reminder_log table deduplicates to one email per user/shift/day; do_send_incomplete_reminders() shared helper in admin_shifts.py; systemd timer fires every 5 min

Pending

• No known pending items. Add new items here as they are identified.