WebChecker--Web-app-/CLAUDE.md

CLAUDE.md — AI Developer Context

This file gives Claude (or any AI assistant) the context needed to continue development on this project without re-reading the entire codebase from scratch.

---

What This Project Is

Website Checker Web is a Flask web application that is a direct port of a Tkinter desktop application. Both apps share the same MySQL database — this is the single most important constraint. Any schema change, query, or encryption logic must remain compatible with what the desktop app writes and reads.

The app helps a small team at LT Services Inc. (Falls Church, VA) track government procurement websites across scheduled shifts, manage bid/opportunity follow-up, and run AI-powered solicitation document analysis.

---

Architecture at a Glance

Browser → Nginx (reverse proxy) → Gunicorn (4 workers) → Flask app
                                                              │
                                                    MySQL (shared with desktop)

• Entry point: wsgi.py → app.py::create_app() — only wsgi.py calls create_app(); the bottom of app.py no longer has a module-level call (that caused double initialisation)
• No ORM — all DB access is raw SQL via mysql-connector-python in models.py
• No frontend framework — vanilla JS, no build step, no npm
• Blueprints: one file per feature area in routes/
• Templates: Jinja2; most extend base.html — login.html is a standalone exception (see CSRF section)
• Static assets: single style.css + app.js — no preprocessor

---

Critical Constraints

1. Shared Database

Never rename columns, drop tables, or change column types without verifying the desktop app still works. Key schema facts:

• activity_log timestamp column is logged_at (DDL and queries both use logged_at; a migration in config.py renames created_at → logged_at for old web-only installs)
• app_log timestamp column is logged_at
• All other tables generally use created_at
• The desktop app writes bid_tracker, bid_updates, ai_analysis_log, ai_criteria, app_settings, users, websites, website_credentials, shifts, shift_users, shift_websites, shift_checks, login_attempts

2. Credential Encryption

utils/crypto.py must stay byte-for-byte compatible with the desktop's utils/crypto.py:

• _APP_SECRET = b"WebsiteChecker-v1-CredentialKey" — never change
• _ITERATIONS = 100_000 — never change
• Salt stored as base64 in app_settings under key "crypto.salt"
• Ciphertext has enc: prefix; values without this prefix are legacy plaintext and returned as-is
• Calling reset_fernet() forces key reload if the salt changes

3. No Inline JS with Dynamic Jinja Values

All button onclick handlers that need dynamic data (site ID, site name, note text) must use data-* attributes on the HTML element and read them in a delegated event listener. Direct onclick="fn({{ value }})" breaks when the value contains quotes, apostrophes, or backslashes.

Example of the correct pattern:

<button class="js-check" data-id="{{ site.id }}" data-name="{{ site.name }}">Check</button>

document.getElementById('list').addEventListener('click', function(e) {
  const btn = e.target.closest('.js-check');
  if (btn) openCheckModal(btn.dataset.id, btn.dataset.name);
});

4. Jinja Macros and {% extends %}

Jinja macros defined in the same file as {% extends "base.html" %} cannot be called with {{ macro_name(...) }} before the macro definition is reached. Do not use macros in child templates. Inline the HTML directly or use JS to build dynamic content.

5. CSS Specificity — Local <style> vs style.css

The global static/css/style.css is loaded in base.html and has equal or higher specificity than local <style> blocks in child templates. Always fix layout issues in style.css — do not rely on local <style> overrides.

6. CSRF Protection (Flask-WTF)

CSRFProtect(app) is initialised inside create_app() in app.py. All non-GET requests are validated automatically.

How tokens reach the server:

Request type	Mechanism
Standard HTML form (in a template that extends base.html)	app.js IIFE injects a hidden csrf_token input into every POST form at page-load time, reading from <meta name="csrf-token"> in base.html
Standalone template (e.g. login.html)	Must include <input type="hidden" name="csrf_token" value="{{ csrf_token() }}"> directly in the form — these pages do not load app.js or base.html
fetch() / AJAX POST with FormData	Append fd.append('csrf_token', getCsrfToken()) OR add header 'X-CSRFToken': getCsrfToken()
fetch() / AJAX POST with JSON body	Add header 'X-CSRFToken': getCsrfToken() (token must be a header; JSON body is not inspected)
JS-built form injected into DOM via innerHTML	Inline the token: '<input type="hidden" name="csrf_token" value="' + getCsrfToken() + '">'

getCsrfToken() is defined in app.js and reads document.querySelector('meta[name="csrf-token"]').content.

SECRET_KEY must be set in .env — if it is absent, each Gunicorn worker generates its own random key, session cookies signed by one worker are rejected by another, and CSRF validation fails for any cross-worker request.

7. SECRET_KEY is Required

# .env
SECRET_KEY=<long-random-hex>   # generate: python -c "import secrets; print(secrets.token_hex(32))"

Without this, sessions and CSRF tokens are broken across the 4 Gunicorn workers. app.py logs a WARNING at startup if it is missing.

---

Key Files Reference

File	Purpose	Notes
app.py	Flask factory	Registers all 11 blueprints; CSRFProtect(app); hhmm template filter; logs WARNING if SECRET_KEY not set
wsgi.py	Gunicorn entry point	Only place that calls create_app() — do not add a second call elsewhere
config.py	DB config, DDL, settings	Calls load_dotenv() at top — must be before DB_CONFIG dict; contains safe re-runnable migrations
models.py	All DB queries	~1,650 lines; no ORM; every function opens/closes its own connection; import datetime at top
utils/crypto.py	Fernet encryption	Must match desktop exactly
utils/decorators.py	@login_required, @admin_required	Simple session checks
utils/email.py	SMTP email helper	send_email(to, subject, body_text) reads smtp settings from app_settings; used by bid reminders and password reset
static/css/style.css	Full design system	Light theme, DM Sans + DM Mono, CSS variables in :root
static/js/app.js	Global JS utilities	openModal(), closeModal(), copyToClipboard(), timeAgo(), mobile sidebar toggle, session timeout warning, getCsrfToken(), CSRF auto-inject IIFE
templates/login.html	Standalone login page	Does NOT extend base.html; has its own <head> and no app.js; CSRF token must be a direct hidden input
templates/forgot_password.html	Standalone forgot-password page	Same constraints as login.html — standalone, direct CSRF hidden input, no app.js
templates/reset_password.html	Standalone reset-password page	Same constraints as login.html — standalone, direct CSRF hidden input, no app.js

---

Session & Auth

• Session key: session["user"] — dict with id, username, full_name, role, email
• Role values: "admin" or "user"
• Session lifetime: 30 minutes (app.permanent_session_lifetime)
• Login rate limit: 5 attempts → 15-minute lockout (enforced in models.check_login_allowed)
• @login_required — redirects to /login if no session
• @admin_required — redirects to login or 403 if not admin
• Session fixation prevention — session.clear() is called in auth.login immediately before setting session["user"] on successful authentication
• Keep-alive endpoint — GET /ping (routes/auth.py, @login_required) touches session.modified = True and returns 204; called by the session-timeout warning in app.js

---

Settings System

app_settings table stores key-value pairs for runtime configuration.

key_name	.env fallback	Used by
groq.api_key	GROQ_API_KEY	AI Summary route
groq.model	GROQ_MODEL	AI Summary route
email.smtp_host	SMTP_HOST	Email reports
email.smtp_port	SMTP_PORT	Email reports
email.smtp_user	SMTP_USER	Email reports
email.smtp_password	SMTP_PASSWORD	Email reports
email.smtp_from	SMTP_FROM	Email reports
crypto.salt	(generated)	Fernet key derivation

get_setting(key, default) reads DB first, then .env fallback, then default.
set_setting(key, value) upserts into app_settings.
On startup, initialize_database() seeds blank DB rows from .env using ON DUPLICATE KEY UPDATE value = IF(value='', VALUES(value), value).

---

AI Summary Feature

Route: /ai-summary/ (routes/ai_summary.py)

• Stage 1 prompt (_EXTRACTION_PROMPT) — extracts 12 fields per document including driving distance from 2815 Hartland Road, Falls Church, VA 22043 to the work site
• Stage 2 prompt (_CRITERIA_PROMPT_SUFFIX) — appended only when active criteria exist; produces a machine-readable RECOMMENDATION: PURSUE|PASS|UNCLEAR line
• File extraction — pypdf for PDF, python-docx (imported as docx) for DOCX/DOC, openpyxl for XLSX, plain decode for TXT/CSV/MD
• API call — direct requests.post() to https://api.groq.com/openai/v1/chat/completions — the groq Python SDK is not installed
• Model — claude-sonnet-4-20250514 should NOT be used here; use llama-3.3-70b-versatile (Groq)
• Max tokens — 4,096; temperature — 0.2
• Text limit — combined document text is truncated to 14,000 characters before being sent; the JSON response includes "truncated": true when this occurs so the UI can warn the user
• sort_order input — always wrap int(request.form.get("sort_order", 0) or 0) in try/except (ValueError, TypeError) — bad input must degrade gracefully to 0, not raise a 500

---

Bid Tracker Feature

Route: /bids/ (routes/bid_tracker.py)

Split-pane UI — left: filterable bid list, right: detail panel with update timeline.

Key AJAX endpoints (all return JSON, no page reload):

• GET /bids/list/json?status= — bid list filtered by status
• GET /bids/<id>/json — bid detail + updates + can_edit flag
• POST /bids/<id>/updates/json — post new update
• POST /bids/updates/<id>/delete/json — delete update

Ownership rule: users can only edit/delete their own bids. Admins can edit/delete any bid. Enforced via can_edit = is_admin or bid.added_by == user.id.

_ser(row) — module-level helper that converts a DB dict to JSON-serialisable form (dates/times → ISO strings). Do not redefine it inline inside individual functions.

All write operations (create, edit, delete bid; add, delete update — both form and JSON variants) call log_action().

---

User Dashboard

Route: /dashboard/ (routes/user_dashboard.py)

• Sites are grouped by check_type (daily / weekly / other) and rendered as collapsible groups
• Each site card has two rows: .sc-row1 (main flex row) and .sc-row2 (note row, only if user_note is set)
• .sc-actions uses margin-left:auto to push buttons to the right edge of .sc-row1
• .site-card is display:block — not flex — so .sc-row1 and .sc-row2 stack vertically
• Health dots probe site reachability using Google's favicon service (cross-origin safe)
• The hhmm Jinja filter (app.py) handles MySQL TIME columns returned as datetime.timedelta

---

Common Patterns

Adding a new admin page

1. Create routes/admin_mypage.py with a Blueprint at /admin/mypage
2. Register it in app.py::create_app()
3. Add a nav link in templates/base.html inside the admin nav section
4. Create templates/admin/mypage.html extending base.html
5. Add model functions to models.py
6. Add any new CSS classes to static/css/style.css

Adding a new modal

<div class="modal-overlay" id="modal-mymodal">
  <div class="modal">                          <!-- or modal-dialog for admin pages -->
    <div class="modal-header">
      <span class="modal-title">Title</span>
      <button class="modal-close" onclick="closeModal('modal-mymodal')">✕</button>
    </div>
    <div class="modal-body">…</div>
    <div class="modal-footer">
      <button class="btn btn-secondary" onclick="closeModal('modal-mymodal')">Cancel</button>
      <button class="btn btn-primary">Save</button>
    </div>
  </div>
</div>

Open with openModal('modal-mymodal') from app.js.

Adding a new POST form (CSRF checklist)

• Template extends base.html → nothing extra needed; app.js IIFE injects the token automatically
• Standalone template (like login.html) → add <input type="hidden" name="csrf_token" value="{{ csrf_token() }}"> inside the <form>
• fetch() POST → add 'X-CSRFToken': getCsrfToken() to the headers object
• JS-built form string → concatenate '<input type="hidden" name="csrf_token" value="' + getCsrfToken() + '">'

Serialising DB rows to JSON

MySQL connector returns datetime, date, timedelta objects which are not JSON-serialisable. Use the shared helper (already defined at module level in bid_tracker.py; replicate it where needed):

def _ser(row):
    return {k: v.isoformat() if hasattr(v, 'isoformat') else v for k, v in row.items()}

Adding a new model query with optional search/filter

Do not use f-strings to interpolate WHERE clauses into SQL. Build the query with string concatenation and always pass user values through the %s parameter list:

sql = "SELECT ... FROM table"
params = []
if search:
    sql += " WHERE col LIKE %s"
    params.append(f"%{search}%")
sql += " ORDER BY id DESC LIMIT %s"
params.append(limit)
cur.execute(sql, params)

---

Known Gotchas

Gotcha	Detail
activity_log.logged_at	DDL now creates logged_at. A migration in config.py renames created_at → logged_at for old web-only installs. Queried as al.logged_at AS created_at in get_activity_log()
TIME columns → timedelta	MySQL returns TIME as datetime.timedelta. Use the |hhmm filter in templates
GROQ_API_KEY in .env	Read directly via os.environ.get("GROQ_API_KEY") in the analyze route — not via get_setting() alone — because get_setting() depends on the DB row being populated
div.modal vs .modal-overlay	All modal backdrops must use class="modal-overlay". Inner dialog boxes use class="modal" or class="modal-dialog". Never use div.modal as a backdrop
Jinja macros in child templates	Cannot call {{ macro_name() }} before the {% macro %} definition is parsed. Inline the HTML instead
INSERT IGNORE for settings seed	Use ON DUPLICATE KEY UPDATE value = IF(value='', VALUES(value), value) — INSERT IGNORE silently skips rows that already exist, even with empty values
bcrypt not in original requirements	Added as bcrypt==4.1.3. Required by models.py for password hashing
login.html is standalone	Does NOT extend base.html. Has no app.js, no CSRF meta tag. Any form on this page needs {{ csrf_token() }} as a direct hidden input
SECRET_KEY must be fixed	os.urandom(32) fallback generates a different key per Gunicorn worker — sessions and CSRF break across workers. Always set SECRET_KEY in .env
create_app() called once	Only wsgi.py calls create_app(). Do not add a module-level call to app.py — it causes double initialisation (double initialize_database(), double blueprint registration)
import datetime in models.py	Already imported at the top of the file. Do not add inline import datetime inside functions

---

Development Workflow

# Activate venv
source /home/webchecker/venv/bin/activate

# Install / update dependencies (e.g. after adding Flask-WTF)
pip install -r requirements.txt

# After changing Python files — reload Gunicorn (zero-downtime)
sudo systemctl reload webchecker

# After changing templates or static files — no restart needed (served live)

# View live logs
sudo journalctl -u webchecker -f

# View Nginx errors
sudo tail -f /var/log/nginx/webchecker_error.log

# Run a quick DB query
mysql -u webchecker_user -p webchecker -e "SELECT key_name, value FROM app_settings;"

---

File Placement on Server

/opt/webchecker/          ← project root (or /home/webchecker/app/)
/opt/webchecker/.env      ← secrets (chmod 600, owned by webchecker)
/opt/webchecker/venv/     ← Python virtual environment
/var/log/webchecker/      ← Gunicorn access.log + error.log
/run/webchecker/          ← Gunicorn UNIX socket (webchecker.sock)
/etc/systemd/system/webchecker.service
/etc/nginx/sites-available/webchecker

---

Change Philosophy

1. Surgical, additive patches — smallest possible change to achieve the goal
2. Preserve all routes, function names, variable names unless explicitly directed otherwise
3. Never remove existing functionality unless explicitly directed
4. Log all create/update/delete actions via log_action()
5. Migration existence checks — all migrations safe to re-run
6. Full file contents for 1–3 file changes; deployment map for larger changesets
7. Explicit deploy instructions — migration steps separated from code steps
8. Root cause analysis on errors — never apply temporary workarounds

---

Pending Improvements (To-Do)

Security

• File upload size cap — MAX_CONTENT_LENGTH = 20 MB set in create_app(); RequestEntityTooLarge handler returns JSON for /ai-summary/ paths and flash+redirect for form routes
• HTTP security headers — @app.after_request in app.py sets X-Frame-Options, X-Content-Type-Options, Referrer-Policy on every response
• Prevent last-admin demotion/deactivation — update_user() in models.py checks that demoting or deactivating an admin won't leave zero active admins; raises ValueError surfaced as a flash message
• File MIME type validation — _validate_magic() in ai_summary.py checks magic bytes for PDF (%PDF), DOCX/XLSX (PK\x03\x04), DOC/XLS (OLE header), and UTF-8 decodability for text types; no extra dependency needed

UI / UX

• Mobile-responsive sidebar — No media queries exist; the sidebar + main-content layout is unusable on phones; implement a hamburger toggle that collapses the sidebar on small screens
• Bid due-date urgency badges — Client-side only: highlight bids due within 7 days with a warning badge and bids past due with a red "Overdue" pill in the bid list
• Relative timestamps in Activity Log — Format raw YYYY-MM-DD HH:MM:SS timestamps as "2 hours ago" / "3 days ago" using a small JS formatter
• Empty state for AI analysis history — When no analyses exist, show a call-to-action ("Upload your first document ↑") instead of a blank panel
• Paginate bid list — get_all_bids() fetches all rows with no limit; add LIMIT/OFFSET to the model query and a "load more" button in the split-pane list

Functionality

• Missed-shift alerting — get_missed_shifts_today() in models.py queries shift+user pairs scheduled today with 0 checks; surfaced as a warning card on the admin dashboard
• Bid deadline email reminders — get_bids_due_soon() + get_admin_emails() in models.py; send_reminders route in bid_tracker.py; "📧 Remind" button in bid tracker toolbar (admin only); uses utils/email.py
• Server-side health checks — /dashboard/health/<id> in user_dashboard.py does a server-side HEAD request; user dashboard JS updated to call this instead of the Google favicon proxy
• Password reset via email — password_reset_tokens table (migration-safe DDL in config.py); model functions in models.py; /forgot-password and /reset-password/<token> routes in auth.py; standalone templates forgot_password.html and reset_password.html; "Forgot password?" link on login page; uses utils/email.py
• "Copy password" button in Credentials modal — Already implemented in the original code via data-copy attribute and delegated click handler in user/dashboard.html
• Shift calendar view — Weekly grid tab added to admin shifts page; Mon–Sun columns, active shifts as rows; server-side rendered with Jinja2 using existing day_map data