WebChecker/CLAUDE.md

# CLAUDE.md — Website Checker: Complete Technical Knowledge Base

This document captures the full architecture, design decisions, database schema, module inventory,
feature set, and operational notes for the **Website Checker** desktop application.

---

## 1. Application Overview

Website Checker is a Python/Tkinter desktop application for shift-based website monitoring.
Regular users log in, view websites assigned to their shift, open each site, and mark it checked.
Administrators manage users, websites, shifts, credentials, and run reports.

- **UI:** Python `tkinter` + `ttk`
- **Database:** Remote MySQL 5.7+ / MariaDB 10.3+ via `mysql-connector-python`
- **Entry point:** `app.py` → `class App(tk.Tk)`
- **Default window:** 1100×700, minimum 960×620
- **Default theme:** Light (toggleable to Dark)

---

## 2. Project File Structure

```
website_checker/
├── app.py                          Entry point, shell, navigation, session management
├── config.py                       DB config, connection pool, schema DDL, migrations
├── models.py                       All database access (data layer)
├── requirements.txt
├── CLAUDE.md                       This file
├── utils/
│   ├── config_crypto.py            Windows DPAPI encryption for config.ini credentials
│   ├── crypto.py                   Fernet credential encryption (website passwords)
│   ├── export.py                   CSV + Excel export
│   ├── scheduler.py                Daily email report background daemon
│   └── ui_helpers.py               ThemeManager, COLOURS proxy, DateEntry, widgets
└── views/
    ├── admin_dashboard_view.py     Admin home — KPI cards + per-user progress table
    ├── admin_log_view.py           Activity log treeview
    ├── admin_shifts_view.py        Shift CRUD + PDF export
    ├── admin_users_view.py         User CRUD
    ├── admin_websites_view.py      Website CRUD + visibility + credentials (collapsible)
    ├── ai_summary_view.py          AI document analysis via Groq API (NEW)
    ├── change_password_view.py     Self-service password change (all roles)
    ├── email_settings_view.py      SMTP / scheduled report configuration
    ├── login_view.py               Login form with rate-limiting countdown
    ├── reports_view.py             Shift Detail / Unchecked / Summary / Chart tabs
    ├── settings_view.py            DB connection settings dialog
    └── user_dashboard_view.py      User checklist — search, bulk check, notifications
```

---

## 3. Runtime Files

### `config.ini`
Created on first launch. Contains [database], [email], [crypto], and [groq] sections.
**Sensitive fields are DPAPI-encrypted** (see §6 Security). Encrypted values have a `dpapi:` prefix.

```ini
[database]
host=your-mysql-host
port=3306
database=website_checker
user=dpapi:<base64-blob>        ; encrypted
password=dpapi:<base64-blob>    ; encrypted

[email]
enabled=false
smtp_host=smtp.example.com
smtp_port=587
smtp_user=sender@example.com
smtp_password=dpapi:<base64-blob>   ; encrypted
use_tls=true
recipients=admin@example.com
send_time=18:00

[groq]
api_key=dpapi:<base64-blob>     ; encrypted
model=llama-3.3-70b-versatile

[crypto]
salt=<base64 32-byte salt — auto-generated>
```

### `app.log`
UTF-8 log file in working directory. All INFO/WARNING/ERROR from every module.

---

## 4. Database Schema

All tables use InnoDB/utf8mb4. Created by `initialize_database()` on first launch.
Safe ALTER TABLE migrations run automatically for columns added post-deployment.

### users
| Column | Type | Notes |
|---|---|---|
| id | INT PK AUTO | |
| username | VARCHAR(100) UNIQUE | |
| password | VARCHAR(255) | bcrypt hash. Legacy SHA-256 (64 hex) auto-migrated on login |
| role | ENUM('admin','user') | |
| full_name | VARCHAR(200) | |
| is_active | TINYINT(1) | |
| failed_attempts | TINYINT UNSIGNED | Incremented on bad login; reset on success |
| locked_until | DATETIME NULL | Set when failed_attempts >= MAX_FAILED_ATTEMPTS (5) |
| created_at / updated_at | DATETIME | |

### websites
| Column | Type | Notes |
|---|---|---|
| id | INT PK AUTO | |
| name | VARCHAR(200) | |
| url | TEXT | |
| check_type | ENUM('daily','weekly') | weekly = once per ISO week |
| visibility | ENUM('all','assigned') | assigned = only users in website_users table |
| note | TEXT | Shown on user dashboard card |
| is_active | TINYINT(1) | Soft-delete |
| created_by | INT FK users SET NULL | |

### website_credentials
| Column | Notes |
|---|---|
| website_id FK | |
| username | Plaintext |
| password | Fernet-encrypted with `enc:` prefix |
| label | e.g. "Admin", "Read-only" |

### shift_checks
One row per user per site per day (upserted). `user_note` optional.

### activity_log
`action` codes: LOGIN, LOGOUT, SESSION_TIMEOUT, ACCOUNT_LOCKED, CREATE/UPDATE/DELETE_USER,
CHANGE_PASSWORD, CHANGE_PASSWORD_FAIL, RESET_PASSWORD, CREATE/UPDATE/DELETE_WEBSITE,
ADD/REMOVE_CREDENTIAL, CREATE/UPDATE/DELETE_SHIFT, CHECK_WEBSITE, UPDATE_NOTE,
EXPORT_CSV, EXPORT_EXCEL, EXPORT_SHIFT_PDF, UPDATE_EMAIL_SETTINGS,
CREATE/UPDATE/DELETE_AI_CRITERION.

### shifts
`days_of_week` is a digit string using MySQL DAYOFWEEK: 1=Sun 2=Mon 3=Tue 4=Wed 5=Thu 6=Fri 7=Sat.
E.g. "23456" = Mon–Fri. Queried with `LOCATE(DAYOFWEEK(CURDATE()), days_of_week) > 0`.

### shift_users (junction): (shift_id, user_id) PK, both CASCADE
### shift_websites (junction): (shift_id, website_id) PK + sort_order
### login_attempts: username, attempted_at, ip_address (indexed)
### website_users (junction): (website_id, user_id) PK — for visibility='assigned'

### ai_criteria
Stores evaluation criteria used by the AI to assess opportunity alignment.
| Column | Type | Notes |
|---|---|---|
| id | INT PK AUTO | |
| title | VARCHAR(200) | Short label |
| description | TEXT | Full criterion text sent to the AI |
| is_active | TINYINT(1) | Inactive criteria excluded from AI prompt |
| sort_order | INT | Controls display and prompt ordering |
| created_by | INT FK users SET NULL | |

### ai_analysis_log
Persists every AI analysis result for history and audit purposes.
| Column | Type | Notes |
|---|---|---|
| id | INT PK AUTO | |
| user_id | INT FK users SET NULL | |
| file_names | TEXT | Comma-separated filenames analyzed |
| model | VARCHAR(100) | Groq model used |
| verdict | ENUM('PURSUE','PASS','UNCLEAR') NULL | NULL when no criteria active |
| criteria_snapshot | TEXT NULL | Active criteria text at time of analysis |
| summary_text | MEDIUMTEXT | Full AI response |
| analyzed_at | DATETIME | |

---

## 5. Module Details

### config.py
- DB_CONFIG loaded from config.ini at import; placeholders if file missing
- get_connection() → pool (pool_size=5)
- initialize_database() → idempotent DDL + ALTER TABLE migrations
- load_config() / save_config() → decrypt/encrypt DPAPI fields transparently
- migrate_plaintext_config() → one-time migration; encrypts any plain-text creds on startup
- config_exists() → True if host, database, user are set
- reload_db_config() → re-reads config.ini, resets pool

### utils/config_crypto.py  *(NEW)*
Windows DPAPI-based encryption for config.ini sensitive values.
- encrypt_value(plaintext) → "dpapi:<base64>" string
- decrypt_value(stored) → plaintext; plain-text pass-through for legacy values
- is_encrypted(value) → True if value starts with "dpapi:"
- Tied to the current Windows user account — blob is unreadable on any other machine/account
- Graceful fallback: if pywin32 not available, values stored/returned as plain text

### models.py
Each function opens+closes its own connection. All writes call log_action().

Password helpers:
- _hash_password(pw) → bcrypt rounds=12
- _verify_password(pw, stored) → handles bcrypt + legacy SHA-256
- _needs_rehash(stored) → True for 64-char hex (SHA-256)

Rate-limit constants: MAX_FAILED_ATTEMPTS=5, LOCKOUT_MINUTES=15
Password strength: PW_MIN_LENGTH=8, requires upper + digit + special char

### utils/crypto.py
- PBKDF2-HMAC-SHA256, 100,000 iterations, 32-byte salt from config.ini [crypto]
- encrypt(plaintext) → "enc:" + base64(Fernet token)
- decrypt(ciphertext) → plaintext; legacy plaintext (no enc: prefix) passes through unchanged
- reset_fernet() → force key reload after config.ini replacement

### utils/ui_helpers.py
- ThemeManager singleton — initial="light". toggle(rebuild_callback) flips theme.
- COLOURS = _ColourProxy(dict) — always delegates to ThemeManager.get(). Import once, always current.
- THEMES["dark"] and THEMES["light"] each have 18 colour keys + 4 cal_* keys
- DateEntry — Frame subclass; .get() → "YYYY-MM-DD", .set(str). Calendar popup with prev/next month+year.
- make_scrollable_frame() — mousewheel scoped to Enter/Leave to prevent stale-widget crashes
- scrolled_text(parent, height, width) → (frame, tk.Text)

### utils/scheduler.py
- Daemon thread; polls every 60 seconds
- Sends HTML email once per day when now >= send_time
- smtp_password is DPAPI-encrypted on save (encrypt_value) / decrypted on load (decrypt_value)
- last_sent_date in-memory (resets on restart)
- start() / stop() called from app.py on login/logout (admin only)

### views/ai_summary_view.py
AI-powered document analysis panel. Accessible from the sidebar for both admin and user roles.

Key internals:
- Supported file types: .txt, .md, .csv, .pdf, .docx, .doc, .xlsx, .xls
- Groq API key and model selection are **visible only to admins** (user role sees neither)
- Regular users use the stored API key transparently
- Background thread for AI calls (prevents UI freeze)
- Extraction prompt focused on procurement/solicitation fields:
  Solicitation Number/Type, Set-Aside, Description, Work Site, Pre-Proposal Conference,
  POC, Square Footage, Driving Distance from office (2815 Hartland Rd, Falls Church VA),
  Last Day for Questions, Due Date
- Overall summary covers: Scope of Work, Contract Period, Proposal Submission Requirements,
  Key Deadlines & Action Items
- Groq API key stored DPAPI-encrypted in config.ini [groq] section
- max_tokens=4096; temperature=0.2; MAX_CHARS_PER_FILE=14,000
- **Evaluation Criteria panel** (collapsible): admin can add/edit/delete criteria;
  users see read-only list. Active criteria are appended to the AI prompt as an
  alignment evaluation section. The AI outputs RECOMMENDATION: PURSUE/PASS/UNCLEAR
  which is parsed and displayed as a colour-coded verdict banner.
- **History tab**: every analysis is saved to ai_analysis_log (with verdict + criteria
  snapshot). History treeview is filterable by verdict. Double-click restores full
  output in the Analyze tab. Admins see all users; regular users see only their own.
- CriterionDialog has a live character counter with 500-char soft limit guidance.
- URL open and health-check probe both guarded by _validate_and_normalise_url.

.doc reading (legacy binary Word format) — 3-tier fallback:
  1. win32com (Word COM automation — requires MS Word installed)
  2. docx2txt (pure Python)
  3. Raw ASCII scrape from binary

### views/admin_websites_view.py
- Website CRUD dialog now has a **collapsible credentials section** (hidden by default)
- "🔑 Show Credentials" / "🔒 Hide Credentials" toggle button
- Auto-expands when editing a site that already has saved credentials
- "＋ Add Credential" auto-expands the section if collapsed
- **Bug fixed:** `_on_visibility_change` previously used `before=self.creds_container`
  which crosses a widget parent boundary causing a TclError; corrected to `before=self._cred_body`

### views/user_dashboard_view.py
Key internals:
- _health_cache: {website_id: ("ok"|"slow"|"down", ms)} — daemon threads, HEAD request, 6s timeout
- Dot colours: green (<=3000ms), amber (>3000ms), red (error)
- Mousewheel: Enter/Leave scoped; unbind_all in destroy()
- Keyboard shortcuts: self.bind() stored in _shortcut_ids; unbound in destroy()
- Notifications: after(60_000) loop; plyer first, fallback to borderless Toplevel toast
- **Credentials popup no longer opens automatically on link click**
- "🔑 Credentials" button appears on each site card only if the site has saved credentials
- CredentialsPopup: 📋 copy button for username (2s flash); 📋 copy button for password
  (2s flash + clipboard auto-cleared after 15s for security); 👁 toggle to reveal password

### views/login_view.py
- check_login_allowed(username) called before authenticate()
- Locked: form disabled; 1-second countdown; re-enables when timer reaches 0
- Not-yet-locked failed attempt: shows "N attempt(s) remaining before lockout"

### views/admin_shifts_view.py
- _export_pdf(): reportlab A4 document; one section per active shift; user+website tables

### views/reports_view.py
- All date fields use DateEntry (no manual text entry)
- Chart: matplotlib Agg + FigureCanvasTkAgg; bars green>=100% / amber>=50% / red<50%

---

## 6. Authentication & Security

| Concern | Implementation |
|---|---|
| Password hashing | bcrypt rounds=12; SHA-256 auto-rehashed on next login |
| Website credential encryption | Fernet (AES-128-CBC + HMAC) via cryptography library |
| Config credential protection | Windows DPAPI (CryptProtectData) — user-account-scoped |
| Encrypted fields | DB user, DB password, SMTP password, Groq API key |
| Login rate limiting | 5 attempts → 15-min lockout in DB |
| Session timeout | 30-min idle; 1-min warning; any mouse/key event resets |
| Password policy | 8+ chars, uppercase, digit, special character |
| Self-service change | Requires current password; strength meter; same-as-current guard |
| Clipboard security | Password copy auto-clears clipboard after 15 seconds |

### DPAPI Migration
`migrate_plaintext_config()` is called automatically on every startup (in `_init_db`).
It is a no-op if all sensitive fields are already encrypted (dpapi: prefix present).
On first run after this feature was added, it encrypts any existing plain-text values in-place.

---

## 7. Key Business Logic

### check_type
- daily: shown every shift day
- weekly: hidden once checked this week (YEARWEEK ISO); reappears Monday

### visibility
- all: every user in shift sees it
- assigned: only users in website_users table see it

### get_today_checks resolution
1. Check if user has active shifts today (DAYOFWEEK match)
2. If yes: union of shift websites filtered by check_type + visibility
3. If no: all active websites filtered by visibility + check_type (legacy fallback)

### Soft deletes
Websites and Shifts: is_active=0. Users: hard-delete (admin can deactivate first).

---

## 8. Configuration Constants (edit in source)

| File | Constant | Default |
|---|---|---|
| app.py | IDLE_TIMEOUT_MS | 1,800,000 (30 min) |
| app.py | IDLE_WARNING_MS | 60,000 (1 min) |
| app.py | APP_VERSION | "1.0.0" |
| app.py | VERSION_CHECK_ENABLED | False |
| models.py | MAX_FAILED_ATTEMPTS | 5 |
| models.py | LOCKOUT_MINUTES | 15 |
| models.py | PW_MIN_LENGTH | 8 |
| user_dashboard_view.py | NOTIFY_MINUTES_BEFORE | 15 |
| admin_dashboard_view.py | REFRESH_INTERVAL_MS | 60,000 |
| utils/crypto.py | _ITERATIONS | 100,000 |
| ai_summary_view.py | MAX_CHARS_PER_FILE | 14,000 |
| ai_summary_view.py | _OFFICE_ADDRESS | "2815 Hartland Road, Falls Church, VA 22043, USA" |
| ai_summary_view.py | CriterionDialog._DESC_SOFT_LIMIT | 500 (chars, soft guidance only) |
| app.py | App._IDLE_THROTTLE_S | 5.0 (seconds between Motion-event timer resets) |
| admin_users_view.py | _PasswordResetDialog._AUTO_CLOSE_S | 120 (seconds before auto-close) |
| admin_users_view.py | _PasswordResetDialog._CLIP_CLEAR_S | 30 (seconds before clipboard wipe) |

---

## 9. Critical Gotchas

1. bind_all("<MouseWheel>") is NEVER used at module level. Always Enter/Leave scoped.
   Every view's destroy() calls unbind_all("<MouseWheel>") and _unbind_shortcuts().

2. COLOURS is a live proxy — delegates to ThemeManager.get() on every access.
   Never snapshot it into a local variable at class-creation time.

3. bcrypt is slow by design (~200-400ms at rounds=12). Expected behaviour.

4. config.ini sensitive fields are DPAPI-encrypted (dpapi: prefix).
   Website credential passwords are Fernet-encrypted (enc: prefix).
   Non-sensitive fields (host, port, db name, smtp_host, recipients) remain plain text.

5. get_today_checks GROUP BY includes sc.id to prevent row collisions when a user
   belongs to multiple shifts sharing the same website.

6. Weekly site logic uses YEARWEEK(..., 1) (ISO week, Monday start).

7. The email scheduler `last_sent_date` is now persisted to `config.ini`
   `[email] last_sent_date` (ISO format). A restart after the configured
   `send_time` will not re-send the report on the same day.

8. All log messages use ASCII only (hyphens not em-dashes/arrows) to prevent
   cp1252 UnicodeEncodeError on Windows consoles.

9. DPAPI encrypted blobs are tied to the Windows user account that created them.
   If config.ini is copied to a different machine or user account, credentials
   cannot be decrypted. Users must re-enter credentials via the Settings dialog.

10. The AI summary view fetches credentials from the DB at card render time (not lazily).
    Avoid having hundreds of sites with credentials as this adds DB round-trips per render.

---

## 10. Dependencies

```
mysql-connector-python>=8.0.0
bcrypt>=4.0.0
openpyxl>=3.1.0
cryptography>=41.0.0
matplotlib>=3.7.0
plyer>=2.1.0
reportlab>=4.0.0
groq>=1.0.0
pypdf>=3.0.0
python-docx>=1.0.0
pywin32>=306
docx2txt>=0.8
```

stdlib used: tkinter, csv, smtplib, urllib.request, configparser, threading, calendar,
datetime, base64, re, tempfile

---

## 11. First-Run Flow

1. _boot() → config_exists() → False → SettingsView (locked modal)
2. User enters DB creds → Test Connection → Save & Connect
3. reload_db_config() → initialize_database() → seeds admin/admin123 if users empty
4. migrate_plaintext_config() → encrypts any plain-text credentials in config.ini
5. Login screen shown
6. ThemeManager(root, initial="light") applied; shell built

---

## 12. Sidebar Navigation

### Admin role
Dashboard · Websites · Users · Shifts · Reports · Activity Log · 🤖 AI Summary · Settings · Sign Out

### User role
My Shift · 🤖 AI Summary · Change Password · Sign Out

---

## 13. Deployment Notes

- Python 3.9+ required. 3.12 tested on Windows.
- tkinter bundled on Windows/macOS; Linux: apt install python3-tk
- Create DB first: CREATE DATABASE website_checker CHARACTER SET utf8mb4;
- MySQL port 3306 must be reachable from client
- Default admin: username=admin / password=admin123 — change immediately
- pywin32 required for DPAPI encryption (Windows only). Install: pip install pywin32
- Microsoft Word recommended for .doc file support in AI Summary (falls back to docx2txt)
- Groq API key required for AI Summary feature — free at https://console.groq.com

---

## 14. AI Summary Feature — Setup & Prompt Details

### Setup (Admin)
1. Obtain a free Groq API key at https://console.groq.com
2. Navigate to **🤖 AI Summary** in the sidebar
3. Enter the API key and select a model
4. Click **💾 Save Settings** — key is stored DPAPI-encrypted in config.ini

### Supported Models
- llama-3.3-70b-versatile (default, recommended)
- llama-3.1-8b-instant
- gemma2-9b-it
- mixtral-8x7b-32768

### Prompt Strategy
The prompt instructs the AI to extract 12 labeled fields per document:
Solicitation Number, Type, Set-Aside, Description/Scope, Work Site, Pre-Proposal Conference,
POC, Square Footage, **Driving Distance from 2815 Hartland Rd Falls Church VA** (calculated by AI),
Last Day for Questions, Due Date, Other requirements.

Then produces an Overall Summary with 4 sections:
A. Scope of Work · B. Contract Period · C. Proposal Submission Requirements · D. Key Deadlines

The driving distance/time is an AI estimate using major highways. Actual times may vary with traffic.