Website Checker — Web Application

A full-stack web application for monitoring government procurement websites across team shifts. Converted from a Tkinter desktop application, sharing the same MySQL database.

Overview

Website Checker helps teams track which government websites need to be checked during each shift, manage bid/procurement opportunities, and run AI-powered document analysis on solicitation files. Administrators manage users, websites, shifts, and view reports; regular users work through their daily or weekly checklists.

Technology Stack

Layer	Technology
Language	Python 3.10+
Web Framework	Flask 3.0
Database	MySQL 8.x (shared with desktop app)
Application Server	Gunicorn
Reverse Proxy	Nginx
Process Manager	systemd
Credential Encryption	Fernet (PBKDF2-HMAC-SHA256, 100k iterations)
Password Hashing	bcrypt (12 rounds)
AI Analysis	Groq REST API (llama-3.3-70b-versatile)
Markdown Rendering	marked.js (CDN, client-side)
Frontend	Vanilla JS + DM Sans/DM Mono (Google Fonts)
OS	Ubuntu 22.04 / 24.04 LTS

Features

All Users

Shift Checklist — Sites grouped by Daily / Weekly frequency, collapsible groups, live progress bar, bulk-check, search filter, server-side health dot per site
Site Credentials — View stored credentials (password masked, copy-to-clipboard)
Per-site Notes — Add or update notes per check, displayed in a dedicated row under the site card
AI Document Analysis — Upload PDF / DOCX / XLSX / TXT solicitation files; AI extracts solicitation number, scope of work, due dates, driving distance from office, and evaluates against active criteria. All documents are analyzed as one combined source. Output rendered as formatted markdown.
AI History — Filter past analyses by verdict (PURSUE / PASS / UNCLEAR / No verdict), view criteria snapshot used, delete own records
Bid Tracker — Split-pane view of all tracked opportunities; post updates, filter by status, search by title / source / solicitation number, urgency badges for bids due within 7 days / overdue, load-more pagination, export current view to CSV
My Profile — Update own full name and email address
Password Reset — Self-service reset via emailed link (1-hour token); "Forgot password?" on login page
Relative Timestamps — Activity log timestamps shown as "2 hours ago" / "3 days ago"
Mobile Sidebar — Hamburger toggle collapses sidebar on small screens

Admins Only

Dashboard — KPI cards (users, active today, total sites, open bids, bids due this week, AI analyses past 30 days); per-user completion progress; missed-shift warning
User Management — Create, edit, activate/deactivate users; role assignment; send password reset link directly to any user with an email address
Website Management — CRUD for monitored sites with credentials, check type (daily/weekly), user assignment; inline search and type filter
Shift Management — Define shifts with days-of-week, time windows, assigned users and sites; weekly calendar view with today's column highlighted
Reports — Shift detail, unchecked sites, and summary reports with date/user/site filters; CSV export
Logs — Activity log (user actions) and application log (system events) with search and purge
Settings — SMTP email configuration (host, port, security, username, password, from address, recipients); Groq AI API key / model selection; one-click Test button for both SMTP and Groq to verify config without saving
AI Criteria Management — Create, edit, reorder, and deactivate evaluation criteria used by AI analysis
Bid Email Reminders — One-click digest email to all admin addresses listing bids due within 7 days
Missed Shift Alerting — Warning card on the admin dashboard showing any user/shift pair with 0 sites checked today

Project Structure

webchecker_web/
├── app.py                      # Flask application factory, blueprint registration
├── config.py                   # DB connection pool, schema DDL, get/set settings
├── models.py                   # All data-access functions (no ORM)
├── wsgi.py                     # Gunicorn entry point
├── requirements.txt
├── .env.example                # Environment variable template
├── CLAUDE.md                   # AI developer context
├── DEPLOY.md                   # Full production deployment guide
│
├── routes/
│   ├── auth.py                 # Login, logout, change password, profile, password reset
│   ├── admin_dashboard.py      # /admin/
│   ├── admin_users.py          # /admin/users/
│   ├── admin_websites.py       # /admin/websites/
│   ├── admin_shifts.py         # /admin/shifts/
│   ├── admin_logs.py           # /admin/logs/
│   ├── admin_reports.py        # /admin/reports/
│   ├── admin_settings.py       # /admin/settings/  (incl. test-email, test-groq)
│   ├── user_dashboard.py       # /dashboard/  (incl. server-side health check)
│   ├── ai_summary.py           # /ai-summary/  (incl. history delete)
│   └── bid_tracker.py          # /bids/  (incl. CSV export, email reminders)
│
├── templates/
│   ├── base.html               # Sidebar layout, flash messages, nav, mobile toggle
│   ├── login.html              # Standalone (no base.html)
│   ├── forgot_password.html    # Standalone password-reset request
│   ├── reset_password.html     # Standalone new-password form
│   ├── profile.html            # User profile editor
│   ├── change_password.html
│   ├── ai_summary.html
│   ├── bid_tracker.html
│   ├── admin/
│   │   ├── dashboard.html
│   │   ├── users.html
│   │   ├── websites.html
│   │   ├── shifts.html
│   │   ├── logs.html
│   │   ├── reports.html
│   │   └── settings.html
│   └── user/
│       └── dashboard.html
│
├── static/
│   ├── css/style.css           # Full light-theme design system
│   └── js/app.js               # Modal helpers, tabs, timeAgo(), mobile sidebar, session timeout
│
└── utils/
    ├── crypto.py               # Fernet encryption (DB-compatible with desktop app)
    ├── decorators.py           # @login_required, @admin_required
    └── email.py                # SMTP helper: send_email(to, subject, body)

URL Routes

Prefix	Description	Access
`/`	Root redirect (role-based)	Any
`/login`	Login page	Public
`/logout`	Session clear + redirect	Authenticated
`/ping`	Session keepalive (returns 204)	Authenticated
`/profile`	Edit own full name and email	Authenticated
`/change-password`	Change own password	Authenticated
`/forgot-password`	Request password reset email	Public
`/reset-password/<token>`	Set new password via token	Public
`/admin/dashboard`	Admin dashboard with KPIs	Admin
`/admin/users/`	User CRUD + send reset link	Admin
`/admin/websites/`	Website CRUD + credentials	Admin
`/admin/shifts/`	Shift CRUD + calendar view	Admin
`/admin/logs/`	Activity & app logs	Admin
`/admin/reports/`	Shift reports + CSV export	Admin
`/admin/settings/`	Email + Groq AI settings + connection tests	Admin
`/dashboard/`	User shift checklist	User
`/dashboard/health/<id>`	Server-side site health check	User
`/ai-summary/`	AI document analysis + history	Any
`/ai-summary/history/<id>`	Analysis detail (JSON)	Any
`/ai-summary/history/<id>/delete`	Delete an analysis record	Any
`/bids/`	Bid / opportunity tracker	Any
`/bids/export.csv`	Export bid list as CSV	Any
`/bids/remind`	Send bid deadline reminder email	Admin

Environment Variables

Copy .env.example to .env and fill in all values:

SECRET_KEY=          # Flask session secret — generate with: python3 -c "import secrets; print(secrets.token_hex(32))"
DB_HOST=127.0.0.1
DB_PORT=3306
DB_NAME=webchecker
DB_USER=webchecker_user
DB_PASSWORD=

GROQ_API_KEY=        # Optional — can also be set via Admin → Settings
GROQ_MODEL=llama-3.3-70b-versatile   # Optional fallback; overridden by DB setting

# SMTP email — optional fallback; values are stored in and overridden by Admin → Settings
SMTP_HOST=
SMTP_PORT=587
SMTP_USER=
SMTP_PASSWORD=
SMTP_FROM=

FLASK_ENV=production

Important: SECRET_KEY must remain stable across restarts — changing it invalidates all active sessions and CSRF tokens across Gunicorn workers.

Database Compatibility

This web application shares the MySQL database with the desktop application. Key compatibility notes:

Encryption — utils/crypto.py uses the exact same key derivation as the desktop (APP_SECRET = b"WebsiteChecker-v1-CredentialKey", 100,000 PBKDF2 iterations, base64 salt, enc: prefix). Credentials are interchangeable between apps.
activity_log — The live database column is logged_at (desktop schema). Queries use al.logged_at AS created_at for template compatibility.
app_settings — On startup, initialize_database() seeds any settings keys that are blank in the DB from the corresponding .env variables, using ON DUPLICATE KEY UPDATE … IF(value = '', …) so admin-saved values are never overwritten.
password_reset_tokens — Web-only table; created by the app's migration-safe DDL on first startup. Tokens expire after 1 hour and are consumed on use. Expired tokens are purged on every successful login.
login_attempts.ip_address — The only additive column the web app creates. Added via ALTER TABLE … ADD COLUMN IF NOT EXISTS on startup.

Development

Run locally

# 1. Clone and enter the project
cd webchecker_web

# 2. Create and activate a virtual environment
python3 -m venv venv
source venv/bin/activate

# 3. Install dependencies
pip install -r requirements.txt

# 4. Configure environment
cp .env.example .env
# Edit .env with your local MySQL credentials

# 5. Run the development server
python app.py
# Open http://localhost:5000

Deploy changes to production

# After copying Python files to the server:
sudo systemctl reload webchecker

# Templates and static files take effect immediately (no reload needed)

# Check for errors after reload:
sudo journalctl -u webchecker -n 30 --no-pager

Security

Passwords are hashed with bcrypt (12 rounds); legacy SHA-256 hashes from the desktop app are automatically rehashed on next login.
Sessions expire after 30 minutes of inactivity. A client-side warning fires 5 minutes before expiry with a keep-alive ping endpoint.
Login is rate-limited: 5 failed attempts locks the account for 15 minutes.
Credentials stored in website_credentials are Fernet-encrypted at rest.
File uploads for AI analysis are validated against magic bytes (PDF %PDF, DOCX/XLSX PK\x03\x04, DOC/XLS OLE header) and UTF-8 decodability for text files — rejecting disguised uploads without extra dependencies.
Upload size is capped at 20 MB (MAX_CONTENT_LENGTH).
HTTP security headers (X-Frame-Options, X-Content-Type-Options, Referrer-Policy) are set on every response.
CSRF protection (Flask-WTF) on all non-GET requests.
Session fixation prevention: session.clear() is called immediately before setting the session on successful login.
Password reset tokens are single-use, expire after 1 hour, and are purged on each login.
Admins cannot demote or deactivate the last remaining active admin.

Deployment

See DEPLOY.md for the full step-by-step guide covering MySQL setup, Gunicorn systemd service, Nginx reverse proxy, Certbot SSL, firewall rules, log rotation, and database backup.

12 KiB Raw Blame History