246 lines
12 KiB
Markdown
246 lines
12 KiB
Markdown
# Website Checker — Web Application
|
|
|
|
A full-stack web application for monitoring government procurement websites across team shifts. Converted from a Tkinter desktop application, sharing the same MySQL database.
|
|
|
|
---
|
|
|
|
## Overview
|
|
|
|
Website Checker helps teams track which government websites need to be checked during each shift, manage bid/procurement opportunities, and run AI-powered document analysis on solicitation files. Administrators manage users, websites, shifts, and view reports; regular users work through their daily or weekly checklists.
|
|
|
|
---
|
|
|
|
## Technology Stack
|
|
|
|
| Layer | Technology |
|
|
|---|---|
|
|
| Language | Python 3.10+ |
|
|
| Web Framework | Flask 3.0 |
|
|
| Database | MySQL 8.x (shared with desktop app) |
|
|
| Application Server | Gunicorn |
|
|
| Reverse Proxy | Nginx |
|
|
| Process Manager | systemd |
|
|
| Credential Encryption | Fernet (PBKDF2-HMAC-SHA256, 100k iterations) |
|
|
| Password Hashing | bcrypt (12 rounds) |
|
|
| AI Analysis | Groq REST API (llama-3.3-70b-versatile) |
|
|
| Markdown Rendering | marked.js (CDN, client-side) |
|
|
| Frontend | Vanilla JS + DM Sans/DM Mono (Google Fonts) |
|
|
| OS | Ubuntu 22.04 / 24.04 LTS |
|
|
|
|
---
|
|
|
|
## Features
|
|
|
|
### All Users
|
|
- **Shift Checklist** — Sites grouped by Daily / Weekly frequency, collapsible groups, live progress bar, bulk-check, search filter, server-side health dot per site
|
|
- **Site Credentials** — View stored credentials (password masked, copy-to-clipboard)
|
|
- **Per-site Notes** — Add or update notes per check, displayed in a dedicated row under the site card
|
|
- **AI Document Analysis** — Upload PDF / DOCX / XLSX / TXT solicitation files; AI extracts solicitation number, scope of work, due dates, driving distance from office, and evaluates against active criteria. All documents are analyzed as one combined source. Output rendered as formatted markdown.
|
|
- **AI History** — Filter past analyses by verdict (PURSUE / PASS / UNCLEAR / No verdict), view criteria snapshot used, delete own records
|
|
- **Bid Tracker** — Split-pane view of all tracked opportunities; post updates, filter by status, search by title / source / solicitation number, urgency badges for bids due within 7 days / overdue, load-more pagination, export current view to CSV
|
|
- **My Profile** — Update own full name and email address
|
|
- **Password Reset** — Self-service reset via emailed link (1-hour token); "Forgot password?" on login page
|
|
- **Relative Timestamps** — Activity log timestamps shown as "2 hours ago" / "3 days ago"
|
|
- **Mobile Sidebar** — Hamburger toggle collapses sidebar on small screens
|
|
|
|
### Admins Only
|
|
- **Dashboard** — KPI cards (users, active today, total sites, open bids, bids due this week, AI analyses past 30 days); per-user completion progress; missed-shift warning
|
|
- **User Management** — Create, edit, activate/deactivate users; role assignment; send password reset link directly to any user with an email address
|
|
- **Website Management** — CRUD for monitored sites with credentials, check type (daily/weekly), user assignment; inline search and type filter
|
|
- **Shift Management** — Define shifts with days-of-week, time windows, assigned users and sites; weekly calendar view with today's column highlighted
|
|
- **Reports** — Shift detail, unchecked sites, and summary reports with date/user/site filters; CSV export
|
|
- **Logs** — Activity log (user actions) and application log (system events) with search and purge
|
|
- **Settings** — SMTP email configuration (host, port, security, username, password, from address, recipients); Groq AI API key / model selection; one-click **Test** button for both SMTP and Groq to verify config without saving
|
|
- **AI Criteria Management** — Create, edit, reorder, and deactivate evaluation criteria used by AI analysis
|
|
- **Bid Email Reminders** — One-click digest email to all admin addresses listing bids due within 7 days
|
|
- **Missed Shift Alerting** — Warning card on the admin dashboard showing any user/shift pair with 0 sites checked today
|
|
|
|
---
|
|
|
|
## Project Structure
|
|
|
|
```
|
|
webchecker_web/
|
|
├── app.py # Flask application factory, blueprint registration
|
|
├── config.py # DB connection pool, schema DDL, get/set settings
|
|
├── models.py # All data-access functions (no ORM)
|
|
├── wsgi.py # Gunicorn entry point
|
|
├── requirements.txt
|
|
├── .env.example # Environment variable template
|
|
├── CLAUDE.md # AI developer context
|
|
├── DEPLOY.md # Full production deployment guide
|
|
│
|
|
├── routes/
|
|
│ ├── auth.py # Login, logout, change password, profile, password reset
|
|
│ ├── admin_dashboard.py # /admin/
|
|
│ ├── admin_users.py # /admin/users/
|
|
│ ├── admin_websites.py # /admin/websites/
|
|
│ ├── admin_shifts.py # /admin/shifts/
|
|
│ ├── admin_logs.py # /admin/logs/
|
|
│ ├── admin_reports.py # /admin/reports/
|
|
│ ├── admin_settings.py # /admin/settings/ (incl. test-email, test-groq)
|
|
│ ├── user_dashboard.py # /dashboard/ (incl. server-side health check)
|
|
│ ├── ai_summary.py # /ai-summary/ (incl. history delete)
|
|
│ └── bid_tracker.py # /bids/ (incl. CSV export, email reminders)
|
|
│
|
|
├── templates/
|
|
│ ├── base.html # Sidebar layout, flash messages, nav, mobile toggle
|
|
│ ├── login.html # Standalone (no base.html)
|
|
│ ├── forgot_password.html # Standalone password-reset request
|
|
│ ├── reset_password.html # Standalone new-password form
|
|
│ ├── profile.html # User profile editor
|
|
│ ├── change_password.html
|
|
│ ├── ai_summary.html
|
|
│ ├── bid_tracker.html
|
|
│ ├── admin/
|
|
│ │ ├── dashboard.html
|
|
│ │ ├── users.html
|
|
│ │ ├── websites.html
|
|
│ │ ├── shifts.html
|
|
│ │ ├── logs.html
|
|
│ │ ├── reports.html
|
|
│ │ └── settings.html
|
|
│ └── user/
|
|
│ └── dashboard.html
|
|
│
|
|
├── static/
|
|
│ ├── css/style.css # Full light-theme design system
|
|
│ └── js/app.js # Modal helpers, tabs, timeAgo(), mobile sidebar, session timeout
|
|
│
|
|
└── utils/
|
|
├── crypto.py # Fernet encryption (DB-compatible with desktop app)
|
|
├── decorators.py # @login_required, @admin_required
|
|
└── email.py # SMTP helper: send_email(to, subject, body)
|
|
```
|
|
|
|
---
|
|
|
|
## URL Routes
|
|
|
|
| Prefix | Description | Access |
|
|
|---|---|---|
|
|
| `/` | Root redirect (role-based) | Any |
|
|
| `/login` | Login page | Public |
|
|
| `/logout` | Session clear + redirect | Authenticated |
|
|
| `/ping` | Session keepalive (returns 204) | Authenticated |
|
|
| `/profile` | Edit own full name and email | Authenticated |
|
|
| `/change-password` | Change own password | Authenticated |
|
|
| `/forgot-password` | Request password reset email | Public |
|
|
| `/reset-password/<token>` | Set new password via token | Public |
|
|
| `/admin/dashboard` | Admin dashboard with KPIs | Admin |
|
|
| `/admin/users/` | User CRUD + send reset link | Admin |
|
|
| `/admin/websites/` | Website CRUD + credentials | Admin |
|
|
| `/admin/shifts/` | Shift CRUD + calendar view | Admin |
|
|
| `/admin/logs/` | Activity & app logs | Admin |
|
|
| `/admin/reports/` | Shift reports + CSV export | Admin |
|
|
| `/admin/settings/` | Email + Groq AI settings + connection tests | Admin |
|
|
| `/dashboard/` | User shift checklist | User |
|
|
| `/dashboard/health/<id>` | Server-side site health check | User |
|
|
| `/ai-summary/` | AI document analysis + history | Any |
|
|
| `/ai-summary/history/<id>` | Analysis detail (JSON) | Any |
|
|
| `/ai-summary/history/<id>/delete` | Delete an analysis record | Any |
|
|
| `/bids/` | Bid / opportunity tracker | Any |
|
|
| `/bids/export.csv` | Export bid list as CSV | Any |
|
|
| `/bids/remind` | Send bid deadline reminder email | Admin |
|
|
|
|
---
|
|
|
|
## Environment Variables
|
|
|
|
Copy `.env.example` to `.env` and fill in all values:
|
|
|
|
```dotenv
|
|
SECRET_KEY= # Flask session secret — generate with: python3 -c "import secrets; print(secrets.token_hex(32))"
|
|
DB_HOST=127.0.0.1
|
|
DB_PORT=3306
|
|
DB_NAME=webchecker
|
|
DB_USER=webchecker_user
|
|
DB_PASSWORD=
|
|
|
|
GROQ_API_KEY= # Optional — can also be set via Admin → Settings
|
|
GROQ_MODEL=llama-3.3-70b-versatile # Optional fallback; overridden by DB setting
|
|
|
|
# SMTP email — optional fallback; values are stored in and overridden by Admin → Settings
|
|
SMTP_HOST=
|
|
SMTP_PORT=587
|
|
SMTP_USER=
|
|
SMTP_PASSWORD=
|
|
SMTP_FROM=
|
|
|
|
FLASK_ENV=production
|
|
```
|
|
|
|
> **Important:** `SECRET_KEY` must remain stable across restarts — changing it invalidates all active sessions and CSRF tokens across Gunicorn workers.
|
|
|
|
---
|
|
|
|
## Database Compatibility
|
|
|
|
This web application shares the MySQL database with the desktop application. Key compatibility notes:
|
|
|
|
- **Encryption** — `utils/crypto.py` uses the exact same key derivation as the desktop (`APP_SECRET = b"WebsiteChecker-v1-CredentialKey"`, 100,000 PBKDF2 iterations, base64 salt, `enc:` prefix). Credentials are interchangeable between apps.
|
|
- **`activity_log`** — The live database column is `logged_at` (desktop schema). Queries use `al.logged_at AS created_at` for template compatibility.
|
|
- **`app_settings`** — On startup, `initialize_database()` seeds any settings keys that are blank in the DB from the corresponding `.env` variables, using `ON DUPLICATE KEY UPDATE … IF(value = '', …)` so admin-saved values are never overwritten.
|
|
- **`password_reset_tokens`** — Web-only table; created by the app's migration-safe DDL on first startup. Tokens expire after 1 hour and are consumed on use. Expired tokens are purged on every successful login.
|
|
- **`login_attempts.ip_address`** — The only additive column the web app creates. Added via `ALTER TABLE … ADD COLUMN IF NOT EXISTS` on startup.
|
|
|
|
---
|
|
|
|
## Development
|
|
|
|
### Run locally
|
|
|
|
```bash
|
|
# 1. Clone and enter the project
|
|
cd webchecker_web
|
|
|
|
# 2. Create and activate a virtual environment
|
|
python3 -m venv venv
|
|
source venv/bin/activate
|
|
|
|
# 3. Install dependencies
|
|
pip install -r requirements.txt
|
|
|
|
# 4. Configure environment
|
|
cp .env.example .env
|
|
# Edit .env with your local MySQL credentials
|
|
|
|
# 5. Run the development server
|
|
python app.py
|
|
# Open http://localhost:5000
|
|
```
|
|
|
|
### Deploy changes to production
|
|
|
|
```bash
|
|
# After copying Python files to the server:
|
|
sudo systemctl reload webchecker
|
|
|
|
# Templates and static files take effect immediately (no reload needed)
|
|
|
|
# Check for errors after reload:
|
|
sudo journalctl -u webchecker -n 30 --no-pager
|
|
```
|
|
|
|
---
|
|
|
|
## Security
|
|
|
|
- Passwords are hashed with bcrypt (12 rounds); legacy SHA-256 hashes from the desktop app are automatically rehashed on next login.
|
|
- Sessions expire after **30 minutes** of inactivity. A client-side warning fires 5 minutes before expiry with a keep-alive ping endpoint.
|
|
- Login is rate-limited: 5 failed attempts locks the account for 15 minutes.
|
|
- Credentials stored in `website_credentials` are Fernet-encrypted at rest.
|
|
- File uploads for AI analysis are validated against magic bytes (PDF `%PDF`, DOCX/XLSX `PK\x03\x04`, DOC/XLS OLE header) and UTF-8 decodability for text files — rejecting disguised uploads without extra dependencies.
|
|
- Upload size is capped at 20 MB (`MAX_CONTENT_LENGTH`).
|
|
- HTTP security headers (`X-Frame-Options`, `X-Content-Type-Options`, `Referrer-Policy`) are set on every response.
|
|
- CSRF protection (Flask-WTF) on all non-GET requests.
|
|
- Session fixation prevention: `session.clear()` is called immediately before setting the session on successful login.
|
|
- Password reset tokens are single-use, expire after 1 hour, and are purged on each login.
|
|
- Admins cannot demote or deactivate the last remaining active admin.
|
|
|
|
---
|
|
|
|
## Deployment
|
|
|
|
See [`DEPLOY.md`](DEPLOY.md) for the full step-by-step guide covering MySQL setup, Gunicorn systemd service, Nginx reverse proxy, Certbot SSL, firewall rules, log rotation, and database backup.
|