05/24 Enhance functionalities 2

2026-05-24 23:04:55 -04:00
parent cb98c7e222
commit 13f8ba124d
11 changed files with 309 additions and 77 deletions
@@ -23,6 +23,7 @@ Website Checker helps teams track which government websites need to be checked d
 | Credential Encryption | Fernet (PBKDF2-HMAC-SHA256, 100k iterations) |
 | Password Hashing | bcrypt (12 rounds) |
 | AI Analysis | Groq REST API (llama-3.3-70b-versatile) |
+| Markdown Rendering | marked.js (CDN, client-side) |
 | Frontend | Vanilla JS + DM Sans/DM Mono (Google Fonts) |
 | OS | Ubuntu 22.04 / 24.04 LTS |

@@ -31,21 +32,28 @@ Website Checker helps teams track which government websites need to be checked d
 ## Features

 ### All Users
- **Shift Checklist** — Sites grouped by Daily / Weekly frequency, collapsible groups, live progress bar, bulk-check, search filter
+- **Shift Checklist** — Sites grouped by Daily / Weekly frequency, collapsible groups, live progress bar, bulk-check, search filter, server-side health dot per site
 - **Site Credentials** — View stored credentials (password masked, copy-to-clipboard)
 - **Per-site Notes** — Add or update notes per check, displayed in a dedicated row under the site card
- **AI Document Analysis** — Upload PDF / DOCX / XLSX / TXT solicitation files; AI extracts solicitation number, scope of work, due dates, driving distance from office, and evaluates against criteria
- **Bid Tracker** — Split-pane view of all tracked opportunities; post updates, filter by status, search by title / source / solicitation number
+- **AI Document Analysis** — Upload PDF / DOCX / XLSX / TXT solicitation files; AI extracts solicitation number, scope of work, due dates, driving distance from office, and evaluates against active criteria. All documents are analyzed as one combined source. Output rendered as formatted markdown.
+- **AI History** — Filter past analyses by verdict (PURSUE / PASS / UNCLEAR / No verdict), view criteria snapshot used, delete own records
+- **Bid Tracker** — Split-pane view of all tracked opportunities; post updates, filter by status, search by title / source / solicitation number, urgency badges for bids due within 7 days / overdue, load-more pagination, export current view to CSV
+- **My Profile** — Update own full name and email address
+- **Password Reset** — Self-service reset via emailed link (1-hour token); "Forgot password?" on login page
+- **Relative Timestamps** — Activity log timestamps shown as "2 hours ago" / "3 days ago"
+- **Mobile Sidebar** — Hamburger toggle collapses sidebar on small screens

 ### Admins Only
- **Dashboard** — KPI cards (users, active today, total sites) and per-user completion progress
- **User Management** — Create, edit, activate/deactivate users; role assignment (admin / user)
- **Website Management** — CRUD for monitored sites with credentials, check type (daily/weekly), and user assignment
- **Shift Management** — Define shifts with days-of-week, time windows, assigned users and sites
+- **Dashboard** — KPI cards (users, active today, total sites, open bids, bids due this week, AI analyses past 30 days); per-user completion progress; missed-shift warning
+- **User Management** — Create, edit, activate/deactivate users; role assignment; send password reset link directly to any user with an email address
+- **Website Management** — CRUD for monitored sites with credentials, check type (daily/weekly), user assignment; inline search and type filter
+- **Shift Management** — Define shifts with days-of-week, time windows, assigned users and sites; weekly calendar view with today's column highlighted
 - **Reports** — Shift detail, unchecked sites, and summary reports with date/user/site filters; CSV export
 - **Logs** — Activity log (user actions) and application log (system events) with search and purge
- **Settings** — SMTP email configuration and Groq AI API key / model selection
+- **Settings** — SMTP email configuration (host, port, security, username, password, from address, recipients); Groq AI API key / model selection; one-click **Test** button for both SMTP and Groq to verify config without saving
 - **AI Criteria Management** — Create, edit, reorder, and deactivate evaluation criteria used by AI analysis
+- **Bid Email Reminders** — One-click digest email to all admin addresses listing bids due within 7 days
+- **Missed Shift Alerting** — Warning card on the admin dashboard showing any user/shift pair with 0 sites checked today

 ---

@@ -53,30 +61,34 @@ Website Checker helps teams track which government websites need to be checked d

 ```
 webchecker_web/
-├── app.py                  # Flask application factory, blueprint registration
-├── config.py               # DB connection pool, schema DDL, get/set settings
-├── models.py               # All data-access functions (no ORM)
-├── wsgi.py                 # Gunicorn entry point
+├── app.py                      # Flask application factory, blueprint registration
+├── config.py                   # DB connection pool, schema DDL, get/set settings
+├── models.py                   # All data-access functions (no ORM)
+├── wsgi.py                     # Gunicorn entry point
 ├── requirements.txt
-├── .env.example            # Environment variable template
-├── DEPLOY.md               # Full production deployment guide
+├── .env.example                # Environment variable template
+├── CLAUDE.md                   # AI developer context
+├── DEPLOY.md                   # Full production deployment guide
 │
 ├── routes/
-│   ├── auth.py             # Login, logout, change password
-│   ├── admin_dashboard.py  # /admin/
-│   ├── admin_users.py      # /admin/users/
-│   ├── admin_websites.py   # /admin/websites/
-│   ├── admin_shifts.py     # /admin/shifts/
-│   ├── admin_logs.py       # /admin/logs/
-│   ├── admin_reports.py    # /admin/reports/
-│   ├── admin_settings.py   # /admin/settings/
-│   ├── user_dashboard.py   # /dashboard/
-│   ├── ai_summary.py       # /ai-summary/
-│   └── bid_tracker.py      # /bids/
+│   ├── auth.py                 # Login, logout, change password, profile, password reset
+│   ├── admin_dashboard.py      # /admin/
+│   ├── admin_users.py          # /admin/users/
+│   ├── admin_websites.py       # /admin/websites/
+│   ├── admin_shifts.py         # /admin/shifts/
+│   ├── admin_logs.py           # /admin/logs/
+│   ├── admin_reports.py        # /admin/reports/
+│   ├── admin_settings.py       # /admin/settings/  (incl. test-email, test-groq)
+│   ├── user_dashboard.py       # /dashboard/  (incl. server-side health check)
+│   ├── ai_summary.py           # /ai-summary/  (incl. history delete)
+│   └── bid_tracker.py          # /bids/  (incl. CSV export, email reminders)
 │
 ├── templates/
-│   ├── base.html           # Sidebar layout, flash messages, nav
-│   ├── login.html
+│   ├── base.html               # Sidebar layout, flash messages, nav, mobile toggle
+│   ├── login.html              # Standalone (no base.html)
+│   ├── forgot_password.html    # Standalone password-reset request
+│   ├── reset_password.html     # Standalone new-password form
+│   ├── profile.html            # User profile editor
 │   ├── change_password.html
 │   ├── ai_summary.html
 │   ├── bid_tracker.html
@@ -92,12 +104,13 @@ webchecker_web/
 │       └── dashboard.html
 │
 ├── static/
-│   ├── css/style.css       # Full light-theme design system
-│   └── js/app.js           # Modal helpers, tabs, session timeout warning
+│   ├── css/style.css           # Full light-theme design system
+│   └── js/app.js               # Modal helpers, tabs, timeAgo(), mobile sidebar, session timeout
 │
 └── utils/
-    ├── crypto.py           # Fernet encryption (DB-compatible with desktop app)
-    └── decorators.py       # @login_required, @admin_required
+    ├── crypto.py               # Fernet encryption (DB-compatible with desktop app)
+    ├── decorators.py           # @login_required, @admin_required
+    └── email.py                # SMTP helper: send_email(to, subject, body)
 ```

 ---
@@ -109,17 +122,26 @@ webchecker_web/
 | `/` | Root redirect (role-based) | Any |
 | `/login` | Login page | Public |
 | `/logout` | Session clear + redirect | Authenticated |
+| `/ping` | Session keepalive (returns 204) | Authenticated |
+| `/profile` | Edit own full name and email | Authenticated |
 | `/change-password` | Change own password | Authenticated |
-| `/admin/` | Admin dashboard | Admin |
-| `/admin/users/` | User CRUD | Admin |
+| `/forgot-password` | Request password reset email | Public |
+| `/reset-password/<token>` | Set new password via token | Public |
+| `/admin/dashboard` | Admin dashboard with KPIs | Admin |
+| `/admin/users/` | User CRUD + send reset link | Admin |
 | `/admin/websites/` | Website CRUD + credentials | Admin |
-| `/admin/shifts/` | Shift CRUD + assignments | Admin |
+| `/admin/shifts/` | Shift CRUD + calendar view | Admin |
 | `/admin/logs/` | Activity & app logs | Admin |
 | `/admin/reports/` | Shift reports + CSV export | Admin |
-| `/admin/settings/` | Email + Groq AI settings | Admin |
+| `/admin/settings/` | Email + Groq AI settings + connection tests | Admin |
 | `/dashboard/` | User shift checklist | User |
-| `/ai-summary/` | AI document analysis | Any |
+| `/dashboard/health/<id>` | Server-side site health check | User |
+| `/ai-summary/` | AI document analysis + history | Any |
+| `/ai-summary/history/<id>` | Analysis detail (JSON) | Any |
+| `/ai-summary/history/<id>/delete` | Delete an analysis record | Any |
 | `/bids/` | Bid / opportunity tracker | Any |
+| `/bids/export.csv` | Export bid list as CSV | Any |
+| `/bids/remind` | Send bid deadline reminder email | Admin |

 ---

@@ -134,12 +156,21 @@ DB_PORT=3306
 DB_NAME=webchecker
 DB_USER=webchecker_user
 DB_PASSWORD=
-CRYPTO_SECRET=       # Fernet key — generate with: python3 -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"
-GROQ_API_KEY=        # Optional — used by AI Summary feature
+
+GROQ_API_KEY=        # Optional — can also be set via Admin → Settings
+GROQ_MODEL=llama-3.3-70b-versatile   # Optional fallback; overridden by DB setting
+
+# SMTP email — optional fallback; values are stored in and overridden by Admin → Settings
+SMTP_HOST=
+SMTP_PORT=587
+SMTP_USER=
+SMTP_PASSWORD=
+SMTP_FROM=
+
 FLASK_ENV=production
 ```

-> **Important:** `CRYPTO_SECRET` must match the key used by the desktop app if you are sharing the database. The salt for encryption is stored in the `app_settings` table under `crypto.salt`.
+> **Important:** `SECRET_KEY` must remain stable across restarts — changing it invalidates all active sessions and CSRF tokens across Gunicorn workers.

 ---

@@ -149,7 +180,8 @@ This web application shares the MySQL database with the desktop application. Key

 - **Encryption** — `utils/crypto.py` uses the exact same key derivation as the desktop (`APP_SECRET = b"WebsiteChecker-v1-CredentialKey"`, 100,000 PBKDF2 iterations, base64 salt, `enc:` prefix). Credentials are interchangeable between apps.
 - **`activity_log`** — The live database column is `logged_at` (desktop schema). Queries use `al.logged_at AS created_at` for template compatibility.
- **`app_settings`** — On startup, `initialize_database()` seeds any settings keys that are blank in the DB from the corresponding `.env` variables (e.g. `GROQ_API_KEY → groq.api_key`), using `ON DUPLICATE KEY UPDATE … IF(value = '', …)` so admin-saved values are never overwritten.
+- **`app_settings`** — On startup, `initialize_database()` seeds any settings keys that are blank in the DB from the corresponding `.env` variables, using `ON DUPLICATE KEY UPDATE … IF(value = '', …)` so admin-saved values are never overwritten.
+- **`password_reset_tokens`** — Web-only table; created by the app's migration-safe DDL on first startup. Tokens expire after 1 hour and are consumed on use. Expired tokens are purged on every successful login.
 - **`login_attempts.ip_address`** — The only additive column the web app creates. Added via `ALTER TABLE … ADD COLUMN IF NOT EXISTS` on startup.

 ---
@@ -178,25 +210,36 @@ python app.py
 # Open http://localhost:5000
 ```

-### Restart the production service
+### Deploy changes to production

 ```bash
-sudo systemctl restart webchecker
-sudo journalctl -u webchecker -f
+# After copying Python files to the server:
+sudo systemctl reload webchecker
+
+# Templates and static files take effect immediately (no reload needed)
+
+# Check for errors after reload:
+sudo journalctl -u webchecker -n 30 --no-pager
 ```

 ---

-## Security Notes
+## Security

 - Passwords are hashed with bcrypt (12 rounds); legacy SHA-256 hashes from the desktop app are automatically rehashed on next login.
- Sessions expire after **30 minutes** of inactivity. A client-side warning fires 5 minutes before expiry.
+- Sessions expire after **30 minutes** of inactivity. A client-side warning fires 5 minutes before expiry with a keep-alive ping endpoint.
 - Login is rate-limited: 5 failed attempts locks the account for 15 minutes.
- Credentials stored in `website_credentials` are Fernet-encrypted at rest. Plaintext legacy values (without the `enc:` prefix) are returned as-is for backward compatibility.
- `SECRET_KEY` must remain stable across restarts — changing it invalidates all active sessions.
+- Credentials stored in `website_credentials` are Fernet-encrypted at rest.
+- File uploads for AI analysis are validated against magic bytes (PDF `%PDF`, DOCX/XLSX `PK\x03\x04`, DOC/XLS OLE header) and UTF-8 decodability for text files — rejecting disguised uploads without extra dependencies.
+- Upload size is capped at 20 MB (`MAX_CONTENT_LENGTH`).
+- HTTP security headers (`X-Frame-Options`, `X-Content-Type-Options`, `Referrer-Policy`) are set on every response.
+- CSRF protection (Flask-WTF) on all non-GET requests.
+- Session fixation prevention: `session.clear()` is called immediately before setting the session on successful login.
+- Password reset tokens are single-use, expire after 1 hour, and are purged on each login.
+- Admins cannot demote or deactivate the last remaining active admin.

 ---

 ## Deployment

-See [`DEPLOY.md`](DEPLOY.md) for the full step-by-step guide covering MySQL setup, Gunicorn systemd service, Nginx reverse proxy, Certbot SSL, firewall rules, log rotation, and database backup.
+See [`DEPLOY.md`](DEPLOY.md) for the full step-by-step guide covering MySQL setup, Gunicorn systemd service, Nginx reverse proxy, Certbot SSL, firewall rules, log rotation, and database backup.