- **Entry point:** `wsgi.py` → `app.py::create_app()` — **only `wsgi.py` calls `create_app()`**; the bottom of `app.py` no longer has a module-level call (that caused double initialisation)
- **No ORM** — all DB access is raw SQL via `mysql-connector-python` in `models.py`
- **No frontend framework** — vanilla JS, no build step, no npm
- **Blueprints:** one file per feature area in `routes/`
- **Templates:** Jinja2, all extend `base.html`
- **Templates:** Jinja2; most extend `base.html` — **`login.html` is a standalone exception** (see CSRF section)
- **Static assets:** single `style.css` + `app.js` — no preprocessor
Never rename columns, drop tables, or change column types without verifying the desktop app still works. Key schema facts:
-`activity_log` timestamp column is **`logged_at`** (not `created_at`)
-`activity_log` timestamp column is **`logged_at`** (DDL and queries both use `logged_at`; a migration in `config.py` renames `created_at` → `logged_at` for old web-only installs)
@@ -66,19 +66,45 @@ Jinja macros defined in the same file as `{% extends "base.html" %}` cannot be c
### 5. CSS Specificity — Local `<style>` vs `style.css`
The global `static/css/style.css` is loaded in `base.html` and has equal or higher specificity than local `<style>` blocks in child templates. Always fix layout issues in `style.css` — do not rely on local `<style>` overrides.
### 6. CSRF Protection (Flask-WTF)
`CSRFProtect(app)` is initialised inside `create_app()` in `app.py`. All non-GET requests are validated automatically.
**How tokens reach the server:**
| Request type | Mechanism |
|---|---|
| Standard HTML form (in a template that extends `base.html`) | `app.js` IIFE injects a hidden `csrf_token` input into every POST form at page-load time, reading from `<meta name="csrf-token">` in `base.html` |
| Standalone template (e.g. `login.html`) | Must include `<input type="hidden" name="csrf_token" value="{{ csrf_token() }}">`**directly in the form** — these pages do not load `app.js` or `base.html` |
| `fetch()` / AJAX POST with `FormData` | Append `fd.append('csrf_token', getCsrfToken())` OR add header `'X-CSRFToken': getCsrfToken()` |
| `fetch()` / AJAX POST with JSON body | Add header `'X-CSRFToken': getCsrfToken()` (token must be a header; JSON body is not inspected) |
| JS-built form injected into DOM via `innerHTML` | Inline the token: `'<input type="hidden" name="csrf_token" value="' + getCsrfToken() + '">'` |
**`getCsrfToken()`** is defined in `app.js` and reads `document.querySelector('meta[name="csrf-token"]').content`.
**`SECRET_KEY` must be set in `.env`** — if it is absent, each Gunicorn worker generates its own random key, session cookies signed by one worker are rejected by another, and CSRF validation fails for any cross-worker request.
| `templates/login.html` | Standalone login page | Does NOT extend `base.html`; has its own `<head>` and no `app.js`; CSRF token must be a direct hidden input |
---
@@ -90,6 +116,8 @@ The global `static/css/style.css` is loaded in `base.html` and has equal or high
-`@login_required` — redirects to `/login` if no session
-`@admin_required` — redirects to login or 403 if not admin
- **Session fixation prevention** — `session.clear()` is called in `auth.login` immediately before setting `session["user"]` on successful authentication
- **Keep-alive endpoint** — `GET /ping` (`routes/auth.py`, `@login_required`) touches `session.modified = True` and returns 204; called by the session-timeout warning in `app.js`
- **API call** — direct `requests.post()` to `https://api.groq.com/openai/v1/chat/completions` — the `groq` Python SDK is **not** installed
- **Model** — `claude-sonnet-4-20250514` should NOT be used here; use `llama-3.3-70b-versatile` (Groq)
- **Max tokens** — 4,096; **temperature** — 0.2
- **Text limit** — combined document text is truncated to 14,000 characters before being sent; the JSON response includes `"truncated": true` when this occurs so the UI can warn the user
- **`sort_order` input** — always wrap `int(request.form.get("sort_order", 0) or 0)` in `try/except (ValueError, TypeError)` — bad input must degrade gracefully to 0, not raise a 500
Ownership rule: users can only edit/delete their own bids. Admins can edit/delete any bid. This is enforced in both the route (`can_edit = is_admin or bid.added_by == user.id`) and the rendered detail panel.
Ownership rule: users can only edit/delete their own bids. Admins can edit/delete any bid. Enforced via `can_edit = is_admin or bid.added_by == user.id`.
**`_ser(row)`** — module-level helper that converts a DB dict to JSON-serialisable form (dates/times → ISO strings). Do not redefine it inline inside individual functions.
All write operations (create, edit, delete bid; add, delete update — both form and JSON variants) call `log_action()`.
MySQL connector returns `datetime`, `date`, `timedelta` objects which are not JSON-serialisable. Always convert:
MySQL connector returns `datetime`, `date`, `timedelta` objects which are not JSON-serialisable. Use the shared helper (already defined at module level in `bid_tracker.py`; replicate it where needed):
### Adding a new model query with optional search/filter
Do **not** use f-strings to interpolate WHERE clauses into SQL. Build the query with string concatenation and always pass user values through the `%s` parameter list:
```python
sql="SELECT ... FROM table"
params=[]
ifsearch:
sql+=" WHERE col LIKE %s"
params.append(f"%{search}%")
sql+=" ORDER BY id DESC LIMIT %s"
params.append(limit)
cur.execute(sql,params)
```
---
## Known Gotchas
| Gotcha | Detail |
|--------|--------|
| `activity_log.logged_at` | Column is `logged_at` in the production DB, not `created_at`. Queried as `al.logged_at AS created_at` |
| `activity_log.logged_at` | DDL now creates `logged_at`. A migration in `config.py` renames `created_at` → `logged_at` for old web-only installs. Queried as `al.logged_at AS created_at` in `get_activity_log()` |
| `TIME` columns → `timedelta` | MySQL returns `TIME` as `datetime.timedelta`. Use the `\|hhmm` filter in templates |
| `GROQ_API_KEY` in `.env` | Read directly via `os.environ.get("GROQ_API_KEY")` in the analyze route — not via `get_setting()` alone — because `get_setting()` depends on the DB row being populated |
| `div.modal` vs `.modal-overlay` | All modal backdrops must use `class="modal-overlay"`. Inner dialog boxes use `class="modal"` or `class="modal-dialog"`. Never use `div.modal` as a backdrop |
| Jinja macros in child templates | Cannot call `{{ macro_name() }}` before the `{% macro %}` definition is parsed. Inline the HTML instead |
| `INSERT IGNORE` for settings seed | Use `ON DUPLICATE KEY UPDATE value = IF(value='', VALUES(value), value)` — `INSERT IGNORE` silently skips rows that already exist, even with empty values |
| `bcrypt` not in original requirements | Added as `bcrypt==4.1.3`. Required by `models.py` for password hashing |
| `login.html` is standalone | Does NOT extend `base.html`. Has no `app.js`, no CSRF meta tag. Any form on this page needs `{{ csrf_token() }}` as a direct hidden input |
| `SECRET_KEY` must be fixed | `os.urandom(32)` fallback generates a different key per Gunicorn worker — sessions and CSRF break across workers. Always set `SECRET_KEY` in `.env` |
| `create_app()` called once | Only `wsgi.py` calls `create_app()`. Do not add a module-level call to `app.py` — it causes double initialisation (double `initialize_database()`, double blueprint registration) |
| `import datetime` in `models.py` | Already imported at the top of the file. Do not add inline `import datetime` inside functions |
---
@@ -213,6 +270,9 @@ def ser(row):
# Activate venv
source /home/webchecker/venv/bin/activate
# Install / update dependencies (e.g. after adding Flask-WTF)
pip install -r requirements.txt
# After changing Python files — reload Gunicorn (zero-downtime)
sudo systemctl reload webchecker
@@ -242,6 +302,8 @@ mysql -u webchecker_user -p webchecker -e "SELECT key_name, value FROM app_setti
/etc/nginx/sites-available/webchecker
```
---
## Change Philosophy
1.**Surgical, additive patches** — smallest possible change to achieve the goal
@@ -251,4 +313,29 @@ mysql -u webchecker_user -p webchecker -e "SELECT key_name, value FROM app_setti
5.**Migration existence checks** — all migrations safe to re-run
6.**Full file contents for 1–3 file changes**; deployment map for larger changesets
7.**Explicit deploy instructions** — migration steps separated from code steps
8.**Root cause analysis** on errors — never apply temporary workarounds
8.**Root cause analysis** on errors — never apply temporary workarounds
---
## Pending Improvements (To-Do)
### Security
- [x]**File upload size cap** — `MAX_CONTENT_LENGTH = 20 MB` set in `create_app()`; `RequestEntityTooLarge` handler returns JSON for `/ai-summary/` paths and flash+redirect for form routes
- [x]**HTTP security headers** — `@app.after_request` in `app.py` sets `X-Frame-Options`, `X-Content-Type-Options`, `Referrer-Policy` on every response
- [x]**Prevent last-admin demotion/deactivation** — `update_user()` in `models.py` checks that demoting or deactivating an admin won't leave zero active admins; raises `ValueError` surfaced as a flash message
- [x]**File MIME type validation** — `_validate_magic()` in `ai_summary.py` checks magic bytes for PDF (`%PDF`), DOCX/XLSX (`PK\x03\x04`), DOC/XLS (OLE header), and UTF-8 decodability for text types; no extra dependency needed
### UI / UX
- [ ]**Mobile-responsive sidebar** — No media queries exist; the sidebar + main-content layout is unusable on phones; implement a hamburger toggle that collapses the sidebar on small screens
- [ ]**Bid due-date urgency badges** — Client-side only: highlight bids due within 7 days with a warning badge and bids past due with a red "Overdue" pill in the bid list
- [ ]**Relative timestamps in Activity Log** — Format raw `YYYY-MM-DD HH:MM:SS` timestamps as "2 hours ago" / "3 days ago" using a small JS formatter
- [ ]**Empty state for AI analysis history** — When no analyses exist, show a call-to-action ("Upload your first document ↑") instead of a blank panel
- [ ]**Paginate bid list** — `get_all_bids()` fetches all rows with no limit; add `LIMIT`/`OFFSET` to the model query and a "load more" button in the split-pane list
### Functionality
- [ ]**Missed-shift alerting** — Query or report that flags shifts where zero `shift_checks` records exist for a given date, surfaced on the admin dashboard or via email
- [ ]**Bid deadline email reminders** — Use the existing `email.smtp_*` settings to send a daily digest of bids with `due_date` within the next 7 days
- [ ]**Server-side health checks** — Replace the Google favicon proxy in the user dashboard with a `/dashboard/health/<id>` route that makes a server-side `HEAD` request (with short timeout) for a real reachability signal
- [ ]**Password reset via email** — Time-limited token flow so users can self-service instead of requiring an admin edit; needs a `password_reset_tokens` table and SMTP integration
- [ ]**"Copy password" button in Credentials modal** — Wire `copyToClipboard()` (already in `app.js`) to the password field in the user dashboard credentials modal
- [ ]**Shift calendar view** — Weekly grid (Mon–Sun columns, shifts as rows) on the admin shifts page to make schedule gaps and overlaps visible at a glance
# Magic-byte signatures for each supported binary format.
# Office Open XML (docx/xlsx) and legacy OLE (doc/xls) share these headers.
_FILE_MAGIC:dict={
".pdf":b"%PDF",
".docx":b"PK\x03\x04",
".xlsx":b"PK\x03\x04",
".doc":b"\xd0\xcf\x11\xe0\xa1\xb1\x1a\xe1",
".xls":b"\xd0\xcf\x11\xe0\xa1\xb1\x1a\xe1",
}
def_validate_magic(data:bytes,ext:str)->None:
"""Raise ValueError if the file's actual bytes don't match its extension."""
magic=_FILE_MAGIC.get(ext)
ifmagic:
ifdata[:len(magic)]!=magic:
raiseValueError(
f"File content does not match the declared {ext} format "
"(possible disguised upload)."
)
elifextin(".txt",".csv",".md"):
# Text files must be decodable; binary data masquerading as text is rejected.
try:
data[:512].decode("utf-8")
exceptUnicodeDecodeError:
raiseValueError(
"Text file does not appear to be valid UTF-8. "
"Ensure the file is a plain text document."
)
def_extract_text(file_obj,ext:str)->str:
"""Extract plain text from an uploaded file object."""
data=file_obj.read()
_validate_magic(data,ext)
ifextin(".txt",".md",".csv"):
returndata.decode("utf-8",errors="replace")
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
