diff --git a/.gitignore b/.gitignore index b8affff..55fef4d 100644 --- a/.gitignore +++ b/.gitignore @@ -50,5 +50,4 @@ Thumbs.db # ─── pytest / coverage ─────────────────────────────────────────────────────── .pytest_cache/ .coverage -htmlcov/ -Claude.md \ No newline at end of file +htmlcov/ \ No newline at end of file diff --git a/Claude.md b/Claude.md new file mode 100644 index 0000000..b76ca56 --- /dev/null +++ b/Claude.md @@ -0,0 +1,765 @@ +# TechDesk — IT Helpdesk System: Knowledge Base for Claude + +This document is the authoritative reference for Claude when working on the +TechDesk codebase. It captures the full architecture, design decisions, known +pitfalls, and hard-won lessons from the development history of this application. + +--- + +## 1. Project Overview + +**TechDesk** is an internal IT helpdesk web application deployed at +`tickets.ltservicesinc.com`. It is built for LT Services Inc. and serves two +audiences: + +- **Employees** — submit and track IT support tickets +- **IT Staff / Admins** — manage, assign, resolve, and report on tickets + +### Technology Stack + +| Layer | Technology | +|---|---| +| Language | Python 3 | +| Web Framework | Flask | +| ORM | SQLAlchemy (via Flask-SQLAlchemy) | +| Database | MySQL (driver: PyMySQL) | +| Migrations | Alembic (via Flask-Migrate) | +| Templates | Jinja2 | +| Real-time | Flask-SocketIO (eventlet async mode) | +| WSGI Server | Gunicorn (single worker, eventlet worker class) | +| Reverse Proxy | nginx (port 7000 upstream, SSL via Certbot) | +| Auth | Flask-Login | +| Forms / CSRF | Flask-WTF | +| Rate Limiting | Flask-Limiter | +| Email | Flask-Mail | +| Background Jobs | APScheduler (BackgroundScheduler, daemon=True) | +| Rich Text Editor | TinyMCE 6 | +| AI Chatbot | Groq API (`llama-3.3-70b-versatile`) | +| HTML Sanitization | bleach | +| Static Files | Served directly by nginx from `/home/it-ticket/myapp/app/static` | + +### Deployment Path + +``` +/home/it-ticket/myapp/ +├── app/ +│ ├── __init__.py # App factory (create_app) +│ ├── models.py # All SQLAlchemy models +│ ├── routes/ +│ │ ├── admin.py # /admin/* — IT staff and admin routes +│ │ ├── api.py # /api/* — JSON endpoints + SocketIO events +│ │ ├── auth.py # /auth/* — login, register, password reset +│ │ ├── chatbot.py # /chatbot/* — AI chatbot endpoint +│ │ └── tickets.py # / and /tickets/* — employee-facing routes +│ ├── services/ +│ │ ├── email_ingestion_service.py +│ │ ├── log_service.py +│ │ ├── notification_service.py +│ │ ├── sla_service.py +│ │ └── validation_service.py +│ ├── templates/ +│ │ ├── base.html +│ │ ├── admin/ # Admin panel templates +│ │ ├── auth/ # Login, register, password reset templates +│ │ └── tickets/ # Employee-facing ticket templates +│ └── static/ +│ └── uploads/ # User-uploaded files (default; overridable via UPLOAD_FOLDER) +├── config/ +│ └── config.py # DevelopmentConfig / ProductionConfig +├── migrations/ +│ └── versions/ # Alembic migration scripts +├── gunicorn.conf.py +├── run.py # Entry point — calls eventlet.monkey_patch() first +└── .env # Secrets (not committed) +``` + +--- + +## 2. User Roles & Access Control + +Three roles are defined in `UserRole`: + +| Role | Constant | Access | +|---|---|---| +| Employee | `employee` | Submit tickets, view own tickets, knowledge base, chatbot | +| IT Staff | `it_staff` | Everything above + manage all tickets, assignment, bulk actions | +| Admin | `admin` | Everything above + user management, system settings, ticket deletion, KB management | + +### Decorators (defined in `admin.py`) + +```python +@admin_required # role == 'admin' only +@it_required # role in ('it_staff', 'admin') +``` + +Both decorators must be applied **after** `@login_required`. + +--- + +## 3. Database Models (`app/models.py`) + +### Core Models + +| Model | Table | Key Notes | +|---|---|---| +| `User` | `users` | Roles: employee / it_staff / admin. `is_admin` and `is_it_staff` are properties. | +| `Ticket` | `tickets` | Central entity. Has `generate_ticket_number()` with `FOR UPDATE` row locking to prevent race conditions. | +| `Comment` | `comments` | Bodies stored as sanitized HTML (rendered at write time via `render_comment_body`). | +| `Attachment` | `attachments` | Tied to both `ticket_id` and `comment_id`. Physical files live in `UPLOAD_FOLDER`. | +| `Notification` | `notifications` | Per-user; read/unread. Cascades delete with ticket. | +| `TicketHistory` | `ticket_history` | Field-level audit trail. `old_value`/`new_value` are TEXT (widened in migration 001). | +| `ActivityLog` | `activity_logs` | System-wide action log. `user_id` has `ondelete='SET NULL'`. | +| `KnowledgeBase` | `knowledge_base` | Articles with TinyMCE body, tags, view count, publish toggle. | +| `KBAttachment` | `kb_attachments` | Files attached to KB articles. Shared `UPLOAD_FOLDER` with ticket attachments. | +| `SystemSetting` | `system_settings` | Key-value config store. Use `SystemSetting.get(key)` / `SystemSetting.set(key, value)`. | +| `TicketLink` | `ticket_links` | Symmetric bidirectional links between tickets (related / duplicate / follow_up). | +| `CannedResponse` | `canned_responses` | Pre-written reply templates for IT staff. | +| `KBFeedback` | `kb_feedback` | One thumbs up/down vote per user per article. | +| `PasswordResetToken` | `password_reset_tokens` | SHA-256 hashed, 1-hour expiry, single-use. | +| `TicketTemplate` | `ticket_templates` | Pre-filled ticket scaffolds selectable on the new ticket form. | +| `TicketSatisfaction` | `ticket_satisfaction` | One survey per resolved ticket; token-authenticated survey URL. | + +### Cascade Rules + +`Ticket` has `cascade='all, delete-orphan'` on: +- `comments` +- `attachments` +- `notifications` +- `history` +- `links_as_source` / `links_as_target` (via `TicketLink`) +- `satisfaction` (via `TicketSatisfaction`) + +**When deleting a ticket, physical files in `UPLOAD_FOLDER` must be removed +manually before the DB delete** — SQLAlchemy cascades handle DB rows only. + +--- + +## 4. Blueprint & Route Structure + +| Blueprint | Prefix | File | Guard | +|---|---|---|---| +| `auth_bp` | (none) | `routes/auth.py` | Public + `@login_required` per route | +| `tickets_bp` | (none) | `routes/tickets.py` | `@login_required` per route | +| `admin_bp` | `/admin` | `routes/admin.py` | `@login_required` + `@it_required` or `@admin_required` | +| `api_bp` | `/api` | `routes/api.py` | `@login_required` per route | +| `chatbot_bp` | `/chatbot` | `routes/chatbot.py` | `@login_required` per route | + +### Key Routes Reference + +**Auth (`auth.py`)** +- `GET/POST /login` — rate limited: 10/min, 50/hr +- `GET/POST /register` — rate limited: 5/min, 20/hr; guarded by `registration_enabled` SystemSetting +- `GET /logout` +- `GET/POST /profile` +- `GET/POST /forgot-password` — rate limited: 5/hr +- `GET/POST /reset-password/` — rate limited: 10/hr +- `GET /avatar/` — login required (prevents unauthenticated access) +- `GET /logo/` — public (used in login page) + +**Tickets (`tickets.py`)** +- `GET /` and `GET /dashboard` — role-aware: employees see `dashboard_employee.html`, IT staff see `dashboard_it.html` +- `GET/POST /tickets/new` +- `GET/POST /tickets/behalf` — IT staff only (file ticket on behalf of employee) +- `GET /tickets` — employee ticket list (own tickets only) +- `GET/POST /tickets/` — ticket detail + comment submission +- `POST /tickets//update` — IT staff update (status, priority, assignment, etc.) +- `POST /comments//delete` +- `GET /attachments/` — file download +- `GET /notifications` / `POST /notifications/mark-read` +- `GET /kb` / `GET /kb/` — knowledge base +- `POST /tickets//link` / `POST /tickets//unlink/` +- `POST /tickets//reopen` +- `GET /canned-responses` — JSON endpoint for IT staff comment box +- `POST /kb//feedback` +- `GET/POST /survey/` — public (no login required) + +**Admin (`admin.py`)** +- `GET /admin/` — IT Operations Overview dashboard +- `GET /admin/users` / `POST /admin/users/new` / `POST /admin/users//edit` / `POST /admin/users//delete` +- `GET /admin/tickets` — all tickets with search, filter, pagination +- `POST /admin/tickets//delete` — **admin only**; deletes ticket + physical files +- `POST /admin/tickets/bulk-action` — actions: resolve, close, assign_me, unassign, **delete** (delete is admin-only) +- `GET /admin/tickets/export` — CSV export with active filters +- `GET /admin/kb` / `GET/POST /admin/kb/new` / `GET/POST /admin/kb//edit` / `POST /admin/kb//delete` / `POST /admin/kb//publish` +- `POST /admin/kb/upload-image` — TinyMCE image upload +- `GET /admin/kb/files/` — file serve (login required) +- `GET/POST /admin/logs` — activity log viewer +- `GET/POST /admin/canned-responses/*` +- `POST /admin/tickets/bulk-action` +- `GET/POST /admin/ticket-templates/*` +- `GET /admin/satisfaction` — satisfaction report +- `GET/POST /admin/settings` + +**API (`api.py`)** +- `GET /api/notifications` — JSON list +- `GET /api/notifications/unread-count` +- `POST /api/notifications//read` +- `POST /api/notifications/mark-all-read` +- `GET /api/tickets//comments` — JSON (used by real-time comment rendering) +- `GET /api/stats/tickets` — dashboard KPI stats +- `GET /api/users/search` — typeahead for ticket assignment +- `GET /api/tickets/search` — typeahead for ticket linking +- SocketIO events: `connect`, `disconnect`, `join_ticket`, `leave_ticket` + +**Chatbot (`chatbot.py`)** +- `POST /chatbot/message` — rate limited: 20/min, 100/hr; calls Groq API + +--- + +## 5. Services + +### `log_service.py` + +```python +log_action(user_id, action, entity_type=None, entity_id=None, details=None) +log_ticket_history(ticket, field_name, old_value, new_value, changed_by_id) +``` + +**Critical rules:** +- Neither function calls `db.session.commit()`. The caller must commit. +- On failure, only the failed log entry is `expunge()`d — **never call `db.session.rollback()`** inside these functions, as that would undo the parent operation. +- Always call `log_action` **before** `db.session.commit()`. +- `log_ticket_history` records granular field-level changes on tickets. + +**Standard action strings used across the codebase:** + +| Action String | Where | +|---|---| +| `ticket_create` | New ticket submitted | +| `ticket_update` | Ticket fields changed | +| `comment_create` | Comment added | +| `comment_delete` | Comment deleted | +| `admin_user_create` | Admin creates a user | +| `admin_user_edit` | Admin edits a user | +| `admin_user_deactivate` | Admin deactivates a user | +| `admin_ticket_delete` | Admin deletes a ticket | +| `ticket_bulk_resolve` | Bulk resolve | +| `ticket_bulk_close` | Bulk close | +| `ticket_bulk_assign_me` | Bulk assign to self | +| `ticket_bulk_unassign` | Bulk unassign | +| `kb_create` / `kb_edit` / `kb_delete` | KB article CRUD | +| `kb_attachment_delete` | KB attachment removed | + +### `notification_service.py` + +Functions: `notify_new_ticket`, `notify_status_change`, `notify_comment_added`, +`notify_assignment`, `send_satisfaction_survey` + +**Critical rule:** Always call notification functions **after** `db.session.commit()`. +Calling them before commit means they act on uncommitted state. + +### `sla_service.py` + +- `set_due_date(ticket, app)` — called at ticket creation; reads SLA hours from `SystemSetting` +- `check_sla_breaches(app)` — runs every 30 minutes via APScheduler; creates notifications for overdue tickets +- `clear_sla_notification(ticket_id)` — called when a ticket is resolved or closed + +### `validation_service.py` + +- `validate_password(password, confirm)` — returns error string or None +- `validate_file(file, allowed_extensions)` — magic-byte MIME validation (not just extension) +- `render_comment_body(raw_text)` — converts Markdown/plain text to sanitized HTML via bleach; called at write time, not display time + +### `email_ingestion_service.py` + +- `check_inbound_email(app)` — APScheduler job; polls IMAP mailbox and converts emails to tickets +- Deduplication via `Message-ID` headers stored in `SystemSetting` +- Unknown senders fall back to the system admin user account + +--- + +## 6. Configuration & Environment + +### `.env` Keys + +``` +SECRET_KEY= +DB_USER= +DB_PASSWORD= +DB_HOST= +DB_PORT=3306 +DB_NAME=it_tickets +MAIL_SERVER= +MAIL_PORT=587 +MAIL_USE_TLS=True +MAIL_USERNAME= +MAIL_PASSWORD= +MAIL_DEFAULT_SENDER= +IT_DEPT_EMAIL= +APP_BASE_URL=https://tickets.ltservicesinc.com +UPLOAD_FOLDER= # Absolute path; defaults to app/static/uploads +GROQ_API_KEY= +GROQ_MODEL=llama-3.3-70b-versatile +ADMIN_EMAIL= +ADMIN_PASSWORD= +RATELIMIT_STORAGE_URI=memory:// +``` + +### `SystemSetting` Keys (runtime config, editable via Admin → Settings) + +| Key | Default | Description | +|---|---|---| +| `registration_enabled` | `true` | Allow self-registration | +| `survey_enabled` | `true` | Send satisfaction survey on resolve | +| `app_timezone` | `America/New_York` | Display timezone for all dates | +| `sla_critical_hours` | `4` | SLA breach threshold for CRITICAL | +| `sla_high_hours` | `8` | SLA breach threshold for HIGH | +| `sla_medium_hours` | `48` | SLA breach threshold for MEDIUM | +| `sla_low_hours` | `120` | SLA breach threshold for LOW | +| `app_name` | `TechDesk` | Sidebar/page title branding | +| `app_subtitle` | `IT Helpdesk System` | Sidebar subtitle | +| `company_name` | `` | Login/register page branding | +| `logo_stored_name` | `` | Uploaded logo filename | +| `logo_initials` | `TD` | Fallback initials when no logo | +| `primary_color` | `#2563eb` | CSS accent color (hex) | +| `email_ingestion_enabled` | `0` | Enable IMAP email-to-ticket | +| `email_ingestion_host/port/user/password/folder/move_to/interval` | various | IMAP settings | + +--- + +## 7. Known Pitfalls & Hard Rules + +These are non-negotiable lessons learned from production incidents and debugging +sessions. Violating any of these will introduce bugs. + +### 7.1 eventlet / Gunicorn Ordering + +- `eventlet.monkey_patch()` **must** run before the app is loaded. +- It is called in `run.py` at module level before `create_app()`. +- `preload_app = False` in `gunicorn.conf.py` is **mandatory**. Setting it to + `True` causes the app to load in the master process before `monkey_patch()` + runs, leaving real OS RLocks un-greened and triggering: + `"1 RLock(s) were not greened"` + +### 7.2 Flask-Limiter Storage + +- Flask-Limiter does **not** support `mysql+pymysql://` as a storage URI. +- Always use `memory://` (or `redis://` if Redis becomes available). +- The `RATELIMIT_STORAGE_URI` config key controls this. + +### 7.3 SQLAlchemy Session & Generators + +- Generators that access SQLAlchemy ORM objects execute **lazily**. +- If a generator is returned from a route and consumed outside the request + context (after the session closes), it silently yields nothing. +- Always collect ORM results into a list **within** the session context before + yielding or returning from a route. This was the cause of empty CSV exports. + +### 7.4 `db.create_all()` and Alembic + +- `db.create_all()` is called in `create_app()` for first-run convenience. +- If tables already exist (normal in production), `create_all()` is a no-op. +- **Do not use `db.create_all()` as a substitute for migrations.** New columns + added in migrations will not be created by `create_all()` on existing tables. +- Alembic revision IDs must fit within `VARCHAR(64)` in `alembic_version` + (widened in migration 003). Do not use revision IDs longer than 64 characters. +- Never break the migration chain: each migration's `Revises:` must point to + the previous revision ID exactly. + +### 7.5 `db.session.get()` vs `Model.query.get()` + +- `Model.query.get()` is **deprecated** in SQLAlchemy 2.x. +- Always use `db.session.get(Model, pk)` for primary-key lookups. +- For 404 handling: `db.session.get(Ticket, ticket_id) or abort(404)` + +### 7.6 CSRF on AJAX Requests + +- All `fetch()` / `XMLHttpRequest` POST requests must include the CSRF token. +- Source it from `` in + `base.html`. +- Header name: `X-CSRFToken` + +### 7.7 WebSocket Emit Timing + +- Never emit a SocketIO event inside a request handler **before** the HTTP→WS + transport upgrade completes. +- Use `socketio.start_background_task()` to defer emits that should not block + the request cycle. +- Emitting synchronously before upgrade causes disconnect storms. + +### 7.8 `@login_required` on File-Serve Routes + +- Routes that serve files via `send_from_directory` (e.g. `kb_serve_file`, + `serve_avatar`) must have `@login_required`. +- **Exception:** Logo serving (`serve_logo`) must be **public** because the + login page renders the logo before authentication. +- `@login_required` on image-serve routes blocks `` tag requests in some + browser/session contexts. Test `` rendering explicitly after + adding auth guards to file routes. + +### 7.9 Notification Timing + +- `notify_assignment()`, `notify_status_change()`, and all notification + functions must be called **after** `db.session.commit()`. +- Calling them before commit means they reference uncommitted state and may + send emails/notifications for operations that subsequently fail. + +### 7.10 `regexp_replace` on MySQL + +- MySQL 8.0's `REGEXP_REPLACE(expr, pat, repl)` accepts **exactly 3 arguments**. +- The 4th `flags` argument (`'g'` for global) is a PostgreSQL/MariaDB extension. +- MySQL applies global replacement by default — the `'g'` flag is unnecessary + and causes: `(1583, "Incorrect parameters in the call to native function 'regexp_replace'")` +- **Always write:** `func.regexp_replace(Comment.body, r'<[^>]+>', '')` + — **never:** `func.regexp_replace(Comment.body, r'<[^>]+>', '', 'g')` + +### 7.11 Chatbot Duplicate Ticket Prevention + +- After a ticket is created via the chatbot, `chatHistory` on the frontend + must be purged to prevent re-submission on the next message. +- The server strips `"action": "create_ticket"` blocks from history before + sending to the Groq API to prevent the model from re-triggering creation. + +### 7.12 Ticket Deletion — Physical File Cleanup + +- SQLAlchemy `cascade='all, delete-orphan'` removes `Attachment` **rows** but + does **not** delete files from disk. +- Always iterate `ticket.attachments.all()` and `os.remove()` each + `attachment.stored_name` from `UPLOAD_FOLDER` before calling `db.session.delete(ticket)`. +- Wrap each `os.remove()` in try/except and log warnings for missing files + (do not abort the delete if a file is already gone). + +### 7.13 Bulk Actions — Delete vs Status Actions + +- Bulk delete must **early-return** after its own commit so the + resolve/close/assign/unassign loop does not run on already-deleted tickets. +- Bulk delete is **admin-only** even though the `bulk_ticket_action` route is + decorated with `@it_required`. Check `current_user.is_admin` inside the + route for the delete branch. + +--- + +## 8. Comment Body Storage + +- Comments are stored as **sanitized HTML** at write time, not display time. +- `render_comment_body(raw_text)` in `validation_service.py` converts + Markdown/plain text to HTML and sanitizes via bleach. +- When searching comment bodies in SQL, the stored HTML must be stripped before + `LIKE` matching: `func.regexp_replace(Comment.body, r'<[^>]+>', '')`. +- Migration 003 backfilled all existing plain-text comment bodies to HTML. + +--- + +## 9. Real-time Comments (SocketIO) + +- Clients join a ticket room on the detail page via `join_ticket` event. +- New comments are submitted via AJAX (`fetch` POST) with `X-CSRFToken` header. +- The server broadcasts the new comment to the room via SocketIO after commit. +- The `get_comments` API endpoint (`/api/tickets//comments`) returns + comment HTML for rendering; it calls `_resolve_mime_type` to handle + NULL/octet-stream MIME types with extension-based fallback. + +--- + +## 10. File Uploads + +- All uploads (ticket attachments, comment images, KB files, avatars, logo) + land in the same `UPLOAD_FOLDER` directory. +- Files are stored with UUID-based `stored_name` values (not the original + filename) to prevent path traversal and collisions. +- `validate_file()` uses magic-byte inspection (not just extension) for MIME + validation. +- Allowed extensions: `png, jpg, jpeg, gif, pdf, doc, docx, txt, zip, log` +- Max upload size: `MAX_CONTENT_LENGTH` (default 16 MB) +- Per-comment attachment limits are enforced both client-side and server-side. + +--- + +## 11. Branding & Theming + +- App name, subtitle, logo, and primary color are all configurable via + `Admin → Settings` and stored in `SystemSetting`. +- Branding values are injected into every template via the `inject_globals` + context processor in `create_app()`. +- The primary color is injected as a CSS variable override so the entire UI + recolors without a code change. +- Logo is served via `GET /logo/` — this route must remain **public** + so the login page can render the logo before authentication. + +--- + +## 12. Migrations + +Migrations live in `migrations/versions/`. The chain is: + +``` +(base) + └── 001_widen_ticket_history_values — VARCHAR(200) → TEXT for old/new_value + └── 002_add_system_settings — creates system_settings table + └── 003_render_comments — backfills comment bodies to HTML; + widens alembic_version.version_num to VARCHAR(64) +``` + +**Rules:** +- Never modify an existing migration that has been applied to production. +- Always set `Revises:` to the previous revision ID exactly. +- Revision IDs must be ≤ 64 characters. +- Run `flask db upgrade` after deploying new migration files. +- `db.create_all()` in `create_app()` will not apply pending migrations — + it only creates tables that do not exist at all. + +--- + +## 13. Security Measures + +| Concern | Implementation | +|---|---| +| CSRF | Flask-WTF `CSRFProtect`; AJAX uses `X-CSRFToken` header | +| Rate limiting | Flask-Limiter on login (10/min, 50/hr), register (5/min, 20/hr), forgot-password (5/hr), chatbot (20/min, 100/hr) | +| Password hashing | Werkzeug `generate_password_hash` / `check_password_hash` | +| Password reset tokens | SHA-256 hashed, 1-hour TTL, single-use | +| File upload validation | Magic-byte MIME inspection + extension allowlist | +| HTML sanitization | bleach on all comment bodies and KB article bodies | +| XSS in KB | bleach allowlist covers TinyMCE-produced tags; `_sanitize_kb_body()` in `admin.py` | +| SQL injection | SQLAlchemy ORM parameterized queries throughout | +| Real client IP | `ProxyFix` middleware trusts one nginx hop; `log_service._get_real_ip()` reads `X-Forwarded-For` | +| Role enforcement | `@admin_required` / `@it_required` decorators; per-route checks where needed | +| Avatar/file access | All file-serve routes require `@login_required` (except logo) | + +--- + +## 14. Activity Logging Reference + +Every create, edit, and delete action must call `log_action()` before the +`db.session.commit()` that finalizes the operation. + +For ticket field changes, also call `log_ticket_history()` for each changed +field. + +Python logger calls (`logger.info(...)`) use structured bracket prefixes for +easy `grep`: + +``` +[TICKET DELETE] +[BULK DELETE] +[BULK ACTION] +[TICKET HISTORY] +[ACTIVITY] +[ACTIVITY LOG ERROR] +[SLA] +[EMAIL INGEST] +[SEED] +[KB IMAGE UPLOAD] +[TZ] +``` + +--- + +## 15. Deployment Checklist + +After any code change: + +```bash +# 1. Copy changed files to deployment path +cp /home/it-ticket/myapp/ + +# 2. If new migration files added: +cd /home/it-ticket/myapp && flask db upgrade + +# 3. Restart Gunicorn +sudo systemctl restart gunicorn + +# 4. Verify — check logs for errors +sudo journalctl -u gunicorn -n 50 +tail -f /home/it-ticket/myapp/logs/gunicorn_error.log +``` + +nginx does **not** need to be restarted for Python code changes (it only +proxies; static files are served directly). + +--- + +## 16. Completed Features (Development History) + +| Feature | Status | +|---|---| +| Core ticket CRUD (create, view, update, comment) | ✅ | +| Role-based access control (employee / IT staff / admin) | ✅ | +| Real-time comments via SocketIO | ✅ | +| Image upload in comments (clipboard paste + file picker) | ✅ | +| User avatar upload with disk cleanup | ✅ | +| Admin-controlled user registration toggle | ✅ | +| CSV export with active filter support | ✅ | +| Markdown rendering with bleach sanitization (stored at write time) | ✅ | +| Full branding customization (logo, name, colors) | ✅ | +| Knowledge Base with TinyMCE, attachments, feedback | ✅ | +| IT Operations Overview dashboard (KPIs, per-staff metrics) | ✅ | +| Canned responses for IT staff | ✅ | +| Ticket templates for common request types | ✅ | +| Ticket linking (related / duplicate / follow-up) | ✅ | +| SLA breach detection with APScheduler | ✅ | +| Satisfaction surveys on ticket resolution | ✅ | +| Email-to-ticket ingestion via IMAP | ✅ | +| IT staff notifications (in-app + individual email) on new ticket | ✅ | +| Browser tab title shows unread notification count | ✅ | +| AI chatbot via Groq (llama-3.3-70b-versatile) | ✅ | +| Self-service password reset with token-authenticated email | ✅ | +| Admin ticket deletion (single + bulk) | ✅ | +| Activity log viewer | ✅ | +| Configurable display timezone | ✅ | + +--- + +## 17. Declined / Rejected Features + +| Feature | Reason | +|---|---| +| WebRTC / Remote desktop control | Architectural complexity too high; prefer integrating existing tools (e.g. AnyDesk, TeamViewer) over building from scratch | + +--- + +*Last updated: 2026-04-17* + +## 18. Notification Architecture + +### `notify_new_ticket` Fan-out (updated 2026-04-17) + +When a ticket is created — via the web form, the chatbot, the "on behalf" flow, +or email ingestion — `notify_new_ticket(ticket)` delivers three tiers of alerts: + +1. **Department alias email** — one email to `IT_DEPT_EMAIL` (the shared inbox). +2. **Individual staff emails** — one email per active `it_staff` / `admin` user + whose `email_notif == True`. Consistent with `notify_comment_added` and + `notify_assignment`. +3. **In-app notifications** — one `Notification` row per active IT staff / admin + user, regardless of `email_notif` setting, pushed via WebSocket. +4. **Creator confirmation** — one in-app notification to the ticket submitter. + +### Background Job Email Delivery (`send_email`) + +`send_email()` spawns a background thread. The daemon mode depends on caller +context: + +- **Inside a web request** → `daemon=True` (process won't be held open by the + thread after the response is sent). +- **Outside a request context** (APScheduler job) → `daemon=False`. Daemon + threads are killed when the parent thread exits; since APScheduler job threads + finish quickly, a daemon SMTP thread is terminated before delivery completes, + causing **silent email loss**. Non-daemon threads survive until SMTP finishes. + +The same logic applies to `send_satisfaction_survey`'s inline thread spawn. + +### SocketIO Emit from Background Jobs + +`create_notification` emits `new_notification` via SocketIO. The emit strategy +branches on `has_request_context()`: + +- **Inside a web request** → `socketio.start_background_task(_emit)` (non-blocking, + avoids racing the polling→WebSocket upgrade handshake). +- **Outside a request context** → `socketio.emit()` called directly. Using + `start_background_task` from a background job causes Flask-SocketIO to invoke + the `on_connect` handler internally, which references `current_user`. With no + session attached, Flask-Login returns `None` → `AttributeError`. + +--- + +## 19. Hard Rules (additions 2026-04-17) + +### 7.14 `inject_globals` Context Processor and Background Rendering + +- `render_template_string()` and `render_template()` trigger **all registered + context processors**, including `inject_globals` in `app/__init__.py`. +- `inject_globals` accesses `current_user.is_authenticated`. Outside a request + context (e.g. APScheduler jobs calling `notify_new_ticket`), Flask-Login + resolves `current_user` to `None`, causing `AttributeError`. +- **Rule:** Always guard `current_user` access in context processors with + `has_request_context()` before calling `current_user.is_authenticated`: + ```python + if has_request_context() and current_user.is_authenticated: + ``` +- **Corollary:** Any service function that calls `render_template_string()` or + `render_template()` from a background context (APScheduler, daemon threads, + CLI commands) will trigger this processor. The guard in `inject_globals` is + the single correct fix — do not work around it at the call site. + +### 7.15 SocketIO `on_connect` / `on_disconnect` Handlers and Sessionless Contexts + +- The `on_connect` and `on_disconnect` SocketIO event handlers in `api.py` + reference `current_user`. When `socketio.emit()` or `start_background_task` + is called without a session-backed request context, these handlers may be + invoked with `current_user = None`. +- **Rule:** All SocketIO event handlers that reference `current_user` must guard + with `if current_user and current_user.is_authenticated:` (not just + `if current_user.is_authenticated:`). + +### 7.16 `SystemSetting.value` Column Width + +- `SystemSetting.value` was `VARCHAR(500)`. The `email_ingested_message_ids` + key accumulates comma-separated Message-IDs; Outlook Message-IDs are ~100 + characters each, so 6+ processed messages overflow the column. +- **Widened to `TEXT` in migration 004** (`004_widen_system_setting_value`). +- **Rule:** Never use `VARCHAR` for any `SystemSetting` value that can accumulate + list data or free-form strings of unbounded length. Use `Text`. + +### 7.17 Diagnosis Protocol — Root Cause Before Fix + +When an error is reported: +1. **Read the full traceback**, not just the exception message. Add `exc_info=True` + to the catching `logger.error()` call if the traceback is being swallowed. +2. **Trace the complete call chain** from the error site back to the originating + caller — do not assume the fix belongs at the first plausible location. +3. **Grep every file** in the call chain for the symbol that is `None` or missing + before writing any code. +4. **Write the fix, verify syntax (`ast.parse`), and assert the changed lines are + present** before packaging for deployment. +5. **Never apply a patch to a symptom site** (e.g. adding a guard to `api.py`) + when the root cause is upstream (e.g. `inject_globals` crashing during + `render_template_string`). Symptom-site patches mask errors without resolving + them and accumulate technical debt. + +--- + +## 20. Migration History (updated) + +| Revision ID | Description | +|---|---| +| `001_widen_ticket_history_values` | Widen `ticket_history.old_value` / `new_value` from `VARCHAR(200)` to `TEXT` | +| `002_add_system_settings` | Create `system_settings` table | +| `003_render_comments` | Backfill comment bodies to HTML; widen `alembic_version.version_num` to `VARCHAR(64)` | +| `004_widen_system_setting_value` | Widen `system_settings.value` from `VARCHAR(500)` to `TEXT` | + +## 21. Browser Tab Notification Counter (added 2026-04-17) + +### Overview + +When the user has TechDesk open in a background tab, the browser tab title +displays the current unread notification count as a prefix so they can see +pending work without switching back. + +Examples: +- `(3) Dashboard — TechDesk` ← 3 unread notifications +- `Dashboard — TechDesk` ← no unread notifications + +### Implementation (`app/templates/base.html`) + +**`` tag** — given `id="page-title-text"` so JavaScript can read the +server-rendered base title without hardcoding it: +```html +<title id="page-title-text">{% block title %}...{% endblock %} — {{ branding.app_name }} +``` + +**`updateDocumentTitle(unreadCount)`** — new JS function that sets +`document.title` to `"(n) "` when `n > 0`, or restores the +original title when `n === 0`. + +**`_baseDocTitle`** — captured once at page load from the `` element so +page-specific titles (e.g. "Ticket Detail", "Dashboard") are preserved exactly. + +**`updateBadge()`** — extended to call `updateDocumentTitle(next)` on every +badge change, covering all update paths: +- Real-time WebSocket `new_notification` event +- Opening the notification panel (`loadNotifications` recalculates count) +- Clicking an individual notification to mark it read +- "Mark all read" button + +**IIFE on page load** — reads the server-rendered badge count and calls +`updateDocumentTitle()` immediately, so the title is correct even before any +WebSocket event fires. + +### Key Design Decisions + +- No new routes, API calls, or Python changes — purely a frontend enhancement. +- `_baseDocTitle` is captured from the DOM, not hardcoded, so it works correctly + for every page that extends `base.html` without any per-template changes. +- The count in the title stays in sync with the bell badge — a single + `updateBadge()` call updates both, so they can never diverge. diff --git a/app/__init__.py b/app/__init__.py index 5a14aaa..524307c 100644 --- a/app/__init__.py +++ b/app/__init__.py @@ -182,6 +182,16 @@ def create_app(config_name=None): return dict(unread_notifications=unread, branding=branding, now_local=now_local, app_tz_name=_tz_name) + # ── Security headers ────────────────────────────────────────────────────── + @app.after_request + def set_security_headers(response): + response.headers.setdefault('X-Frame-Options', 'SAMEORIGIN') + response.headers.setdefault('X-Content-Type-Options', 'nosniff') + response.headers.setdefault('Referrer-Policy', 'strict-origin-when-cross-origin') + response.headers.setdefault('X-XSS-Protection', '1; mode=block') + response.headers.setdefault('Permissions-Policy', 'geolocation=(), microphone=(), camera=()') + return response + # ── DB initialisation (first run) ───────────────────────────────────────── with app.app_context(): db.create_all() diff --git a/app/templates/admin/activity_logs.html b/app/templates/admin/activity_logs.html index 325f271..9bd7acc 100644 --- a/app/templates/admin/activity_logs.html +++ b/app/templates/admin/activity_logs.html @@ -60,7 +60,7 @@ {% for days, colour, icon, label, hint in options %} <div class="col-6 col-lg-3"> <form method="POST" action="{{ url_for('admin.activity_logs') }}" - onsubmit="return confirmCleanup({{ days }}, {{ counts_by_retention[days] }})"> + onsubmit="return confirmCleanup(this, {{ days }}, {{ counts_by_retention[days] }})"> <input type="hidden" name="csrf_token" value="{{ csrf_token() }}"/> <input type="hidden" name="days" value="{{ days }}"/> <button type="submit" class="btn btn-sm w-100 d-flex flex-column align-items-center py-3 gap-1" @@ -170,12 +170,17 @@ </div> <script> -function confirmCleanup(days, count) { +function confirmCleanup(form, days, count) { if (count === 0) return false; - return confirm( + confirmModal( + 'Clean Up Activity Logs', 'Delete ' + count.toLocaleString() + ' log entr' + (count === 1 ? 'y' : 'ies') + - ' older than ' + days + ' days?\n\nThis cannot be undone.' + ' older than ' + days + ' days? This action cannot be undone.', + 'Delete Logs', + 'btn-danger', + () => { form._confirmed = true; form.submit(); } ); + return false; } </script> {% endblock %} \ No newline at end of file diff --git a/app/templates/admin/canned_responses.html b/app/templates/admin/canned_responses.html index d225dd1..7903a19 100644 --- a/app/templates/admin/canned_responses.html +++ b/app/templates/admin/canned_responses.html @@ -44,7 +44,7 @@ class="btn btn-secondary btn-sm"><i class="bi bi-pencil"></i></a> <form method="POST" action="{{ url_for('admin.canned_response_delete', item_id=item.id) }}" - onsubmit="return confirm('Delete this quick reply?')"> + data-confirm="Delete this quick reply?" data-confirm-title="Delete Quick Reply" data-confirm-ok="Delete"> <input type="hidden" name="csrf_token" value="{{ csrf_token() }}"/> <button type="submit" class="btn btn-sm btn-outline-danger"><i class="bi bi-trash"></i></button> </form> diff --git a/app/templates/admin/kb_edit.html b/app/templates/admin/kb_edit.html index 3c09a45..b848834 100644 --- a/app/templates/admin/kb_edit.html +++ b/app/templates/admin/kb_edit.html @@ -369,12 +369,10 @@ function loadExistingAtts() { } function deleteAtt(attId, deleteUrl, filename) { - if (!confirm('Delete "' + filename + '"?')) return; - - const row = document.getElementById('att-row-' + attId); - if (row) { row.style.opacity = '0.4'; row.style.pointerEvents = 'none'; } - - fetch(deleteUrl, { method: 'POST', credentials: 'same-origin', headers: { 'X-CSRFToken': document.querySelector('meta[name="csrf-token"]').content } }) + confirmModal('Delete Attachment', 'Delete "' + filename + '"? This cannot be undone.', 'Delete', 'btn-danger', () => { + const row = document.getElementById('att-row-' + attId); + if (row) { row.style.opacity = '0.4'; row.style.pointerEvents = 'none'; } + fetch(deleteUrl, { method: 'POST', credentials: 'same-origin', headers: { 'X-CSRFToken': document.querySelector('meta[name="csrf-token"]').content } }) .then(r => r.json()) .then(data => { if (data.ok) { @@ -395,6 +393,7 @@ function deleteAtt(attId, deleteUrl, filename) { if (row) { row.style.opacity = ''; row.style.pointerEvents = ''; } alert('Network error. Please try again.'); }); + }); } // Load existing attachments on page load (only for edit mode) diff --git a/app/templates/admin/kb_list.html b/app/templates/admin/kb_list.html index a510acc..cfbb2bb 100644 --- a/app/templates/admin/kb_list.html +++ b/app/templates/admin/kb_list.html @@ -110,7 +110,7 @@ </button> {% endif %} </form> - <form method="POST" action="{{ url_for('admin.kb_delete', article_id=art.id) }}" onsubmit="return confirm('Delete this article?');" style="margin:0;"> + <form method="POST" action="{{ url_for('admin.kb_delete', article_id=art.id) }}" data-confirm="Delete this article? This cannot be undone." data-confirm-title="Delete Article" data-confirm-ok="Delete" style="margin:0;"> <input type="hidden" name="csrf_token" value="{{ csrf_token() }}"/> <button type="submit" class="btn btn-sm" style="background:rgba(248,113,113,.1);border:1px solid rgba(248,113,113,.2);color:var(--danger);" title="Delete"><i class="bi bi-trash"></i></button> </form> diff --git a/app/templates/admin/settings.html b/app/templates/admin/settings.html index e405fae..007a013 100644 --- a/app/templates/admin/settings.html +++ b/app/templates/admin/settings.html @@ -368,7 +368,8 @@ {% if survey_enabled %} <button type="submit" name="survey_enabled" value="0" class="btn btn-sm btn-outline-danger" - onclick="return confirm('Disable satisfaction surveys? No survey emails will be sent when tickets are resolved.')"> + data-confirm="Disable satisfaction surveys? No survey emails will be sent when tickets are resolved." + data-confirm-title="Disable Satisfaction Surveys" data-confirm-ok="Disable"> <i class="bi bi-toggle-on me-1"></i>Disable </button> {% else %} @@ -416,7 +417,8 @@ {% if registration_enabled %} <button type="submit" name="registration_enabled" value="0" class="btn btn-sm btn-outline-danger" - onclick="return confirm('Disable self-registration? New users will not be able to sign up independently.')"> + data-confirm="Disable self-registration? New users will not be able to sign up independently." + data-confirm-title="Disable Self-Registration" data-confirm-ok="Disable"> <i class="bi bi-toggle-on me-1"></i>Disable </button> {% else %} @@ -461,13 +463,14 @@ function syncColorPicker(val) { } function removeLogo() { - if (!confirm('Remove the current logo?')) return; - document.getElementById('remove-logo-input').value = '1'; - const initials = document.getElementById('logo-initials').value.substring(0, 2).toUpperCase(); - document.getElementById('logo-preview-wrap').innerHTML = - `<span id="logo-preview-initials" style="font-family:'Space Mono',monospace;font-weight:700;font-size:18px;color:#fff;">${initials}</span>`; - const row = document.getElementById('remove-logo-row'); - if (row) row.style.display = 'none'; + confirmModal('Remove Logo', 'Remove the current logo? The initials fallback will be displayed instead.', 'Remove', 'btn-danger', () => { + document.getElementById('remove-logo-input').value = '1'; + const initials = document.getElementById('logo-initials').value.substring(0, 2).toUpperCase(); + document.getElementById('logo-preview-wrap').innerHTML = + `<span id="logo-preview-initials" style="font-family:'Space Mono',monospace;font-weight:700;font-size:18px;color:#fff;">${initials}</span>`; + const row = document.getElementById('remove-logo-row'); + if (row) row.style.display = 'none'; + }); } </script> {% endblock %} diff --git a/app/templates/admin/ticket_templates.html b/app/templates/admin/ticket_templates.html index 93bce0e..23e2c17 100644 --- a/app/templates/admin/ticket_templates.html +++ b/app/templates/admin/ticket_templates.html @@ -55,7 +55,8 @@ <a href="{{ url_for('admin.ticket_template_edit', tmpl_id=t.id) }}" class="btn btn-secondary btn-sm" title="Edit"><i class="bi bi-pencil"></i></a> <form method="POST" action="{{ url_for('admin.ticket_template_delete', tmpl_id=t.id) }}" - onsubmit="return confirm('Delete template "{{ t.name }}"?');" style="margin:0;"> + data-confirm="Delete template {{ t.name }}? This cannot be undone." + data-confirm-title="Delete Template" data-confirm-ok="Delete" style="margin:0;"> <input type="hidden" name="csrf_token" value="{{ csrf_token() }}"/> <button type="submit" class="btn btn-sm" style="background:rgba(248,113,113,.1);border:1px solid rgba(248,113,113,.2);color:var(--danger);" diff --git a/app/templates/admin/users.html b/app/templates/admin/users.html index 5c47a9b..3f73f53 100644 --- a/app/templates/admin/users.html +++ b/app/templates/admin/users.html @@ -60,7 +60,8 @@ </a> {% if u.id != current_user.id and u.is_active %} <form method="POST" action="{{ url_for('admin.delete_user', user_id=u.id) }}" - onsubmit="return confirm('Deactivate {{ u.full_name }}?');"> + data-confirm="Deactivate {{ u.full_name }}? They will no longer be able to log in." + data-confirm-title="Deactivate User" data-confirm-ok="Deactivate" data-confirm-class="btn-warning"> <input type="hidden" name="csrf_token" value="{{ csrf_token() }}"/> <button type="submit" class="btn btn-sm" style="background:rgba(248,113,113,.1);border:1px solid rgba(248,113,113,.2);color:var(--danger);"> <i class="bi bi-person-dash"></i> </button> diff --git a/app/templates/base.html b/app/templates/base.html index 2af853e..27c0f98 100644 --- a/app/templates/base.html +++ b/app/templates/base.html @@ -8,21 +8,8 @@ <title id="page-title-text">{% block title %}IT Helpdesk{% endblock %} — {{ branding.app_name }} @@ -1270,6 +1257,25 @@ {% endif %} + + +