diff --git a/.gitignore b/.gitignore index 466ae7f..60d53c9 100644 --- a/.gitignore +++ b/.gitignore @@ -207,5 +207,3 @@ cython_debug/ marimo/_static/ marimo/_lsp/ __marimo__/ -README.md -Claude.md \ No newline at end of file diff --git a/Claude.md b/Claude.md new file mode 100644 index 0000000..73ff703 --- /dev/null +++ b/Claude.md @@ -0,0 +1,949 @@ +# CLAUDE.md — QR Code Attendance Management System +# Unified reference — LT Services, Inc. & GOV Services, Inc. + +This file is the authoritative reference for Claude across all sessions. +Every session must treat this file as ground truth for architecture, conventions, +domain rules, and developer preferences. Read it fully before making any changes. + +--- + +## 1. Project Overview + +A Flask/Python web application for **employee attendance tracking via QR codes** with +GPS validation, Excel import/export, payroll calculations, role-based access control, +and per-QR photo verification toggle. + +**Deployed for two companies on separate servers:** +- **LT Services, Inc.** — `THEME_NAME` empty (blue theme). Production: `test.ltservicesinc.com`. Gitea: `gitea.ngodanguyen.tech/nngo/LT_QR_Codes_Management`. +- **GOV Services, Inc.** — `THEME_NAME=gov` (green theme). Production: `qr.govservicesinc.com`. Key `.env` differences: `COMPANY_NAME=GOV. Services, Inc`, `THEME_NAME=gov`, `QR_BASE_URL=https://qr.govservicesinc.com`. + +**Both codebases are structurally identical** in all Python files, routes, models, and templates. +They differ only in `.env` values and `static/css/theme-gov.css`. +**Every code fix must be applied to both instances.** + +- **Stack:** Flask 3.1, SQLAlchemy 2.0 / MySQL (PyMySQL only — `mysql-connector-python` removed), openpyxl 3.1, pandas 2.2, Python 3.12 +- **Frontend:** Jinja2 templates + vanilla JS + Font Awesome 6 icons +- **Auth guard:** Cloudflare Turnstile (optional, toggled via `.env`) +- **Production server:** Gunicorn + gevent workers, Nginx reverse proxy, Ubuntu Server + +--- + +## 2. Folder Structure + +``` +QR_Code_Management/ +│ +├── app.py # Application factory (create_app) + startup entry point +├── config.py # All env-var reads — single source of truth +├── extensions.py # Shared singletons: db (SQLAlchemy) + logger_handler (AppLogger) +├── logger_handler.py # AppLogger class + log_user_activity / log_database_operations decorators +├── location_logging.py # Android GPS debug routes (/api/log-location-action, /api/location-debug-info) +├── turnstile_utils.py # Cloudflare Turnstile verification helper (TurnstileUtils class) +├── advanced_security_middleware.py # SecurityManager (CSRF token gen, rate limiting, suspicious IP tracking) +│ # Wired into create_app() — do not import separately in blueprints +├── working_hours_calculator.py # CANONICAL calculator for Excel exports (WorkingHoursCalculator) +├── single_checkin_calculator.py # Legacy calculator — NOT used for exports; never switch to this +├── time_attendance_import_service.py # Excel import pipeline with duplicate detection +├── qr_code_import_service.py # Bulk QR code import from Excel +├── address_normalization_fix.py # normalize_address() + addresses_are_similar() helpers +├── app_performance_middleware.py # PerformanceMonitor — dev-mode only; lazy-imported inside __main__ +├── db_audit_tables.py # DB audit table helpers (standalone operational script) +├── db_health_check.py # DB connectivity health check (standalone) +├── db_maintenance.py # DB maintenance utilities (standalone) +├── db_performance_optimization.py # Index/query optimization helpers +├── employee_data_merger.py # Merge duplicate employee records (standalone) +├── employee_duplicate_remove.py # Remove employee duplicates (standalone) +├── employee_sync_scheduler.py # Scheduled employee sync (schedule library) +├── employee_table_sync.py # Employee table sync logic +├── requirements.txt # All pinned Python dependencies (single MySQL driver: PyMySQL) +│ +├── models/ +│ ├── __init__.py # set_db(db) → unpacks and returns all model classes +│ ├── base.py # Shared db reference (base.db); all models import from here +│ ├── user.py # User model (table: users) +│ ├── employee.py # Employee model (table: employee) +│ ├── attendance.py # AttendanceData model (table: attendance_data) +│ ├── time_attendance.py # TimeAttendance model (table: time_attendance) +│ ├── qrcode.py # QRCode, QRCodeStyle, QRCodeLocation models +│ ├── project.py # Project model (table: projects) +│ └── permissions.py # UserProjectPermission, UserLocationPermission models +│ +├── routes/ +│ ├── __init__.py # Empty (intentional) +│ ├── auth.py # Blueprint 'auth': /, /register (admin-only), /login, /logout, /profile +│ ├── dashboard.py # Blueprint 'dashboard': /dashboard, project QR views, stats APIs +│ ├── users.py # Blueprint 'users': /users/*, user management APIs +│ ├── admin.py # Blueprint 'admin': /admin/logs, /api/logs/* +│ ├── projects.py # Blueprint 'projects': /projects/* +│ ├── qr_codes.py # Blueprint 'qr_codes': /qr-codes/*, /qr/ check-in flow +│ ├── attendance.py # Blueprint 'attendance': /attendance report + API endpoints +│ ├── attendance_edit.py # Side-effect module: edit, add manual, save, delete routes +│ │ # Imports bp FROM attendance.py — does NOT define its own blueprint +│ ├── verification.py # Side-effect module: /verification-review/* routes +│ │ # Imports bp FROM attendance.py — does NOT define its own blueprint +│ ├── attendance_export.py # Side-effect module: /export-configuration, /generate-excel-export +│ │ # Imports bp FROM attendance.py — does NOT define its own blueprint +│ ├── statistics.py # Blueprint 'statistics': /statistics +│ ├── employees.py # Blueprint 'employees': /employees/* +│ ├── time_attendance.py # Blueprint 'time_attendance': /time-attendance/* (all TA routes) +│ └── time_attendance_export.py # Plain module (NO Blueprint): export helper functions only +│ +├── utils/ +│ ├── __init__.py +│ ├── helpers.py # Role decorators, QR generation, role/permission helpers +│ ├── geocoding.py # Haversine distance, Google Maps client, reverse geocode +│ └── template_helpers.py # Context processors: get_employee_name, format_hours, etc. +│ +├── static/ +│ ├── css/ +│ │ ├── style.css # MASTER stylesheet — all CSS variables and layout +│ │ ├── theme-gov.css # GOV Services brand theme overrides (loaded when THEME_NAME=gov) +│ │ └── *.css # Page-specific stylesheets +│ └── js/ +│ ├── script.js # Global JS — includes CSRF fetch wrapper +│ ├── qr_destination.js # Check-in page: GPS, bilingual UI, camera — CSRF-EXEMPT +│ ├── attendance_report.js # Attendance report: pagination, sorting, charts +│ ├── dashboard.js +│ ├── export_configuration.js +│ ├── users.js +│ ├── attendance_fullscreen.js +│ └── android_location_handler.js # Android-specific GPS workaround +│ +├── templates/ +│ ├── base.html # Base for unauthenticated pages — loads theme CSS conditionally +│ ├── base_authenticated.html # Base for protected pages — loads theme CSS conditionally +│ │ # Brand text uses {{ COMPANY_NAME }} — no hardcoded strings +│ ├── [all other page templates] # All POST forms include {{ csrf_token() }} hidden field +│ └── errors/ +│ ├── 403.html / 404.html / 500.html +│ +└── tools/ + ├── migration_photo_verification_toggle.py # Adds photo_verification_enabled to qr_codes + ├── migration_PM_permissions.py # One-time: user_project/location_permissions tables + ├── migration_dynamic_qr_locations.py # One-time: qr_type column + qr_code_locations table + └── optimize_time_attendance_db.py # Standalone DB index optimization script +``` + +--- + +## 3. Database Tables + +| Table | Model | Purpose | +|---|---|---| +| `users` | `User` | System user accounts with RBAC | +| `employee` | `Employee` | Employee master data (firstName, lastName, title, contractId) | +| `attendance_data` | `AttendanceData` | QR check-in records with GPS, photo verification | +| `time_attendance` | `TimeAttendance` | Imported time-clock records from Excel | +| `qr_codes` | `QRCode` | QR code definitions (standard and dynamic types) | +| `qr_code_styles` | `QRCodeStyle` | Reusable QR code visual styles | +| `qr_code_locations` | `QRCodeLocation` | Selectable locations for dynamic QR codes | +| `projects` | `Project` | Projects that group QR codes and employees | +| `user_project_permissions` | `UserProjectPermission` | Project-level access for Project Managers | +| `user_location_permissions` | `UserLocationPermission` | Location-level access for Project Managers | +| `log_events` | (raw SQL) | Application event log (created by AppLogger) | + +### `qr_codes` — key columns + +| Column | Type | Notes | +|---|---|---| +| `qr_type` | VARCHAR(20) | `'standard'` or `'dynamic'` | +| `photo_verification_enabled` | TINYINT(1) DEFAULT 1 | Per-QR photo verification toggle (added May 2026) | +| `active_status` | BOOLEAN | | +| `project_id` | FK → projects | | + +### `attendance_data` — key columns + +| Column | Type | Notes | +|---|---|---| +| `employee_id` | VARCHAR(50) | Always stored uppercased | +| `location_name` | VARCHAR(100) | Resolved location — never stores `'Dynamic'` | +| `location_accuracy` | FLOAT | Haversine distance in miles | +| `is_dynamic_qr` | BOOLEAN | True when checked in via dynamic QR | +| `verification_photo` | TEXT | Base64 encoded image | +| `verification_required` | BOOLEAN | | +| `verification_status` | VARCHAR(20) | `pending` / `approved` / `rejected` | + +**Key relationships:** +- `Employee.contractId` → `Project.id` +- `QRCode.project_id` → `Project.id` +- `AttendanceData.qr_code_id` → `QRCode.id` (CASCADE DELETE) +- `TimeAttendance.project_id` → `Project.id` + +--- + +## 4. User Roles & Access Control + +```python +VALID_ROLES = ['admin', 'staff', 'payroll', 'project_manager', 'accounting'] +STAFF_LEVEL_ROLES = ['staff', 'payroll', 'project_manager', 'accounting'] +``` + +| Role | Key Access | +|---|---| +| `admin` | Full access; only role that sees System Logs, Users, all export tools | +| `staff` | Create/edit QR codes; view dashboard and reports; no delete, no admin | +| `payroll` | Same as staff + Time Attendance section | +| `accounting` | Same as payroll (identical menu items) | +| `project_manager` | Reports only; scoped to assigned projects and locations via permission tables | + +**Auth decorators** (in `utils/helpers.py`): +- `@login_required` — redirects to `/login` if no session +- `@admin_required` — 403 if role is not `admin` +- `@staff_or_admin_required` — 403 if not admin or staff-level + +**`/register` is restricted to `@admin_required`** — public self-registration is disabled. + +--- + +## 5. Application Factory & Initialization Order + +``` +1. load_dotenv() +2. from extensions import db, init_logger +3. create_app(): + a. app.config.from_object(get_config()) + b. SECRET_KEY guard — sys.exit(1) if default value in non-debug mode + c. db.init_app(app) + d. set_db(db) → unpacks all model classes + e. init_logger(app, db) → binds logger_handler in extensions.py + f. register_blueprints() — in canonical order (see §6) + + side-effect import of attendance_edit, verification, attendance_export + g. create_location_logging_routes() ← from location_logging.py + h. SecurityManager.init_app() ← wires CSRF before_request + rate limiting + i. inject_csrf_token() context processor + j. inject_company_name() context processor (COMPANY_NAME, THEME_NAME, CURRENT_YEAR) + k. inject_logging_status(), inject_turnstile() context processors + l. template_filters (strftime, days_since, time_ago) + m. before_request: adjust_session_lifetime + g.start_time + suspicious UA scan + n. after_request: slow-query detection + error response logging + o. Error handlers: 403, 404, 500 + p. Startup init: create_tables() + update_existing_qr_codes() ← runs under Gunicorn too +4. if __name__ == '__main__': + a. lazy import PerformanceMonitor (dev-mode only) + b. initialize_performance_optimizations() + c. app.run() +``` + +**Critical notes:** +- `create_tables()` and `update_existing_qr_codes()` run inside `with app.app_context()` inside `create_app()` — they execute under gunicorn, not only under `__main__`. +- `PerformanceMonitor` is imported lazily inside `__main__` only — gunicorn workers never load it. +- `update_existing_qr_codes()` prefers `QR_BASE_URL` from `.env`. Falls back to `FLASK_HOST`/`FLASK_PORT`. Never uses `request.url_root`. + +--- + +## 6. Blueprint Registration Order + +```python +auth_bp, dashboard_bp, users_bp, admin_bp, projects_bp, +qr_codes_bp, attendance_bp, statistics_bp, +employees_bp, time_attendance_bp +``` + +**Attendance blueprint split:** +`attendance.py` defines `bp = Blueprint('attendance', __name__)` once. +`attendance_edit.py`, `verification.py`, and `attendance_export.py` each do: +```python +from routes.attendance import bp # shared blueprint — do not redefine +``` +`app.py` registers only `attendance_bp` once. Sub-modules are loaded as side-effect imports: +```python +import routes.attendance_edit # noqa: F401 +import routes.verification # noqa: F401 +import routes.attendance_export # noqa: F401 +``` + +`time_attendance_export.py` is **not a Blueprint**. Plain module providing export helper functions. Never register it separately. + +--- + +## 7. All Routes + +### auth (Blueprint: `auth`) +| URL | Endpoint | Notes | +|---|---|---| +| `/` | `auth.index` | | +| `/register` | `auth.register` | `@admin_required` — not public | +| `/login` | `auth.login` | Rate-limited via SecurityManager | +| `/logout` | `auth.logout` | | +| `/profile` | `auth.profile` | | + +### dashboard (Blueprint: `dashboard`) +| URL | Endpoint | +|---|---| +| `/dashboard` | `dashboard.dashboard` | +| `/project//qr-codes` | `dashboard.project_qr_codes` | +| `/dashboard/search` | `dashboard.search_qr_codes` | +| `/api/dashboard/stats` | `dashboard.dashboard_stats_api` | +| `/api/dashboard/realtime` | `dashboard.dashboard_realtime_api` | + +### users (Blueprint: `users`) +| URL | Endpoint | +|---|---| +| `/users` | `users.users` | +| `/users/create` | `users.create_user` | +| `/users//edit` | `users.edit_user` | +| `/users//delete` | `users.delete_user` | +| `/users//reactivate` | `users.reactivate_user` | +| `/users//promote` | `users.promote_user` | +| `/users//demote` | `users.demote_user` | +| `/users//toggle-status` | `users.toggle_user_status` | +| `/users//activate` | `users.activate_user` | +| `/users//deactivate` | `users.deactivate_user` | +| `/users//permanently-delete` | `users.permanently_delete_user` | +| `/api/users/stats` | `users.user_stats_api` | +| `/api/locations-by-projects` | `users.get_locations_by_projects` | +| `/api/roles/permissions` | `users.role_permissions_api` | +| `/api/geocode` | `users.geocode_address_api` | +| `/api/reverse-geocode` | `users.reverse_geocode_api` | + +### admin (Blueprint: `admin`) +| URL | Endpoint | +|---|---| +| `/admin/logs` | `admin.admin_logs` | +| `/admin/health/google-maps` | `admin.google_maps_health` | +| `/api/logs/recent` | `admin.api_recent_logs` | +| `/api/logs/stats` | `admin.api_log_stats` | +| `/api/logs/cleanup` | `admin.api_cleanup_logs` | +| `/api/logs/clear` | `admin.api_clear_logs` | +| `/api/logs/clear-old` | `admin.api_clear_old_logs` | +| `/api/logs/export` | `admin.api_export_logs` | + +### projects (Blueprint: `projects`) +| URL | Endpoint | +|---|---| +| `/projects` | `projects.projects` | +| `/projects/create` | `projects.create_project` | +| `/projects//edit` | `projects.edit_project` | +| `/projects//toggle` | `projects.toggle_project` | +| `/api/projects/active` | `projects.api_active_projects` | + +### qr_codes (Blueprint: `qr_codes`) +| URL | Endpoint | Notes | +|---|---|---| +| `/qr-codes/create` | `qr_codes.create_qr_code` | | +| `/qr-codes/bulk-import` | `qr_codes.import_bulk_qr_codes` | | +| `/qr-codes/bulk-import/template` | `qr_codes.download_qr_import_template` | | +| `/qr-codes//edit` | `qr_codes.edit_qr_code` | | +| `/qr-codes//delete` | `qr_codes.delete_qr_code` | | +| `/qr-codes//toggle-status` | `qr_codes.toggle_qr_status` | | +| `/qr-codes//activate` | `qr_codes.activate_qr_code` | | +| `/qr-codes//deactivate` | `qr_codes.deactivate_qr_code` | | +| `/qr-codes//copy-url` | `qr_codes.copy_qr_url` | | +| `/qr-codes//open-link` | `qr_codes.open_qr_link` | | +| `/qr/` | `qr_codes.qr_destination` | | +| `/qr//checkin` | `qr_codes.qr_checkin` | **CSRF-exempt** — public unauthenticated | +| `/qr//locations` | `qr_codes.qr_get_locations` | | + +### attendance (Blueprint: `attendance` — split across 4 files) +| URL | Endpoint | File | +|---|---|---| +| `/attendance` | `attendance.attendance_report` | `attendance.py` | +| `/api/attendance/locations` | `attendance.attendance_locations_api` | `attendance.py` | +| `/api/attendance/stats` | `attendance.attendance_stats_api` | `attendance.py` | +| `/api/search_employees` | `attendance.search_employees_api` | `attendance.py` | +| `/api/get_project_locations` | `attendance.get_project_locations_api` | `attendance.py` | +| `/api/time-attendance/locations` | `attendance.time_attendance_locations_api` | `attendance.py` | +| `/attendance//edit` | `attendance.edit_attendance` | `attendance_edit.py` | +| `/attendance//delete` | `attendance.delete_attendance` | `attendance_edit.py` | +| `/attendance/add` | `attendance.add_manual_attendance` | `attendance_edit.py` | +| `/attendance/save_manual` | `attendance.save_manual_attendance` | `attendance_edit.py` | +| `/verification-review` | `attendance.verification_review` | `verification.py` | +| `/verification-review/` | `attendance.verification_review_detail` | `verification.py` | +| `/verification-review//update` | `attendance.update_verification_status` | `verification.py` | +| `/api/attendance//verification-details` | `attendance.get_verification_details` | `verification.py` | +| `/export-configuration` | `attendance.export_configuration` | `attendance_export.py` | +| `/generate-excel-export` | `attendance.generate_excel_export` | `attendance_export.py` | + +### statistics (Blueprint: `statistics`) +| URL | Endpoint | +|---|---| +| `/statistics` | `statistics.qr_statistics` | +| `/api/statistics/export` | `statistics.export_statistics` | + +### employees (Blueprint: `employees`) +| URL | Endpoint | +|---|---| +| `/employees` | `employees.employees` | +| `/employees/create` | `employees.create_employee` | +| `/employees//edit` | `employees.edit_employee` | +| `/employees//delete` | `employees.delete_employee` | +| `/employees/` | `employees.employee_detail` | +| `/api/employees/search` | `employees.api_employees_search` | + +### time_attendance (Blueprint: `time_attendance`) +| URL | Endpoint | +|---|---| +| `/time-attendance` | `time_attendance.time_attendance_dashboard` | +| `/time-attendance/import` | `time_attendance.import_time_attendance` | +| `/time-attendance/import/analyze-duplicates` | `time_attendance.analyze_import_duplicates` | +| `/time-attendance/import/analyze-invalid` | `time_attendance.analyze_import_invalid` | +| `/time-attendance/import/start` | `time_attendance.start_import_job` | +| `/time-attendance/import/stream/` | `time_attendance.stream_import_progress` | +| `/time-attendance/import/cancel-pending` | `time_attendance.cancel_pending_import` | +| `/time-attendance/import/validate` | `time_attendance.validate_import_file` | +| `/time-attendance/import/batch/` | `time_attendance.view_import_batch` | +| `/time-attendance/import/batch//delete` | `time_attendance.delete_import_batch` | +| `/time-attendance/import/download-template` | `time_attendance.download_import_template` | +| `/time-attendance/export` | `time_attendance.export_time_attendance` | +| `/time-attendance/export/excel` | `time_attendance.excel_export_time_attendance` | +| `/time-attendance/export-by-building` | `time_attendance.export_time_attendance_by_building` | +| `/time-attendance/records` | `time_attendance.time_attendance_records` | +| `/time-attendance/record/` | `time_attendance.time_attendance_record_detail` | +| `/time-attendance/delete/` | `time_attendance.delete_time_attendance_record` | +| `/api/time-attendance/employee/` | `time_attendance.api_time_attendance_by_employee` | +| `/api/time-attendance/location/` | `time_attendance.api_time_attendance_by_location` | + +### Location logging (registered directly on `app`, not a Blueprint) +| URL | Purpose | +|---|---| +| `/api/log-location-action` | Android GPS debug logging | +| `/api/location-debug-info` | Return GPS debug info | + +### Security API (registered by SecurityManager on `app`) +| URL | Purpose | +|---|---| +| `/api/security/status` | Admin-only security dashboard stats | +| `/api/security/clear-blocks` | Admin-only: clear rate-limit blocks | + +--- + +## 8. Template System + +### Two Base Templates +- **`base.html`** — unauthenticated pages (login, register, QR scan, errors). Body class: `login-layout`. +- **`base_authenticated.html`** — all protected pages. Body class: `has-sidebar`. Fixed collapsible sidebar, top header with user/role badge, flash message rendering. + +### Template Block Names +| Block | Purpose | +|---|---| +| `{% block title %}` | Page `` text only — **no CSS or JS here** | +| `{% block page_title %}` | Top header `<h1>` (authenticated only) | +| `{% block extra_head %}` | Page-specific CSS — inside `<head>` | +| `{% block content %}` | Main page body | +| `{% block extra_scripts %}` | Page-specific JS — before `</body>` | + +**Rule:** CSS always goes in `extra_head`. JS always goes in `extra_scripts`. **Never inject either into `{% block title %}`.** + +### Theme System + +Both `base.html` and `base_authenticated.html` load the theme override after `style.css`: +```html +<!-- Theme Override (set THEME_NAME in .env to activate, e.g. THEME_NAME=gov) --> +{% if THEME_NAME %} +<link rel="stylesheet" href="{{ url_for('static', filename='css/theme-' + THEME_NAME + '.css') }}" /> +{% endif %} +``` +Loaded **before** `{% block extra_head %}` so page-specific CSS loads after and can override. + +**`theme-gov.css` overrides:** `--primary-color: #16a34a` / `--primary-hover: #15803d` (institutional green). No CSS text hacks — brand names handled via `{{ COMPANY_NAME }}`. + +**Adding a third company theme:** Create `static/css/theme-{name}.css` and set `THEME_NAME={name}` in `.env`. No code changes required. + +### Global Context Variables +| Variable | Source | +|---|---| +| `COMPANY_NAME` | `.env` (`COMPANY_NAME`) | +| `THEME_NAME` | `.env` (`THEME_NAME`) — empty string when not set | +| `CURRENT_YEAR` | `datetime.now().year` | +| `csrf_token` | `generate_csrf_token()` from `advanced_security_middleware` | +| `is_admin` | `bool` from session role | +| `turnstile_enabled`, `turnstile_site_key` | Turnstile config | + +### CSRF in Templates +Every POST form: +```html +<input type="hidden" name="csrf_token" value="{{ csrf_token() }}"> +``` +Every AJAX POST: +```js +'X-CSRF-Token': (window.qrConfig && window.qrConfig.csrfToken) || '' +``` +Exception: `/qr/<url>/checkin` is CSRF-exempt (public, unauthenticated). + +--- + +## 9. Design System + +### CSS Custom Properties (defined in `style.css`) +```css +--primary-color: #2563eb /* Blue — LT default */ +--primary-hover: #1d4ed8 +--secondary-color: #64748b +--success-color: #10b981 +--warning-color: #f59e0b +--danger-color: #ef4444 +--info-color: #0891b2 +--gray-50: #f8fafc /* Page background */ +--gray-700: #334155 /* Body text */ +--gray-900: #0f172a /* Headings */ +``` + +**Light mode only** — `color-scheme: light only !important` enforced globally. + +### Layout +- Sidebar: `280px` expanded, `64px` collapsed. Gradient `#2563eb → #1d4ed8` (LT). GOV theme: `#16a34a → #15803d`. +- Header height: `64px`. +- Z-index: dropdown 1000, modal 1050, sidebar 1100, overlay 1200. + +### Excel Export Styling +- **Header rows:** white bold text on solid black fill (`000000`) +- **Miss-punch / amber:** `FFC000` fill +- **Font:** Aptos Narrow 11pt everywhere; 14pt report title, 12pt section summary + +--- + +## 10. Security Architecture + +### CSRF Protection +- `SecurityManager` from `advanced_security_middleware.py` wired via `init_app()` in `create_app()` +- `before_request` validates `csrf_token` form field or `X-CSRF-Token` header on every `POST/PUT/PATCH/DELETE` +- Token stored in `session['csrf_token']`; compared with `hmac.compare_digest()` +- **CSRF-exempt:** `auth.login`, `auth.register`, `qr_codes.qr_checkin`, `static` + +### Rate Limiting +- 5 failed login attempts within 15 minutes → IP blocked 15 minutes (in-memory per worker) +- `SecurityManager.create_secure_session()` called on login success (clears failed-attempt counter) + +### Session Security +- `session.clear()` before setting new keys on login (prevents session fixation) +- `before_request` hook `adjust_session_lifetime()`: `remember_me` → 30 days, default → 10 hours +- `SESSION_COOKIE_SECURE=True` requires HTTPS — HTTP-only deployments must set `false` or login loops + +### validate_session_security() — DO NOT USE in before_request +In-memory dict per worker — breaks under multi-worker gunicorn (login worker A, next request hits worker B with empty dict → 401). Intentionally not called. Flask signed cookie + CSRF handles integrity. + +### SQL Injection Prevention +All dynamic SQL uses SQLAlchemy parameterized queries: +```python +conditions, params = [], {} +conditions.append("ad.check_in_date >= :date_from") +params["date_from"] = date_from +db.session.execute(text("... WHERE 1=1 " + filter_clause), params) +``` + +--- + +## 11. QR Code System + +### QR Types +- **`standard`** (default) — fixed single location +- **`dynamic`** — employee selects location at scan time from list auto-generated from all active standard QR codes. No manual management UI — always queried live via `SELECT DISTINCT location, location_address FROM qr_codes WHERE qr_type='standard'`. + +### Photo Verification — Two-Layer Toggle +Both must be `True` for photo verification to trigger: +1. **Global:** `PHOTO_VERIFICATION_ENABLED` in `.env` +2. **Per-QR:** `qr_codes.photo_verification_enabled` (TINYINT(1) DEFAULT 1) + +```python +qr_photo_verification = getattr(qr_code, 'photo_verification_enabled', True) +if qr_photo_verification and current_app.config.get('PHOTO_VERIFICATION_ENABLED', True) \ + and location_accuracy > threshold: + # require photo +``` + +Toggle UI in `create_qr_code.html` and `edit_qr_code.html` — uses `addEventListener('change', ...)` in `extra_scripts` block. + +### Check-In Flow +1. Employee scans QR → `qr_destination.html` +2. Enters ID; GPS captured by browser +3. 30-min interval guard (configurable via `TIME_INTERVAL`) +4. Dynamic QR: server rejects if `selected_location_name` is empty; `location_name` in record is always the resolved name, never `'Dynamic'` +5. Haversine distance calculated; photo required if beyond threshold (and both toggles enabled) +6. Server-side photo size check: rejects > `VERIFICATION_PHOTO_MAX_SIZE` with HTTP 413 +7. Record saved to `attendance_data` + +### Check-In Page Features +- Bilingual: English/Vietnamese toggle (`qr_destination.js`) +- Staff ID persistence: `localStorage` remembers last employee ID +- Android GPS: special handler (`android_location_handler.js`) +- QR URLs must be full `https://` absolute URLs — relative URLs parsed as search queries by phones +- Base URL constructed from `QR_BASE_URL` env var — never from `request.url_root` + +--- + +## 12. Time Attendance Import Pipeline + +### Excel File Requirements +- **Required columns:** `ID`, `Date`, `Time`, `Location Name`, `Action Description` +- **Optional columns:** `Name`, `Platform`, `Event Description`, `Recorded Address`, `Distance` + +### Import Flow +1. Upload → `validate_import_file` +2. Duplicate analysis → `analyze_import_duplicates` → user reviews +3. Invalid record analysis → `analyze_import_invalid` +4. Start job → `start_import_job` (SSE streaming progress via `stream_import_progress`) +5. Result in `time_attendance_import_result.html` + +### Special Handling +- `Recorded Address`: read via openpyxl directly (not pandas) to preserve HYPERLINK formulas +- Duplicate detection: hash of `employee_id + date + time + action_description` +- Import tracked by `import_batch_id` (UUID) +- `self.db` not `db` — in `TimeAttendanceImportService`, always access SQLAlchemy via `self.db` +- Validation `except` blocks must not silently swallow exceptions (fail-open prevention) + +--- + +## 13. Time Attendance Excel Export + +### Calculator — CRITICAL +**Always use `WorkingHoursCalculator`** from `working_hours_calculator.py`. +**Never switch to `SingleCheckInCalculator`** — legacy, causes silent calculation errors. + +### Rounding Rules +- **Daily / Weekly / Grand Total + SP/PW/PT summary columns:** `_qtr()` — quarter-hour rounding +- **Individual Hours/Building entry cells:** raw `round(..., 2)` — no quarter rounding +- `_qtr()` pipeline: decimal hours → minutes → `round_time_to_quarter_hour()` → `convert_minutes_to_base100()` → `round_base100_hours()` + +### Work Type Codes +- **SP** = Special Project, **PW** = Periodic Work, **PT** = Project Team (Part-Time) +- Parsed from `employee_id` via `parse_employee_id_for_work_type()` + +### Overnight Shift Handling +**Rule 1 — Sort key:** early-morning OUTs (`hour <= 3`) use `_overnight_aware_sort_key()` which adds 86400 seconds — pushes them past midnight so they sort after same-day evening INs. + +**Rule 2 — Detection threshold:** `hour >= 12` (noon). Any IN at or after 12:00 PM is an overnight IN candidate if no matching OUT exists same day AND early-morning OUT (`hour <= 3`) exists next calendar day. + +**Orphan guard:** early-morning OUTs with `check_in_date <= current_day` are orphans — skip. Only OUTs with `check_in_date > current_day` (moved by overnight detection) pair with evening INs. + +### Cross-Type SP Pairs +`is_cross_type = True` when SP IN pairs with non-SP OUT (or vice versa). Accumulates separately in `cross_type_sp/pw/pt_hours` to prevent double-counting in summary rows. + +### Summary Rows (SP / PW / PT + Regular) +``` +col8='SP' col9=_qtr(sp_hours) ← only if sp_hours > 0 +col8='PW' col9=_qtr(pw_hours) ← only if pw_hours > 0 +col8='PT' col9=_qtr(pt_hours) ← only if pt_hours > 0 +col8='Regular' col9=_qtr(regular_only_hours) ← always when any special type exists +col7='GRAND TOTAL:' col9=_qtr(grand_regular) col10=_qtr(grand_ot) +``` + +### Export Date Range +`_resolve_date_range()` enforces a **14-day cap by default**. Both export functions accept `unlimited=False`; pass `unlimited=True` to bypass. + +**UI:** "Unlimited" checkbox in `time_attendance_records.html` before export buttons. + +### Employee Name Format +`"Lastname, Firstname"` — `f"{emp.lastName}, {emp.firstName}"` + +### Export Date Iteration +Cap `sorted_dates` to `<= end_date` — prevents overnight buffer day from rendering as a display row. + +### Cross-Building Pairing +Same-day IN at Building A + OUT at Building B → pair and label `"IN: Building A → OUT: Building B"`. Suppress spurious Missed Punch rows. + +--- + +## 14. Logging System (`logger_handler.py`) + +Single `AppLogger` instance in `extensions.py`. Import everywhere as: +```python +from extensions import logger_handler +logger_handler.logger.info("...") +logger_handler.logger.error("...", exc_info=True) # always pass exc_info=True in except blocks +``` + +### Log Destinations +- `logs/application.log` — rotating 10MB/5 backups +- `logs/errors.log` — rotating 5MB/10 backups +- `logs/security.log` — rotating 2MB/20 backups +- `log_events` DB table — admin dashboard at `/admin/logs` + +### Log Every Action +All create, edit, and delete operations must include a log entry: +```python +logger_handler.logger.info(f"User {session['username']} created employee {new_employee.id}") +``` + +### traceback Convention +- Use `exc_info=True` on `logger.error()` — never `import traceback` inline +- `traceback.format_exc()` only acceptable when passing `stack_trace=` to `log_flask_error()` + +--- + +## 15. Configuration (`config.py` + `.env`) + +All env-var reads centralized in `config.py`. Blueprints use `current_app.config['KEY']`. + +### Key `.env` Variables +``` +DATABASE_URL # mysql+pymysql://user:pass@host/db — special chars in password OK +SECRET_KEY # MUST differ from default +COMPANY_NAME / CONTRACT_NAME +FLASK_HOST / FLASK_PORT / FLASK_ENV / DEBUG +SESSION_COOKIE_SECURE # MUST be 'false' for HTTP-only deployments +SESSION_COOKIE_HTTPONLY / SESSION_COOKIE_SAMESITE +TIME_INTERVAL # Check-in cooldown in minutes (default 30) +GOOGLE_MAPS_API_KEY # Optional; falls back to Haversine-only +TURNSTILE_ENABLED / TURNSTILE_SITE_KEY / TURNSTILE_SECRET_KEY +ENABLE_PHOTO_VERIFICATION # Global toggle (default 'true') +PHOTO_VERIFICATION_DISTANCE_THRESHOLD # Miles (default 0.3) +VERIFICATION_PHOTO_MAX_SIZE # Bytes (default 5MB) +UPLOAD_FOLDER # Temp path (default /tmp) +DEFAULT_ADMIN_PASSWORD # CHANGE IN PRODUCTION +QR_BASE_URL # Public-facing domain for QR links — required behind reverse proxy +THEME_NAME # Activates static/css/theme-{THEME_NAME}.css +SYNC_INTERVAL_MINUTES +SQLALCHEMY_ENGINE_OPTIONS_POOL_SIZE / POOL_TIMEOUT / POOL_RECYCLE / MAX_OVERFLOW +``` + +--- + +## 16. Distance Calculation + +Uses **Haversine formula** only (straight-line geodesic distance). Google Maps Distance Matrix API removed. + +`utils/geocoding.py` initializes `gmaps_client` for geocoding (address → coordinates) if `GOOGLE_MAPS_API_KEY` is set. + +### Google Maps Guard — ALWAYS USE +```python +from utils.geocoding import is_gmaps_available, gmaps_client +if is_gmaps_available(): + result = gmaps_client.geocode(address) +else: + # fall back to OpenStreetMap / Haversine +``` +Never call `gmaps_client.method()` without a `None` guard. + +### Address Normalization +Multi-strategy matching: core street extraction → component matching → fuzzy fallback. Handles geocoding drift for near-identical addresses. + +--- + +## 17. Migration Scripts + +All in `tools/`. Always use **`pymysql` directly** — never import the Flask app or SQLAlchemy ORM. +ORM loads models at import time; if target column doesn't exist yet, it crashes on startup. + +**Pattern:** +```python +from dotenv import load_dotenv +load_dotenv() +import pymysql, re + +def parse_db_url(url): + """Use regex — urlparse breaks on special chars (@, :) in passwords.""" + url = re.sub(r'^mysql\+pymysql://', '', url) + m = re.match( + r'^(?P<user>[^:]+):(?P<password>.+)@(?P<host>[^@:/]+)(?::(?P<port>\d+))?/(?P<db>[^?]+)', + url + ) + return {'host': m.group('host'), 'port': int(m.group('port') or 3306), + 'user': m.group('user'), 'password': m.group('password'), 'database': m.group('db')} + +def run(): + conn = pymysql.connect(**parse_db_url(os.environ['DATABASE_URL']), charset='utf8mb4', autocommit=False) + with conn.cursor() as cur: + cur.execute("SELECT COUNT(*) FROM information_schema.COLUMNS WHERE ...") + if cur.fetchone()[0] > 0: + print("[SKIP] already exists"); return + cur.execute("ALTER TABLE `table` ADD COLUMN `col` TINYINT(1) NOT NULL DEFAULT 1") + conn.commit() +``` + +### Completed Migrations +| Script | What it adds | +|---|---| +| `migration_photo_verification_toggle.py` | `qr_codes.photo_verification_enabled` TINYINT(1) DEFAULT 1 | +| `migration_dynamic_qr_locations.py` | `qr_type` column + `qr_code_locations` table | +| `migration_PM_permissions.py` | `user_project_permissions` + `user_location_permissions` tables | + +--- + +## 18. Coding Conventions & Developer Preferences + +### Change Philosophy +- **Minimal, additive changes only** — preserve all route names, function names, endpoint names, variable names, URL patterns +- New functionality added alongside existing code, not replacing it +- Never rename routes, functions, endpoints, or variables +- **Every code fix applies to both LT and GOV instances** + +### File Delivery +- **≤ 4 files changed** → present each file individually +- **≥ 5 files changed** → deliver as a single zip archive + +### File Editing Approach +- Pull latest code at start of each session +- CRLF normalization first if needed: `content.replace('\r\n', '\n')` +- Use surgical `str_replace` edits — never rewrite large blocks wholesale +- AST parse check after every Python edit: `python3 -c "import ast; ast.parse(open(f).read())"` +- Simulate logic before and after any calculator or pairing logic changes + +### SQLAlchemy Patterns +```python +# Correct (SQLAlchemy 2.0): +record = db.session.get(Model, record_id) +if record is None: + abort(404) + +# Deprecated — do not use: +record = Model.query.get_or_404(record_id) +record = Model.query.get(record_id) + +# Raw SQL requires text(): +db.session.execute(text("SELECT ..."), params) +# Never pass raw strings to conn.execute() — ObjectNotExecutableError in SQLAlchemy 2.0 +``` + +### JavaScript Patterns +- Use `createElement` + `addEventListener` — never inline `onchange`, `onclick`, etc. +- Wrap in `DOMContentLoaded` +- CSS → `{% block extra_head %}`, JS → `{% block extra_scripts %}`, **never into `{% block title %}`** +- All AJAX POST: `'X-CSRF-Token': (window.qrConfig && window.qrConfig.csrfToken) || ''` +- No `localStorage` / `sessionStorage` in artifacts + +### Error Handling Pattern +Every route `except` block must: +1. `db.session.rollback()` +2. `logger_handler.logger.error(f"...: {e}", exc_info=True)` +3. `flash(...)` a user-facing message +4. Return redirect or error response + +No bare `except:` — always `except Exception as e:`. + +### Employee Autocomplete (attendance and time-attendance) +- Visible text input + hidden `employee_id` field synced on numeric input +- Fetches `/api/search_employees` (includes unregistered IDs from `attendance_data`, not only Employee table) +- CSS inlined in `extra_head`; JS uses `createElement` + `addEventListener` + +### Timestamp Convention +Use `datetime.now()` (local time) throughout — **not** `datetime.utcnow()`. + +### Context Safety +- Use `has_request_context()` (not `if not request:`) to check Flask request context +- Capture `current_app._get_current_object()` in route body, not inside lazy generators + +--- + +## 19. Attendance Report + +Query fetches **1,001 rows**, trims to 1,000 if extra row present, sets `records_truncated = True`. +Template displays yellow banner when truncated: +``` +Showing the most recent 1,000 records. Narrow the date range or apply additional filters. +``` + +--- + +## 20. Known Bugs Fixed — Do Not Reintroduce + +### Set 1 — Critical +| File | Fix | +|---|---| +| `routes/time_attendance_export.py` | Removed stray second docstring | +| `app.py` | Error handlers wired to `render_template('errors/*.html')`; 403 added | +| `templates/errors/*.html` | `url_for('dashboard')` → `url_for('dashboard.dashboard')` | +| `routes/payroll.py` | Removed duplicate `get_employee_name`, `get_qr_code_checkin_count`, `@bp.context_processor` | +| `models/attendance.py` | `check_in_time` default: `datetime.now().time` → `lambda: datetime.now().time()` | +| `models/user.py` | Added `@staticmethod` to `has_export_permissions()` | + +### Set 2 — Moderate / Minor +| File | Fix | +|---|---| +| `app.py` | Merged duplicate before/after request hooks | +| `app.py` | `update_existing_qr_codes()` uses `QR_BASE_URL` first, not `request.url_root` | +| `config.py` | `SQLALCHEMY_ENGINE_OPTIONS` pool wiring | +| `utils/geocoding.py` | `is_gmaps_available()` helper | +| `working_hours_calculator.py` | CRLF → LF | + +### Set 3 — Structural +| File | Fix | +|---|---| +| `routes/time_attendance_export.py` | Removed 14 unused imports | +| `routes/dashboard.py`, `statistics.py`, `payroll.py` | `db.session.rollback()` in all except blocks | +| `models/employee.py` | `cls.id.like()` on BigInteger → `cast(cls.id, String).like()` | + +### Set 4 — Export Pipeline +| File | Fix | +|---|---| +| `routes/time_attendance_export.py` | Overnight IN threshold: `>= 19` → `>= 12` | +| `routes/time_attendance_export.py` | SP/PW/PT summary rows + `Regular` row | +| `routes/time_attendance_export.py` | `unlimited=False` parameter throughout | +| `templates/time_attendance_records.html` | "Unlimited" checkbox | + +### Set 5 — GOV Deployment +| File | Fix | +|---|---| +| `routes/qr_codes.py` | `_get_qr_base_url()` helper; replaced all `request.url_root` | +| `config.py` | `QR_BASE_URL` env var | +| nginx (GOV) | Removed duplicate `proxy_set_header Host` (doubled hostname in QR links) | + +### Set 6 — Theme System +| File | Fix | +|---|---| +| `static/css/theme-gov.css` | New file — GOV brand green overrides | +| `app.py` | `inject_company_name()` returns `THEME_NAME` and `CURRENT_YEAR` | +| `templates/base.html` + `base_authenticated.html` | Conditional theme CSS; `{{ COMPANY_NAME }}` for brand text; `{{ CURRENT_YEAR }}` in footer | + +### Set 7 — Security Hardening +| File | Fix | +|---|---| +| `routes/statistics.py`, `routes/payroll.py` | SQL injection: f-string filters → parameterized queries | +| `routes/auth.py` | Open redirect: `_is_safe_url()` on `next` param | +| `app.py` | `SECRET_KEY` default guard: `sys.exit(1)` | +| `routes/auth.py` | Session fixation: `session.clear()` before new keys | +| `routes/auth.py` | `@login_required` + `@admin_required` on `/register` | +| `app.py` | CSRF `SecurityManager` wired; `before_request` validator | +| All POST forms (22 templates) | `{{ csrf_token() }}` hidden field | +| All AJAX POST calls | `X-CSRF-Token` header | +| `advanced_security_middleware.py` | `request.json` guarded with content-type check (fixes 415) | +| `advanced_security_middleware.py` | Stable key from `SECRET_KEY` via SHA-256 (fixes per-worker warning) | +| `advanced_security_middleware.py` | `validate_session_security()` removed from `security_check()` (fixes 401 under multi-worker) | +| All route files | Bare `except:` → `except Exception` (11 locations) | +| `routes/qr_codes.py` | Server-side photo size enforcement; HTTP 413 on oversized payload | + +### Set 8 — Code Quality +| File | Fix | +|---|---| +| 7 route files | `Model.query.get_or_404()` → `db.session.get()` + `abort(404)` (21 call sites) | +| `config.py` | `PERMANENT_SESSION_LIFETIME` → `timedelta(hours=10)` | +| `routes/attendance.py` | Split into 4 files sharing one blueprint | +| `requirements.txt` | `mysql-connector-python` removed | +| `routes/attendance.py` | LIMIT 1001 + `records_truncated` flag + yellow banner | + +### Set 9 — Template Sync (May 2026) +| File | Fix | +|---|---| +| `templates/base.html` (GOV) | Footer hardcoded `2025 QR Code Management System` → `{{ CURRENT_YEAR }} {{ COMPANY_NAME }}` | +| `templates/base.html` (GOV) | Theme CSS block moved to after `style.css`, before Font Awesome (matches LT order) | +| `templates/base_authenticated.html` (GOV) | Indentation sync with LT | +| `templates/projects.html` (GOV) | Confirm dialog on project toggle + loading state on Edit buttons (sync with LT) | + +### Set 10 — Per-QR Photo Verification Toggle (May 2026) +| File | Change | +|---|---| +| `models/qrcode.py` | Added `photo_verification_enabled` TINYINT(1) DEFAULT 1 | +| `routes/qr_codes.py` | Create: reads toggle from form, passes to constructor + logs | +| `routes/qr_codes.py` | Edit: reads toggle from form, assigns to record + logs | +| `routes/qr_codes.py` | Checkin: checks per-QR AND global flag (both must be True) | +| `templates/create_qr_code.html` | Photo Verification toggle section added | +| `templates/edit_qr_code.html` | Photo Verification toggle section added (CSS in `extra_head`, JS in `extra_scripts` with `addEventListener`) | +| `tools/migration_photo_verification_toggle.py` | pymysql migration — adds column safely | + +--- + +## 21. Infrastructure & Deployment + +- **Deploy user:** `qrcode` on both servers +- **Restart:** `sudo supervisorctl restart qrcode` +- **Template-only changes:** no restart needed +- **Python/model changes:** always run migration first, then deploy files, then restart +- **Gunicorn:** gevent workers — avoid threading-unsafe patterns + +### Deploy Order for DB Column Additions +1. Run `python3 tools/migration_<name>.py` (pymysql, safe to re-run) +2. Deploy updated Python files +3. Restart service + +--- + +## 22. Session Changelog + +### Sessions 1–8 — Foundational Work +See §20 Sets 1–8 for detailed bug fix history. Covers: initial critical fixes, security hardening (CSRF, SQL injection, session fixation, rate limiting, open redirect), export pipeline enhancements, GOV deployment, theme system, code quality sprint. + +### Session (May 2026) — Template Sync + Photo Verification Toggle +- Diffed LT and GOV codebases — confirmed only 3 template files differed (whitespace + minor logic) +- Synced `base.html`, `base_authenticated.html`, `projects.html` — GOV brought to parity with LT +- Added per-QR `photo_verification_enabled` toggle (DB column + model + routes + UI + migration) +- Fixed migration script: replaced `urlparse` with regex parser (handles special chars in DB passwords), replaced SQLAlchemy `conn.execute(str)` with `pymysql` direct (avoids ORM loading model before column exists) +- Fixed `edit_qr_code.html` toggle bug: JS was injected into `{% block title %}` (corrupted by earlier injection) — moved CSS to `extra_head`, JS to `extra_scripts` using `addEventListener`, removed inline `onchange` attribute \ No newline at end of file