GOV_QR_Codes_Management/Claude.md

CLAUDE.md — QR Code Attendance Management System

Unified reference — LT Services, Inc. & GOV Services, Inc.

This file is the authoritative reference for Claude across all sessions. Every session must treat this file as ground truth for architecture, conventions, domain rules, and developer preferences. Read it fully before making any changes.

---

1. Project Overview

A Flask/Python web application for employee attendance tracking via QR codes with GPS validation, Excel import/export, payroll calculations, role-based access control, and per-QR photo verification toggle.

Deployed for two companies on separate servers:

• LT Services, Inc. — THEME_NAME empty (blue theme). Production: test.ltservicesinc.com. Gitea: gitea.ngodanguyen.tech/nngo/LT_QR_Codes_Management.
• GOV Services, Inc. — THEME_NAME=gov (green theme). Production: qr.govservicesinc.com. Key .env differences: COMPANY_NAME=GOV. Services, Inc, THEME_NAME=gov, QR_BASE_URL=https://qr.govservicesinc.com.

Both codebases are structurally identical in all Python files, routes, models, and templates. They differ only in .env values and static/css/theme-gov.css. Every code fix must be applied to both instances.

• Stack: Flask 3.1, SQLAlchemy 2.0 / MySQL (PyMySQL only — mysql-connector-python removed), openpyxl 3.1, pandas 2.2, Python 3.12
• Frontend: Jinja2 templates + vanilla JS + Font Awesome 6 icons
• Auth guard: Cloudflare Turnstile (optional, toggled via .env)
• Production server: Gunicorn + gevent workers, Nginx reverse proxy, Ubuntu Server

---

2. Folder Structure

QR_Code_Management/
│
├── app.py                              # Application factory (create_app) + startup entry point
├── config.py                           # All env-var reads — single source of truth
├── extensions.py                       # Shared singletons: db (SQLAlchemy) + logger_handler (AppLogger)
├── logger_handler.py                   # AppLogger class + log_user_activity / log_database_operations decorators
├── location_logging.py                 # Android GPS debug routes (/api/log-location-action, /api/location-debug-info)
├── turnstile_utils.py                  # Cloudflare Turnstile verification helper (TurnstileUtils class)
├── advanced_security_middleware.py     # SecurityManager (CSRF token gen, rate limiting, suspicious IP tracking)
│                                       # Wired into create_app() — do not import separately in blueprints
├── working_hours_calculator.py         # CANONICAL calculator for Excel exports (WorkingHoursCalculator)
├── single_checkin_calculator.py        # Legacy calculator — NOT used for exports; never switch to this
├── time_attendance_import_service.py   # Excel import pipeline with duplicate detection
├── qr_code_import_service.py           # Bulk QR code import from Excel
├── address_normalization_fix.py        # normalize_address() + addresses_are_similar() helpers
├── app_performance_middleware.py       # PerformanceMonitor — dev-mode only; lazy-imported inside __main__
├── db_audit_tables.py                  # DB audit table helpers (standalone operational script)
├── db_health_check.py                  # DB connectivity health check (standalone)
├── db_maintenance.py                   # DB maintenance utilities (standalone)
├── db_performance_optimization.py      # Index/query optimization helpers
├── employee_data_merger.py             # Merge duplicate employee records (standalone)
├── employee_duplicate_remove.py        # Remove employee duplicates (standalone)
├── employee_sync_scheduler.py          # Scheduled employee sync (schedule library)
├── employee_table_sync.py              # Employee table sync logic
├── requirements.txt                    # All pinned Python dependencies (single MySQL driver: PyMySQL)
│
├── models/
│   ├── __init__.py                     # set_db(db) → unpacks and returns all model classes
│   ├── base.py                         # Shared db reference (base.db); all models import from here
│   ├── user.py                         # User model (table: users)
│   ├── employee.py                     # Employee model (table: employee)
│   ├── attendance.py                   # AttendanceData model (table: attendance_data)
│   ├── time_attendance.py              # TimeAttendance model (table: time_attendance)
│   ├── qrcode.py                       # QRCode, QRCodeStyle, QRCodeLocation models
│   ├── project.py                      # Project model (table: projects)
│   └── permissions.py                  # UserProjectPermission, UserLocationPermission models
│
├── routes/
│   ├── __init__.py                     # Empty (intentional)
│   ├── auth.py                         # Blueprint 'auth': /, /register (admin-only), /login, /logout, /profile
│   ├── dashboard.py                    # Blueprint 'dashboard': /dashboard, project QR views, stats APIs
│   ├── users.py                        # Blueprint 'users': /users/*, user management APIs
│   ├── admin.py                        # Blueprint 'admin': /admin/logs, /api/logs/*
│   ├── projects.py                     # Blueprint 'projects': /projects/*
│   ├── qr_codes.py                     # Blueprint 'qr_codes': /qr-codes/*, /qr/<url> check-in flow
│   ├── attendance.py                   # Blueprint 'attendance': /attendance report + API endpoints
│   ├── attendance_edit.py              # Side-effect module: edit, add manual, save, delete routes
│   │                                   # Imports bp FROM attendance.py — does NOT define its own blueprint
│   ├── verification.py                 # Side-effect module: /verification-review/* routes
│   │                                   # Imports bp FROM attendance.py — does NOT define its own blueprint
│   ├── attendance_export.py            # Side-effect module: /export-configuration, /generate-excel-export
│   │                                   # Imports bp FROM attendance.py — does NOT define its own blueprint
│   ├── statistics.py                   # Blueprint 'statistics': /statistics
│   ├── employees.py                    # Blueprint 'employees': /employees/*
│   ├── time_attendance.py              # Blueprint 'time_attendance': /time-attendance/* (all TA routes)
│   └── time_attendance_export.py       # Plain module (NO Blueprint): export helper functions only
│
├── utils/
│   ├── __init__.py
│   ├── helpers.py                      # Role decorators, QR generation, role/permission helpers
│   ├── geocoding.py                    # Haversine distance, Google Maps client, reverse geocode
│   └── template_helpers.py             # Context processors: get_employee_name, format_hours, etc.
│
├── static/
│   ├── css/
│   │   ├── style.css                   # MASTER stylesheet — all CSS variables and layout
│   │   ├── theme-gov.css               # GOV Services brand theme overrides (loaded when THEME_NAME=gov)
│   │   └── *.css                       # Page-specific stylesheets
│   └── js/
│       ├── script.js                   # Global JS — includes CSRF fetch wrapper
│       ├── qr_destination.js           # Check-in page: GPS, bilingual UI, camera — CSRF-EXEMPT
│       ├── attendance_report.js        # Attendance report: pagination, sorting, charts
│       ├── dashboard.js
│       ├── export_configuration.js
│       ├── users.js
│       ├── attendance_fullscreen.js
│       └── android_location_handler.js # Android-specific GPS workaround
│
├── templates/
│   ├── base.html                       # Base for unauthenticated pages — loads theme CSS conditionally
│   ├── base_authenticated.html         # Base for protected pages — loads theme CSS conditionally
│   │                                   # Brand text uses {{ COMPANY_NAME }} — no hardcoded strings
│   ├── [all other page templates]      # All POST forms include {{ csrf_token() }} hidden field
│   └── errors/
│       ├── 403.html / 404.html / 500.html
│
└── tools/
    ├── migration_photo_verification_toggle.py  # Adds photo_verification_enabled to qr_codes
    ├── migration_PM_permissions.py             # One-time: user_project/location_permissions tables
    ├── migration_dynamic_qr_locations.py       # One-time: qr_type column + qr_code_locations table
    └── optimize_time_attendance_db.py          # Standalone DB index optimization script

---

3. Database Tables

Table	Model	Purpose
users	User	System user accounts with RBAC
employee	Employee	Employee master data (firstName, lastName, title, contractId)
attendance_data	AttendanceData	QR check-in records with GPS, photo verification
time_attendance	TimeAttendance	Imported time-clock records from Excel
qr_codes	QRCode	QR code definitions (standard and dynamic types)
qr_code_styles	QRCodeStyle	Reusable QR code visual styles
qr_code_locations	QRCodeLocation	Selectable locations for dynamic QR codes
projects	Project	Projects that group QR codes and employees
user_project_permissions	UserProjectPermission	Project-level access for Project Managers
user_location_permissions	UserLocationPermission	Location-level access for Project Managers
log_events	(raw SQL)	Application event log (created by AppLogger)

qr_codes — key columns

Column	Type	Notes
qr_type	VARCHAR(20)	'standard' or 'dynamic'
photo_verification_enabled	TINYINT(1) DEFAULT 1	Per-QR photo verification toggle (added May 2026)
active_status	BOOLEAN
project_id	FK → projects

attendance_data — key columns

Column	Type	Notes
employee_id	VARCHAR(50)	Always stored uppercased
location_name	VARCHAR(100)	Resolved location — never stores 'Dynamic'
location_accuracy	FLOAT	Haversine distance in miles
is_dynamic_qr	BOOLEAN	True when checked in via dynamic QR
verification_photo	TEXT	Base64 encoded image
verification_required	BOOLEAN
verification_status	VARCHAR(20)	pending / approved / rejected

Key relationships:

• Employee.contractId → Project.id
• QRCode.project_id → Project.id
• AttendanceData.qr_code_id → QRCode.id (CASCADE DELETE)
• TimeAttendance.project_id → Project.id

---

4. User Roles & Access Control

VALID_ROLES       = ['admin', 'staff', 'payroll', 'project_manager', 'accounting']
STAFF_LEVEL_ROLES = ['staff', 'payroll', 'project_manager', 'accounting']

Role	Key Access
admin	Full access; only role that sees System Logs, Users, all export tools
staff	Create/edit QR codes; view dashboard and reports; no delete, no admin
payroll	Same as staff + Time Attendance section
accounting	Same as payroll (identical menu items)
project_manager	Reports only; scoped to assigned projects and locations via permission tables

Auth decorators (in utils/helpers.py):

• @login_required — redirects to /login if no session
• @admin_required — 403 if role is not admin
• @staff_or_admin_required — 403 if not admin or staff-level

/register is restricted to @admin_required — public self-registration is disabled.

---

5. Application Factory & Initialization Order

1. load_dotenv()
2. from extensions import db, init_logger
3. create_app():
   a. app.config.from_object(get_config())
   b. SECRET_KEY guard — sys.exit(1) if default value in non-debug mode
   c. db.init_app(app)
   d. set_db(db) → unpacks all model classes
   e. init_logger(app, db) → binds logger_handler in extensions.py
   f. register_blueprints() — in canonical order (see §6)
      + side-effect import of attendance_edit, verification, attendance_export
   g. create_location_logging_routes() ← from location_logging.py
   h. SecurityManager.init_app() ← wires CSRF before_request + rate limiting
   i. inject_csrf_token() context processor
   j. inject_company_name() context processor (COMPANY_NAME, THEME_NAME, CURRENT_YEAR)
   k. inject_logging_status(), inject_turnstile() context processors
   l. template_filters (strftime, days_since, time_ago)
   m. before_request: adjust_session_lifetime + g.start_time + suspicious UA scan
   n. after_request: slow-query detection + error response logging
   o. Error handlers: 403, 404, 500
   p. Startup init: create_tables() + update_existing_qr_codes() ← runs under Gunicorn too
4. if __name__ == '__main__':
   a. lazy import PerformanceMonitor (dev-mode only)
   b. initialize_performance_optimizations()
   c. app.run()

Critical notes:

• create_tables() and update_existing_qr_codes() run inside with app.app_context() inside create_app() — they execute under gunicorn, not only under __main__.
• PerformanceMonitor is imported lazily inside __main__ only — gunicorn workers never load it.
• update_existing_qr_codes() prefers QR_BASE_URL from .env. Falls back to FLASK_HOST/FLASK_PORT. Never uses request.url_root.

---

6. Blueprint Registration Order

auth_bp, dashboard_bp, users_bp, admin_bp, projects_bp,
qr_codes_bp, attendance_bp, statistics_bp,
employees_bp, time_attendance_bp

Attendance blueprint split: attendance.py defines bp = Blueprint('attendance', __name__) once. attendance_edit.py, verification.py, and attendance_export.py each do:

from routes.attendance import bp  # shared blueprint — do not redefine

app.py registers only attendance_bp once. Sub-modules are loaded as side-effect imports:

import routes.attendance_edit    # noqa: F401
import routes.verification       # noqa: F401
import routes.attendance_export  # noqa: F401

time_attendance_export.py is not a Blueprint. Plain module providing export helper functions. Never register it separately.

---

7. All Routes

auth (Blueprint: auth)

URL	Endpoint	Notes
/	auth.index
/register	auth.register	@admin_required — not public
/login	auth.login	Rate-limited via SecurityManager
/logout	auth.logout
/profile	auth.profile

dashboard (Blueprint: dashboard)

URL	Endpoint
/dashboard	dashboard.dashboard
/project/<id>/qr-codes	dashboard.project_qr_codes
/dashboard/search	dashboard.search_qr_codes
/api/dashboard/stats	dashboard.dashboard_stats_api
/api/dashboard/realtime	dashboard.dashboard_realtime_api

users (Blueprint: users)

URL	Endpoint
/users	users.users
/users/create	users.create_user
/users/<id>/edit	users.edit_user
/users/<id>/delete	users.delete_user
/users/<id>/reactivate	users.reactivate_user
/users/<id>/promote	users.promote_user
/users/<id>/demote	users.demote_user
/users/<id>/toggle-status	users.toggle_user_status
/users/<id>/activate	users.activate_user
/users/<id>/deactivate	users.deactivate_user
/users/<id>/permanently-delete	users.permanently_delete_user
/api/users/stats	users.user_stats_api
/api/locations-by-projects	users.get_locations_by_projects
/api/roles/permissions	users.role_permissions_api
/api/geocode	users.geocode_address_api
/api/reverse-geocode	users.reverse_geocode_api

admin (Blueprint: admin)

URL	Endpoint
/admin/logs	admin.admin_logs
/admin/health/google-maps	admin.google_maps_health
/api/logs/recent	admin.api_recent_logs
/api/logs/stats	admin.api_log_stats
/api/logs/cleanup	admin.api_cleanup_logs
/api/logs/clear	admin.api_clear_logs
/api/logs/clear-old	admin.api_clear_old_logs
/api/logs/export	admin.api_export_logs

projects (Blueprint: projects)

URL	Endpoint
/projects	projects.projects
/projects/create	projects.create_project
/projects/<id>/edit	projects.edit_project
/projects/<id>/toggle	projects.toggle_project
/api/projects/active	projects.api_active_projects

qr_codes (Blueprint: qr_codes)

URL	Endpoint	Notes
/qr-codes/create	qr_codes.create_qr_code
/qr-codes/bulk-import	qr_codes.import_bulk_qr_codes
/qr-codes/bulk-import/template	qr_codes.download_qr_import_template
/qr-codes/<id>/edit	qr_codes.edit_qr_code
/qr-codes/<id>/delete	qr_codes.delete_qr_code
/qr-codes/<id>/toggle-status	qr_codes.toggle_qr_status
/qr-codes/<id>/activate	qr_codes.activate_qr_code
/qr-codes/<id>/deactivate	qr_codes.deactivate_qr_code
/qr-codes/<id>/copy-url	qr_codes.copy_qr_url
/qr-codes/<id>/open-link	qr_codes.open_qr_link
/qr/<url>	qr_codes.qr_destination
/qr/<url>/checkin	qr_codes.qr_checkin	CSRF-exempt — public unauthenticated
/qr/<url>/locations	qr_codes.qr_get_locations

attendance (Blueprint: attendance — split across 4 files)

URL	Endpoint	File
/attendance	attendance.attendance_report	attendance.py
/api/attendance/locations	attendance.attendance_locations_api	attendance.py
/api/attendance/stats	attendance.attendance_stats_api	attendance.py
/api/search_employees	attendance.search_employees_api	attendance.py
/api/get_project_locations	attendance.get_project_locations_api	attendance.py
/api/time-attendance/locations	attendance.time_attendance_locations_api	attendance.py
/attendance/<id>/edit	attendance.edit_attendance	attendance_edit.py
/attendance/<id>/delete	attendance.delete_attendance	attendance_edit.py
/attendance/add	attendance.add_manual_attendance	attendance_edit.py
/attendance/save_manual	attendance.save_manual_attendance	attendance_edit.py
/verification-review	attendance.verification_review	verification.py
/verification-review/<id>	attendance.verification_review_detail	verification.py
/verification-review/<id>/update	attendance.update_verification_status	verification.py
/api/attendance/<id>/verification-details	attendance.get_verification_details	verification.py
/export-configuration	attendance.export_configuration	attendance_export.py
/generate-excel-export	attendance.generate_excel_export	attendance_export.py

statistics (Blueprint: statistics)

URL	Endpoint
/statistics	statistics.qr_statistics
/api/statistics/export	statistics.export_statistics

employees (Blueprint: employees)

URL	Endpoint
/employees	employees.employees
/employees/create	employees.create_employee
/employees/<idx>/edit	employees.edit_employee
/employees/<idx>/delete	employees.delete_employee
/employees/<idx>	employees.employee_detail
/api/employees/search	employees.api_employees_search

time_attendance (Blueprint: time_attendance)

URL	Endpoint
/time-attendance	time_attendance.time_attendance_dashboard
/time-attendance/import	time_attendance.import_time_attendance
/time-attendance/import/analyze-duplicates	time_attendance.analyze_import_duplicates
/time-attendance/import/analyze-invalid	time_attendance.analyze_import_invalid
/time-attendance/import/start	time_attendance.start_import_job
/time-attendance/import/stream/<job_id>	time_attendance.stream_import_progress
/time-attendance/import/cancel-pending	time_attendance.cancel_pending_import
/time-attendance/import/validate	time_attendance.validate_import_file
/time-attendance/import/batch/<batch_id>	time_attendance.view_import_batch
/time-attendance/import/batch/<batch_id>/delete	time_attendance.delete_import_batch
/time-attendance/import/download-template	time_attendance.download_import_template
/time-attendance/export	time_attendance.export_time_attendance
/time-attendance/export/excel	time_attendance.excel_export_time_attendance
/time-attendance/export-by-building	time_attendance.export_time_attendance_by_building
/time-attendance/records	time_attendance.time_attendance_records
/time-attendance/record/<id>	time_attendance.time_attendance_record_detail
/time-attendance/delete/<id>	time_attendance.delete_time_attendance_record
/api/time-attendance/employee/<id>	time_attendance.api_time_attendance_by_employee
/api/time-attendance/location/<n>	time_attendance.api_time_attendance_by_location

Location logging (registered directly on app, not a Blueprint)

URL	Purpose
/api/log-location-action	Android GPS debug logging
/api/location-debug-info	Return GPS debug info

Security API (registered by SecurityManager on app)

URL	Purpose
/api/security/status	Admin-only security dashboard stats
/api/security/clear-blocks	Admin-only: clear rate-limit blocks

---

8. Template System

Two Base Templates

• base.html — unauthenticated pages (login, register, QR scan, errors). Body class: login-layout.
• base_authenticated.html — all protected pages. Body class: has-sidebar. Fixed collapsible sidebar, top header with user/role badge, flash message rendering.

Template Block Names

Block	Purpose
{% block title %}	Page <title> text only — no CSS or JS here
{% block page_title %}	Top header <h1> (authenticated only)
{% block extra_head %}	Page-specific CSS — inside <head>
{% block content %}	Main page body
{% block extra_scripts %}	Page-specific JS — before </body>

Rule: CSS always goes in extra_head. JS always goes in extra_scripts. Never inject either into {% block title %}.

Theme System

Both base.html and base_authenticated.html load the theme override after style.css:

<!-- Theme Override (set THEME_NAME in .env to activate, e.g. THEME_NAME=gov) -->
{% if THEME_NAME %}
<link rel="stylesheet" href="{{ url_for('static', filename='css/theme-' + THEME_NAME + '.css') }}" />
{% endif %}

Loaded before {% block extra_head %} so page-specific CSS loads after and can override.

theme-gov.css overrides: --primary-color: #16a34a / --primary-hover: #15803d (institutional green). No CSS text hacks — brand names handled via {{ COMPANY_NAME }}.

Adding a third company theme: Create static/css/theme-{name}.css and set THEME_NAME={name} in .env. No code changes required.

Global Context Variables

Variable	Source
COMPANY_NAME	.env (COMPANY_NAME)
THEME_NAME	.env (THEME_NAME) — empty string when not set
CURRENT_YEAR	datetime.now().year
csrf_token	generate_csrf_token() from advanced_security_middleware
is_admin	bool from session role
turnstile_enabled, turnstile_site_key	Turnstile config

CSRF in Templates

Every POST form:

<input type="hidden" name="csrf_token" value="{{ csrf_token() }}">

Every AJAX POST:

'X-CSRF-Token': (window.qrConfig && window.qrConfig.csrfToken) || ''

Exception: /qr/<url>/checkin is CSRF-exempt (public, unauthenticated).

---

9. Design System

CSS Custom Properties (defined in style.css)

--primary-color:   #2563eb   /* Blue — LT default */
--primary-hover:   #1d4ed8
--secondary-color: #64748b
--success-color:   #10b981
--warning-color:   #f59e0b
--danger-color:    #ef4444
--info-color:      #0891b2
--gray-50:  #f8fafc          /* Page background */
--gray-700: #334155          /* Body text */
--gray-900: #0f172a          /* Headings */

Light mode only — color-scheme: light only !important enforced globally.

Layout

• Sidebar: 280px expanded, 64px collapsed. Gradient #2563eb → #1d4ed8 (LT). GOV theme: #16a34a → #15803d.
• Header height: 64px.
• Z-index: dropdown 1000, modal 1050, sidebar 1100, overlay 1200.

Excel Export Styling

• Header rows: white bold text on solid black fill (000000)
• Miss-punch / amber: FFC000 fill
• Font: Aptos Narrow 11pt everywhere; 14pt report title, 12pt section summary

---

10. Security Architecture

CSRF Protection

• SecurityManager from advanced_security_middleware.py wired via init_app() in create_app()
• before_request validates csrf_token form field or X-CSRF-Token header on every POST/PUT/PATCH/DELETE
• Token stored in session['csrf_token']; compared with hmac.compare_digest()
• CSRF-exempt: auth.login, auth.register, qr_codes.qr_checkin, static

Rate Limiting

• 5 failed login attempts within 15 minutes → IP blocked 15 minutes (in-memory per worker)
• SecurityManager.create_secure_session() called on login success (clears failed-attempt counter)

Session Security

• session.clear() before setting new keys on login (prevents session fixation)
• before_request hook adjust_session_lifetime(): remember_me → 30 days, default → 10 hours
• SESSION_COOKIE_SECURE=True requires HTTPS — HTTP-only deployments must set false or login loops

validate_session_security() — DO NOT USE in before_request

In-memory dict per worker — breaks under multi-worker gunicorn (login worker A, next request hits worker B with empty dict → 401). Intentionally not called. Flask signed cookie + CSRF handles integrity.

SQL Injection Prevention

All dynamic SQL uses SQLAlchemy parameterized queries:

conditions, params = [], {}
conditions.append("ad.check_in_date >= :date_from")
params["date_from"] = date_from
db.session.execute(text("... WHERE 1=1 " + filter_clause), params)

---

11. QR Code System

QR Types

• standard (default) — fixed single location
• dynamic — employee selects location at scan time from list auto-generated from all active standard QR codes. No manual management UI — always queried live via SELECT DISTINCT location, location_address FROM qr_codes WHERE qr_type='standard'.

Photo Verification — Two-Layer Toggle

Both must be True for photo verification to trigger:

1. Global: PHOTO_VERIFICATION_ENABLED in .env
2. Per-QR: qr_codes.photo_verification_enabled (TINYINT(1) DEFAULT 1)

qr_photo_verification = getattr(qr_code, 'photo_verification_enabled', True)
if qr_photo_verification and current_app.config.get('PHOTO_VERIFICATION_ENABLED', True) \
        and location_accuracy > threshold:
    # require photo

Toggle UI in create_qr_code.html and edit_qr_code.html — uses addEventListener('change', ...) in extra_scripts block.

Check-In Flow

1. Employee scans QR → qr_destination.html
2. Enters ID; GPS captured by browser
3. 30-min interval guard (configurable via TIME_INTERVAL)
4. Dynamic QR: server rejects if selected_location_name is empty; location_name in record is always the resolved name, never 'Dynamic'
5. Haversine distance calculated; photo required if beyond threshold (and both toggles enabled)
6. Server-side photo size check: rejects > VERIFICATION_PHOTO_MAX_SIZE with HTTP 413
7. Record saved to attendance_data

Check-In Page Features

• Bilingual: English/Vietnamese toggle (qr_destination.js)
• Staff ID persistence: localStorage remembers last employee ID
• Android GPS: special handler (android_location_handler.js)
• QR URLs must be full https:// absolute URLs — relative URLs parsed as search queries by phones
• Base URL constructed from QR_BASE_URL env var — never from request.url_root

---

12. Time Attendance Import Pipeline

Excel File Requirements

• Required columns: ID, Date, Time, Location Name, Action Description
• Optional columns: Name, Platform, Event Description, Recorded Address, Distance

Import Flow

1. Upload → validate_import_file
2. Duplicate analysis → analyze_import_duplicates → user reviews
3. Invalid record analysis → analyze_import_invalid
4. Start job → start_import_job (SSE streaming progress via stream_import_progress)
5. Result in time_attendance_import_result.html

Special Handling

• Recorded Address: read via openpyxl directly (not pandas) to preserve HYPERLINK formulas
• Duplicate detection: hash of employee_id + date + time + action_description
• Import tracked by import_batch_id (UUID)
• self.db not db — in TimeAttendanceImportService, always access SQLAlchemy via self.db
• Validation except blocks must not silently swallow exceptions (fail-open prevention)

---

13. Time Attendance Excel Export

Calculator — CRITICAL

Always use WorkingHoursCalculator from working_hours_calculator.py. Never switch to SingleCheckInCalculator — legacy, causes silent calculation errors.

Rounding Rules

• Daily / Weekly / Grand Total + SP/PW/PT summary columns: _qtr() — quarter-hour rounding
• Individual Hours/Building entry cells: raw round(..., 2) — no quarter rounding
• _qtr() pipeline: decimal hours → minutes → round_time_to_quarter_hour() → convert_minutes_to_base100() → round_base100_hours()

Work Type Codes

• SP = Special Project, PW = Periodic Work, PT = Project Team (Part-Time)
• Parsed from employee_id via parse_employee_id_for_work_type()

Overnight Shift Handling

Rule 1 — Sort key: early-morning OUTs (hour <= 3) use _overnight_aware_sort_key() which adds 86400 seconds — pushes them past midnight so they sort after same-day evening INs.

Rule 2 — Detection threshold: hour >= 12 (noon). Any IN at or after 12:00 PM is an overnight IN candidate if no matching OUT exists same day AND early-morning OUT (hour <= 3) exists next calendar day.

Orphan guard: early-morning OUTs with check_in_date <= current_day are orphans — skip. Only OUTs with check_in_date > current_day (moved by overnight detection) pair with evening INs.

Cross-Type SP Pairs

is_cross_type = True when SP IN pairs with non-SP OUT (or vice versa). Accumulates separately in cross_type_sp/pw/pt_hours to prevent double-counting in summary rows.

Summary Rows (SP / PW / PT + Regular)

col8='SP'       col9=_qtr(sp_hours)           ← only if sp_hours > 0
col8='PW'       col9=_qtr(pw_hours)           ← only if pw_hours > 0
col8='PT'       col9=_qtr(pt_hours)           ← only if pt_hours > 0
col8='Regular'  col9=_qtr(regular_only_hours) ← always when any special type exists
col7='GRAND TOTAL:' col9=_qtr(grand_regular) col10=_qtr(grand_ot)

Export Date Range

_resolve_date_range() enforces a 14-day cap by default. Both export functions accept unlimited=False; pass unlimited=True to bypass.

UI: "Unlimited" checkbox in time_attendance_records.html before export buttons.

Employee Name Format

"Lastname, Firstname" — f"{emp.lastName}, {emp.firstName}"

Export Date Iteration

Cap sorted_dates to <= end_date — prevents overnight buffer day from rendering as a display row.

Cross-Building Pairing

Same-day IN at Building A + OUT at Building B → pair and label "IN: Building A → OUT: Building B". Suppress spurious Missed Punch rows.

---

14. Logging System (logger_handler.py)

Single AppLogger instance in extensions.py. Import everywhere as:

from extensions import logger_handler
logger_handler.logger.info("...")
logger_handler.logger.error("...", exc_info=True)  # always pass exc_info=True in except blocks

Log Destinations

• logs/application.log — rotating 10MB/5 backups
• logs/errors.log — rotating 5MB/10 backups
• logs/security.log — rotating 2MB/20 backups
• log_events DB table — admin dashboard at /admin/logs

Log Every Action

All create, edit, and delete operations must include a log entry:

logger_handler.logger.info(f"User {session['username']} created employee {new_employee.id}")

traceback Convention

• Use exc_info=True on logger.error() — never import traceback inline
• traceback.format_exc() only acceptable when passing stack_trace= to log_flask_error()

---

15. Configuration (config.py + .env)

All env-var reads centralized in config.py. Blueprints use current_app.config['KEY'].

Key .env Variables

DATABASE_URL                           # mysql+pymysql://user:pass@host/db — special chars in password OK
SECRET_KEY                             # MUST differ from default
COMPANY_NAME / CONTRACT_NAME
FLASK_HOST / FLASK_PORT / FLASK_ENV / DEBUG
SESSION_COOKIE_SECURE                  # MUST be 'false' for HTTP-only deployments
SESSION_COOKIE_HTTPONLY / SESSION_COOKIE_SAMESITE
TIME_INTERVAL                          # Check-in cooldown in minutes (default 30)
GOOGLE_MAPS_API_KEY                    # Optional; falls back to Haversine-only
TURNSTILE_ENABLED / TURNSTILE_SITE_KEY / TURNSTILE_SECRET_KEY
ENABLE_PHOTO_VERIFICATION              # Global toggle (default 'true')
PHOTO_VERIFICATION_DISTANCE_THRESHOLD  # Miles (default 0.3)
VERIFICATION_PHOTO_MAX_SIZE            # Bytes (default 5MB)
UPLOAD_FOLDER                          # Temp path (default /tmp)
DEFAULT_ADMIN_PASSWORD                 # CHANGE IN PRODUCTION
QR_BASE_URL                            # Public-facing domain for QR links — required behind reverse proxy
THEME_NAME                             # Activates static/css/theme-{THEME_NAME}.css
SYNC_INTERVAL_MINUTES
SQLALCHEMY_ENGINE_OPTIONS_POOL_SIZE / POOL_TIMEOUT / POOL_RECYCLE / MAX_OVERFLOW

---

16. Distance Calculation

Uses Haversine formula only (straight-line geodesic distance). Google Maps Distance Matrix API removed.

utils/geocoding.py initializes gmaps_client for geocoding (address → coordinates) if GOOGLE_MAPS_API_KEY is set.

Google Maps Guard — ALWAYS USE

from utils.geocoding import is_gmaps_available, gmaps_client
if is_gmaps_available():
    result = gmaps_client.geocode(address)
else:
    # fall back to OpenStreetMap / Haversine

Never call gmaps_client.method() without a None guard.

Address Normalization

Multi-strategy matching: core street extraction → component matching → fuzzy fallback. Handles geocoding drift for near-identical addresses.

---

17. Migration Scripts

All in tools/. Always use pymysql directly — never import the Flask app or SQLAlchemy ORM. ORM loads models at import time; if target column doesn't exist yet, it crashes on startup.

Pattern:

from dotenv import load_dotenv
load_dotenv()
import pymysql, re

def parse_db_url(url):
    """Use regex — urlparse breaks on special chars (@, :) in passwords."""
    url = re.sub(r'^mysql\+pymysql://', '', url)
    m = re.match(
        r'^(?P<user>[^:]+):(?P<password>.+)@(?P<host>[^@:/]+)(?::(?P<port>\d+))?/(?P<db>[^?]+)',
        url
    )
    return {'host': m.group('host'), 'port': int(m.group('port') or 3306),
            'user': m.group('user'), 'password': m.group('password'), 'database': m.group('db')}

def run():
    conn = pymysql.connect(**parse_db_url(os.environ['DATABASE_URL']), charset='utf8mb4', autocommit=False)
    with conn.cursor() as cur:
        cur.execute("SELECT COUNT(*) FROM information_schema.COLUMNS WHERE ...")
        if cur.fetchone()[0] > 0:
            print("[SKIP] already exists"); return
        cur.execute("ALTER TABLE `table` ADD COLUMN `col` TINYINT(1) NOT NULL DEFAULT 1")
        conn.commit()

Completed Migrations

Script	What it adds
migration_photo_verification_toggle.py	qr_codes.photo_verification_enabled TINYINT(1) DEFAULT 1
migration_dynamic_qr_locations.py	qr_type column + qr_code_locations table
migration_PM_permissions.py	user_project_permissions + user_location_permissions tables

---

18. Coding Conventions & Developer Preferences

Change Philosophy

• Minimal, additive changes only — preserve all route names, function names, endpoint names, variable names, URL patterns
• New functionality added alongside existing code, not replacing it
• Never rename routes, functions, endpoints, or variables
• Every code fix applies to both LT and GOV instances

File Delivery

• ≤ 4 files changed → present each file individually
• ≥ 5 files changed → deliver as a single zip archive

File Editing Approach

• Pull latest code at start of each session
• CRLF normalization first if needed: content.replace('\r\n', '\n')
• Use surgical str_replace edits — never rewrite large blocks wholesale
• AST parse check after every Python edit: python3 -c "import ast; ast.parse(open(f).read())"
• Simulate logic before and after any calculator or pairing logic changes

SQLAlchemy Patterns

# Correct (SQLAlchemy 2.0):
record = db.session.get(Model, record_id)
if record is None:
    abort(404)

# Deprecated — do not use:
record = Model.query.get_or_404(record_id)
record = Model.query.get(record_id)

# Raw SQL requires text():
db.session.execute(text("SELECT ..."), params)
# Never pass raw strings to conn.execute() — ObjectNotExecutableError in SQLAlchemy 2.0

JavaScript Patterns

• Use createElement + addEventListener — never inline onchange, onclick, etc.
• Wrap in DOMContentLoaded
• CSS → {% block extra_head %}, JS → {% block extra_scripts %}, never into {% block title %}
• All AJAX POST: 'X-CSRF-Token': (window.qrConfig && window.qrConfig.csrfToken) || ''
• No localStorage / sessionStorage in artifacts

Error Handling Pattern

Every route except block must:

1. db.session.rollback()
2. logger_handler.logger.error(f"...: {e}", exc_info=True)
3. flash(...) a user-facing message
4. Return redirect or error response

No bare except: — always except Exception as e:.

QR Code Name — Locked After Creation

The QR code name cannot be changed after creation because it is used to generate the qr_url slug. Changing the name would break all existing printed/distributed QR codes.

• Template (edit_qr_code.html): name <input> has readonly attribute + cursor: not-allowed style + lock icon in label
• Route (routes/qr_codes.py): edit POST ignores request.form['name'] — sets new_name = qr_code.name (original value). Do NOT change this back to reading from the form.

# CORRECT — name locked after creation:
new_name = qr_code.name  # always use existing name, never request.form['name']

# WRONG — never do this in the edit route:
# new_name = request.form['name']

Employee Autocomplete (attendance and time-attendance)

• Visible text input + hidden employee_id field synced on numeric input
• Fetches /api/search_employees (includes unregistered IDs from attendance_data, not only Employee table)
• CSS inlined in extra_head; JS uses createElement + addEventListener

Timestamp Convention

Use datetime.now() (local time) throughout — not datetime.utcnow().

Context Safety

• Use has_request_context() (not if not request:) to check Flask request context
• Capture current_app._get_current_object() in route body, not inside lazy generators

---

19. Attendance Report

Query fetches 1,001 rows, trims to 1,000 if extra row present, sets records_truncated = True. Template displays yellow banner when truncated:

Showing the most recent 1,000 records. Narrow the date range or apply additional filters.

---

20. Known Bugs Fixed — Do Not Reintroduce

Set 1 — Critical

File	Fix
routes/time_attendance_export.py	Removed stray second docstring
app.py	Error handlers wired to render_template('errors/*.html'); 403 added
templates/errors/*.html	url_for('dashboard') → url_for('dashboard.dashboard')
routes/payroll.py	Removed duplicate get_employee_name, get_qr_code_checkin_count, @bp.context_processor
models/attendance.py	check_in_time default: datetime.now().time → lambda: datetime.now().time()
models/user.py	Added @staticmethod to has_export_permissions()

Set 2 — Moderate / Minor

File	Fix
app.py	Merged duplicate before/after request hooks
app.py	update_existing_qr_codes() uses QR_BASE_URL first, not request.url_root
config.py	SQLALCHEMY_ENGINE_OPTIONS pool wiring
utils/geocoding.py	is_gmaps_available() helper
working_hours_calculator.py	CRLF → LF

Set 3 — Structural

File	Fix
routes/time_attendance_export.py	Removed 14 unused imports
routes/dashboard.py, statistics.py, payroll.py	db.session.rollback() in all except blocks
models/employee.py	cls.id.like() on BigInteger → cast(cls.id, String).like()

Set 4 — Export Pipeline

File	Fix
routes/time_attendance_export.py	Overnight IN threshold: >= 19 → >= 12
routes/time_attendance_export.py	SP/PW/PT summary rows + Regular row
routes/time_attendance_export.py	unlimited=False parameter throughout
templates/time_attendance_records.html	"Unlimited" checkbox

Set 5 — GOV Deployment

File	Fix
routes/qr_codes.py	_get_qr_base_url() helper; replaced all request.url_root
config.py	QR_BASE_URL env var
nginx (GOV)	Removed duplicate proxy_set_header Host (doubled hostname in QR links)

Set 6 — Theme System

File	Fix
static/css/theme-gov.css	New file — GOV brand green overrides
app.py	inject_company_name() returns THEME_NAME and CURRENT_YEAR
templates/base.html + base_authenticated.html	Conditional theme CSS; {{ COMPANY_NAME }} for brand text; {{ CURRENT_YEAR }} in footer

Set 7 — Security Hardening

File	Fix
routes/statistics.py, routes/payroll.py	SQL injection: f-string filters → parameterized queries
routes/auth.py	Open redirect: _is_safe_url() on next param
app.py	SECRET_KEY default guard: sys.exit(1)
routes/auth.py	Session fixation: session.clear() before new keys
routes/auth.py	@login_required + @admin_required on /register
app.py	CSRF SecurityManager wired; before_request validator
All POST forms (22 templates)	{{ csrf_token() }} hidden field
All AJAX POST calls	X-CSRF-Token header
advanced_security_middleware.py	request.json guarded with content-type check (fixes 415)
advanced_security_middleware.py	Stable key from SECRET_KEY via SHA-256 (fixes per-worker warning)
advanced_security_middleware.py	validate_session_security() removed from security_check() (fixes 401 under multi-worker)
All route files	Bare except: → except Exception (11 locations)
routes/qr_codes.py	Server-side photo size enforcement; HTTP 413 on oversized payload

Set 8 — Code Quality

File	Fix
7 route files	Model.query.get_or_404() → db.session.get() + abort(404) (21 call sites)
config.py	PERMANENT_SESSION_LIFETIME → timedelta(hours=10)
routes/attendance.py	Split into 4 files sharing one blueprint
requirements.txt	mysql-connector-python removed
routes/attendance.py	LIMIT 1001 + records_truncated flag + yellow banner

Set 9 — Template Sync (May 2026)

File	Fix
templates/base.html (GOV)	Footer hardcoded 2025 QR Code Management System → {{ CURRENT_YEAR }} {{ COMPANY_NAME }}
templates/base.html (GOV)	Theme CSS block moved to after style.css, before Font Awesome (matches LT order)
templates/base_authenticated.html (GOV)	Indentation sync with LT
templates/projects.html (GOV)	Confirm dialog on project toggle + loading state on Edit buttons (sync with LT)

Set 10 — Per-QR Photo Verification Toggle (May 2026)

File	Change
models/qrcode.py	Added photo_verification_enabled TINYINT(1) DEFAULT 1
routes/qr_codes.py	Create: reads toggle from form, passes to constructor + logs
routes/qr_codes.py	Edit: reads toggle from form, assigns to record + logs
routes/qr_codes.py	Checkin: checks per-QR AND global flag (both must be True)
templates/create_qr_code.html	Photo Verification toggle section added
templates/edit_qr_code.html	Photo Verification toggle section added (CSS in extra_head, JS in extra_scripts with addEventListener)
tools/migration_photo_verification_toggle.py	pymysql migration — adds column safely

Set 11 — QR Name Lock Restored + CSRF Fix (May 2026)

File	Change
templates/edit_qr_code.html	Name field restored to readonly — lock icon in label, cursor: not-allowed, help text explaining why
routes/qr_codes.py	Edit route: new_name = qr_code.name (never reads from form) — prevents name change server-side
templates/create_qr_code.html	window.qrConfig with csrfToken injected in <head> — fixes 403 on /api/geocode (standalone template, doesn't extend base)
templates/base_authenticated.html	window.qrConfig block moved before {% block extra_scripts %} — fixes token availability for all pages extending base

---

21. Infrastructure & Deployment

• Deploy user: qrcode on both servers
• Restart: sudo supervisorctl restart qrcode
• Template-only changes: no restart needed
• Python/model changes: always run migration first, then deploy files, then restart
• Gunicorn: gevent workers — avoid threading-unsafe patterns

Deploy Order for DB Column Additions

1. Run python3 tools/migration_<name>.py (pymysql, safe to re-run)
2. Deploy updated Python files
3. Restart service

---

22. Session Changelog

Sessions 1–8 — Foundational Work

See §20 Sets 1–8 for detailed bug fix history. Covers: initial critical fixes, security hardening (CSRF, SQL injection, session fixation, rate limiting, open redirect), export pipeline enhancements, GOV deployment, theme system, code quality sprint.

Session (May 2026) — Template Sync + Photo Verification Toggle

• Diffed LT and GOV codebases — confirmed only 3 template files differed (whitespace + minor logic)
• Synced base.html, base_authenticated.html, projects.html — GOV brought to parity with LT
• Added per-QR photo_verification_enabled toggle (DB column + model + routes + UI + migration)
• Fixed migration script: replaced urlparse with regex parser (handles special chars in DB passwords), replaced SQLAlchemy conn.execute(str) with pymysql direct (avoids ORM loading model before column exists)
• Fixed edit_qr_code.html toggle bug: JS was injected into {% block title %} (corrupted by earlier injection) — moved CSS to extra_head, JS to extra_scripts using addEventListener, removed inline onchange attribute
• Restored QR code name lock on edit page: field is readonly in template; route ignores submitted name (new_name = qr_code.name)
• Fixed /api/geocode 403: create_qr_code.html is standalone HTML (no {% extends %}), so window.qrConfig was never injected — added direct injection in <head>; also fixed ordering in base_authenticated.html for pages that do extend it