Sep 15 - Update the attendance records table to update every 20 min

Claude.md

@@ -342,6 +342,7 @@ import routes.attendance_export  # noqa: F401
 | `/api/attendance/locations` | `attendance.attendance_locations_api` | `attendance.py` |
 | `/api/attendance/stats` | `attendance.attendance_stats_api` | `attendance.py` |
 | `/api/search_employees` | `attendance.search_employees_api` | `attendance.py` |
+| `/api/attendance/live-updates` | `attendance.attendance_live_updates_api` | `attendance.py` — live table polling, see §19 |
 | `/api/get_project_locations` | `attendance.get_project_locations_api` | `attendance.py` |
 | `/api/time-attendance/locations` | `attendance.time_attendance_locations_api` | `attendance.py` |
 | `/attendance/<id>/edit` | `attendance.edit_attendance` | `attendance_edit.py` |
@@ -1034,6 +1035,37 @@ Semantics (identical on both sides, mirroring `_work_type_codes_for()`):
 **Do not reintroduce** `record.employeeId.toLowerCase() === id` in `applyFilters()` — that
 exact-match test is what dropped every SP/PW/PT row the query had already returned.
 
+### Live Updates (Sept 15, 2026)
+
+The records table picks up new check-ins without a reload. It uses **polling, not SSE/WebSockets**:
+a held-open stream per tab would pin a gevent worker connection and still need a DB poll behind
+it (the workers share no pub/sub).
+
+- **Endpoint:** `GET /api/attendance/live-updates?since_id=&date_from=&date_to=&location=&employee=&project=`
+  (`attendance_live_updates_api`, `@login_required`), JSON with `no-store` headers.
+- **Cheap path:** `SELECT COALESCE(MAX(id), 0) FROM attendance_data` (primary key). Not newer than
+  `since_id` → empty answer, no other query. Otherwise ONE query over `ad.id > since_id AND
+  ad.id <= latest` plus the page's filters, `ORDER BY ad.id LIMIT 201` (200 per batch; `has_more`
+  → the page fetches the next batch after 1 s). `verification_photo` (base64) is never selected; the
+  `location_accuracy` information_schema check is cached per process.
+- **Same filters as the page:** `_attendance_filter_conditions()` builds the WHERE clause for both
+  the report route and the endpoint, Project Manager scope included. **Never duplicate that logic.**
+- **Cursor:** the route reads `MAX(id)` **before** the report query and renders it as
+  `data-live-since-id` on `#attendanceReportContainer`. No attribute (PM without access, lookup
+  failed) → live updates stay off.
+- **Payload shape:** `_live_record_payload()` returns exactly what `loadTableData()` reads from the
+  server-rendered `<tbody>` (text, truncation, accuracy parsing, verification badge).
+  **If the row markup in `attendance_report.html` changes, update `_live_record_payload()` too.**
+- **Client** (`attendance_report.js`, LIVE UPDATES section): polls every 20 s, never overlapping;
+  paused while `document.hidden`, one immediate check when the tab is shown; exponential backoff
+  40 s → 5 min after errors; stops for good on redirect / 401 / 403 (session ended). New rows are
+  prepended newest-first, de-duplicated by id, highlighted for 8 s, and `refreshTableKeepingView()`
+  keeps the user's search, sort and page. A page showing the empty state reloads once when a
+  matching record arrives. A green **Live** pill in the table header shows the state.
+- **Not live:** edits, deletions and the summary statistics — they refresh on reload.
+- `createTableRow()` HTML-escapes every value (`escapeHtml`; `escapeJsString` inside `onclick`;
+  `truncateText` runs before escaping) — device and address text come from the public check-in page.
+
 ---
 
 ## 20. Known Bugs Fixed — Do Not Reintroduce
@@ -1205,6 +1237,15 @@ exact-match test is what dropped every SP/PW/PT row the query had already return
 | — | Verified by loading the real hook (before/after) into a Flask app with a controlled clock: Remember Me now survives 11 h, 25 days of daily use and 29 idle days, and expires after 31 idle days; non-Remember-Me still ends at 10 h |
 | — | No forced re-login on deploy: valid sessions keep working; non-Remember-Me sessions from before the deploy (no `login_epoch`) get their 10 hours counted from their first request after it |
 
+### Set 20 — Attendance Report Live Updates (Sept 15, 2026)
+| File | Change |
+|---|---|
+| `routes/attendance.py` | `_attendance_filter_conditions()` extracted from `attendance_report()`; `live_latest_id` cursor; `GET /api/attendance/live-updates` + `_live_record_payload()` (see §19) |
+| `templates/attendance_report.html` | `data-live-since-id` on the container; Live status pill + new-row highlight CSS |
+| `static/js/attendance_report.js` | LIVE UPDATES module; `filterRecords()` split out of `applyFilters()`, `sortFilteredData()` split out of `sortTable()` |
+| `static/js/attendance_report.js` | **Security:** `createTableRow()` put raw device / address / name text into `innerHTML`. Device comes from the check-in User-Agent, so a crafted UA could inject markup into the report — every value is now escaped |
+| — | Verified: filter helper produces identical SQL + params to the previous inline block for 180 input combinations; `_live_record_payload()` matches what `loadTableData()` reads from the Jinja-rendered `<tbody>` for 20 row variants; the real `attendance_report.js` driven with a fake DOM/timers/fetch passes 26 checks (insert, de-dup, page + sort kept, hidden-tab pause, no overlap, backoff + cap, stop on logout, escaping). Not yet exercised against a live MySQL server or a real browser |
+
 ---
 
 ## 21. Infrastructure & Deployment

routes/attendance.py

@@ -45,6 +45,99 @@ bp = Blueprint('attendance', __name__)
 UPPER_EMPLOYEE_ID_SQL = "UPPER(ad.employee_id)"
 
 
+def _attendance_filter_conditions(user_role, allowed_project_ids, allowed_location_names,
+                                  date_from, date_to, location_filter, employee_ids, project_filter):
+    """
+    Build the WHERE conditions + bound parameters for attendance_data queries.
+
+    Shared by the Attendance Report page and its live-update endpoint so both
+    always apply the same Project Manager scope and the same user filters.
+    Every value is a bound parameter. Returns (filter_conditions, query_params).
+    """
+    filter_conditions = []
+    query_params = {}
+
+    # ============================================================
+    # APPLY PROJECT MANAGER FILTERS TO SQL QUERY
+    # ============================================================
+    if user_role == 'project_manager':
+        # Filter by allowed projects
+        if allowed_project_ids:
+            project_placeholders = ','.join([f':project_{i}' for i in range(len(allowed_project_ids))])
+            filter_conditions.append(f"qc.project_id IN ({project_placeholders})")
+            for i, pid in enumerate(allowed_project_ids):
+                query_params[f'project_{i}'] = pid
+
+        # Filter by allowed locations
+        if allowed_location_names:
+            location_placeholders = ','.join([f':location_{i}' for i in range(len(allowed_location_names))])
+            filter_conditions.append(f"ad.location_name IN ({location_placeholders})")
+            for i, loc in enumerate(allowed_location_names):
+                query_params[f'location_{i}'] = loc
+    # ============================================================
+    # END: APPLY PROJECT MANAGER FILTERS
+    # ============================================================
+
+    # Apply user-selected filters
+    if date_from:
+        filter_conditions.append("ad.check_in_date >= :date_from")
+        query_params['date_from'] = date_from
+
+    if date_to:
+        filter_conditions.append("ad.check_in_date <= :date_to")
+        query_params['date_to'] = date_to
+
+    if location_filter:
+        # Exact match — dropdown value IS the exact location_name string
+        filter_conditions.append("ad.location_name = :location")
+        query_params['location'] = location_filter
+
+    if employee_ids:
+        # Expand each selected ID into every stored spelling so extra-work
+        # check-ins (SP / PW / PT) are included alongside regular records.
+        # Two branches: exact match on the raw column (uses the index), plus a
+        # REGEXP match that catches any separator style ("1234.PW", "1234-SP").
+        exact_variants, regex_patterns = expand_employee_id_filter(employee_ids)
+        # Never emit an empty IN () — fall back to the raw selection if expansion
+        # somehow produced nothing, so the filter can't degrade into "match all".
+        if not exact_variants:
+            exact_variants = list(employee_ids)
+
+        exact_placeholders = ', '.join([f':employee_{i}' for i in range(len(exact_variants))])
+        exact_condition = f"ad.employee_id IN ({exact_placeholders})"
+        for i, variant in enumerate(exact_variants):
+            query_params[f'employee_{i}'] = variant
+
+        if regex_patterns:
+            regex_clauses = ' OR '.join([
+                f"{UPPER_EMPLOYEE_ID_SQL} REGEXP :employee_re_{i}"
+                for i in range(len(regex_patterns))
+            ])
+            filter_conditions.append(f"({exact_condition} OR {regex_clauses})")
+            for i, pattern in enumerate(regex_patterns):
+                query_params[f'employee_re_{i}'] = pattern
+        else:
+            filter_conditions.append(exact_condition)
+
+    if project_filter:
+        # For standard QR records: match by the QR code's project_id directly.
+        # For dynamic QR records: the dynamic QR itself may not be in any project,
+        # but the employee-selected location corresponds to a standard QR in that
+        # project. Match those by checking if attendance_data.location_name
+        # appears in the locations of QR codes belonging to the selected project.
+        filter_conditions.append(
+            "(qc.project_id = :project OR "
+            "(ad.is_dynamic_qr = 1 AND ad.location_name IN ("
+            "  SELECT DISTINCT qc2.location FROM qr_codes qc2 "
+            "  WHERE qc2.project_id = :project AND qc2.qr_type = 'standard' "
+            "  AND qc2.location IS NOT NULL AND qc2.location != ''"
+            ")))"
+        )
+        query_params['project'] = project_filter
+
+    return filter_conditions, query_params
+
+
 
 @bp.route('/attendance', endpoint='attendance_report')
 @login_required
@@ -225,93 +318,22 @@ def attendance_report():
                 WHERE 1=1
             """
 
-        # Prepare filter conditions and parameters
+        # Prepare filter conditions and parameters — built by the helper shared
-        filter_conditions = []
+        # with the live-update endpoint, so the Project Manager scope and the
-        query_params = {}
+        # filters can never differ between the page and its live rows.
-
+        filter_conditions, query_params = _attendance_filter_conditions(
-        # ============================================================
+            user_role, allowed_project_ids, allowed_location_names,
-        # APPLY PROJECT MANAGER FILTERS TO SQL QUERY
+            date_from, date_to, location_filter, employee_ids, project_filter)
-        # ============================================================
-        if user_role == 'project_manager':
-            # Filter by allowed projects
-            if allowed_project_ids:
-                project_placeholders = ','.join([f':project_{i}' for i in range(len(allowed_project_ids))])
-                filter_conditions.append(f"qc.project_id IN ({project_placeholders})")
-                for i, pid in enumerate(allowed_project_ids):
-                    query_params[f'project_{i}'] = pid
-            
-            # Filter by allowed locations
-            if allowed_location_names:
-                location_placeholders = ','.join([f':location_{i}' for i in range(len(allowed_location_names))])
-                filter_conditions.append(f"ad.location_name IN ({location_placeholders})")
-                for i, loc in enumerate(allowed_location_names):
-                    query_params[f'location_{i}'] = loc
-        # ============================================================
-        # END: APPLY PROJECT MANAGER FILTERS
-        # ============================================================
-
-        # Apply user-selected filters
-        if date_from:
-            filter_conditions.append("ad.check_in_date >= :date_from")
-            query_params['date_from'] = date_from
-
-        if date_to:
-            filter_conditions.append("ad.check_in_date <= :date_to")
-            query_params['date_to'] = date_to
-
-        if location_filter:
-            # Exact match — dropdown value IS the exact location_name string
-            filter_conditions.append("ad.location_name = :location")
-            query_params['location'] = location_filter
 
         if employee_ids:
-            # Expand each selected ID into every stored spelling so extra-work
+            exact_variant_count = len([k for k in query_params
-            # check-ins (SP / PW / PT) are included alongside regular records.
+                                       if k.startswith('employee_') and not k.startswith('employee_re_')])
-            # Two branches: exact match on the raw column (uses the index), plus a
-            # REGEXP match that catches any separator style ("1234.PW", "1234-SP").
-            exact_variants, regex_patterns = expand_employee_id_filter(employee_ids)
-            # Never emit an empty IN () — fall back to the raw selection if expansion
-            # somehow produced nothing, so the filter can't degrade into "match all".
-            if not exact_variants:
-                exact_variants = list(employee_ids)
-
-            exact_placeholders = ', '.join([f':employee_{i}' for i in range(len(exact_variants))])
-            exact_condition = f"ad.employee_id IN ({exact_placeholders})"
-            for i, variant in enumerate(exact_variants):
-                query_params[f'employee_{i}'] = variant
-
-            if regex_patterns:
-                regex_clauses = ' OR '.join([
-                    f"{UPPER_EMPLOYEE_ID_SQL} REGEXP :employee_re_{i}"
-                    for i in range(len(regex_patterns))
-                ])
-                filter_conditions.append(f"({exact_condition} OR {regex_clauses})")
-                for i, pattern in enumerate(regex_patterns):
-                    query_params[f'employee_re_{i}'] = pattern
-            else:
-                filter_conditions.append(exact_condition)
             logger_handler.logger.info(
                 f"Attendance report filtered by employee IDs: {employee_ids} "
-                f"(matching {len(exact_variants)} ID variants incl. SP/PW/PT) "
+                f"(matching {exact_variant_count} ID variants incl. SP/PW/PT) "
                 f"by user {session.get('username', 'unknown')}"
             )
 
-        if project_filter:
-            # For standard QR records: match by the QR code's project_id directly.
-            # For dynamic QR records: the dynamic QR itself may not be in any project,
-            # but the employee-selected location corresponds to a standard QR in that
-            # project. Match those by checking if attendance_data.location_name
-            # appears in the locations of QR codes belonging to the selected project.
-            filter_conditions.append(
-                "(qc.project_id = :project OR "
-                "(ad.is_dynamic_qr = 1 AND ad.location_name IN ("
-                "  SELECT DISTINCT qc2.location FROM qr_codes qc2 "
-                "  WHERE qc2.project_id = :project AND qc2.qr_type = 'standard' "
-                "  AND qc2.location IS NOT NULL AND qc2.location != ''"
-                ")))"
-            )
-            query_params['project'] = project_filter
-
         # Combine query with filters
         if filter_conditions:
             base_query += " AND " + " AND ".join(filter_conditions)
@@ -320,6 +342,16 @@ def attendance_report():
         ATTENDANCE_PAGE_LIMIT = 1000
         base_query += f" ORDER BY ad.check_in_date DESC, ad.check_in_time DESC LIMIT {ATTENDANCE_PAGE_LIMIT + 1}"
 
+        # Starting cursor for the live table: the newest attendance id right now.
+        # Read BEFORE the report query, so a check-in saved in between is picked
+        # up by the first poll (the browser de-duplicates by id). Primary-key MAX.
+        try:
+            live_latest_id = int(db.session.execute(
+                text("SELECT COALESCE(MAX(id), 0) FROM attendance_data")).scalar() or 0)
+        except Exception as live_error:
+            logger_handler.logger.warning(f"Could not read latest attendance id for live updates: {live_error}")
+            live_latest_id = None  # page still renders; live updates stay off
+
         logger_handler.logger.debug(f"Executing attendance query with filters: {list(query_params.keys())}")
 
         # Execute query
@@ -556,6 +588,7 @@ def attendance_report():
                      attendance_records=processed_records,
                      records_truncated=records_truncated,
                      records_limit=ATTENDANCE_PAGE_LIMIT,
+                     live_latest_id=live_latest_id,
                      locations=locations,
                      projects=projects,
                      stats=stats,
@@ -587,6 +620,216 @@ def attendance_report():
         flash('Error loading attendance report. Please check the server logs for details.', 'error')
         return redirect(url_for('dashboard.dashboard'))
 
+# ============================================================
+# LIVE UPDATES for the Attendance Report table
+# ============================================================
+# The report page polls attendance_live_updates_api(). It is designed to cost
+# almost nothing: every poll is one primary-key MAX(id) lookup, and the filtered
+# query only runs when a newer record exists, scanning only ids above since_id.
+LIVE_UPDATE_MAX_RECORDS = 200
+# Cached once the column is confirmed; information_schema is not queried per poll
+_live_has_location_accuracy = None
+
+
+def _live_no_store_json(payload, status=200):
+    """JSON response that browsers and proxies must never cache."""
+    response = jsonify(payload)
+    response.status_code = status
+    response.headers['Cache-Control'] = 'no-store, no-cache, max-age=0, must-revalidate'
+    response.headers['Pragma'] = 'no-cache'
+    return response
+
+
+def _live_text(value):
+    """Text as Jinja renders it into the report (None renders as 'None')."""
+    return 'None' if value is None else str(value)
+
+
+def _live_format_time(check_in_time):
+    """The HH:MM text the report template renders for a time / timedelta / str."""
+    if not check_in_time:
+        return 'N/A'
+    if isinstance(check_in_time, str):
+        return check_in_time
+    if hasattr(check_in_time, 'strftime'):
+        return check_in_time.strftime('%H:%M')
+    if hasattr(check_in_time, 'total_seconds'):
+        total_seconds = int(check_in_time.total_seconds())
+        return f"{(total_seconds // 3600) % 24:02d}:{(total_seconds % 3600) // 60:02d}"
+    return str(check_in_time)
+
+
+def _live_record_payload(row, has_location_accuracy):
+    """
+    One attendance row in exactly the shape loadTableData() in
+    static/js/attendance_report.js reads from the server-rendered table, so a
+    live row sorts, filters, paginates and renders like the rows loaded with
+    the page. Mirrors the <tbody> markup of templates/attendance_report.html —
+    keep the two in sync.
+    """
+    (record_id, employee_id, check_in_date, check_in_time, location_name, location_event,
+     qr_address, checked_in_address, location_accuracy, gps_accuracy, device_info,
+     created_timestamp, updated_timestamp, employee_name, verification_required,
+     verification_status, is_dynamic_qr) = row
+
+    # Verification badge, as extractVerificationData() reads it
+    if verification_required and verification_status == 'pending':
+        verification = (True, 'pending')
+    elif verification_status in ('approved', 'rejected'):
+        verification = (True, verification_status)
+    else:
+        verification = (False, None)
+
+    # Accuracy number, as extractLocationAccuracy() parses the rendered badge text
+    if verification[0]:
+        accuracy = round(float(location_accuracy), 3) if location_accuracy is not None else None
+    elif has_location_accuracy and location_accuracy:
+        accuracy = round(float(location_accuracy), 3)
+    elif gps_accuracy:
+        accuracy = round(float(gps_accuracy), 1) * 0.000621371  # badge shows metres
+    else:
+        accuracy = None
+
+    if isinstance(check_in_date, str):
+        date_text = check_in_date
+    else:
+        date_text = check_in_date.strftime('%m/%d/%Y') if check_in_date else ''
+
+    checked_in_text = (checked_in_address or 'N/A')[:50]
+    if len(checked_in_address or '') > 50:
+        checked_in_text += '...'
+    device_text = device_info or ''
+    device_text = device_text[:20] + ('...' if len(device_text) > 20 else '')
+
+    return {
+        'id': str(record_id),
+        'employeeId': _live_text(employee_id).strip(),
+        'employeeName': (employee_name or 'Unknown Employee').strip(),
+        'location': ('(No location recorded)' if location_name == 'Dynamic'
+                     else _live_text(location_name)).strip(),
+        'event': _live_text(location_event).strip(),
+        'date': date_text.strip(),
+        'time': _live_format_time(check_in_time).strip(),
+        'qr_address': ((qr_address[:50] + ('...' if len(qr_address) > 50 else ''))
+                       if qr_address else 'N/A').strip(),
+        'checked_in_address': checked_in_text.strip(),
+        'location_accuracy': accuracy,
+        'accuracy_level': ('unknown' if accuracy is None
+                           else ('accurate' if accuracy < 0.3 else 'inaccurate')),
+        'device': device_text.strip(),
+        'isModified': bool(updated_timestamp and created_timestamp
+                           and updated_timestamp > created_timestamp),
+        'isDynamic': bool(is_dynamic_qr) or location_name == 'Dynamic',
+        'verification_required': verification[0],
+        'verification_status': verification[1],
+    }
+
+
+@bp.route('/api/attendance/live-updates', endpoint='attendance_live_updates_api')
+@login_required
+def attendance_live_updates_api():
+    """
+    New attendance records for the Attendance Report's live table.
+
+    GET params: since_id (newest attendance id the page already knows) plus the
+    page's own filters: date_from, date_to, location, employee, project.
+    Returns {success, latest_id, records, has_more}; the page sends latest_id
+    back as since_id on its next poll.
+    """
+    global _live_has_location_accuracy
+    try:
+        try:
+            since_id = int(request.args.get('since_id', ''))
+        except (TypeError, ValueError):
+            return _live_no_store_json({'success': False, 'message': 'since_id is required.'}, 400)
+
+        # Cheap path — taken by almost every poll: nothing newer than since_id
+        latest_id = int(db.session.execute(
+            text("SELECT COALESCE(MAX(id), 0) FROM attendance_data")).scalar() or 0)
+        empty = {'success': True, 'latest_id': max(latest_id, since_id), 'records': [], 'has_more': False}
+        if latest_id <= since_id:
+            return _live_no_store_json(empty)
+
+        # Same Project Manager scope as the report page
+        user_role = session.get('role')
+        allowed_project_ids, allowed_location_names = [], []
+        if user_role == 'project_manager':
+            user_id = session.get('user_id')
+            allowed_project_ids = [p.project_id for p in
+                                   UserProjectPermission.query.filter_by(user_id=user_id).all()]
+            allowed_location_names = [l.location_name for l in
+                                      UserLocationPermission.query.filter_by(user_id=user_id).all()]
+            if not allowed_project_ids and not allowed_location_names:
+                return _live_no_store_json(empty)
+
+        employee_filter = request.args.get('employee', '')
+        employee_ids = [e.strip() for e in employee_filter.split(',') if e.strip()]
+        filter_conditions, query_params = _attendance_filter_conditions(
+            user_role, allowed_project_ids, allowed_location_names,
+            request.args.get('date_from', ''), request.args.get('date_to', ''),
+            request.args.get('location', ''), employee_ids, request.args.get('project', ''))
+
+        # Only ids the page has not seen yet (a primary-key range scan)
+        filter_conditions = ["ad.id > :live_since_id", "ad.id <= :live_latest_id"] + filter_conditions
+        query_params['live_since_id'] = since_id
+        query_params['live_latest_id'] = latest_id
+
+        if _live_has_location_accuracy is None and check_location_accuracy_column_exists():
+            _live_has_location_accuracy = True
+        has_location_accuracy = bool(_live_has_location_accuracy)
+        accuracy_column = "ad.location_accuracy" if has_location_accuracy else "NULL"
+
+        # Same columns as the report query minus verification_photo (base64 image)
+        where_clause = " AND ".join(filter_conditions)
+        rows = db.session.execute(text(f"""
+            SELECT
+                ad.id,
+                ad.employee_id,
+                ad.check_in_date,
+                ad.check_in_time,
+                ad.location_name,
+                qc.location_event,
+                COALESCE(ad.qr_address, qc.location_address) AS qr_address,
+                ad.address AS checked_in_address,
+                {accuracy_column} AS location_accuracy,
+                ad.accuracy AS gps_accuracy,
+                ad.device_info,
+                ad.created_timestamp,
+                ad.updated_timestamp,
+                CONCAT(e.firstName, ' ', e.lastName) AS employee_name,
+                ad.verification_required,
+                ad.verification_status,
+                COALESCE(ad.is_dynamic_qr, 0) AS is_dynamic_qr
+            FROM attendance_data ad
+            LEFT JOIN qr_codes qc ON ad.qr_code_id = qc.id
+            LEFT JOIN employee e ON CAST(ad.employee_id AS UNSIGNED) = e.id
+            WHERE {where_clause}
+            ORDER BY ad.id ASC
+            LIMIT {LIVE_UPDATE_MAX_RECORDS + 1}
+        """), query_params).fetchall()
+
+        has_more = len(rows) > LIVE_UPDATE_MAX_RECORDS
+        rows = rows[:LIVE_UPDATE_MAX_RECORDS]
+        # Capped batch: continue after the last row sent. Otherwise every id up
+        # to latest_id has been examined.
+        cursor = rows[-1][0] if has_more else latest_id
+
+        records = []
+        for row in rows:
+            try:
+                records.append(_live_record_payload(row, has_location_accuracy))
+            except Exception as rec_error:
+                logger_handler.logger.warning(f"Live updates: skipped attendance record {row[0]}: {rec_error}")
+
+        return _live_no_store_json({'success': True, 'latest_id': cursor,
+                                    'records': records, 'has_more': has_more})
+
+    except Exception as e:
+        db.session.rollback()
+        logger_handler.logger.error(f"Error loading attendance live updates: {e}", exc_info=True)
+        return _live_no_store_json({'success': False, 'message': 'Could not load new records.'}, 500)
+
+
 @bp.route('/api/time-attendance/locations', endpoint='time_attendance_locations_api')
 @login_required
 def time_attendance_locations_api():

static/js/attendance_report.js

@@ -24,6 +24,7 @@ document.addEventListener("DOMContentLoaded", function () {
   initializeCharts();
   setupEventListeners();
   initializeDateRangeFilters();
+  initializeLiveUpdates();
 });
 
 function initializeReport() {
@@ -164,6 +165,20 @@ function stripLeadingZeros(value) {
 }
 
 function applyFilters() {
+  filteredData = filterRecords();
+
+  currentPage = 1;
+  updateTable();
+  updatePagination();
+  updateFilterStats();
+}
+
+/**
+ * Records matching the on-page search / location / employee filters.
+ * Shared by applyFilters() and refreshTableKeepingView() (live updates), which
+ * must keep the user's current page and sort instead of jumping to page 1.
+ */
+function filterRecords() {
   const searchTerm =
     document.getElementById("searchInput")?.value.toLowerCase() || "";
   const locationFilter = document.getElementById("location")?.value || "";
@@ -184,7 +199,7 @@ function applyFilters() {
         .map(parseEmployeeIdWorkType)
     : [];
 
-  filteredData = attendanceData.filter((record) => {
+  return attendanceData.filter((record) => {
     const matchesSearch =
       !searchTerm ||
       record.employeeId.toLowerCase().includes(searchTerm) ||
@@ -209,11 +224,6 @@ function applyFilters() {
 
     return matchesSearch && matchesLocation && matchesEmployee;
   });
-
-  currentPage = 1;
-  updateTable();
-  updatePagination();
-  updateFilterStats();
 }
 
 function sortTable(columnIndex) {
@@ -687,6 +697,10 @@ function createTableRow(record, displayIndex) {
   if (record.isDynamic) {
       row.classList.add('dynamic-qr-record');
   }
+  // Brief highlight for rows delivered by live updates
+  if (record.isLiveNew) {
+      row.classList.add('live-new-record');
+  }
 
   // Debug logging for first few records
   if (displayIndex <= 3) {
@@ -704,7 +718,7 @@ function createTableRow(record, displayIndex) {
   
   if (record.verification_required && record.verification_status === 'pending') {
     // Show Review Needed badge for pending verification - LINK to review page
-    locationAccuracyBadge = `<a href="/verification-review/${record.id}" 
+    locationAccuracyBadge = `<a href="/verification-review/${encodeURIComponent(record.id)}" 
             class="location-accuracy-badge badge-review-needed" 
             style="cursor: pointer; text-decoration: none;"
             title="Click to review verification photo - Distance: ${record.location_accuracy ? record.location_accuracy.toFixed(3) : 'N/A'} miles">
@@ -782,12 +796,8 @@ function createTableRow(record, displayIndex) {
       addressDisplayHTML = `
         <i class="${addressIcon}" style="color: #059669; margin-right: 4px;" 
            title="High accuracy - showing QR location"></i>
-        <span title="${addressTitle}" class="${addressClass}">
+        <span title="${escapeHtml(addressTitle)}" class="${addressClass}">
-          ${
+          ${escapeHtml(truncateText(addressToShow, 45))}
-            addressToShow.length > 45
-              ? addressToShow.substring(0, 45) + "..."
-              : addressToShow
-          }
         </span>
       `;
     } else {
@@ -799,14 +809,10 @@ function createTableRow(record, displayIndex) {
       addressDisplayHTML = `
         <i class="fas fa-exclamation-triangle" style="color: #f59e0b; margin-right: 4px;" 
            title="Lower accuracy - showing actual check-in location"></i>
-        <span title="${addressTitle} (Accuracy: ${accuracy.toFixed(
+        <span title="${escapeHtml(addressTitle)} (Accuracy: ${accuracy.toFixed(
         3
       )} mi)" class="${addressClass}">
-          ${
+          ${escapeHtml(truncateText(addressToShow, 45))}
-            addressToShow.length > 45
-              ? addressToShow.substring(0, 45) + "..."
-              : addressToShow
-          }
         </span>
       `;
     }
@@ -820,12 +826,8 @@ function createTableRow(record, displayIndex) {
 
     addressDisplayHTML = `
       <i class="${addressIcon}"></i>
-      <span title="${addressTitle}" class="${addressClass}">
+      <span title="${escapeHtml(addressTitle)}" class="${addressClass}">
-        ${
+        ${escapeHtml(truncateText(addressToShow, 45))}
-          addressToShow.length > 45
-            ? addressToShow.substring(0, 45) + "..."
-            : addressToShow
-        }
       </span>
     `;
   }
@@ -834,45 +836,41 @@ function createTableRow(record, displayIndex) {
         <td>${displayIndex}</td>
         <td>
             <div class="employee-info">
-                <span class="employee-id">${record.employeeId}</span>
+                <span class="employee-id">${escapeHtml(record.employeeId)}</span>
             </div>
         </td>
         <td>
             <div class="employee-name">
                 <i class="fas fa-user"></i>
-                <span>${record.employeeName || 'Unknown'}</span>
+                <span>${escapeHtml(record.employeeName || 'Unknown')}</span>
             </div>
         </td>
         <td>
             <div class="location-info">
                 <i class="fas fa-map-marker-alt"></i>
-                ${record.location}
+                ${escapeHtml(record.location)}
             </div>
         </td>
         <td>
             <div class="event-info">
-                ${record.event}
+                ${escapeHtml(record.event)}
             </div>
         </td>
         <td>
             <div class="date-info">
-                ${record.date}
+                ${escapeHtml(record.date)}
             </div>
         </td>
         <td>
             <div class="time-info">
-                ${record.time}
+                ${escapeHtml(record.time)}
             </div>
         </td>
         <td>
             <div class="address-info qr-address">
                 <i class="fas fa-qrcode" style="color: #6366f1; margin-right: 4px;" title="QR Code Address (Fixed)"></i>
-                <span title="QR Address: ${record.qr_address}">
+                <span title="QR Address: ${escapeHtml(record.qr_address)}">
-                    ${
+                    ${escapeHtml(truncateText(record.qr_address, 50))}
-                      record.qr_address.length > 50
-                        ? record.qr_address.substring(0, 50) + "..."
-                        : record.qr_address
-                    }
                 </span>
             </div>
         </td>
@@ -889,12 +887,8 @@ function createTableRow(record, displayIndex) {
         <td>
             <div class="device-info">
                 <i class="fas fa-mobile-alt"></i>
-                <span title="${record.device}">
+                <span title="${escapeHtml(record.device)}">
-                    ${
+                    ${escapeHtml(truncateText(record.device, 20))}
-                      record.device.length > 20
-                        ? record.device.substring(0, 20) + "..."
-                        : record.device
-                    }
                 </span>
             </div>
         </td>
@@ -902,7 +896,7 @@ function createTableRow(record, displayIndex) {
             <div class="record-actions">
                 ${
                   record.verification_required && record.verification_status === 'pending'
-                    ? `<a href="/verification-review/${record.id}" 
+                    ? `<a href="/verification-review/${encodeURIComponent(record.id)}" 
                               class="action-btn btn-review"
                               title="Review Verification Photo">
                           <i class="fas fa-camera"></i>
@@ -912,12 +906,12 @@ function createTableRow(record, displayIndex) {
                 ${
                   hasEditPermission
                     ? `
-                <button onclick="editRecord('${record.id}')" 
+                <button onclick="editRecord('${escapeHtml(escapeJsString(record.id))}')"
                         class="action-btn btn-edit"
                         title="Edit Record">
                     <i class="fas fa-edit"></i>
                 </button>
-                <button onclick="deleteRecord('${record.id}', '${record.employeeId}')" 
+                <button onclick="deleteRecord('${escapeHtml(escapeJsString(record.id))}', '${escapeHtml(escapeJsString(record.employeeId))}')"
                         class="action-btn btn-delete"
                         title="Delete Record">
                     <i class="fas fa-trash"></i>
@@ -961,6 +955,18 @@ function sortTable(columnIndex) {
     sortDirection = "asc";
   }
 
+  sortFilteredData();
+
+  updateTable();
+  updateSortIndicators(columnIndex);
+}
+
+/**
+ * Sort filteredData by the current sortColumn / sortDirection without toggling
+ * the direction — reused when live updates re-apply the user's chosen sort.
+ */
+function sortFilteredData() {
+  const columnIndex = sortColumn;
   const sortKey = getSortKey(columnIndex);
 
   filteredData.sort((a, b) => {
@@ -993,9 +999,6 @@ function sortTable(columnIndex) {
 
     return sortDirection === "asc" ? result : -result;
   });
-
-  updateTable();
-  updateSortIndicators(columnIndex);
 }
 
 // Enhanced statistics display for location accuracy
@@ -1241,3 +1244,268 @@ document.addEventListener("DOMContentLoaded", function () {
 
   console.log("Enhanced export functionality initialized");
 });
+
+// ============================================================
+// HTML SAFETY HELPERS
+// ============================================================
+// Row values are plain text (read with textContent, or JSON from the server)
+// but createTableRow() builds markup with innerHTML. Device and address text
+// come from the public check-in page, so every value is escaped first.
+function escapeHtml(value) {
+  return String(value === null || value === undefined ? "" : value)
+    .replace(/&/g, "&amp;")
+    .replace(/</g, "&lt;")
+    .replace(/>/g, "&gt;")
+    .replace(/"/g, "&quot;")
+    .replace(/'/g, "&#39;");
+}
+
+// For a value placed inside a single-quoted JS string in an onclick attribute:
+// escape for JS here, then escapeHtml() for the attribute.
+function escapeJsString(value) {
+  return String(value === null || value === undefined ? "" : value)
+    .replace(/\\/g, "\\\\")
+    .replace(/'/g, "\\'")
+    .replace(/[\r\n]+/g, " ");
+}
+
+// Truncate BEFORE escaping, so an entity like &amp; is never cut in half
+function truncateText(value, maxLength) {
+  const text = String(value === null || value === undefined ? "" : value);
+  return text.length > maxLength ? text.substring(0, maxLength) + "..." : text;
+}
+
+// ============================================================
+// LIVE UPDATES — new check-ins appear without reloading the page
+// ============================================================
+// Polls GET /api/attendance/live-updates (routes/attendance.py), which answers
+// from a primary-key MAX(id) lookup when nothing is new. To keep the load
+// negligible the poll:
+//   - runs every LIVE_POLL_INTERVAL_MS and never overlaps itself
+//   - pauses while the tab is hidden, and checks once as soon as it is shown
+//   - backs off exponentially after errors, up to LIVE_MAX_BACKOFF_MS
+//   - stops for good when the session has ended (redirect / 401 / 403)
+// New rows keep the user's current page, sort and search.
+const LIVE_POLL_INTERVAL_MS = 20000;
+const LIVE_MAX_BACKOFF_MS = 5 * 60 * 1000;
+const LIVE_HIGHLIGHT_MS = 8000;
+const LIVE_MAX_RECORDS = 5000; // rows kept in memory on a page left open for days
+const LIVE_FILTER_PARAMS = ["date_from", "date_to", "location", "employee", "project"];
+
+const liveUpdates = {
+  enabled: false,
+  sinceId: 0,
+  timer: null,
+  inFlight: false,
+  failures: 0,
+  stopped: false,
+  newSinceLoad: 0,
+  statusEl: null,
+};
+
+function initializeLiveUpdates() {
+  const container = document.getElementById("attendanceReportContainer");
+  const sinceId = container ? parseInt(container.dataset.liveSinceId, 10) : NaN;
+  if (isNaN(sinceId)) {
+    return; // rendered without a cursor (no access / id lookup failed): stay static
+  }
+
+  liveUpdates.enabled = true;
+  liveUpdates.sinceId = sinceId;
+  createLiveStatusIndicator();
+  setLiveStatus("live");
+
+  document.addEventListener("visibilitychange", function () {
+    if (!liveUpdates.enabled || liveUpdates.stopped) return;
+    if (document.hidden) {
+      clearTimeout(liveUpdates.timer); // no requests from background tabs
+    } else {
+      pollLiveUpdates(); // catch up straight away when the tab is shown again
+    }
+  });
+
+  scheduleLivePoll(LIVE_POLL_INTERVAL_MS);
+}
+
+function scheduleLivePoll(delayMs) {
+  clearTimeout(liveUpdates.timer);
+  if (liveUpdates.stopped || document.hidden) return;
+  liveUpdates.timer = setTimeout(pollLiveUpdates, delayMs);
+}
+
+function pollLiveUpdates() {
+  if (!liveUpdates.enabled || liveUpdates.stopped || liveUpdates.inFlight || document.hidden) {
+    return;
+  }
+  clearTimeout(liveUpdates.timer);
+  liveUpdates.inFlight = true;
+
+  // The server applies the same filters the page was loaded with
+  const pageParams = new URLSearchParams(window.location.search);
+  const params = new URLSearchParams();
+  LIVE_FILTER_PARAMS.forEach(function (name) {
+    const value = pageParams.get(name);
+    if (value) params.set(name, value);
+  });
+  params.set("since_id", String(liveUpdates.sinceId));
+
+  let nextDelay = LIVE_POLL_INTERVAL_MS;
+
+  fetch("/api/attendance/live-updates?" + params.toString(), {
+    cache: "no-store",
+    credentials: "same-origin",
+    headers: { "X-Requested-With": "XMLHttpRequest", Accept: "application/json" },
+  })
+    .then(function (response) {
+      const contentType = response.headers.get("Content-Type") || "";
+      if (
+        response.redirected ||
+        response.status === 401 ||
+        response.status === 403 ||
+        (response.ok && contentType.indexOf("application/json") === -1)
+      ) {
+        // login_required redirected to the login page: the session is over
+        stopLiveUpdates();
+        return null;
+      }
+      if (!response.ok) {
+        throw new Error("HTTP " + response.status);
+      }
+      return response.json();
+    })
+    .then(function (data) {
+      if (!data) return;
+      if (!data.success) {
+        throw new Error(data.message || "live update failed");
+      }
+      liveUpdates.failures = 0;
+      if (Array.isArray(data.records) && data.records.length > 0) {
+        insertLiveRecords(data.records);
+      }
+      const latestId = parseInt(data.latest_id, 10);
+      if (!isNaN(latestId) && latestId > liveUpdates.sinceId) {
+        liveUpdates.sinceId = latestId;
+      }
+      if (data.has_more) {
+        nextDelay = 1000; // a large burst arrived: fetch the next batch promptly
+      }
+      if (!liveUpdates.stopped) setLiveStatus("live");
+    })
+    .catch(function (error) {
+      liveUpdates.failures += 1;
+      nextDelay = Math.min(
+        LIVE_POLL_INTERVAL_MS * Math.pow(2, liveUpdates.failures),
+        LIVE_MAX_BACKOFF_MS
+      );
+      console.log(
+        "Live updates: check failed (" + error.message + "), retrying in " +
+          Math.round(nextDelay / 1000) + "s"
+      );
+      setLiveStatus("retry");
+    })
+    .then(function () {
+      liveUpdates.inFlight = false;
+      if (!liveUpdates.stopped) scheduleLivePoll(nextDelay);
+    });
+}
+
+function stopLiveUpdates() {
+  liveUpdates.stopped = true;
+  clearTimeout(liveUpdates.timer);
+  setLiveStatus("stopped");
+}
+
+function insertLiveRecords(records) {
+  // The page showed "No Attendance Records Found" (no table to add to):
+  // reload once, keeping the filters, so the table renders normally.
+  if (!document.getElementById("attendanceTable")) {
+    stopLiveUpdates();
+    if (typeof refreshReport === "function") {
+      refreshReport();
+    } else {
+      window.location.reload();
+    }
+    return;
+  }
+
+  const knownIds = new Set(attendanceData.map(function (r) { return String(r.id); }));
+  const fresh = records
+    .filter(function (r) { return !knownIds.has(String(r.id)); })
+    .sort(function (a, b) { return Number(b.id) - Number(a.id); }); // newest first
+  if (fresh.length === 0) return;
+
+  fresh.forEach(function (r) { r.isLiveNew = true; });
+  attendanceData = fresh.concat(attendanceData);
+  if (attendanceData.length > LIVE_MAX_RECORDS) {
+    attendanceData.length = LIVE_MAX_RECORDS;
+  }
+  attendanceData.forEach(function (r, i) { r.index = i + 1; });
+
+  liveUpdates.newSinceLoad += fresh.length;
+  refreshTableKeepingView();
+
+  // Rows re-rendered after this (page change, sort) no longer highlight
+  setTimeout(function () {
+    fresh.forEach(function (r) { r.isLiveNew = false; });
+  }, LIVE_HIGHLIGHT_MS);
+}
+
+// Re-render after new rows arrive WITHOUT resetting the user's view:
+// same search/filters, same sort, same page (clamped if it no longer exists).
+function refreshTableKeepingView() {
+  filteredData = filterRecords();
+  if (sortColumn !== -1) {
+    sortFilteredData();
+  }
+  if (entriesPerPage !== "all") {
+    const totalPages = Math.max(1, Math.ceil(filteredData.length / entriesPerPage));
+    if (currentPage > totalPages) currentPage = totalPages;
+  }
+  updateTable();
+  updatePagination();
+}
+
+function createLiveStatusIndicator() {
+  const heading = document.querySelector(".attendance-table-section .table-header h3");
+  if (!heading) return;
+
+  const status = document.createElement("span");
+  status.id = "liveUpdateStatus";
+  status.className = "live-status";
+
+  const dot = document.createElement("span");
+  dot.className = "live-dot";
+  const text = document.createElement("span");
+  text.className = "live-text";
+
+  status.appendChild(dot);
+  status.appendChild(text);
+  heading.appendChild(status);
+  liveUpdates.statusEl = status;
+}
+
+function setLiveStatus(state) {
+  const status = liveUpdates.statusEl;
+  if (!status) return;
+  const text = status.querySelector(".live-text");
+
+  status.classList.remove("live-status--retry", "live-status--stopped");
+  if (state === "retry") {
+    status.classList.add("live-status--retry");
+    text.textContent = "Reconnecting…";
+    status.title = "Could not check for new records. Retrying automatically.";
+  } else if (state === "stopped") {
+    status.classList.add("live-status--stopped");
+    text.textContent = "Live updates paused";
+    status.title = "Live updates stopped. Refresh the page to resume.";
+  } else {
+    text.textContent = liveUpdates.newSinceLoad > 0
+      ? "Live · " + liveUpdates.newSinceLoad + " new"
+      : "Live";
+    status.title =
+      "New check-ins appear automatically (checked every " +
+      LIVE_POLL_INTERVAL_MS / 1000 +
+      " seconds, paused while this tab is hidden). Last checked " +
+      new Date().toLocaleTimeString() + ".";
+  }
+}

templates/attendance_report.html

@@ -14,6 +14,65 @@ tr.dynamic-qr-record {
 tr.dynamic-qr-record:hover {
     background-color: rgba(59, 91, 219, 0.07) !important;
 }
+
+/* Live updates: status pill in the table header + highlight for rows that
+   arrived while the page was open (see LIVE UPDATES in attendance_report.js) */
+.live-status {
+    display: inline-flex;
+    align-items: center;
+    gap: 0.4rem;
+    margin-left: 0.75rem;
+    padding: 0.15rem 0.6rem;
+    border-radius: 999px;
+    font-size: 0.75rem;
+    font-weight: 600;
+    vertical-align: middle;
+    background: #dcfce7;
+    color: #166534;
+    cursor: default;
+}
+.live-status .live-dot {
+    width: 0.5rem;
+    height: 0.5rem;
+    border-radius: 50%;
+    background: #16a34a;
+    animation: live-pulse 2s ease-in-out infinite;
+}
+.live-status.live-status--retry {
+    background: #fef3c7;
+    color: #92400e;
+}
+.live-status.live-status--retry .live-dot {
+    background: #f59e0b;
+}
+.live-status.live-status--stopped {
+    background: #f1f5f9;
+    color: #475569;
+}
+.live-status.live-status--stopped .live-dot {
+    background: #94a3b8;
+    animation: none;
+}
+@keyframes live-pulse {
+    0%, 100% { opacity: 1; }
+    50% { opacity: 0.35; }
+}
+tr.live-new-record > td {
+    animation: live-row-in 6s ease-out;
+}
+@keyframes live-row-in {
+    0% { background-color: #bbf7d0; }
+    100% { background-color: transparent; }
+}
+@media (prefers-reduced-motion: reduce) {
+    .live-status .live-dot,
+    tr.live-new-record > td {
+        animation: none;
+    }
+    tr.live-new-record > td {
+        background-color: #f0fdf4;
+    }
+}
 </style>
 <!-- Fullscreen CSS -->
 <link rel="stylesheet" href="{{ url_for('static', filename='css/attendance_fullscreen.css') }}">
@@ -30,7 +89,7 @@ tr.dynamic-qr-record:hover {
 </button>
 
 <!-- Attendance Report Container with Fullscreen Support -->
-<div id="attendanceReportContainer" class="attendance-page" data-user-role="{{ session.role }}">
+<div id="attendanceReportContainer" class="attendance-page" data-user-role="{{ session.role }}"{% if live_latest_id is defined and live_latest_id is not none %} data-live-since-id="{{ live_latest_id }}"{% endif %}>
     <!-- Header Section -->
     <div class="attendance-header">
         <div class="header-content">