Sep 16 - Optimize code, part 2

2026-09-16 14:31:41 -04:00
parent 2c5627354e
commit 13b56fb1d1
9 changed files with 632 additions and 140 deletions
@@ -210,6 +210,18 @@ is the reference for who may use what; routes enforce the same rules:
 - New routes in a gated blueprint are covered automatically; new routes elsewhere need an
  explicit decorator.
 **Project Manager data scoping (Set 27)** — role gates decide which *pages* a PM may open;
 `load_project_manager_scope()` (`utils/helpers.py`) decides which *rows* they see. It returns
 `(is_project_manager, allowed_project_ids, allowed_location_names)` and **fails closed** (a PM with
 no assignments, or a permission lookup error, sees nothing). Applied in:
 - `routes/attendance.py`: the report, live updates, `attendance_locations_api`,
  `time_attendance_locations_api`, `search_employees_api`, `get_project_locations_api`
 - `routes/dashboard.py`: dashboard QR list + project list, `project_qr_codes` (403-style redirect
  for someone else's project), `dashboard_stats_api`, `dashboard_realtime_api`
 - A PM scoped only by **locations** is resolved to the projects behind those locations when
  employee names are searched, so their filter still works without exposing other projects.
 - **Any new endpoint returning attendance, employee, QR or project rows must call it.**
 ---
 ## 5. Application Factory & Initialization Order
@@ -756,8 +768,19 @@ within the interval. This is pre-existing behaviour for suffixed IDs.
 ### Special Handling
 - `Recorded Address`: read via openpyxl directly (not pandas) to preserve HYPERLINK formulas
 - Duplicate detection: hash of `employee_id + date + time + action_description`
 - `_clean_employee_id()` normalises IDs to the stored form (Set 26): `1234.0` → `1234`,
  `1759.PW` / `1759 - PW` / `PW.1759` → `1759PW`. Zero padding is **preserved** (`01234`) and a
  real fraction (`1234.5`) is **never truncated** — both would change the duplicate hash of rows
  already imported, or silently move hours to another employee
 - Import tracked by `import_batch_id` (UUID)
 - `self.db` not `db` — in `TimeAttendanceImportService`, always access SQLAlchemy via `self.db`
 - **Dates go through `_parse_date_field()` (Set 27) — never `pd.to_datetime()` directly.** It handles
  real date cells, Excel serial numbers (a General-formatted column: `45123` → 2023-07-16, which
  `pd.to_datetime()` read as 1970-01-01), and text dates **month-first** (US time clocks), falling
  back to day-first only when the first number cannot be a month (`13/04/2026`). Unparseable dates
  fail the row instead of importing a wrong one
 - **A failed import removes its own rows** (`_rollback_partial_batch()`): the loop commits every 50
  records, so a failure part-way used to leave a partial batch with nothing marking it incomplete
 - Validation `except` blocks must not silently swallow exceptions (fail-open prevention)
 ---
@@ -777,8 +800,15 @@ within the interval. This is pre-existing behaviour for suffixed IDs.
 - **SP** = Special Project, **PW** = Periodic Work, **PT** = Project Team (Part-Time),
  **C** = Covering (added Sept 2026)
 - Parsed from `employee_id` via `parse_employee_id_for_work_type()`
- Suffix and prefix forms, with or without a separator: `1234SP`, `1234 SP`, `SP1234`,
+- Suffix and prefix forms, separated by **any run of non-alphanumeric characters, or nothing**:
-  `1234.PW`. Two-letter codes are matched **before** the single-letter `C`
+  `1234SP`, `1234 SP`, `1234.PW`, `1234-PT`, `1234 . C`, `SP1234`, `SP 1234`, `PW.1234`.
  Two-letter codes are matched **before** the single-letter `C`
 - The separator class is `[^0-9A-Z]*` in all three parsers — `parse_employee_id_for_work_type()`,
  `build_employee_id_regex()` (SQL filters) and `parseEmployeeIdWorkType()` (report JS). Keep them
  in sync: until Sept 16 2026 the Python one accepted only a space, so `1759.PW` was counted as a
  separate Regular employee in the exports while the report showed it as PW (Set 26)
 - **Not** work types: `1234.5` (no code) and `1234SPX` (code runs into a word) — both stay regular
  with the ID unchanged
 - Like SP/PW/PT, `C` hours are excluded from the 40-hour overtime rule
 - **Codes are declared in four places — keep them in sync:**
  | File | Symbol |
@@ -791,6 +821,26 @@ within the interval. This is pre-existing behaviour for suffixed IDs.
  aggregation dicts are keyed `regular/SP/PW/PT`, so adding `C` to its parser alone
  would raise `KeyError`
 ### Overtime — 40-Hour Rule (Sept 16, 2026, confirmed by the user)
 **SP / PW / PT / C hours are paid but never build toward overtime.** Both exports compute the
 weekly Regular and OT columns from regular hours only:
 ```python
 week_regular  = min(weekly_regular_hours, 40.0)   # NOT weekly_total_hours
 week_overtime = max(0, weekly_regular_hours - 40.0)
 ```
 - `weekly_regular_hours` accumulates `_qtr()` of each day's regular-only pair hours;
  `_pair_is_regular(in, out)` decides, preferring the **OUT** record's work type (a Regular IN
  paired with an SP OUT counts as SP, mirroring `effective_work_type`).
 - **Column H (Weekly Total) still shows every hour worked**, so H ≠ Regular + OT whenever the
  employee has special hours — the SP/PW/PT/C and Regular rows below break that down.
 - Example: 36 h Regular + 8 h SP → H 44, Regular 36, **OT 0** (before Sept 16: OT 4).
 - **Export by Building applies the same rule per BUILDING**, so an employee with 30 h at two
  buildings shows 0 OT in each block while the main export shows 20 h. A note row under the date
  range says so. Treat that sheet as review-only; pay from the main export (§20 Set 25).
 ### Overnight Shift Handling
 **Rule 1 — Sort key:** early-morning OUTs (`hour <= 3`) use `_overnight_aware_sort_key()` which adds 86400 seconds — pushes them past midnight so they sort after same-day evening INs.
@@ -1053,6 +1103,12 @@ new_name = qr_code.name  # always use existing name, never request.form['name']
 ### Timestamp Convention
 Use `datetime.now()` (local time) throughout — **not** `datetime.utcnow()`.
 Check-in dates/times are stored local, so anything compared against them must be local too.
 Fixed in Set 27: `routes/attendance_edit.py` (record timestamps + audit note), `routes/dashboard.py`
 ("today" / "last 30 days" stats, which rolled over mid-evening) and `app.py` (`CURRENT_YEAR`).
 Still UTC on purpose or harmlessly: `users.last_login_date`, security-middleware event stamps,
 `qr_codes.coordinates_updated_date`, import batch `import_date`.
 ### Context Safety
 - Use `has_request_context()` (not `if not request:`) to check Flask request context
 - Capture `current_app._get_current_object()` in route body, not inside lazy generators
@@ -1333,6 +1389,34 @@ it (the workers share no pub/sub).
 | `extensions.py` | `logger_handler` is now a proxy (see §14). `utils/helpers.py` imports it at module import time, which happens BEFORE `init_logger()`, so it was permanently `None`: the Set 23 role check logged a warning when denying access and raised `AttributeError: 'NoneType' object has no attribute 'logger'` → 500 for a project manager opening `/time-attendance` or editing a QR code. The same latent bug affected `generate_qr_code()` logging in `utils/helpers.py` and every call in `utils/template_helpers.py` / `utils/geocoding.py` |
 | — | The step-1 tests had stubbed a working logger into `extensions`, which hid it. They now import the real module and run the role checks BEFORE `init_logger()`, exactly like a gunicorn worker |
 ### Set 25 — Overtime Counted SP/PW/PT/C Hours (Sept 16, 2026)
 | File | Fix |
 |---|---|
 | `routes/time_attendance_export.py` | Both exports added EVERY paired hour to the weekly total that overtime is calculated from, so 36 h Regular + 8 h SP produced 4 h OT. Added `_pair_is_regular()` and a `weekly_regular_hours` counter; the weekly-boundary and final weekly rows now take Regular/OT from it (§13). Column H is unchanged (all hours worked) |
 | `routes/time_attendance_export.py` | Export by Building: a note row under the date range states that its Weekly Total / Regular / OT are **per building** and exclude SP/PW/PT/C — the user was unsure whether that sheet is used for pay, so it is marked rather than changed. Shifts every Sheet0 row down by one (the Filtered Report is built from row bookkeeping, so it follows) |
 | — | Verified by running both real export functions over the same fixture week, before (HEAD) vs after: 36 h Reg + 8 h SP → OT 4 → 0; 40 h Reg + 4 h C → 4 → 0; Regular-IN/SP-OUT cross-type pair → 4 → 0; 44 h plain Regular unchanged at 4; two-building employee unchanged (20 h in the main export, 0 per building block); punch rows identical; SP/C summary rows unchanged |
 | — | **Payroll impact:** weeks where an employee had both special-type hours and 40+ total hours will now show less overtime than the same export produced before |
 ### Set 26 — Work-Type IDs With a Separator (`1759.PW`) (Sept 16, 2026)
 | File | Fix |
 |---|---|
 | `working_hours_calculator.py` | `parse_employee_id_for_work_type()` accepted only a space (`\s*`), so `1759.PW` — a spelling that really exists in imported data — parsed as base `1759.PW` / regular. The exports then showed a separate REGULAR employee "1759.PW" while the report and SQL filters (which use `[^0-9A-Z]*`) treated it as PW hours for 1759. Separator class aligned with the other two parsers (§13) |
 | `time_attendance_import_service.py` | `_clean_employee_id()` now stores work-type IDs canonically (`1759.PW` → `1759PW`), keeps zero padding, and no longer truncates `1234.5` to `1234` (silent data loss) |
 | — | Verified: 17 ID spellings through the parser (before vs after), agreement with the report's JS parser, 11 cleaner cases (incl. the two old bugs), and an end-to-end export where a 7 h `1759.PW` shift now appears as PW hours for employee 1759 (39 h worked / 32 regular / 0 OT) instead of its own block. Overtime (24) and step-1 (71) suites still pass |
 | — | **No data migration:** existing rows keep their stored spelling and are read correctly now. Exports for past weeks will move those hours from "Employee 1759.PW" Regular into 1759's PW row — and out of the overtime base (Set 25) |
 ### Set 27 — Import Dates, PM Data Scoping, Manual Entry (Sept 16, 2026)
 | File | Fix |
 |---|---|
 | `time_attendance_import_service.py` | `_parse_date_field()` replaces `pd.to_datetime()` at all four call sites: Excel serial numbers imported as **1970-01-01**, and `03/04/2026` was read month-first with no rule written down (§12) |
 | `time_attendance_import_service.py` | `_rollback_partial_batch()` — a mid-import failure left a partial batch (commits every 50 rows); the batch is now deleted and reported, so an import is all-or-nothing |
 | `utils/helpers.py` | `load_project_manager_scope()` — one fail-closed source for a PM's projects/locations (§4) |
 | `routes/attendance.py` | Scope applied to `attendance_locations_api`, `time_attendance_locations_api`, `search_employees_api` (location-only PMs resolved to their projects), `get_project_locations_api` |
 | `routes/dashboard.py` | Scope applied to the dashboard QR + project lists, `project_qr_codes` (redirect for another project), `dashboard_stats_api`, `dashboard_realtime_api`. "Today" now uses local time |
 | `routes/attendance_edit.py` | `_normalize_manual_employee_id()` — manual add did `int(employee_id)`, so `1234SP` could not be entered at all, and edit accepted any text (pseudo-employees in the exports). Both now accept 1–4 digits with an optional work type and store it canonically; timestamps local |
 | `app.py` | `CURRENT_YEAR` from local time |
 | — | Verified offline (42 checks): 16 date cases + 6 rejections, partial-batch rollback against a fake session, PM scope loading, scoped vs unscoped SQL for both locations APIs (values bound as parameters), empty result for an unassigned PM, 9 manual-ID cases, and no `utcnow()` left in the two route files. Suites still green: 71 (step 1), 24 (overtime), 38 (work-type IDs). Not run against MySQL or a browser |
 ---
 ## 21. Infrastructure & Deployment
@@ -161,7 +161,7 @@ def create_app() -> Flask:
        return {
            'COMPANY_NAME': os.environ.get('COMPANY_NAME', 'QR Code Management System'),
            'THEME_NAME':   os.environ.get('THEME_NAME', ''),
-            'CURRENT_YEAR': datetime.utcnow().year,
+            'CURRENT_YEAR': datetime.now().year,  # local time (§18)
        }
    @app.context_processor
@@ -24,6 +24,7 @@ from sqlalchemy import text, or_, and_
 from logger_handler import log_user_activity, log_database_operations
 from utils.helpers import (
                           admin_required,
                           load_project_manager_scope,
                           expand_employee_id_filter,
                           get_base_employee_id,
                           get_client_ip,
@@ -837,27 +838,38 @@ def time_attendance_locations_api():
    Used by the time attendance records page to dynamically scope the location dropdown."""
    try:
        project_id = request.args.get('project_id', '').strip()
        conditions = ["location_name IS NOT NULL"]
        params = {}
        if project_id:
            try:
-                project_id_int = int(project_id)
+                params['project_id'] = int(project_id)
            except (ValueError, TypeError):
                return jsonify({'success': False, 'error': 'Invalid project_id'}), 400
            conditions.append("project_id = :project_id")
-            result = db.session.execute(text("""
+        # Project Managers see only their assigned projects / locations (§4)
-                SELECT DISTINCT location_name
+        is_pm, allowed_project_ids, allowed_location_names = load_project_manager_scope()
-                FROM time_attendance
+        if is_pm:
-                WHERE project_id = :project_id
+            if not (allowed_project_ids or allowed_location_names):
-                  AND location_name IS NOT NULL
+                return jsonify({'success': True, 'locations': []})
-                ORDER BY location_name
+            scope = []
-            """), {'project_id': project_id_int})
+            if allowed_project_ids:
-        else:
+                placeholders = ', '.join(f':pm_project_{i}' for i in range(len(allowed_project_ids)))
-            result = db.session.execute(text("""
+                scope.append(f"project_id IN ({placeholders})")
-                SELECT DISTINCT location_name
+                params.update({f'pm_project_{i}': pid for i, pid in enumerate(allowed_project_ids)})
-                FROM time_attendance
+            if allowed_location_names:
-                WHERE location_name IS NOT NULL
+                placeholders = ', '.join(f':pm_location_{i}' for i in range(len(allowed_location_names)))
-                ORDER BY location_name
+                scope.append(f"location_name IN ({placeholders})")
-            """))
+                params.update({f'pm_location_{i}': loc for i, loc in enumerate(allowed_location_names)})
            conditions.append('(' + ' OR '.join(scope) + ')')
        result = db.session.execute(text(f"""
            SELECT DISTINCT location_name
            FROM time_attendance
            WHERE {' AND '.join(conditions)}
            ORDER BY location_name
        """), params)
        locations = [row[0] for row in result.fetchall()]
        logger_handler.logger.info(
@@ -878,28 +890,39 @@ def attendance_locations_api():
    Used by the attendance report page to dynamically scope the location dropdown when a project is selected."""
    try:
        project_id = request.args.get('project_id', '').strip()
        conditions = ["ad.location_name IS NOT NULL"]
        params = {}
        if project_id:
            try:
-                project_id_int = int(project_id)
+                params['project_id'] = int(project_id)
            except (ValueError, TypeError):
                return jsonify({'success': False, 'error': 'Invalid project_id'}), 400
            conditions.append("qc.project_id = :project_id")
-            result = db.session.execute(text("""
+        # Project Managers see only their assigned projects / locations (§4)
-                SELECT DISTINCT ad.location_name
+        is_pm, allowed_project_ids, allowed_location_names = load_project_manager_scope()
-                FROM attendance_data ad
+        if is_pm:
-                INNER JOIN qr_codes qc ON ad.qr_code_id = qc.id
+            if not (allowed_project_ids or allowed_location_names):
-                WHERE qc.project_id = :project_id
+                return jsonify({'success': True, 'locations': []})
-                  AND ad.location_name IS NOT NULL
+            scope = []
-                ORDER BY ad.location_name
+            if allowed_project_ids:
-            """), {'project_id': project_id_int})
+                placeholders = ', '.join(f':pm_project_{i}' for i in range(len(allowed_project_ids)))
-        else:
+                scope.append(f"qc.project_id IN ({placeholders})")
-            result = db.session.execute(text("""
+                params.update({f'pm_project_{i}': pid for i, pid in enumerate(allowed_project_ids)})
-                SELECT DISTINCT location_name
+            if allowed_location_names:
-                FROM attendance_data
+                placeholders = ', '.join(f':pm_location_{i}' for i in range(len(allowed_location_names)))
-                WHERE location_name IS NOT NULL
+                scope.append(f"ad.location_name IN ({placeholders})")
-                ORDER BY location_name
+                params.update({f'pm_location_{i}': loc for i, loc in enumerate(allowed_location_names)})
-            """))
+            conditions.append('(' + ' OR '.join(scope) + ')')
        result = db.session.execute(text(f"""
            SELECT DISTINCT ad.location_name
            FROM attendance_data ad
            LEFT JOIN qr_codes qc ON ad.qr_code_id = qc.id
            WHERE {' AND '.join(conditions)}
            ORDER BY ad.location_name
        """), params)
        locations = [row[0] for row in result.fetchall()]
        logger_handler.logger.info(
@@ -930,15 +953,35 @@ def search_employees_api():
        search_pattern = f"%{search_query}%"
        # Project Managers may only search within their own projects (§4). A PM
        # scoped by locations is resolved to the projects those locations belong
        # to, so their report filter still works without exposing other projects.
        is_pm, allowed_project_ids, allowed_location_names = load_project_manager_scope()
        pm_project_ids = []
        if is_pm:
            pm_project_ids = list(allowed_project_ids)
            if allowed_location_names:
                location_projects = db.session.query(QRCode.project_id).filter(
                    QRCode.location.in_(allowed_location_names),
                    QRCode.project_id.isnot(None)
                ).distinct().all()
                pm_project_ids.extend(row[0] for row in location_projects)
            pm_project_ids = sorted(set(pm_project_ids))
            if not pm_project_ids:
                return jsonify({'employees': []})
        # 1. Registered employees — search by ID or name
-        employees = Employee.query.filter(
+        employee_query = Employee.query.filter(
            db.or_(
                Employee.id.like(search_pattern),
                Employee.firstName.like(search_pattern),
                Employee.lastName.like(search_pattern),
                db.func.concat(Employee.firstName, ' ', Employee.lastName).like(search_pattern)
            )
-        ).limit(10).all()
+        )
        if is_pm:
            employee_query = employee_query.filter(Employee.contractId.in_(pm_project_ids))
        employees = employee_query.limit(10).all()
        employee_list = [{
            'id': emp.id,
@@ -955,17 +998,25 @@ def search_employees_api():
        if len(employee_list) < 10:
            remaining_slots = 10 - len(employee_list)
            try:
                unregistered_conditions = ["e.id IS NULL", "ad.employee_id LIKE :pattern"]
                unregistered_params = {'pattern': search_pattern, 'lim': remaining_slots}
                if is_pm:
                    placeholders = ', '.join(f':pm_project_{i}' for i in range(len(pm_project_ids)))
                    unregistered_conditions.append(f"qc.project_id IN ({placeholders})")
                    unregistered_params.update(
                        {f'pm_project_{i}': pid for i, pid in enumerate(pm_project_ids)})
                unregistered_rows = db.session.execute(
-                    text("""
+                    text(f"""
                        SELECT DISTINCT ad.employee_id
                        FROM attendance_data ad
                        LEFT JOIN employee e ON CAST(ad.employee_id AS UNSIGNED) = e.id
-                        WHERE e.id IS NULL
+                        LEFT JOIN qr_codes qc ON ad.qr_code_id = qc.id
-                          AND ad.employee_id LIKE :pattern
+                        WHERE {' AND '.join(unregistered_conditions)}
                        ORDER BY ad.employee_id
                        LIMIT :lim
                    """),
-                    {'pattern': search_pattern, 'lim': remaining_slots}
+                    unregistered_params
                ).fetchall()
                for row in unregistered_rows:
@@ -999,7 +1050,16 @@ def get_project_locations_api():
        if not project_id:
            return jsonify({'success': False, 'locations': [], 'error': 'Project ID required'})
-        
+
        # Project Managers may only ask about their own projects (§4)
        is_pm, allowed_project_ids, _ = load_project_manager_scope()
        if is_pm and int(project_id) not in allowed_project_ids:
            logger_handler.logger.warning(
                f"Project Manager {session.get('username')} requested locations for "
                f"project {project_id}, which is not assigned to them"
            )
            return jsonify({'success': True, 'locations': []})
        # Get active QR codes for this project
        qr_codes = QRCode.query.filter_by(
            project_id=int(project_id),
@@ -35,6 +35,38 @@ from openpyxl.utils import get_column_letter
 from routes.attendance import bp  # shared blueprint — do not redefine
 # Employee IDs typed on the edit / manual-add forms follow the same rule as a
 # check-in: 1 to 4 digits, optionally carrying a work-type code.
 MANUAL_EMPLOYEE_ID_MAX_DIGITS = 4
 def _normalize_manual_employee_id(raw_employee_id):
    """
    (canonical_id, base_id, work_type) for an ID typed on a form, or
    (None, None, None) when it is not a valid employee ID.
        "1234"     -> ("1234",   "1234", "regular")
        "1234sp"   -> ("1234SP", "1234", "SP")
        "1234 . PW"-> ("1234PW", "1234", "PW")
        "12345", "abc", "" -> (None, None, None)
    Manual entry used to do int(employee_id), so an extra-work record such as
    "1234SP" could not be added by hand at all, and edit accepted any text,
    which created pseudo-employees in the exports (§13).
    """
    from working_hours_calculator import parse_employee_id_for_work_type
    raw = str(raw_employee_id or '').strip()
    if not raw:
        return None, None, None
    base_id, work_type = parse_employee_id_for_work_type(raw)
    if not re.fullmatch(r'[0-9]+', base_id) or len(base_id) > MANUAL_EMPLOYEE_ID_MAX_DIGITS:
        return None, None, None
    canonical = base_id if work_type == 'regular' else f"{base_id}{work_type}"
    return canonical, base_id, work_type
@bp.route('/attendance/<int:record_id>/edit', methods=['GET', 'POST'], endpoint='edit_attendance')
@login_required
@@ -74,7 +106,16 @@ def edit_attendance(record_id):
            }
            # Update attendance record fields
-            new_employee_id = request.form['employee_id'].strip().upper()
+            new_employee_id, _base_id, _work_type = _normalize_manual_employee_id(
                request.form.get('employee_id', ''))
            if not new_employee_id:
                flash('Employee ID must be 1 to 4 digits, optionally with a work type '
                      '(for example 1234 or 1234SP).', 'error')
                projects = Project.query.filter_by(active_status=True).order_by(Project.name).all()
                return render_template('edit_attendance.html',
                                     attendance_record=attendance_record,
                                     projects=projects,
                                     qr_codes=QRCode.query.filter_by(active_status=True).all())
            new_check_in_date = datetime.strptime(request.form['check_in_date'], '%Y-%m-%d').date()
            new_check_in_time = datetime.strptime(request.form['check_in_time'], '%H:%M').time()
            new_location_name = request.form['location_name'].strip()
@@ -120,10 +161,12 @@ def edit_attendance(record_id):
            attendance_record.check_in_time = new_check_in_time
            attendance_record.location_name = new_location_name
            attendance_record.qr_code_id = int(new_qr_code_id)
-            attendance_record.updated_timestamp = datetime.utcnow()
+            # Local time throughout (§18): check-in dates and times are local, so
-            
+            # a UTC audit stamp read hours apart from the record it describes.
            attendance_record.updated_timestamp = datetime.now()
            # Store the audit note with timestamp and user info
-            timestamp = datetime.utcnow().strftime('%Y-%m-%d %H:%M:%S UTC')
+            timestamp = datetime.now().strftime('%Y-%m-%d %H:%M:%S')
            username = session.get('username', 'Unknown')
            role = session.get('role', 'unknown')
@@ -262,10 +305,17 @@ def save_manual_attendance():
            flash('All fields are required.', 'error')
            return redirect(url_for('attendance.add_manual_attendance'))
-        # Validate employee exists
+        # Validate the ID, and keep any work-type code ("1234SP") on the stored value
-        employee = Employee.query.filter_by(id=int(employee_id)).first()
+        canonical_employee_id, base_employee_id, work_type = _normalize_manual_employee_id(employee_id)
        if not canonical_employee_id:
            flash('Employee ID must be 1 to 4 digits, optionally with a work type '
                  '(for example 1234 or 1234SP).', 'error')
            return redirect(url_for('attendance.add_manual_attendance'))
        # Validate employee exists (by the numeric base ID)
        employee = Employee.query.filter_by(id=int(base_employee_id)).first()
        if not employee:
-            flash(f'Employee with ID {employee_id} not found.', 'error')
+            flash(f'Employee with ID {base_employee_id} not found.', 'error')
            return redirect(url_for('attendance.add_manual_attendance'))
        # Get QR code (location)
@@ -285,7 +335,7 @@ def save_manual_attendance():
        # Check if record already exists for this employee, location, date, and time
        existing_record = AttendanceData.query.filter_by(
-            employee_id=str(employee_id),
+            employee_id=canonical_employee_id,
            qr_code_id=qr_code.id,
            check_in_date=check_date_obj,
            check_in_time=check_time_obj
@@ -300,7 +350,7 @@ def save_manual_attendance():
        # Set fixed distance of 0.010 miles
        new_attendance = AttendanceData(
            qr_code_id=qr_code.id,
-            employee_id=str(employee_id),
+            employee_id=canonical_employee_id,
            check_in_date=check_date_obj,
            check_in_time=check_time_obj,
            location_name=qr_code.location,
@@ -320,8 +370,8 @@ def save_manual_attendance():
            status='present',
            verification_required=False,
            verification_status='approved',
-            created_timestamp=datetime.utcnow(),
+            created_timestamp=datetime.now(),
-            updated_timestamp=datetime.utcnow()
+            updated_timestamp=datetime.now()
        )
        db.session.add(new_attendance)
@@ -330,7 +380,7 @@ def save_manual_attendance():
        # Log the manual entry
        logger_handler.logger.info(
            f"Manual attendance record created by {session.get('username')} ({user_role}): "
-            f"Employee {employee.firstName} {employee.lastName} (ID: {employee_id}), "
+            f"Employee {employee.firstName} {employee.lastName} (ID: {canonical_employee_id}), "
            f"Location: {qr_code.location}, Event: {qr_code.location_event}, "
            f"Date: {check_date}, Time: {check_time}"
        )
@@ -8,6 +8,7 @@ Routes: /dashboard, /project/<id>/qr-codes, /dashboard/search,
 """
 from flask import abort, Blueprint, render_template, request, redirect, flash, session, jsonify, url_for
 from datetime import datetime, timedelta, date, time
 from sqlalchemy import or_
 from extensions import db, logger_handler
 from models.attendance import AttendanceData
@@ -15,7 +16,27 @@ from models.project import Project
 from models.qrcode import QRCode
 from models.user import User
 from logger_handler import log_user_activity, log_database_operations
-from utils.helpers import login_required
+from utils.helpers import login_required, load_project_manager_scope
 def _project_manager_qr_filter():
    """
    (is_pm, qr_filter) for the session user — the QR codes a Project Manager may
    see (§4). qr_filter is None for everyone else, and a never-true condition for
    a PM with no assignments.
    """
    is_pm, allowed_project_ids, allowed_location_names = load_project_manager_scope()
    if not is_pm:
        return False, None
    scope = []
    if allowed_project_ids:
        scope.append(QRCode.project_id.in_(allowed_project_ids))
    if allowed_location_names:
        scope.append(QRCode.location.in_(allowed_location_names))
    if not scope:
        return True, QRCode.id.is_(None)  # assigned nothing → sees nothing
    return True, or_(*scope)
 bp = Blueprint('dashboard', __name__)
@@ -34,7 +55,12 @@ def dashboard():
        # Build QR codes query with filters
        qr_query = QRCode.query
-        
+
        # Project Managers only see QR codes in their assigned projects/locations
        is_pm, pm_qr_filter = _project_manager_qr_filter()
        if pm_qr_filter is not None:
            qr_query = qr_query.filter(pm_qr_filter)
        # Apply name filter if provided
        if search_name:
            qr_query = qr_query.filter(QRCode.name.ilike(f'%{search_name}%'))
@@ -47,7 +73,16 @@ def dashboard():
        # Execute query
        qr_codes = qr_query.order_by(QRCode.created_date.desc()).all()
-        projects = Project.query.order_by(Project.name.asc()).all()
+
        project_query = Project.query
        if is_pm:
            # Only the projects the PM is assigned to, plus those behind their locations
            visible_project_ids = {qr.project_id for qr in qr_codes if qr.project_id}
            _, allowed_project_ids, _ = load_project_manager_scope()
            visible_project_ids.update(allowed_project_ids)
            project_query = (project_query.filter(Project.id.in_(sorted(visible_project_ids)))
                             if visible_project_ids else project_query.filter(Project.id.is_(None)))
        projects = project_query.order_by(Project.name.asc()).all()
        # Log dashboard access with filter info
        filter_info = []
@@ -87,7 +122,16 @@ def project_qr_codes(project_id):
        project = db.session.get(Project, project_id)
        if project is None:
            abort(404)
-        
+
        # Project Managers may only open their own projects (§4)
        is_pm, allowed_project_ids, _ = load_project_manager_scope()
        if is_pm and project_id not in allowed_project_ids:
            logger_handler.logger.warning(
                f"Project Manager {session.get('username')} tried to open project {project_id}"
            )
            flash('You do not have permission to view that project.', 'error')
            return redirect(url_for('dashboard.dashboard'))
        # Get search parameters from URL
        search_name = request.args.get('search_name', '').strip()
        search_status = request.args.get('search_status', '').strip()
@@ -154,36 +198,59 @@ def search_qr_codes():
 def dashboard_stats_api():
    """API endpoint for dashboard statistics"""
    try:
        # Project Manager scope (§4): every count below is restricted to the QR
        # codes / locations they are assigned to.
        is_pm, allowed_project_ids, allowed_location_names = load_project_manager_scope()
        _, pm_qr_filter = _project_manager_qr_filter()
        qr_base_query = QRCode.query.filter_by(active_status=True)
        attendance_base_query = AttendanceData.query
        location_base_query = db.session.query(AttendanceData.location_name)
        project_base_query = Project.query.filter_by(active_status=True)
        if is_pm:
            qr_base_query = qr_base_query.filter(pm_qr_filter)
            attendance_scope = []
            if allowed_project_ids:
                attendance_scope.append(AttendanceData.qr_code_id.in_(
                    db.session.query(QRCode.id).filter(QRCode.project_id.in_(allowed_project_ids))
                ))
                project_base_query = project_base_query.filter(Project.id.in_(allowed_project_ids))
            else:
                project_base_query = project_base_query.filter(Project.id.is_(None))
            if allowed_location_names:
                attendance_scope.append(AttendanceData.location_name.in_(allowed_location_names))
            attendance_condition = or_(*attendance_scope) if attendance_scope else AttendanceData.id.is_(None)
            attendance_base_query = attendance_base_query.filter(attendance_condition)
            location_base_query = location_base_query.filter(attendance_condition)
        # Get current stats
-        total_qr_codes = QRCode.query.filter_by(active_status=True).count()
+        total_qr_codes = qr_base_query.count()
-        
+
-        # Today's check-ins
+        # Today's check-ins — local date, matching how check-ins are stored
-        today = datetime.utcnow().date()
+        today = datetime.now().date()
-        today_checkins = AttendanceData.query.filter(
+        today_checkins = attendance_base_query.filter(
            AttendanceData.check_in_date == today
        ).count()
-        
+
        # Active projects
-        active_projects = Project.query.filter_by(active_status=True).count()
+        active_projects = project_base_query.count()
-        
+
        # Unique locations
-        unique_locations = db.session.query(
+        unique_locations = location_base_query.distinct().count()
-            AttendanceData.location_name
+
        ).distinct().count()
        # Calculate trends (compared to last month)
-        last_month = datetime.utcnow() - timedelta(days=30)
+        last_month = datetime.now() - timedelta(days=30)
-        
+
        # QR codes trend
-        old_qr_count = QRCode.query.filter(
+        old_qr_count = qr_base_query.filter(
-            QRCode.created_date <= last_month,
+            QRCode.created_date <= last_month
            QRCode.active_status == True
        ).count()
        qr_change = ((total_qr_codes - old_qr_count) / max(old_qr_count, 1)) * 100
        # Check-ins trend (yesterday)
        yesterday = today - timedelta(days=1)
-        yesterday_checkins = AttendanceData.query.filter(
+        yesterday_checkins = attendance_base_query.filter(
            AttendanceData.check_in_date == yesterday
        ).count()
        checkin_change = ((today_checkins - yesterday_checkins) / max(yesterday_checkins, 1)) * 100
@@ -214,12 +281,27 @@ def dashboard_realtime_api():
    """API endpoint for real-time dashboard data"""
    try:
        # Get recent activity (last 10 check-ins)
-        recent_activity = db.session.query(
+        recent_query = db.session.query(
            AttendanceData.employee_id,
            AttendanceData.location_name,
            AttendanceData.check_in_time,
            AttendanceData.check_in_date
-        ).order_by(
+        )
        # Project Managers only see activity at their own projects/locations (§4)
        is_pm, allowed_project_ids, allowed_location_names = load_project_manager_scope()
        if is_pm:
            recent_scope = []
            if allowed_project_ids:
                recent_scope.append(AttendanceData.qr_code_id.in_(
                    db.session.query(QRCode.id).filter(QRCode.project_id.in_(allowed_project_ids))
                ))
            if allowed_location_names:
                recent_scope.append(AttendanceData.location_name.in_(allowed_location_names))
            recent_query = recent_query.filter(or_(*recent_scope) if recent_scope
                                               else AttendanceData.id.is_(None))
        recent_activity = recent_query.order_by(
            AttendanceData.check_in_date.desc(),
            AttendanceData.check_in_time.desc()
        ).limit(10).all()
@@ -85,6 +85,20 @@ def _overnight_aware_sort_key(record):
        seconds_total += 24 * 3600
    return seconds_total
 def _pair_is_regular(check_in_record, check_out_record) -> bool:
    """
    True when a completed pair counts toward the 40-hour overtime rule.
    SP / PW / PT / C hours are paid but never build toward overtime (§13). The
    OUT record's work type wins, mirroring effective_work_type elsewhere, so a
    Regular IN paired with an SP OUT counts as SP.
    """
    in_work_type = getattr(check_in_record, 'work_type', None)
    out_work_type = getattr(check_out_record, 'work_type', None)
    effective = out_work_type or in_work_type
    return effective not in ('SP', 'PW', 'PT', 'C')
 def _qtr(decimal_hours: float) -> float:
    """
    Round a decimal-hours value to the nearest quarter hour (.00/.25/.50/.75).
@@ -596,6 +610,10 @@ def export_time_attendance_excel(records, project_name_for_filename, date_range_
        current_week_start = None
        grand_regular_hours = 0
        grand_ot_hours = 0
        # Regular-only hours feeding the 40-hour overtime rule. SP/PW/PT/C hours
        # are worked and paid, but they do NOT build toward overtime (§13), so
        # they are excluded here while column H still shows every hour worked.
        weekly_regular_hours = 0
        # Accumulate SP/PW/PT/C hours from cross-type pairs (where the calculator
        # could not detect them because it processes each work-type stream independently).
        cross_type_sp_hours = 0.0
@@ -624,20 +642,22 @@ def export_time_attendance_excel(records, project_name_for_filename, date_range_
            _report_start = start_date
            week_start = (_report_start + timedelta(days=((date_obj.date() - _report_start).days // 7) * 7))
            if current_week_start is not None and week_start != current_week_start:
-                # Write weekly total row
+                # Write weekly total row. Column H = every hour worked; Regular
-                week_regular = min(weekly_total_hours, 40.0)
+                # and OT are computed from regular hours only (SP/PW/PT/C excluded).
-                week_overtime = max(0, weekly_total_hours - 40.0)
+                week_regular = min(weekly_regular_hours, 40.0)
-                
+                week_overtime = max(0, weekly_regular_hours - 40.0)
                ws.cell(row=current_row, column=7, value='Weekly Total: ').font = bold_font
                ws.cell(row=current_row, column=8, value=_qtr(weekly_total_hours)).font = bold_font
                ws.cell(row=current_row, column=9, value=_qtr(week_regular)).font = bold_font
                ws.cell(row=current_row, column=10, value=_qtr(week_overtime)).font = bold_font
-                
+
                grand_regular_hours += week_regular
                grand_ot_hours += week_overtime
                current_row += 1
-                
+
                weekly_total_hours = 0
                weekly_regular_hours = 0
            current_week_start = week_start
@@ -666,6 +686,8 @@ def export_time_attendance_excel(records, project_name_for_filename, date_range_
            # group, and sum only complete pairs.  This ensures the daily total in
            # column H matches exactly the pairs rendered in the export rows.
            _day_total_hours = 0.0
            # Same pairs, minus SP/PW/PT/C: the overtime base (§13)
            _day_regular_hours = 0.0
            # Track which records are consumed by same-building pairing so the
            # cross-building pass only considers true orphans.
            _same_building_used_ids = set()
@@ -704,6 +726,8 @@ def export_time_attendance_excel(records, project_name_for_filename, date_range_
                        if _duration > 24:
                            continue
                        _day_total_hours += _duration
                        if _pair_is_regular(_in_r, _out_r):
                            _day_regular_hours += _duration
                        _out_used[_oi2] = True
                        _same_building_used_ids.add(id(_in_r))
                        _same_building_used_ids.add(id(_out_r))
@@ -768,6 +792,8 @@ def export_time_attendance_excel(records, project_name_for_filename, date_range_
                        'hours':     _cb_dur,
                    })
                    _day_total_hours += _cb_dur
                    if _pair_is_regular(_cb_in, _cb_out):
                        _day_regular_hours += _cb_dur
                    logger_handler.logger.info(
                        f"[TA Export] Cross-building pair for employee {employee_id} on {date_str}: "
                        f"IN {_cb_in.location_name} @ {_cb_in.check_in_time} → "
@@ -788,6 +814,7 @@ def export_time_attendance_excel(records, project_name_for_filename, date_range_
            total_hours = _qtr(_day_total_hours)
            weekly_total_hours += total_hours
            weekly_regular_hours += _qtr(_day_regular_hours)
            # Daily total display (only shown on last location's last row)
            daily_total_display = _qtr(total_hours) if total_hours > 0 else ''
@@ -1282,10 +1309,10 @@ def export_time_attendance_excel(records, project_name_for_filename, date_range_
            # END CROSS-BUILDING PAIR ROW WRITING
            # -------------------------------------------------------------------
-        # Write final weekly total for this employee
+        # Write final weekly total for this employee (Regular/OT from regular hours only)
        if weekly_total_hours > 0:
-            week_regular = min(weekly_total_hours, 40.0)
+            week_regular = min(weekly_regular_hours, 40.0)
-            week_overtime = max(0, weekly_total_hours - 40.0)
+            week_overtime = max(0, weekly_regular_hours - 40.0)
            ws.cell(row=current_row, column=7, value='Weekly Total: ').font = bold_font
            ws.cell(row=current_row, column=8, value=_qtr(weekly_total_hours)).font = bold_font
@@ -1757,7 +1784,21 @@ def export_time_attendance_by_building_excel(records, project_name_for_filename,
    date_cell.font = Font(name='Aptos Narrow', size=11)
    date_cell.alignment = Alignment(horizontal='left')
    current_row += 1
-    
+
    # Row 5: how the totals in this sheet are calculated. Overtime here is per
    # BUILDING (each block counts only that building's hours), so an employee
    # split across two buildings shows no overtime even past 40 hours — the main
    # Time Attendance export totals the week across all buildings.
    ws.merge_cells(f'A{current_row}:N{current_row}')
    ot_note_cell = ws.cell(
        row=current_row, column=1,
        value=('Note: Weekly Total / Regular / OT below are per building, and overtime excludes '
               'SP / PW / PT / C hours. Use the main Time Attendance export for payroll totals.')
    )
    ot_note_cell.font = Font(name='Aptos Narrow', size=10, italic=True)
    ot_note_cell.alignment = Alignment(horizontal='left')
    current_row += 1
    # Empty rows before first building
    current_row += 2
@@ -1967,6 +2008,9 @@ def export_time_attendance_by_building_excel(records, project_name_for_filename,
            current_week_start = None
            grand_regular_hours = 0
            grand_ot_hours = 0
            # Regular-only hours for the 40-hour rule (SP/PW/PT/C excluded, §13).
            # Note: per BUILDING in this sheet — see the note under the date range.
            weekly_regular_hours = 0
            # Accumulate raw regular-only (non-SP/PW/PT) pair hours.
            # Used for the "Regular" summary row when the employee also has
            # special work-type hours.
@@ -1993,9 +2037,10 @@ def export_time_attendance_by_building_excel(records, project_name_for_filename,
                _report_start = start_date
                week_start = (_report_start + timedelta(days=((date_obj.date() - _report_start).days // 7) * 7))
                if current_week_start is not None and week_start != current_week_start:
-                    # Write weekly total row
+                    # Write weekly total row. Column H = every hour worked at this
-                    week_regular = min(weekly_total_hours, 40.0)
+                    # building; Regular and OT come from regular hours only.
-                    week_overtime = max(0, weekly_total_hours - 40.0)
+                    week_regular = min(weekly_regular_hours, 40.0)
                    week_overtime = max(0, weekly_regular_hours - 40.0)
                    ws.cell(row=current_row, column=7, value='Weekly Total: ').font = bold_font
                    ws.cell(row=current_row, column=8, value=_qtr(weekly_total_hours)).font = bold_font
@@ -2005,8 +2050,9 @@ def export_time_attendance_by_building_excel(records, project_name_for_filename,
                    grand_regular_hours += week_regular
                    grand_ot_hours += week_overtime
                    current_row += 1
-                    
+
                    weekly_total_hours = 0
                    weekly_regular_hours = 0
                current_week_start = week_start
@@ -2145,6 +2191,7 @@ def export_time_attendance_by_building_excel(records, project_name_for_filename,
                # Calculate daily hours
                daily_hours = 0
                _bb_day_non_sp_hours = 0.0  # weekly summary sheet excludes SP time
                _bb_day_regular_hours = 0.0  # overtime base: excludes SP/PW/PT/C
                for pair in pairs:
                    if pair['check_in'] and pair['check_out'] and not pair['is_miss_punch']:
                        pair_in  = datetime.combine(date_obj, pair['check_in'].check_in_time)
@@ -2165,11 +2212,13 @@ def export_time_attendance_by_building_excel(records, project_name_for_filename,
                            _bb_eff_wt = _bb_out_wt or _bb_in_wt  # prefer OUT's type (mirrors main export)
                            if _bb_eff_wt not in ('SP', 'PW', 'PT', 'C'):
                                regular_only_hours += _bb_dur
                                _bb_day_regular_hours += _bb_dur
                            if _bb_eff_wt != 'SP':
                                _bb_day_non_sp_hours += _bb_dur
                daily_hours = _qtr(daily_hours)
                weekly_total_hours += daily_hours
                weekly_regular_hours += _qtr(_bb_day_regular_hours)
                # Same week anchoring as the Weekly Total rows (report start date)
                _bb_week_idx = (date_obj.date() - start_date).days // 7
@@ -2250,10 +2299,10 @@ def export_time_attendance_by_building_excel(records, project_name_for_filename,
                    current_row += 1
-            # Write final weekly total
+            # Write final weekly total (Regular/OT from regular hours only)
            if weekly_total_hours > 0:
-                week_regular = min(weekly_total_hours, 40.0)
+                week_regular = min(weekly_regular_hours, 40.0)
-                week_overtime = max(0, weekly_total_hours - 40.0)
+                week_overtime = max(0, weekly_regular_hours - 40.0)
                ws.cell(row=current_row, column=7, value='Weekly Total: ').font = bold_font
                ws.cell(row=current_row, column=8, value=_qtr(weekly_total_hours)).font = bold_font
@@ -6,8 +6,9 @@ Added functionality to detect and present duplicates for user review.
 """
 import pandas as pd
 import re
 import uuid
-from datetime import datetime, time
+from datetime import datetime, date, time
 from sqlalchemy.exc import SQLAlchemyError
 from typing import Dict, List, Any, Optional, Tuple
 import traceback
@@ -433,7 +434,7 @@ class TimeAttendanceImportService:
                        employee_name = self._get_employee_name(clean_id)
                    # Parse date and time
-                    attendance_date = pd.to_datetime(row['Date']).date()
+                    attendance_date = self._parse_date_field(row['Date'])
                    attendance_time = self._parse_time_field(row['Time'])
                    # Prepare record data
@@ -593,10 +594,10 @@ class TimeAttendanceImportService:
                    row_errors.append("Missing Date")
                else:
                    try:
-                        attendance_date = pd.to_datetime(row['Date']).date()
+                        attendance_date = self._parse_date_field(row['Date'])
                        row_data['attendance_date'] = attendance_date
-                    except Exception:
+                    except Exception as date_error:
-                        row_errors.append(f"Invalid date format: {row['Date']}")
+                        row_errors.append(f"Invalid date format: {row['Date']} ({date_error})")
                # Check and parse Time
                if pd.isna(row['Time']):
@@ -832,7 +833,7 @@ class TimeAttendanceImportService:
                    # Validate and parse date
                    try:
-                        attendance_date = pd.to_datetime(row['Date']).date()
+                        attendance_date = self._parse_date_field(row['Date'])
                    except Exception as date_error:
                        import_results['failed_records'] += 1
                        import_results['errors'].append(f"Row {index + 2}: Invalid date format - {str(date_error)}")
@@ -936,17 +937,20 @@ class TimeAttendanceImportService:
                )
        except SQLAlchemyError as e:
            self.db.session.rollback()
            error_msg = f"Database error during import: {str(e)}"
            import_results['errors'].append(error_msg)
-            
+            # Rows commit in batches of 50 — drop the partial batch (all-or-nothing)
            self._rollback_partial_batch(batch_id, import_results)
            import_results['success'] = False
            if self.logger:
                self.logger.log_database_error('time_attendance_import', e)
-        
+
        except Exception as e:
            self.db.session.rollback()
            error_msg = f"Unexpected error during import: {str(e)}"
            import_results['errors'].append(error_msg)
            self._rollback_partial_batch(batch_id, import_results)
            import_results['success'] = False
            import_results['traceback'] = traceback.format_exc()
            if self.logger:
@@ -955,6 +959,111 @@ class TimeAttendanceImportService:
        return import_results
    # Excel stores dates as days since 1899-12-30. A column formatted as General
    # arrives as that number instead of a date, and pd.to_datetime() then read it
    # as nanoseconds since 1970 — every such row imported as 1970-01-01.
    _EXCEL_EPOCH = '1899-12-30'
    _EXCEL_SERIAL_MIN = 20000   # 1954-10-03
    _EXCEL_SERIAL_MAX = 60000   # 2064-04-05
    def _parse_date_field(self, date_value) -> date:
        """
        Parse the Date column into a date.
        Handles:
        - real date / datetime / pandas Timestamp cells
        - Excel serial numbers (45123 -> 2023-07-04), the General-format case
        - text dates, read **month-first** (US time clocks): "03/04/2026" is
          4 March 2026. When the first number cannot be a month ("13/04/2026")
          it is read day-first instead.
        Raises ValueError with a readable message — the caller records the row as
        failed instead of importing a wrong date.
        """
        if date_value is None or (isinstance(date_value, float) and pd.isna(date_value)):
            raise ValueError('Missing date')
        # Already a date/datetime (openpyxl / pandas parsed the cell)
        if isinstance(date_value, datetime):
            return date_value.date()
        if isinstance(date_value, date):
            return date_value
        to_pydatetime = getattr(date_value, 'to_pydatetime', None)  # pandas Timestamp
        if callable(to_pydatetime):
            return to_pydatetime().date()
        # Excel serial number, as a number or as text
        serial = None
        if isinstance(date_value, bool):
            raise ValueError(f"'{date_value}' is not a date")
        if isinstance(date_value, (int, float)):
            serial = float(date_value)
        else:
            text_value = str(date_value).strip()
            if re.fullmatch(r'\d{4,6}(\.\d+)?', text_value):
                serial = float(text_value)
        if serial is not None:
            if not (self._EXCEL_SERIAL_MIN <= serial <= self._EXCEL_SERIAL_MAX):
                raise ValueError(f"'{date_value}' is not a valid date (number out of range)")
            return pd.to_datetime(serial, unit='D', origin=self._EXCEL_EPOCH).date()
        text_value = str(date_value).strip()
        if not text_value:
            raise ValueError('Missing date')
        # Month-first unless the first component cannot be a month
        day_first = False
        parts = re.match(r'^(\d{1,2})[/\-.](\d{1,2})[/\-.](\d{2,4})$', text_value)
        if parts and int(parts.group(1)) > 12:
            day_first = True
        parsed = pd.to_datetime(text_value, dayfirst=day_first, errors='coerce')
        if pd.isna(parsed):
            raise ValueError(f"'{date_value}' is not a recognised date")
        return parsed.date()
    def _rollback_partial_batch(self, batch_id, import_results=None):
        """
        Remove rows already committed for a failed import batch.
        The row loop commits every 50 records to keep memory flat, so a failure
        part-way through used to leave a partial batch in the table with nothing
        to show it was incomplete. An import is all-or-nothing from the user's
        point of view, so the batch is deleted and the failure reported.
        """
        try:
            from models.time_attendance import TimeAttendance
            self.db.session.rollback()
            removed = TimeAttendance.query.filter_by(import_batch_id=batch_id).delete(
                synchronize_session=False)
            self.db.session.commit()
            if removed and self.logger:
                self.logger.logger.warning(
                    f"Import batch {batch_id} failed — removed {removed} partially imported records"
                )
            if import_results is not None:
                import_results['imported_records'] = 0
                import_results['rolled_back_records'] = removed
                import_results['errors'].append(
                    f"Import failed — {removed} partially imported records were removed. "
                    f"Nothing from this file was kept."
                )
            return removed
        except Exception as cleanup_error:
            if self.logger:
                self.logger.logger.error(
                    f"Could not remove partial import batch {batch_id}: {cleanup_error}", exc_info=True
                )
            if import_results is not None:
                import_results['errors'].append(
                    f"Import failed and the partial batch {batch_id} could not be removed "
                    f"automatically — delete it from the import history."
                )
            return 0
    def _parse_time_field(self, time_value) -> time:
        """
        Parse various time formats from Excel
@@ -1050,30 +1159,48 @@ class TimeAttendanceImportService:
    def _clean_employee_id(self, employee_id) -> str:
        """
-        Clean employee ID to handle various formats
+        Normalise an employee ID from an Excel cell into the form the app stores.
-        
+
-        Args:
+            1234 / "1234" ............. "1234"     (unchanged)
-            employee_id: Raw employee ID value
+            1234.0 (Excel numeric) .... "1234"
-            
+            "1759.PW", "1759 - PW" .... "1759PW"   canonical work-type spelling
-        Returns:
+            "1234.5" .................. "1234.5"   NOT an ID — never truncated
-            Cleaned employee ID string
+            "01234" ................... "01234"    zero padding preserved
        Work-type IDs are stored canonically so the calculator, the exports and
        the report filters all read the same spelling (§13). Before this, an
        imported "1759.PW" stayed as-is and the exports counted it as a separate
        REGULAR employee called "1759.PW" instead of PW hours for employee 1759.
        """
-        try:
+        from working_hours_calculator import parse_employee_id_for_work_type
-            # Convert to string first
+
-            id_str = str(employee_id).strip()
+        id_str = str(employee_id).strip()
-            
+        if not id_str:
            # Handle float values like 1234.0 or '1234.0'
            if '.' in id_str:
                # Convert to float, then to int, then back to string
                # This removes the decimal part: 1234.0 -> 1234
                id_str = str(int(float(id_str)))
            return id_str
-        except (ValueError, TypeError) as e:
+
-            # If conversion fails, return original string
+        # Excel hands whole numbers over as floats: 1234.0 -> "1234". A real
-            if self.logger:
+        # fraction is not an employee ID, so keep it as typed and let validation
-                self.logger.logger.warning(f"Could not clean employee ID '{employee_id}': {e}")
+        # flag it rather than silently importing hours for employee 1234.
-            return str(employee_id).strip()
+        if '.' in id_str:
            try:
                numeric = float(id_str)
            except (TypeError, ValueError):
                numeric = None
            if numeric is not None:
                if numeric.is_integer():
                    return str(int(numeric))
                if self.logger:
                    self.logger.logger.warning(
                        f"Employee ID '{id_str}' is not a whole number — imported unchanged"
                    )
                return id_str
        # "1759.PW" / "1759 - PW" / "PW.1759" -> "1759PW"
        base_id, work_type = parse_employee_id_for_work_type(id_str)
        if work_type != 'regular' and base_id.isdigit():
            return f"{base_id}{work_type}"
        return id_str
    def validate_excel_file(self, file_path: str) -> Dict[str, Any]:
        """
@@ -1167,8 +1294,8 @@ class TimeAttendanceImportService:
                for idx, date_val in df['Date'].items():
                    if pd.notna(date_val):
                        try:
-                            pd.to_datetime(date_val)
+                            self._parse_date_field(date_val)
-                        except:
+                        except Exception:
                            invalid_dates += 1
                if invalid_dates > 0:
@@ -371,6 +371,35 @@ def roles_required(*allowed_roles):
    return decorator
 def load_project_manager_scope():
    """
    (is_project_manager, allowed_project_ids, allowed_location_names) for the
    session user. Non-PMs get (False, [], []) and must NOT be filtered.
    A PM may be scoped by projects, by locations, or both; empty lists for a PM
    mean "no access at all" (fail closed), which is also what a permission
    lookup failure returns. Used by the attendance APIs and the dashboard so a
    PM cannot see other projects' locations, employees or scan activity (§4).
    """
    if session.get('role') != 'project_manager':
        return False, [], []
    from models.permissions import UserProjectPermission, UserLocationPermission
    user_id = session.get('user_id')
    try:
        project_ids = [p.project_id for p in
                       UserProjectPermission.query.filter_by(user_id=user_id).all()]
        location_names = [l.location_name for l in
                          UserLocationPermission.query.filter_by(user_id=user_id).all()]
        return True, project_ids, location_names
    except Exception as e:
        logger_handler.logger.error(
            f"Could not load Project Manager permissions for user {user_id}: {e}", exc_info=True
        )
        return True, [], []  # fail closed
 def restrict_blueprint_to_roles(blueprint, allowed_roles):
    """Apply the role check to every route of a blueprint (one line per module)."""
    @blueprint.before_request
@@ -126,11 +126,19 @@ def parse_employee_id_for_work_type(employee_id: str) -> Tuple[str, str]:
    """
    Parse employee ID to extract base ID and work type (supports SP, PW, PT, C)
-    Handles multiple formats:
+    Handles multiple formats — the separator may be anything that is not a letter
-    - Suffix with space: "1234 SP", "1234 PW", "1234 PT", "1234 C"
+    or a digit (space, dot, dash, underscore, slash, or nothing at all), because
-    - Suffix without space: "1234SP", "1234PW", "1234PT", "1234C"
+    imported IDs come straight from customer Excel files:
-    - Prefix with space: "SP 1234", "PW 1234", "PT 1234", "C 1234"
+    - Suffix: "1234SP", "1234 SP", "1234.PW", "1234-PT", "1234 . C"
-    - Prefix without space: "SP1234", "PW1234", "PT1234", "C1234"
+    - Prefix: "SP1234", "SP 1234", "PW.1234", "PT-1234"
    This mirrors build_employee_id_regex() in utils/helpers.py (SQL filters) and
    parseEmployeeIdWorkType() in static/js/attendance_report.js — keep the three
    in sync, or the report, the filters and the exports disagree about who worked
    those hours (§13).
    Not a work type: "1234.5" (no code) and "1234SPX" (code runs into a word) —
    both come back as regular with the ID unchanged.
    Args:
        employee_id: Employee ID string in any of the above formats
@@ -147,16 +155,19 @@ def parse_employee_id_for_work_type(employee_id: str) -> Tuple[str, str]:
    # Define work type codes
    work_type_codes = ['SP', 'PW', 'PT', 'C']
    # Any run of non-alphanumeric characters may separate the ID from the code
    # (or nothing at all): "1234SP", "1234 SP", "1234.PW", "1234 - PT".
    # The class excludes letters, so "1234SPX" never matches.
    separator = r'[^0-9A-Z]*'
    for work_type in work_type_codes:
-        # Pattern 1: Suffix with optional space - "1234 SP" or "1234SP"
+        # Pattern 1: Suffix - "1234SP", "1234 SP", "1234.SP"
-        suffix_pattern = rf'^(\d+)\s*{work_type}$'
+        suffix_match = re.match(rf'^(\d+){separator}{work_type}$', employee_id_clean)
        suffix_match = re.match(suffix_pattern, employee_id_clean)
        if suffix_match:
            return suffix_match.group(1), work_type
-        
+
-        # Pattern 2: Prefix with optional space - "SP 1234" or "SP1234"
+        # Pattern 2: Prefix - "SP1234", "SP 1234", "SP.1234"
-        prefix_pattern = rf'^{work_type}\s*(\d+)$'
+        prefix_match = re.match(rf'^{work_type}{separator}(\d+)$', employee_id_clean)
        prefix_match = re.match(prefix_pattern, employee_id_clean)
        if prefix_match:
            return prefix_match.group(1), work_type