Sep 16 - Optimize code, part 2
This commit is contained in:
@@ -210,6 +210,18 @@ is the reference for who may use what; routes enforce the same rules:
|
||||
- New routes in a gated blueprint are covered automatically; new routes elsewhere need an
|
||||
explicit decorator.
|
||||
|
||||
**Project Manager data scoping (Set 27)** — role gates decide which *pages* a PM may open;
|
||||
`load_project_manager_scope()` (`utils/helpers.py`) decides which *rows* they see. It returns
|
||||
`(is_project_manager, allowed_project_ids, allowed_location_names)` and **fails closed** (a PM with
|
||||
no assignments, or a permission lookup error, sees nothing). Applied in:
|
||||
- `routes/attendance.py`: the report, live updates, `attendance_locations_api`,
|
||||
`time_attendance_locations_api`, `search_employees_api`, `get_project_locations_api`
|
||||
- `routes/dashboard.py`: dashboard QR list + project list, `project_qr_codes` (403-style redirect
|
||||
for someone else's project), `dashboard_stats_api`, `dashboard_realtime_api`
|
||||
- A PM scoped only by **locations** is resolved to the projects behind those locations when
|
||||
employee names are searched, so their filter still works without exposing other projects.
|
||||
- **Any new endpoint returning attendance, employee, QR or project rows must call it.**
|
||||
|
||||
---
|
||||
|
||||
## 5. Application Factory & Initialization Order
|
||||
@@ -756,8 +768,19 @@ within the interval. This is pre-existing behaviour for suffixed IDs.
|
||||
### Special Handling
|
||||
- `Recorded Address`: read via openpyxl directly (not pandas) to preserve HYPERLINK formulas
|
||||
- Duplicate detection: hash of `employee_id + date + time + action_description`
|
||||
- `_clean_employee_id()` normalises IDs to the stored form (Set 26): `1234.0` → `1234`,
|
||||
`1759.PW` / `1759 - PW` / `PW.1759` → `1759PW`. Zero padding is **preserved** (`01234`) and a
|
||||
real fraction (`1234.5`) is **never truncated** — both would change the duplicate hash of rows
|
||||
already imported, or silently move hours to another employee
|
||||
- Import tracked by `import_batch_id` (UUID)
|
||||
- `self.db` not `db` — in `TimeAttendanceImportService`, always access SQLAlchemy via `self.db`
|
||||
- **Dates go through `_parse_date_field()` (Set 27) — never `pd.to_datetime()` directly.** It handles
|
||||
real date cells, Excel serial numbers (a General-formatted column: `45123` → 2023-07-16, which
|
||||
`pd.to_datetime()` read as 1970-01-01), and text dates **month-first** (US time clocks), falling
|
||||
back to day-first only when the first number cannot be a month (`13/04/2026`). Unparseable dates
|
||||
fail the row instead of importing a wrong one
|
||||
- **A failed import removes its own rows** (`_rollback_partial_batch()`): the loop commits every 50
|
||||
records, so a failure part-way used to leave a partial batch with nothing marking it incomplete
|
||||
- Validation `except` blocks must not silently swallow exceptions (fail-open prevention)
|
||||
|
||||
---
|
||||
@@ -777,8 +800,15 @@ within the interval. This is pre-existing behaviour for suffixed IDs.
|
||||
- **SP** = Special Project, **PW** = Periodic Work, **PT** = Project Team (Part-Time),
|
||||
**C** = Covering (added Sept 2026)
|
||||
- Parsed from `employee_id` via `parse_employee_id_for_work_type()`
|
||||
- Suffix and prefix forms, with or without a separator: `1234SP`, `1234 SP`, `SP1234`,
|
||||
`1234.PW`. Two-letter codes are matched **before** the single-letter `C`
|
||||
- Suffix and prefix forms, separated by **any run of non-alphanumeric characters, or nothing**:
|
||||
`1234SP`, `1234 SP`, `1234.PW`, `1234-PT`, `1234 . C`, `SP1234`, `SP 1234`, `PW.1234`.
|
||||
Two-letter codes are matched **before** the single-letter `C`
|
||||
- The separator class is `[^0-9A-Z]*` in all three parsers — `parse_employee_id_for_work_type()`,
|
||||
`build_employee_id_regex()` (SQL filters) and `parseEmployeeIdWorkType()` (report JS). Keep them
|
||||
in sync: until Sept 16 2026 the Python one accepted only a space, so `1759.PW` was counted as a
|
||||
separate Regular employee in the exports while the report showed it as PW (Set 26)
|
||||
- **Not** work types: `1234.5` (no code) and `1234SPX` (code runs into a word) — both stay regular
|
||||
with the ID unchanged
|
||||
- Like SP/PW/PT, `C` hours are excluded from the 40-hour overtime rule
|
||||
- **Codes are declared in four places — keep them in sync:**
|
||||
| File | Symbol |
|
||||
@@ -791,6 +821,26 @@ within the interval. This is pre-existing behaviour for suffixed IDs.
|
||||
aggregation dicts are keyed `regular/SP/PW/PT`, so adding `C` to its parser alone
|
||||
would raise `KeyError`
|
||||
|
||||
### Overtime — 40-Hour Rule (Sept 16, 2026, confirmed by the user)
|
||||
|
||||
**SP / PW / PT / C hours are paid but never build toward overtime.** Both exports compute the
|
||||
weekly Regular and OT columns from regular hours only:
|
||||
|
||||
```python
|
||||
week_regular = min(weekly_regular_hours, 40.0) # NOT weekly_total_hours
|
||||
week_overtime = max(0, weekly_regular_hours - 40.0)
|
||||
```
|
||||
|
||||
- `weekly_regular_hours` accumulates `_qtr()` of each day's regular-only pair hours;
|
||||
`_pair_is_regular(in, out)` decides, preferring the **OUT** record's work type (a Regular IN
|
||||
paired with an SP OUT counts as SP, mirroring `effective_work_type`).
|
||||
- **Column H (Weekly Total) still shows every hour worked**, so H ≠ Regular + OT whenever the
|
||||
employee has special hours — the SP/PW/PT/C and Regular rows below break that down.
|
||||
- Example: 36 h Regular + 8 h SP → H 44, Regular 36, **OT 0** (before Sept 16: OT 4).
|
||||
- **Export by Building applies the same rule per BUILDING**, so an employee with 30 h at two
|
||||
buildings shows 0 OT in each block while the main export shows 20 h. A note row under the date
|
||||
range says so. Treat that sheet as review-only; pay from the main export (§20 Set 25).
|
||||
|
||||
### Overnight Shift Handling
|
||||
**Rule 1 — Sort key:** early-morning OUTs (`hour <= 3`) use `_overnight_aware_sort_key()` which adds 86400 seconds — pushes them past midnight so they sort after same-day evening INs.
|
||||
|
||||
@@ -1053,6 +1103,12 @@ new_name = qr_code.name # always use existing name, never request.form['name']
|
||||
### Timestamp Convention
|
||||
Use `datetime.now()` (local time) throughout — **not** `datetime.utcnow()`.
|
||||
|
||||
Check-in dates/times are stored local, so anything compared against them must be local too.
|
||||
Fixed in Set 27: `routes/attendance_edit.py` (record timestamps + audit note), `routes/dashboard.py`
|
||||
("today" / "last 30 days" stats, which rolled over mid-evening) and `app.py` (`CURRENT_YEAR`).
|
||||
Still UTC on purpose or harmlessly: `users.last_login_date`, security-middleware event stamps,
|
||||
`qr_codes.coordinates_updated_date`, import batch `import_date`.
|
||||
|
||||
### Context Safety
|
||||
- Use `has_request_context()` (not `if not request:`) to check Flask request context
|
||||
- Capture `current_app._get_current_object()` in route body, not inside lazy generators
|
||||
@@ -1333,6 +1389,34 @@ it (the workers share no pub/sub).
|
||||
| `extensions.py` | `logger_handler` is now a proxy (see §14). `utils/helpers.py` imports it at module import time, which happens BEFORE `init_logger()`, so it was permanently `None`: the Set 23 role check logged a warning when denying access and raised `AttributeError: 'NoneType' object has no attribute 'logger'` → 500 for a project manager opening `/time-attendance` or editing a QR code. The same latent bug affected `generate_qr_code()` logging in `utils/helpers.py` and every call in `utils/template_helpers.py` / `utils/geocoding.py` |
|
||||
| — | The step-1 tests had stubbed a working logger into `extensions`, which hid it. They now import the real module and run the role checks BEFORE `init_logger()`, exactly like a gunicorn worker |
|
||||
|
||||
### Set 25 — Overtime Counted SP/PW/PT/C Hours (Sept 16, 2026)
|
||||
| File | Fix |
|
||||
|---|---|
|
||||
| `routes/time_attendance_export.py` | Both exports added EVERY paired hour to the weekly total that overtime is calculated from, so 36 h Regular + 8 h SP produced 4 h OT. Added `_pair_is_regular()` and a `weekly_regular_hours` counter; the weekly-boundary and final weekly rows now take Regular/OT from it (§13). Column H is unchanged (all hours worked) |
|
||||
| `routes/time_attendance_export.py` | Export by Building: a note row under the date range states that its Weekly Total / Regular / OT are **per building** and exclude SP/PW/PT/C — the user was unsure whether that sheet is used for pay, so it is marked rather than changed. Shifts every Sheet0 row down by one (the Filtered Report is built from row bookkeeping, so it follows) |
|
||||
| — | Verified by running both real export functions over the same fixture week, before (HEAD) vs after: 36 h Reg + 8 h SP → OT 4 → 0; 40 h Reg + 4 h C → 4 → 0; Regular-IN/SP-OUT cross-type pair → 4 → 0; 44 h plain Regular unchanged at 4; two-building employee unchanged (20 h in the main export, 0 per building block); punch rows identical; SP/C summary rows unchanged |
|
||||
| — | **Payroll impact:** weeks where an employee had both special-type hours and 40+ total hours will now show less overtime than the same export produced before |
|
||||
|
||||
### Set 26 — Work-Type IDs With a Separator (`1759.PW`) (Sept 16, 2026)
|
||||
| File | Fix |
|
||||
|---|---|
|
||||
| `working_hours_calculator.py` | `parse_employee_id_for_work_type()` accepted only a space (`\s*`), so `1759.PW` — a spelling that really exists in imported data — parsed as base `1759.PW` / regular. The exports then showed a separate REGULAR employee "1759.PW" while the report and SQL filters (which use `[^0-9A-Z]*`) treated it as PW hours for 1759. Separator class aligned with the other two parsers (§13) |
|
||||
| `time_attendance_import_service.py` | `_clean_employee_id()` now stores work-type IDs canonically (`1759.PW` → `1759PW`), keeps zero padding, and no longer truncates `1234.5` to `1234` (silent data loss) |
|
||||
| — | Verified: 17 ID spellings through the parser (before vs after), agreement with the report's JS parser, 11 cleaner cases (incl. the two old bugs), and an end-to-end export where a 7 h `1759.PW` shift now appears as PW hours for employee 1759 (39 h worked / 32 regular / 0 OT) instead of its own block. Overtime (24) and step-1 (71) suites still pass |
|
||||
| — | **No data migration:** existing rows keep their stored spelling and are read correctly now. Exports for past weeks will move those hours from "Employee 1759.PW" Regular into 1759's PW row — and out of the overtime base (Set 25) |
|
||||
|
||||
### Set 27 — Import Dates, PM Data Scoping, Manual Entry (Sept 16, 2026)
|
||||
| File | Fix |
|
||||
|---|---|
|
||||
| `time_attendance_import_service.py` | `_parse_date_field()` replaces `pd.to_datetime()` at all four call sites: Excel serial numbers imported as **1970-01-01**, and `03/04/2026` was read month-first with no rule written down (§12) |
|
||||
| `time_attendance_import_service.py` | `_rollback_partial_batch()` — a mid-import failure left a partial batch (commits every 50 rows); the batch is now deleted and reported, so an import is all-or-nothing |
|
||||
| `utils/helpers.py` | `load_project_manager_scope()` — one fail-closed source for a PM's projects/locations (§4) |
|
||||
| `routes/attendance.py` | Scope applied to `attendance_locations_api`, `time_attendance_locations_api`, `search_employees_api` (location-only PMs resolved to their projects), `get_project_locations_api` |
|
||||
| `routes/dashboard.py` | Scope applied to the dashboard QR + project lists, `project_qr_codes` (redirect for another project), `dashboard_stats_api`, `dashboard_realtime_api`. "Today" now uses local time |
|
||||
| `routes/attendance_edit.py` | `_normalize_manual_employee_id()` — manual add did `int(employee_id)`, so `1234SP` could not be entered at all, and edit accepted any text (pseudo-employees in the exports). Both now accept 1–4 digits with an optional work type and store it canonically; timestamps local |
|
||||
| `app.py` | `CURRENT_YEAR` from local time |
|
||||
| — | Verified offline (42 checks): 16 date cases + 6 rejections, partial-batch rollback against a fake session, PM scope loading, scoped vs unscoped SQL for both locations APIs (values bound as parameters), empty result for an unassigned PM, 9 manual-ID cases, and no `utcnow()` left in the two route files. Suites still green: 71 (step 1), 24 (overtime), 38 (work-type IDs). Not run against MySQL or a browser |
|
||||
|
||||
---
|
||||
|
||||
## 21. Infrastructure & Deployment
|
||||
|
||||
@@ -161,7 +161,7 @@ def create_app() -> Flask:
|
||||
return {
|
||||
'COMPANY_NAME': os.environ.get('COMPANY_NAME', 'QR Code Management System'),
|
||||
'THEME_NAME': os.environ.get('THEME_NAME', ''),
|
||||
'CURRENT_YEAR': datetime.utcnow().year,
|
||||
'CURRENT_YEAR': datetime.now().year, # local time (§18)
|
||||
}
|
||||
|
||||
@app.context_processor
|
||||
|
||||
+91
-31
@@ -24,6 +24,7 @@ from sqlalchemy import text, or_, and_
|
||||
from logger_handler import log_user_activity, log_database_operations
|
||||
from utils.helpers import (
|
||||
admin_required,
|
||||
load_project_manager_scope,
|
||||
expand_employee_id_filter,
|
||||
get_base_employee_id,
|
||||
get_client_ip,
|
||||
@@ -837,27 +838,38 @@ def time_attendance_locations_api():
|
||||
Used by the time attendance records page to dynamically scope the location dropdown."""
|
||||
try:
|
||||
project_id = request.args.get('project_id', '').strip()
|
||||
conditions = ["location_name IS NOT NULL"]
|
||||
params = {}
|
||||
|
||||
if project_id:
|
||||
try:
|
||||
project_id_int = int(project_id)
|
||||
params['project_id'] = int(project_id)
|
||||
except (ValueError, TypeError):
|
||||
return jsonify({'success': False, 'error': 'Invalid project_id'}), 400
|
||||
conditions.append("project_id = :project_id")
|
||||
|
||||
result = db.session.execute(text("""
|
||||
# Project Managers see only their assigned projects / locations (§4)
|
||||
is_pm, allowed_project_ids, allowed_location_names = load_project_manager_scope()
|
||||
if is_pm:
|
||||
if not (allowed_project_ids or allowed_location_names):
|
||||
return jsonify({'success': True, 'locations': []})
|
||||
scope = []
|
||||
if allowed_project_ids:
|
||||
placeholders = ', '.join(f':pm_project_{i}' for i in range(len(allowed_project_ids)))
|
||||
scope.append(f"project_id IN ({placeholders})")
|
||||
params.update({f'pm_project_{i}': pid for i, pid in enumerate(allowed_project_ids)})
|
||||
if allowed_location_names:
|
||||
placeholders = ', '.join(f':pm_location_{i}' for i in range(len(allowed_location_names)))
|
||||
scope.append(f"location_name IN ({placeholders})")
|
||||
params.update({f'pm_location_{i}': loc for i, loc in enumerate(allowed_location_names)})
|
||||
conditions.append('(' + ' OR '.join(scope) + ')')
|
||||
|
||||
result = db.session.execute(text(f"""
|
||||
SELECT DISTINCT location_name
|
||||
FROM time_attendance
|
||||
WHERE project_id = :project_id
|
||||
AND location_name IS NOT NULL
|
||||
WHERE {' AND '.join(conditions)}
|
||||
ORDER BY location_name
|
||||
"""), {'project_id': project_id_int})
|
||||
else:
|
||||
result = db.session.execute(text("""
|
||||
SELECT DISTINCT location_name
|
||||
FROM time_attendance
|
||||
WHERE location_name IS NOT NULL
|
||||
ORDER BY location_name
|
||||
"""))
|
||||
"""), params)
|
||||
|
||||
locations = [row[0] for row in result.fetchall()]
|
||||
logger_handler.logger.info(
|
||||
@@ -878,28 +890,39 @@ def attendance_locations_api():
|
||||
Used by the attendance report page to dynamically scope the location dropdown when a project is selected."""
|
||||
try:
|
||||
project_id = request.args.get('project_id', '').strip()
|
||||
conditions = ["ad.location_name IS NOT NULL"]
|
||||
params = {}
|
||||
|
||||
if project_id:
|
||||
try:
|
||||
project_id_int = int(project_id)
|
||||
params['project_id'] = int(project_id)
|
||||
except (ValueError, TypeError):
|
||||
return jsonify({'success': False, 'error': 'Invalid project_id'}), 400
|
||||
conditions.append("qc.project_id = :project_id")
|
||||
|
||||
result = db.session.execute(text("""
|
||||
# Project Managers see only their assigned projects / locations (§4)
|
||||
is_pm, allowed_project_ids, allowed_location_names = load_project_manager_scope()
|
||||
if is_pm:
|
||||
if not (allowed_project_ids or allowed_location_names):
|
||||
return jsonify({'success': True, 'locations': []})
|
||||
scope = []
|
||||
if allowed_project_ids:
|
||||
placeholders = ', '.join(f':pm_project_{i}' for i in range(len(allowed_project_ids)))
|
||||
scope.append(f"qc.project_id IN ({placeholders})")
|
||||
params.update({f'pm_project_{i}': pid for i, pid in enumerate(allowed_project_ids)})
|
||||
if allowed_location_names:
|
||||
placeholders = ', '.join(f':pm_location_{i}' for i in range(len(allowed_location_names)))
|
||||
scope.append(f"ad.location_name IN ({placeholders})")
|
||||
params.update({f'pm_location_{i}': loc for i, loc in enumerate(allowed_location_names)})
|
||||
conditions.append('(' + ' OR '.join(scope) + ')')
|
||||
|
||||
result = db.session.execute(text(f"""
|
||||
SELECT DISTINCT ad.location_name
|
||||
FROM attendance_data ad
|
||||
INNER JOIN qr_codes qc ON ad.qr_code_id = qc.id
|
||||
WHERE qc.project_id = :project_id
|
||||
AND ad.location_name IS NOT NULL
|
||||
LEFT JOIN qr_codes qc ON ad.qr_code_id = qc.id
|
||||
WHERE {' AND '.join(conditions)}
|
||||
ORDER BY ad.location_name
|
||||
"""), {'project_id': project_id_int})
|
||||
else:
|
||||
result = db.session.execute(text("""
|
||||
SELECT DISTINCT location_name
|
||||
FROM attendance_data
|
||||
WHERE location_name IS NOT NULL
|
||||
ORDER BY location_name
|
||||
"""))
|
||||
"""), params)
|
||||
|
||||
locations = [row[0] for row in result.fetchall()]
|
||||
logger_handler.logger.info(
|
||||
@@ -930,15 +953,35 @@ def search_employees_api():
|
||||
|
||||
search_pattern = f"%{search_query}%"
|
||||
|
||||
# Project Managers may only search within their own projects (§4). A PM
|
||||
# scoped by locations is resolved to the projects those locations belong
|
||||
# to, so their report filter still works without exposing other projects.
|
||||
is_pm, allowed_project_ids, allowed_location_names = load_project_manager_scope()
|
||||
pm_project_ids = []
|
||||
if is_pm:
|
||||
pm_project_ids = list(allowed_project_ids)
|
||||
if allowed_location_names:
|
||||
location_projects = db.session.query(QRCode.project_id).filter(
|
||||
QRCode.location.in_(allowed_location_names),
|
||||
QRCode.project_id.isnot(None)
|
||||
).distinct().all()
|
||||
pm_project_ids.extend(row[0] for row in location_projects)
|
||||
pm_project_ids = sorted(set(pm_project_ids))
|
||||
if not pm_project_ids:
|
||||
return jsonify({'employees': []})
|
||||
|
||||
# 1. Registered employees — search by ID or name
|
||||
employees = Employee.query.filter(
|
||||
employee_query = Employee.query.filter(
|
||||
db.or_(
|
||||
Employee.id.like(search_pattern),
|
||||
Employee.firstName.like(search_pattern),
|
||||
Employee.lastName.like(search_pattern),
|
||||
db.func.concat(Employee.firstName, ' ', Employee.lastName).like(search_pattern)
|
||||
)
|
||||
).limit(10).all()
|
||||
)
|
||||
if is_pm:
|
||||
employee_query = employee_query.filter(Employee.contractId.in_(pm_project_ids))
|
||||
employees = employee_query.limit(10).all()
|
||||
|
||||
employee_list = [{
|
||||
'id': emp.id,
|
||||
@@ -955,17 +998,25 @@ def search_employees_api():
|
||||
if len(employee_list) < 10:
|
||||
remaining_slots = 10 - len(employee_list)
|
||||
try:
|
||||
unregistered_conditions = ["e.id IS NULL", "ad.employee_id LIKE :pattern"]
|
||||
unregistered_params = {'pattern': search_pattern, 'lim': remaining_slots}
|
||||
if is_pm:
|
||||
placeholders = ', '.join(f':pm_project_{i}' for i in range(len(pm_project_ids)))
|
||||
unregistered_conditions.append(f"qc.project_id IN ({placeholders})")
|
||||
unregistered_params.update(
|
||||
{f'pm_project_{i}': pid for i, pid in enumerate(pm_project_ids)})
|
||||
|
||||
unregistered_rows = db.session.execute(
|
||||
text("""
|
||||
text(f"""
|
||||
SELECT DISTINCT ad.employee_id
|
||||
FROM attendance_data ad
|
||||
LEFT JOIN employee e ON CAST(ad.employee_id AS UNSIGNED) = e.id
|
||||
WHERE e.id IS NULL
|
||||
AND ad.employee_id LIKE :pattern
|
||||
LEFT JOIN qr_codes qc ON ad.qr_code_id = qc.id
|
||||
WHERE {' AND '.join(unregistered_conditions)}
|
||||
ORDER BY ad.employee_id
|
||||
LIMIT :lim
|
||||
"""),
|
||||
{'pattern': search_pattern, 'lim': remaining_slots}
|
||||
unregistered_params
|
||||
).fetchall()
|
||||
|
||||
for row in unregistered_rows:
|
||||
@@ -1000,6 +1051,15 @@ def get_project_locations_api():
|
||||
if not project_id:
|
||||
return jsonify({'success': False, 'locations': [], 'error': 'Project ID required'})
|
||||
|
||||
# Project Managers may only ask about their own projects (§4)
|
||||
is_pm, allowed_project_ids, _ = load_project_manager_scope()
|
||||
if is_pm and int(project_id) not in allowed_project_ids:
|
||||
logger_handler.logger.warning(
|
||||
f"Project Manager {session.get('username')} requested locations for "
|
||||
f"project {project_id}, which is not assigned to them"
|
||||
)
|
||||
return jsonify({'success': True, 'locations': []})
|
||||
|
||||
# Get active QR codes for this project
|
||||
qr_codes = QRCode.query.filter_by(
|
||||
project_id=int(project_id),
|
||||
|
||||
+61
-11
@@ -35,6 +35,38 @@ from openpyxl.utils import get_column_letter
|
||||
|
||||
from routes.attendance import bp # shared blueprint — do not redefine
|
||||
|
||||
# Employee IDs typed on the edit / manual-add forms follow the same rule as a
|
||||
# check-in: 1 to 4 digits, optionally carrying a work-type code.
|
||||
MANUAL_EMPLOYEE_ID_MAX_DIGITS = 4
|
||||
|
||||
|
||||
def _normalize_manual_employee_id(raw_employee_id):
|
||||
"""
|
||||
(canonical_id, base_id, work_type) for an ID typed on a form, or
|
||||
(None, None, None) when it is not a valid employee ID.
|
||||
|
||||
"1234" -> ("1234", "1234", "regular")
|
||||
"1234sp" -> ("1234SP", "1234", "SP")
|
||||
"1234 . PW"-> ("1234PW", "1234", "PW")
|
||||
"12345", "abc", "" -> (None, None, None)
|
||||
|
||||
Manual entry used to do int(employee_id), so an extra-work record such as
|
||||
"1234SP" could not be added by hand at all, and edit accepted any text,
|
||||
which created pseudo-employees in the exports (§13).
|
||||
"""
|
||||
from working_hours_calculator import parse_employee_id_for_work_type
|
||||
|
||||
raw = str(raw_employee_id or '').strip()
|
||||
if not raw:
|
||||
return None, None, None
|
||||
|
||||
base_id, work_type = parse_employee_id_for_work_type(raw)
|
||||
if not re.fullmatch(r'[0-9]+', base_id) or len(base_id) > MANUAL_EMPLOYEE_ID_MAX_DIGITS:
|
||||
return None, None, None
|
||||
|
||||
canonical = base_id if work_type == 'regular' else f"{base_id}{work_type}"
|
||||
return canonical, base_id, work_type
|
||||
|
||||
|
||||
@bp.route('/attendance/<int:record_id>/edit', methods=['GET', 'POST'], endpoint='edit_attendance')
|
||||
@login_required
|
||||
@@ -74,7 +106,16 @@ def edit_attendance(record_id):
|
||||
}
|
||||
|
||||
# Update attendance record fields
|
||||
new_employee_id = request.form['employee_id'].strip().upper()
|
||||
new_employee_id, _base_id, _work_type = _normalize_manual_employee_id(
|
||||
request.form.get('employee_id', ''))
|
||||
if not new_employee_id:
|
||||
flash('Employee ID must be 1 to 4 digits, optionally with a work type '
|
||||
'(for example 1234 or 1234SP).', 'error')
|
||||
projects = Project.query.filter_by(active_status=True).order_by(Project.name).all()
|
||||
return render_template('edit_attendance.html',
|
||||
attendance_record=attendance_record,
|
||||
projects=projects,
|
||||
qr_codes=QRCode.query.filter_by(active_status=True).all())
|
||||
new_check_in_date = datetime.strptime(request.form['check_in_date'], '%Y-%m-%d').date()
|
||||
new_check_in_time = datetime.strptime(request.form['check_in_time'], '%H:%M').time()
|
||||
new_location_name = request.form['location_name'].strip()
|
||||
@@ -120,10 +161,12 @@ def edit_attendance(record_id):
|
||||
attendance_record.check_in_time = new_check_in_time
|
||||
attendance_record.location_name = new_location_name
|
||||
attendance_record.qr_code_id = int(new_qr_code_id)
|
||||
attendance_record.updated_timestamp = datetime.utcnow()
|
||||
# Local time throughout (§18): check-in dates and times are local, so
|
||||
# a UTC audit stamp read hours apart from the record it describes.
|
||||
attendance_record.updated_timestamp = datetime.now()
|
||||
|
||||
# Store the audit note with timestamp and user info
|
||||
timestamp = datetime.utcnow().strftime('%Y-%m-%d %H:%M:%S UTC')
|
||||
timestamp = datetime.now().strftime('%Y-%m-%d %H:%M:%S')
|
||||
username = session.get('username', 'Unknown')
|
||||
role = session.get('role', 'unknown')
|
||||
|
||||
@@ -262,10 +305,17 @@ def save_manual_attendance():
|
||||
flash('All fields are required.', 'error')
|
||||
return redirect(url_for('attendance.add_manual_attendance'))
|
||||
|
||||
# Validate employee exists
|
||||
employee = Employee.query.filter_by(id=int(employee_id)).first()
|
||||
# Validate the ID, and keep any work-type code ("1234SP") on the stored value
|
||||
canonical_employee_id, base_employee_id, work_type = _normalize_manual_employee_id(employee_id)
|
||||
if not canonical_employee_id:
|
||||
flash('Employee ID must be 1 to 4 digits, optionally with a work type '
|
||||
'(for example 1234 or 1234SP).', 'error')
|
||||
return redirect(url_for('attendance.add_manual_attendance'))
|
||||
|
||||
# Validate employee exists (by the numeric base ID)
|
||||
employee = Employee.query.filter_by(id=int(base_employee_id)).first()
|
||||
if not employee:
|
||||
flash(f'Employee with ID {employee_id} not found.', 'error')
|
||||
flash(f'Employee with ID {base_employee_id} not found.', 'error')
|
||||
return redirect(url_for('attendance.add_manual_attendance'))
|
||||
|
||||
# Get QR code (location)
|
||||
@@ -285,7 +335,7 @@ def save_manual_attendance():
|
||||
|
||||
# Check if record already exists for this employee, location, date, and time
|
||||
existing_record = AttendanceData.query.filter_by(
|
||||
employee_id=str(employee_id),
|
||||
employee_id=canonical_employee_id,
|
||||
qr_code_id=qr_code.id,
|
||||
check_in_date=check_date_obj,
|
||||
check_in_time=check_time_obj
|
||||
@@ -300,7 +350,7 @@ def save_manual_attendance():
|
||||
# Set fixed distance of 0.010 miles
|
||||
new_attendance = AttendanceData(
|
||||
qr_code_id=qr_code.id,
|
||||
employee_id=str(employee_id),
|
||||
employee_id=canonical_employee_id,
|
||||
check_in_date=check_date_obj,
|
||||
check_in_time=check_time_obj,
|
||||
location_name=qr_code.location,
|
||||
@@ -320,8 +370,8 @@ def save_manual_attendance():
|
||||
status='present',
|
||||
verification_required=False,
|
||||
verification_status='approved',
|
||||
created_timestamp=datetime.utcnow(),
|
||||
updated_timestamp=datetime.utcnow()
|
||||
created_timestamp=datetime.now(),
|
||||
updated_timestamp=datetime.now()
|
||||
)
|
||||
|
||||
db.session.add(new_attendance)
|
||||
@@ -330,7 +380,7 @@ def save_manual_attendance():
|
||||
# Log the manual entry
|
||||
logger_handler.logger.info(
|
||||
f"Manual attendance record created by {session.get('username')} ({user_role}): "
|
||||
f"Employee {employee.firstName} {employee.lastName} (ID: {employee_id}), "
|
||||
f"Employee {employee.firstName} {employee.lastName} (ID: {canonical_employee_id}), "
|
||||
f"Location: {qr_code.location}, Event: {qr_code.location_event}, "
|
||||
f"Date: {check_date}, Time: {check_time}"
|
||||
)
|
||||
|
||||
+100
-18
@@ -8,6 +8,7 @@ Routes: /dashboard, /project/<id>/qr-codes, /dashboard/search,
|
||||
"""
|
||||
from flask import abort, Blueprint, render_template, request, redirect, flash, session, jsonify, url_for
|
||||
from datetime import datetime, timedelta, date, time
|
||||
from sqlalchemy import or_
|
||||
|
||||
from extensions import db, logger_handler
|
||||
from models.attendance import AttendanceData
|
||||
@@ -15,7 +16,27 @@ from models.project import Project
|
||||
from models.qrcode import QRCode
|
||||
from models.user import User
|
||||
from logger_handler import log_user_activity, log_database_operations
|
||||
from utils.helpers import login_required
|
||||
from utils.helpers import login_required, load_project_manager_scope
|
||||
|
||||
|
||||
def _project_manager_qr_filter():
|
||||
"""
|
||||
(is_pm, qr_filter) for the session user — the QR codes a Project Manager may
|
||||
see (§4). qr_filter is None for everyone else, and a never-true condition for
|
||||
a PM with no assignments.
|
||||
"""
|
||||
is_pm, allowed_project_ids, allowed_location_names = load_project_manager_scope()
|
||||
if not is_pm:
|
||||
return False, None
|
||||
|
||||
scope = []
|
||||
if allowed_project_ids:
|
||||
scope.append(QRCode.project_id.in_(allowed_project_ids))
|
||||
if allowed_location_names:
|
||||
scope.append(QRCode.location.in_(allowed_location_names))
|
||||
if not scope:
|
||||
return True, QRCode.id.is_(None) # assigned nothing → sees nothing
|
||||
return True, or_(*scope)
|
||||
|
||||
bp = Blueprint('dashboard', __name__)
|
||||
|
||||
@@ -35,6 +56,11 @@ def dashboard():
|
||||
# Build QR codes query with filters
|
||||
qr_query = QRCode.query
|
||||
|
||||
# Project Managers only see QR codes in their assigned projects/locations
|
||||
is_pm, pm_qr_filter = _project_manager_qr_filter()
|
||||
if pm_qr_filter is not None:
|
||||
qr_query = qr_query.filter(pm_qr_filter)
|
||||
|
||||
# Apply name filter if provided
|
||||
if search_name:
|
||||
qr_query = qr_query.filter(QRCode.name.ilike(f'%{search_name}%'))
|
||||
@@ -47,7 +73,16 @@ def dashboard():
|
||||
|
||||
# Execute query
|
||||
qr_codes = qr_query.order_by(QRCode.created_date.desc()).all()
|
||||
projects = Project.query.order_by(Project.name.asc()).all()
|
||||
|
||||
project_query = Project.query
|
||||
if is_pm:
|
||||
# Only the projects the PM is assigned to, plus those behind their locations
|
||||
visible_project_ids = {qr.project_id for qr in qr_codes if qr.project_id}
|
||||
_, allowed_project_ids, _ = load_project_manager_scope()
|
||||
visible_project_ids.update(allowed_project_ids)
|
||||
project_query = (project_query.filter(Project.id.in_(sorted(visible_project_ids)))
|
||||
if visible_project_ids else project_query.filter(Project.id.is_(None)))
|
||||
projects = project_query.order_by(Project.name.asc()).all()
|
||||
|
||||
# Log dashboard access with filter info
|
||||
filter_info = []
|
||||
@@ -88,6 +123,15 @@ def project_qr_codes(project_id):
|
||||
if project is None:
|
||||
abort(404)
|
||||
|
||||
# Project Managers may only open their own projects (§4)
|
||||
is_pm, allowed_project_ids, _ = load_project_manager_scope()
|
||||
if is_pm and project_id not in allowed_project_ids:
|
||||
logger_handler.logger.warning(
|
||||
f"Project Manager {session.get('username')} tried to open project {project_id}"
|
||||
)
|
||||
flash('You do not have permission to view that project.', 'error')
|
||||
return redirect(url_for('dashboard.dashboard'))
|
||||
|
||||
# Get search parameters from URL
|
||||
search_name = request.args.get('search_name', '').strip()
|
||||
search_status = request.args.get('search_status', '').strip()
|
||||
@@ -154,36 +198,59 @@ def search_qr_codes():
|
||||
def dashboard_stats_api():
|
||||
"""API endpoint for dashboard statistics"""
|
||||
try:
|
||||
# Get current stats
|
||||
total_qr_codes = QRCode.query.filter_by(active_status=True).count()
|
||||
# Project Manager scope (§4): every count below is restricted to the QR
|
||||
# codes / locations they are assigned to.
|
||||
is_pm, allowed_project_ids, allowed_location_names = load_project_manager_scope()
|
||||
_, pm_qr_filter = _project_manager_qr_filter()
|
||||
|
||||
# Today's check-ins
|
||||
today = datetime.utcnow().date()
|
||||
today_checkins = AttendanceData.query.filter(
|
||||
qr_base_query = QRCode.query.filter_by(active_status=True)
|
||||
attendance_base_query = AttendanceData.query
|
||||
location_base_query = db.session.query(AttendanceData.location_name)
|
||||
project_base_query = Project.query.filter_by(active_status=True)
|
||||
|
||||
if is_pm:
|
||||
qr_base_query = qr_base_query.filter(pm_qr_filter)
|
||||
attendance_scope = []
|
||||
if allowed_project_ids:
|
||||
attendance_scope.append(AttendanceData.qr_code_id.in_(
|
||||
db.session.query(QRCode.id).filter(QRCode.project_id.in_(allowed_project_ids))
|
||||
))
|
||||
project_base_query = project_base_query.filter(Project.id.in_(allowed_project_ids))
|
||||
else:
|
||||
project_base_query = project_base_query.filter(Project.id.is_(None))
|
||||
if allowed_location_names:
|
||||
attendance_scope.append(AttendanceData.location_name.in_(allowed_location_names))
|
||||
attendance_condition = or_(*attendance_scope) if attendance_scope else AttendanceData.id.is_(None)
|
||||
attendance_base_query = attendance_base_query.filter(attendance_condition)
|
||||
location_base_query = location_base_query.filter(attendance_condition)
|
||||
|
||||
# Get current stats
|
||||
total_qr_codes = qr_base_query.count()
|
||||
|
||||
# Today's check-ins — local date, matching how check-ins are stored
|
||||
today = datetime.now().date()
|
||||
today_checkins = attendance_base_query.filter(
|
||||
AttendanceData.check_in_date == today
|
||||
).count()
|
||||
|
||||
# Active projects
|
||||
active_projects = Project.query.filter_by(active_status=True).count()
|
||||
active_projects = project_base_query.count()
|
||||
|
||||
# Unique locations
|
||||
unique_locations = db.session.query(
|
||||
AttendanceData.location_name
|
||||
).distinct().count()
|
||||
unique_locations = location_base_query.distinct().count()
|
||||
|
||||
# Calculate trends (compared to last month)
|
||||
last_month = datetime.utcnow() - timedelta(days=30)
|
||||
last_month = datetime.now() - timedelta(days=30)
|
||||
|
||||
# QR codes trend
|
||||
old_qr_count = QRCode.query.filter(
|
||||
QRCode.created_date <= last_month,
|
||||
QRCode.active_status == True
|
||||
old_qr_count = qr_base_query.filter(
|
||||
QRCode.created_date <= last_month
|
||||
).count()
|
||||
qr_change = ((total_qr_codes - old_qr_count) / max(old_qr_count, 1)) * 100
|
||||
|
||||
# Check-ins trend (yesterday)
|
||||
yesterday = today - timedelta(days=1)
|
||||
yesterday_checkins = AttendanceData.query.filter(
|
||||
yesterday_checkins = attendance_base_query.filter(
|
||||
AttendanceData.check_in_date == yesterday
|
||||
).count()
|
||||
checkin_change = ((today_checkins - yesterday_checkins) / max(yesterday_checkins, 1)) * 100
|
||||
@@ -214,12 +281,27 @@ def dashboard_realtime_api():
|
||||
"""API endpoint for real-time dashboard data"""
|
||||
try:
|
||||
# Get recent activity (last 10 check-ins)
|
||||
recent_activity = db.session.query(
|
||||
recent_query = db.session.query(
|
||||
AttendanceData.employee_id,
|
||||
AttendanceData.location_name,
|
||||
AttendanceData.check_in_time,
|
||||
AttendanceData.check_in_date
|
||||
).order_by(
|
||||
)
|
||||
|
||||
# Project Managers only see activity at their own projects/locations (§4)
|
||||
is_pm, allowed_project_ids, allowed_location_names = load_project_manager_scope()
|
||||
if is_pm:
|
||||
recent_scope = []
|
||||
if allowed_project_ids:
|
||||
recent_scope.append(AttendanceData.qr_code_id.in_(
|
||||
db.session.query(QRCode.id).filter(QRCode.project_id.in_(allowed_project_ids))
|
||||
))
|
||||
if allowed_location_names:
|
||||
recent_scope.append(AttendanceData.location_name.in_(allowed_location_names))
|
||||
recent_query = recent_query.filter(or_(*recent_scope) if recent_scope
|
||||
else AttendanceData.id.is_(None))
|
||||
|
||||
recent_activity = recent_query.order_by(
|
||||
AttendanceData.check_in_date.desc(),
|
||||
AttendanceData.check_in_time.desc()
|
||||
).limit(10).all()
|
||||
|
||||
@@ -85,6 +85,20 @@ def _overnight_aware_sort_key(record):
|
||||
seconds_total += 24 * 3600
|
||||
return seconds_total
|
||||
|
||||
def _pair_is_regular(check_in_record, check_out_record) -> bool:
|
||||
"""
|
||||
True when a completed pair counts toward the 40-hour overtime rule.
|
||||
|
||||
SP / PW / PT / C hours are paid but never build toward overtime (§13). The
|
||||
OUT record's work type wins, mirroring effective_work_type elsewhere, so a
|
||||
Regular IN paired with an SP OUT counts as SP.
|
||||
"""
|
||||
in_work_type = getattr(check_in_record, 'work_type', None)
|
||||
out_work_type = getattr(check_out_record, 'work_type', None)
|
||||
effective = out_work_type or in_work_type
|
||||
return effective not in ('SP', 'PW', 'PT', 'C')
|
||||
|
||||
|
||||
def _qtr(decimal_hours: float) -> float:
|
||||
"""
|
||||
Round a decimal-hours value to the nearest quarter hour (.00/.25/.50/.75).
|
||||
@@ -596,6 +610,10 @@ def export_time_attendance_excel(records, project_name_for_filename, date_range_
|
||||
current_week_start = None
|
||||
grand_regular_hours = 0
|
||||
grand_ot_hours = 0
|
||||
# Regular-only hours feeding the 40-hour overtime rule. SP/PW/PT/C hours
|
||||
# are worked and paid, but they do NOT build toward overtime (§13), so
|
||||
# they are excluded here while column H still shows every hour worked.
|
||||
weekly_regular_hours = 0
|
||||
# Accumulate SP/PW/PT/C hours from cross-type pairs (where the calculator
|
||||
# could not detect them because it processes each work-type stream independently).
|
||||
cross_type_sp_hours = 0.0
|
||||
@@ -624,9 +642,10 @@ def export_time_attendance_excel(records, project_name_for_filename, date_range_
|
||||
_report_start = start_date
|
||||
week_start = (_report_start + timedelta(days=((date_obj.date() - _report_start).days // 7) * 7))
|
||||
if current_week_start is not None and week_start != current_week_start:
|
||||
# Write weekly total row
|
||||
week_regular = min(weekly_total_hours, 40.0)
|
||||
week_overtime = max(0, weekly_total_hours - 40.0)
|
||||
# Write weekly total row. Column H = every hour worked; Regular
|
||||
# and OT are computed from regular hours only (SP/PW/PT/C excluded).
|
||||
week_regular = min(weekly_regular_hours, 40.0)
|
||||
week_overtime = max(0, weekly_regular_hours - 40.0)
|
||||
|
||||
ws.cell(row=current_row, column=7, value='Weekly Total: ').font = bold_font
|
||||
ws.cell(row=current_row, column=8, value=_qtr(weekly_total_hours)).font = bold_font
|
||||
@@ -638,6 +657,7 @@ def export_time_attendance_excel(records, project_name_for_filename, date_range_
|
||||
current_row += 1
|
||||
|
||||
weekly_total_hours = 0
|
||||
weekly_regular_hours = 0
|
||||
|
||||
current_week_start = week_start
|
||||
|
||||
@@ -666,6 +686,8 @@ def export_time_attendance_excel(records, project_name_for_filename, date_range_
|
||||
# group, and sum only complete pairs. This ensures the daily total in
|
||||
# column H matches exactly the pairs rendered in the export rows.
|
||||
_day_total_hours = 0.0
|
||||
# Same pairs, minus SP/PW/PT/C: the overtime base (§13)
|
||||
_day_regular_hours = 0.0
|
||||
# Track which records are consumed by same-building pairing so the
|
||||
# cross-building pass only considers true orphans.
|
||||
_same_building_used_ids = set()
|
||||
@@ -704,6 +726,8 @@ def export_time_attendance_excel(records, project_name_for_filename, date_range_
|
||||
if _duration > 24:
|
||||
continue
|
||||
_day_total_hours += _duration
|
||||
if _pair_is_regular(_in_r, _out_r):
|
||||
_day_regular_hours += _duration
|
||||
_out_used[_oi2] = True
|
||||
_same_building_used_ids.add(id(_in_r))
|
||||
_same_building_used_ids.add(id(_out_r))
|
||||
@@ -768,6 +792,8 @@ def export_time_attendance_excel(records, project_name_for_filename, date_range_
|
||||
'hours': _cb_dur,
|
||||
})
|
||||
_day_total_hours += _cb_dur
|
||||
if _pair_is_regular(_cb_in, _cb_out):
|
||||
_day_regular_hours += _cb_dur
|
||||
logger_handler.logger.info(
|
||||
f"[TA Export] Cross-building pair for employee {employee_id} on {date_str}: "
|
||||
f"IN {_cb_in.location_name} @ {_cb_in.check_in_time} → "
|
||||
@@ -788,6 +814,7 @@ def export_time_attendance_excel(records, project_name_for_filename, date_range_
|
||||
|
||||
total_hours = _qtr(_day_total_hours)
|
||||
weekly_total_hours += total_hours
|
||||
weekly_regular_hours += _qtr(_day_regular_hours)
|
||||
|
||||
# Daily total display (only shown on last location's last row)
|
||||
daily_total_display = _qtr(total_hours) if total_hours > 0 else ''
|
||||
@@ -1282,10 +1309,10 @@ def export_time_attendance_excel(records, project_name_for_filename, date_range_
|
||||
# END CROSS-BUILDING PAIR ROW WRITING
|
||||
# -------------------------------------------------------------------
|
||||
|
||||
# Write final weekly total for this employee
|
||||
# Write final weekly total for this employee (Regular/OT from regular hours only)
|
||||
if weekly_total_hours > 0:
|
||||
week_regular = min(weekly_total_hours, 40.0)
|
||||
week_overtime = max(0, weekly_total_hours - 40.0)
|
||||
week_regular = min(weekly_regular_hours, 40.0)
|
||||
week_overtime = max(0, weekly_regular_hours - 40.0)
|
||||
|
||||
ws.cell(row=current_row, column=7, value='Weekly Total: ').font = bold_font
|
||||
ws.cell(row=current_row, column=8, value=_qtr(weekly_total_hours)).font = bold_font
|
||||
@@ -1758,6 +1785,20 @@ def export_time_attendance_by_building_excel(records, project_name_for_filename,
|
||||
date_cell.alignment = Alignment(horizontal='left')
|
||||
current_row += 1
|
||||
|
||||
# Row 5: how the totals in this sheet are calculated. Overtime here is per
|
||||
# BUILDING (each block counts only that building's hours), so an employee
|
||||
# split across two buildings shows no overtime even past 40 hours — the main
|
||||
# Time Attendance export totals the week across all buildings.
|
||||
ws.merge_cells(f'A{current_row}:N{current_row}')
|
||||
ot_note_cell = ws.cell(
|
||||
row=current_row, column=1,
|
||||
value=('Note: Weekly Total / Regular / OT below are per building, and overtime excludes '
|
||||
'SP / PW / PT / C hours. Use the main Time Attendance export for payroll totals.')
|
||||
)
|
||||
ot_note_cell.font = Font(name='Aptos Narrow', size=10, italic=True)
|
||||
ot_note_cell.alignment = Alignment(horizontal='left')
|
||||
current_row += 1
|
||||
|
||||
# Empty rows before first building
|
||||
current_row += 2
|
||||
|
||||
@@ -1967,6 +2008,9 @@ def export_time_attendance_by_building_excel(records, project_name_for_filename,
|
||||
current_week_start = None
|
||||
grand_regular_hours = 0
|
||||
grand_ot_hours = 0
|
||||
# Regular-only hours for the 40-hour rule (SP/PW/PT/C excluded, §13).
|
||||
# Note: per BUILDING in this sheet — see the note under the date range.
|
||||
weekly_regular_hours = 0
|
||||
# Accumulate raw regular-only (non-SP/PW/PT) pair hours.
|
||||
# Used for the "Regular" summary row when the employee also has
|
||||
# special work-type hours.
|
||||
@@ -1993,9 +2037,10 @@ def export_time_attendance_by_building_excel(records, project_name_for_filename,
|
||||
_report_start = start_date
|
||||
week_start = (_report_start + timedelta(days=((date_obj.date() - _report_start).days // 7) * 7))
|
||||
if current_week_start is not None and week_start != current_week_start:
|
||||
# Write weekly total row
|
||||
week_regular = min(weekly_total_hours, 40.0)
|
||||
week_overtime = max(0, weekly_total_hours - 40.0)
|
||||
# Write weekly total row. Column H = every hour worked at this
|
||||
# building; Regular and OT come from regular hours only.
|
||||
week_regular = min(weekly_regular_hours, 40.0)
|
||||
week_overtime = max(0, weekly_regular_hours - 40.0)
|
||||
|
||||
ws.cell(row=current_row, column=7, value='Weekly Total: ').font = bold_font
|
||||
ws.cell(row=current_row, column=8, value=_qtr(weekly_total_hours)).font = bold_font
|
||||
@@ -2007,6 +2052,7 @@ def export_time_attendance_by_building_excel(records, project_name_for_filename,
|
||||
current_row += 1
|
||||
|
||||
weekly_total_hours = 0
|
||||
weekly_regular_hours = 0
|
||||
|
||||
current_week_start = week_start
|
||||
|
||||
@@ -2145,6 +2191,7 @@ def export_time_attendance_by_building_excel(records, project_name_for_filename,
|
||||
# Calculate daily hours
|
||||
daily_hours = 0
|
||||
_bb_day_non_sp_hours = 0.0 # weekly summary sheet excludes SP time
|
||||
_bb_day_regular_hours = 0.0 # overtime base: excludes SP/PW/PT/C
|
||||
for pair in pairs:
|
||||
if pair['check_in'] and pair['check_out'] and not pair['is_miss_punch']:
|
||||
pair_in = datetime.combine(date_obj, pair['check_in'].check_in_time)
|
||||
@@ -2165,11 +2212,13 @@ def export_time_attendance_by_building_excel(records, project_name_for_filename,
|
||||
_bb_eff_wt = _bb_out_wt or _bb_in_wt # prefer OUT's type (mirrors main export)
|
||||
if _bb_eff_wt not in ('SP', 'PW', 'PT', 'C'):
|
||||
regular_only_hours += _bb_dur
|
||||
_bb_day_regular_hours += _bb_dur
|
||||
if _bb_eff_wt != 'SP':
|
||||
_bb_day_non_sp_hours += _bb_dur
|
||||
|
||||
daily_hours = _qtr(daily_hours)
|
||||
weekly_total_hours += daily_hours
|
||||
weekly_regular_hours += _qtr(_bb_day_regular_hours)
|
||||
|
||||
# Same week anchoring as the Weekly Total rows (report start date)
|
||||
_bb_week_idx = (date_obj.date() - start_date).days // 7
|
||||
@@ -2250,10 +2299,10 @@ def export_time_attendance_by_building_excel(records, project_name_for_filename,
|
||||
|
||||
current_row += 1
|
||||
|
||||
# Write final weekly total
|
||||
# Write final weekly total (Regular/OT from regular hours only)
|
||||
if weekly_total_hours > 0:
|
||||
week_regular = min(weekly_total_hours, 40.0)
|
||||
week_overtime = max(0, weekly_total_hours - 40.0)
|
||||
week_regular = min(weekly_regular_hours, 40.0)
|
||||
week_overtime = max(0, weekly_regular_hours - 40.0)
|
||||
|
||||
ws.cell(row=current_row, column=7, value='Weekly Total: ').font = bold_font
|
||||
ws.cell(row=current_row, column=8, value=_qtr(weekly_total_hours)).font = bold_font
|
||||
|
||||
@@ -6,8 +6,9 @@ Added functionality to detect and present duplicates for user review.
|
||||
"""
|
||||
|
||||
import pandas as pd
|
||||
import re
|
||||
import uuid
|
||||
from datetime import datetime, time
|
||||
from datetime import datetime, date, time
|
||||
from sqlalchemy.exc import SQLAlchemyError
|
||||
from typing import Dict, List, Any, Optional, Tuple
|
||||
import traceback
|
||||
@@ -433,7 +434,7 @@ class TimeAttendanceImportService:
|
||||
employee_name = self._get_employee_name(clean_id)
|
||||
|
||||
# Parse date and time
|
||||
attendance_date = pd.to_datetime(row['Date']).date()
|
||||
attendance_date = self._parse_date_field(row['Date'])
|
||||
attendance_time = self._parse_time_field(row['Time'])
|
||||
|
||||
# Prepare record data
|
||||
@@ -593,10 +594,10 @@ class TimeAttendanceImportService:
|
||||
row_errors.append("Missing Date")
|
||||
else:
|
||||
try:
|
||||
attendance_date = pd.to_datetime(row['Date']).date()
|
||||
attendance_date = self._parse_date_field(row['Date'])
|
||||
row_data['attendance_date'] = attendance_date
|
||||
except Exception:
|
||||
row_errors.append(f"Invalid date format: {row['Date']}")
|
||||
except Exception as date_error:
|
||||
row_errors.append(f"Invalid date format: {row['Date']} ({date_error})")
|
||||
|
||||
# Check and parse Time
|
||||
if pd.isna(row['Time']):
|
||||
@@ -832,7 +833,7 @@ class TimeAttendanceImportService:
|
||||
|
||||
# Validate and parse date
|
||||
try:
|
||||
attendance_date = pd.to_datetime(row['Date']).date()
|
||||
attendance_date = self._parse_date_field(row['Date'])
|
||||
except Exception as date_error:
|
||||
import_results['failed_records'] += 1
|
||||
import_results['errors'].append(f"Row {index + 2}: Invalid date format - {str(date_error)}")
|
||||
@@ -936,17 +937,20 @@ class TimeAttendanceImportService:
|
||||
)
|
||||
|
||||
except SQLAlchemyError as e:
|
||||
self.db.session.rollback()
|
||||
error_msg = f"Database error during import: {str(e)}"
|
||||
import_results['errors'].append(error_msg)
|
||||
# Rows commit in batches of 50 — drop the partial batch (all-or-nothing)
|
||||
self._rollback_partial_batch(batch_id, import_results)
|
||||
import_results['success'] = False
|
||||
|
||||
if self.logger:
|
||||
self.logger.log_database_error('time_attendance_import', e)
|
||||
|
||||
except Exception as e:
|
||||
self.db.session.rollback()
|
||||
error_msg = f"Unexpected error during import: {str(e)}"
|
||||
import_results['errors'].append(error_msg)
|
||||
self._rollback_partial_batch(batch_id, import_results)
|
||||
import_results['success'] = False
|
||||
import_results['traceback'] = traceback.format_exc()
|
||||
|
||||
if self.logger:
|
||||
@@ -955,6 +959,111 @@ class TimeAttendanceImportService:
|
||||
|
||||
return import_results
|
||||
|
||||
# Excel stores dates as days since 1899-12-30. A column formatted as General
|
||||
# arrives as that number instead of a date, and pd.to_datetime() then read it
|
||||
# as nanoseconds since 1970 — every such row imported as 1970-01-01.
|
||||
_EXCEL_EPOCH = '1899-12-30'
|
||||
_EXCEL_SERIAL_MIN = 20000 # 1954-10-03
|
||||
_EXCEL_SERIAL_MAX = 60000 # 2064-04-05
|
||||
|
||||
def _parse_date_field(self, date_value) -> date:
|
||||
"""
|
||||
Parse the Date column into a date.
|
||||
|
||||
Handles:
|
||||
- real date / datetime / pandas Timestamp cells
|
||||
- Excel serial numbers (45123 -> 2023-07-04), the General-format case
|
||||
- text dates, read **month-first** (US time clocks): "03/04/2026" is
|
||||
4 March 2026. When the first number cannot be a month ("13/04/2026")
|
||||
it is read day-first instead.
|
||||
|
||||
Raises ValueError with a readable message — the caller records the row as
|
||||
failed instead of importing a wrong date.
|
||||
"""
|
||||
if date_value is None or (isinstance(date_value, float) and pd.isna(date_value)):
|
||||
raise ValueError('Missing date')
|
||||
|
||||
# Already a date/datetime (openpyxl / pandas parsed the cell)
|
||||
if isinstance(date_value, datetime):
|
||||
return date_value.date()
|
||||
if isinstance(date_value, date):
|
||||
return date_value
|
||||
to_pydatetime = getattr(date_value, 'to_pydatetime', None) # pandas Timestamp
|
||||
if callable(to_pydatetime):
|
||||
return to_pydatetime().date()
|
||||
|
||||
# Excel serial number, as a number or as text
|
||||
serial = None
|
||||
if isinstance(date_value, bool):
|
||||
raise ValueError(f"'{date_value}' is not a date")
|
||||
if isinstance(date_value, (int, float)):
|
||||
serial = float(date_value)
|
||||
else:
|
||||
text_value = str(date_value).strip()
|
||||
if re.fullmatch(r'\d{4,6}(\.\d+)?', text_value):
|
||||
serial = float(text_value)
|
||||
|
||||
if serial is not None:
|
||||
if not (self._EXCEL_SERIAL_MIN <= serial <= self._EXCEL_SERIAL_MAX):
|
||||
raise ValueError(f"'{date_value}' is not a valid date (number out of range)")
|
||||
return pd.to_datetime(serial, unit='D', origin=self._EXCEL_EPOCH).date()
|
||||
|
||||
text_value = str(date_value).strip()
|
||||
if not text_value:
|
||||
raise ValueError('Missing date')
|
||||
|
||||
# Month-first unless the first component cannot be a month
|
||||
day_first = False
|
||||
parts = re.match(r'^(\d{1,2})[/\-.](\d{1,2})[/\-.](\d{2,4})$', text_value)
|
||||
if parts and int(parts.group(1)) > 12:
|
||||
day_first = True
|
||||
|
||||
parsed = pd.to_datetime(text_value, dayfirst=day_first, errors='coerce')
|
||||
if pd.isna(parsed):
|
||||
raise ValueError(f"'{date_value}' is not a recognised date")
|
||||
return parsed.date()
|
||||
|
||||
def _rollback_partial_batch(self, batch_id, import_results=None):
|
||||
"""
|
||||
Remove rows already committed for a failed import batch.
|
||||
|
||||
The row loop commits every 50 records to keep memory flat, so a failure
|
||||
part-way through used to leave a partial batch in the table with nothing
|
||||
to show it was incomplete. An import is all-or-nothing from the user's
|
||||
point of view, so the batch is deleted and the failure reported.
|
||||
"""
|
||||
try:
|
||||
from models.time_attendance import TimeAttendance
|
||||
|
||||
self.db.session.rollback()
|
||||
removed = TimeAttendance.query.filter_by(import_batch_id=batch_id).delete(
|
||||
synchronize_session=False)
|
||||
self.db.session.commit()
|
||||
|
||||
if removed and self.logger:
|
||||
self.logger.logger.warning(
|
||||
f"Import batch {batch_id} failed — removed {removed} partially imported records"
|
||||
)
|
||||
if import_results is not None:
|
||||
import_results['imported_records'] = 0
|
||||
import_results['rolled_back_records'] = removed
|
||||
import_results['errors'].append(
|
||||
f"Import failed — {removed} partially imported records were removed. "
|
||||
f"Nothing from this file was kept."
|
||||
)
|
||||
return removed
|
||||
except Exception as cleanup_error:
|
||||
if self.logger:
|
||||
self.logger.logger.error(
|
||||
f"Could not remove partial import batch {batch_id}: {cleanup_error}", exc_info=True
|
||||
)
|
||||
if import_results is not None:
|
||||
import_results['errors'].append(
|
||||
f"Import failed and the partial batch {batch_id} could not be removed "
|
||||
f"automatically — delete it from the import history."
|
||||
)
|
||||
return 0
|
||||
|
||||
def _parse_time_field(self, time_value) -> time:
|
||||
"""
|
||||
Parse various time formats from Excel
|
||||
@@ -1050,30 +1159,48 @@ class TimeAttendanceImportService:
|
||||
|
||||
def _clean_employee_id(self, employee_id) -> str:
|
||||
"""
|
||||
Clean employee ID to handle various formats
|
||||
Normalise an employee ID from an Excel cell into the form the app stores.
|
||||
|
||||
Args:
|
||||
employee_id: Raw employee ID value
|
||||
1234 / "1234" ............. "1234" (unchanged)
|
||||
1234.0 (Excel numeric) .... "1234"
|
||||
"1759.PW", "1759 - PW" .... "1759PW" canonical work-type spelling
|
||||
"1234.5" .................. "1234.5" NOT an ID — never truncated
|
||||
"01234" ................... "01234" zero padding preserved
|
||||
|
||||
Returns:
|
||||
Cleaned employee ID string
|
||||
Work-type IDs are stored canonically so the calculator, the exports and
|
||||
the report filters all read the same spelling (§13). Before this, an
|
||||
imported "1759.PW" stayed as-is and the exports counted it as a separate
|
||||
REGULAR employee called "1759.PW" instead of PW hours for employee 1759.
|
||||
"""
|
||||
try:
|
||||
# Convert to string first
|
||||
id_str = str(employee_id).strip()
|
||||
from working_hours_calculator import parse_employee_id_for_work_type
|
||||
|
||||
# Handle float values like 1234.0 or '1234.0'
|
||||
id_str = str(employee_id).strip()
|
||||
if not id_str:
|
||||
return id_str
|
||||
|
||||
# Excel hands whole numbers over as floats: 1234.0 -> "1234". A real
|
||||
# fraction is not an employee ID, so keep it as typed and let validation
|
||||
# flag it rather than silently importing hours for employee 1234.
|
||||
if '.' in id_str:
|
||||
# Convert to float, then to int, then back to string
|
||||
# This removes the decimal part: 1234.0 -> 1234
|
||||
id_str = str(int(float(id_str)))
|
||||
try:
|
||||
numeric = float(id_str)
|
||||
except (TypeError, ValueError):
|
||||
numeric = None
|
||||
if numeric is not None:
|
||||
if numeric.is_integer():
|
||||
return str(int(numeric))
|
||||
if self.logger:
|
||||
self.logger.logger.warning(
|
||||
f"Employee ID '{id_str}' is not a whole number — imported unchanged"
|
||||
)
|
||||
return id_str
|
||||
|
||||
# "1759.PW" / "1759 - PW" / "PW.1759" -> "1759PW"
|
||||
base_id, work_type = parse_employee_id_for_work_type(id_str)
|
||||
if work_type != 'regular' and base_id.isdigit():
|
||||
return f"{base_id}{work_type}"
|
||||
|
||||
return id_str
|
||||
except (ValueError, TypeError) as e:
|
||||
# If conversion fails, return original string
|
||||
if self.logger:
|
||||
self.logger.logger.warning(f"Could not clean employee ID '{employee_id}': {e}")
|
||||
return str(employee_id).strip()
|
||||
|
||||
def validate_excel_file(self, file_path: str) -> Dict[str, Any]:
|
||||
"""
|
||||
@@ -1167,8 +1294,8 @@ class TimeAttendanceImportService:
|
||||
for idx, date_val in df['Date'].items():
|
||||
if pd.notna(date_val):
|
||||
try:
|
||||
pd.to_datetime(date_val)
|
||||
except:
|
||||
self._parse_date_field(date_val)
|
||||
except Exception:
|
||||
invalid_dates += 1
|
||||
|
||||
if invalid_dates > 0:
|
||||
|
||||
@@ -371,6 +371,35 @@ def roles_required(*allowed_roles):
|
||||
return decorator
|
||||
|
||||
|
||||
def load_project_manager_scope():
|
||||
"""
|
||||
(is_project_manager, allowed_project_ids, allowed_location_names) for the
|
||||
session user. Non-PMs get (False, [], []) and must NOT be filtered.
|
||||
|
||||
A PM may be scoped by projects, by locations, or both; empty lists for a PM
|
||||
mean "no access at all" (fail closed), which is also what a permission
|
||||
lookup failure returns. Used by the attendance APIs and the dashboard so a
|
||||
PM cannot see other projects' locations, employees or scan activity (§4).
|
||||
"""
|
||||
if session.get('role') != 'project_manager':
|
||||
return False, [], []
|
||||
|
||||
from models.permissions import UserProjectPermission, UserLocationPermission
|
||||
|
||||
user_id = session.get('user_id')
|
||||
try:
|
||||
project_ids = [p.project_id for p in
|
||||
UserProjectPermission.query.filter_by(user_id=user_id).all()]
|
||||
location_names = [l.location_name for l in
|
||||
UserLocationPermission.query.filter_by(user_id=user_id).all()]
|
||||
return True, project_ids, location_names
|
||||
except Exception as e:
|
||||
logger_handler.logger.error(
|
||||
f"Could not load Project Manager permissions for user {user_id}: {e}", exc_info=True
|
||||
)
|
||||
return True, [], [] # fail closed
|
||||
|
||||
|
||||
def restrict_blueprint_to_roles(blueprint, allowed_roles):
|
||||
"""Apply the role check to every route of a blueprint (one line per module)."""
|
||||
@blueprint.before_request
|
||||
|
||||
+22
-11
@@ -126,11 +126,19 @@ def parse_employee_id_for_work_type(employee_id: str) -> Tuple[str, str]:
|
||||
"""
|
||||
Parse employee ID to extract base ID and work type (supports SP, PW, PT, C)
|
||||
|
||||
Handles multiple formats:
|
||||
- Suffix with space: "1234 SP", "1234 PW", "1234 PT", "1234 C"
|
||||
- Suffix without space: "1234SP", "1234PW", "1234PT", "1234C"
|
||||
- Prefix with space: "SP 1234", "PW 1234", "PT 1234", "C 1234"
|
||||
- Prefix without space: "SP1234", "PW1234", "PT1234", "C1234"
|
||||
Handles multiple formats — the separator may be anything that is not a letter
|
||||
or a digit (space, dot, dash, underscore, slash, or nothing at all), because
|
||||
imported IDs come straight from customer Excel files:
|
||||
- Suffix: "1234SP", "1234 SP", "1234.PW", "1234-PT", "1234 . C"
|
||||
- Prefix: "SP1234", "SP 1234", "PW.1234", "PT-1234"
|
||||
|
||||
This mirrors build_employee_id_regex() in utils/helpers.py (SQL filters) and
|
||||
parseEmployeeIdWorkType() in static/js/attendance_report.js — keep the three
|
||||
in sync, or the report, the filters and the exports disagree about who worked
|
||||
those hours (§13).
|
||||
|
||||
Not a work type: "1234.5" (no code) and "1234SPX" (code runs into a word) —
|
||||
both come back as regular with the ID unchanged.
|
||||
|
||||
Args:
|
||||
employee_id: Employee ID string in any of the above formats
|
||||
@@ -147,16 +155,19 @@ def parse_employee_id_for_work_type(employee_id: str) -> Tuple[str, str]:
|
||||
# Define work type codes
|
||||
work_type_codes = ['SP', 'PW', 'PT', 'C']
|
||||
|
||||
# Any run of non-alphanumeric characters may separate the ID from the code
|
||||
# (or nothing at all): "1234SP", "1234 SP", "1234.PW", "1234 - PT".
|
||||
# The class excludes letters, so "1234SPX" never matches.
|
||||
separator = r'[^0-9A-Z]*'
|
||||
|
||||
for work_type in work_type_codes:
|
||||
# Pattern 1: Suffix with optional space - "1234 SP" or "1234SP"
|
||||
suffix_pattern = rf'^(\d+)\s*{work_type}$'
|
||||
suffix_match = re.match(suffix_pattern, employee_id_clean)
|
||||
# Pattern 1: Suffix - "1234SP", "1234 SP", "1234.SP"
|
||||
suffix_match = re.match(rf'^(\d+){separator}{work_type}$', employee_id_clean)
|
||||
if suffix_match:
|
||||
return suffix_match.group(1), work_type
|
||||
|
||||
# Pattern 2: Prefix with optional space - "SP 1234" or "SP1234"
|
||||
prefix_pattern = rf'^{work_type}\s*(\d+)$'
|
||||
prefix_match = re.match(prefix_pattern, employee_id_clean)
|
||||
# Pattern 2: Prefix - "SP1234", "SP 1234", "SP.1234"
|
||||
prefix_match = re.match(rf'^{work_type}{separator}(\d+)$', employee_id_clean)
|
||||
if prefix_match:
|
||||
return prefix_match.group(1), work_type
|
||||
|
||||
|
||||
Reference in New Issue
Block a user