Aug 17 - Update with utilities management
This commit is contained in:
@@ -253,11 +253,35 @@ Self-hosted personal finance web app. Tracks income, expenses, investments. AI a
|
||||
- **Update webhook for existing items** (`POST /plaid/update-webhook`) — calls Plaid `/item/webhook/update` for all active items; "Apply to Existing Items" button shown on Plaid page when URL is configured
|
||||
- **Not in sidebar** — accessed via Settings page only
|
||||
|
||||
### 2.20 Utilities (Electricity / Water / Gas / Internet)
|
||||
- Sidebar link: "Utilities" under Money section; amber badge counts bills due within 7 days
|
||||
- **Providers** (`/utilities/providers`) — one record per utility company
|
||||
- Types: electricity, water, gas, internet, phone, trash, other (each with a default icon/color/usage unit)
|
||||
- Fields: name, type, account number, usage unit, pay-from account, expense category, billing day, color, icon, notes
|
||||
- `usage_unit` blank = no consumption tracking for that provider (the bill form dims those fields)
|
||||
- Changing the type on a **new** provider auto-fills unit/icon/color; editing an existing one never overwrites choices
|
||||
- Archive (hides from dashboard, keeps history) or delete (cascades to bills; payment transactions are left in place)
|
||||
- **Bills** (`/utilities/bills`) — one record per billing period
|
||||
- Fields: period start/end, amount, due date, usage, meter start/end, notes
|
||||
- Usage entered directly **or** derived from meter readings — readings win (`sync_usage_from_meter`)
|
||||
- Live unit-rate hint on the form; `usage_unit` snapshotted from the provider at entry time
|
||||
- `UNIQUE (provider_id, period_start)` — duplicate periods rejected with a flash, not a 500
|
||||
- Validation: period end ≥ period start; meter end ≥ meter start
|
||||
- Filters: provider, type, status (unpaid / overdue / paid), year; pagination 30/page
|
||||
- **Status** is derived, not stored: `paid` → `overdue` (past due) → `due_soon` (≤ 7 days) → `unpaid`
|
||||
- **Payment — two paths** (both set `utility_bills.transaction_id`):
|
||||
- **Mark Paid** (`/utilities/bills/<id>/pay`) — creates an expense transaction (notes `Utility:<bill_id>`), defaults account/category from the provider, then runs `calc_balance` + the budget alert check
|
||||
- **Link Existing** (`/utilities/bills/<id>/link`) — attaches an already-imported Plaid/Teller/Schwab transaction; candidates are unlinked expenses within 45 days of the due date, closest amount first; linking one transaction to two bills is refused
|
||||
- **Reopen** (`unpay`) — deletes the transaction **only** if PFM generated it (`Utility:<id>` marker); externally linked ones are left alone. Same rule when deleting a bill.
|
||||
- **Dashboard** (`/utilities/`) — this month / 12-month average / YTD / unpaid+overdue stat cards, bills-due table, 12-month stacked bar chart by type, spend-by-type breakdown, per-provider cards with period-over-period Δ
|
||||
- **Provider detail** — latest/average/12-month/usage stats plus a chart toggling Amount / Usage / Unit Rate over 24 months
|
||||
- Service: `app/services/utility_service.py` — `dashboard_summary`, `monthly_series`, `type_totals`, `provider_summary`, `usage_series`, `candidate_transactions`, `build_payment_transaction`, `is_generated_payment`
|
||||
|
||||
---
|
||||
|
||||
## 3. Database Schema (MySQL)
|
||||
|
||||
### All 23 Tables
|
||||
### All 25 Tables
|
||||
```
|
||||
users — single user, hashed password, currency/timezone prefs, totp_secret, totp_enabled
|
||||
accounts — bank/wallet accounts (balance managed per provider rules)
|
||||
@@ -282,6 +306,8 @@ schwab_accounts — Schwab account ↔ PFM account mapping, account_hash
|
||||
plaid_items — Plaid item: item_id, access_token (EncryptedText), institution_name, cursor, last_synced_at
|
||||
plaid_accounts — Plaid account ↔ PFM account mapping; cc_due_date, cc_minimum_payment, cc_last_statement_balance, cc_is_overdue
|
||||
plaid_sync_previews — temporary preview data: item_id (UNIQUE), data_json (TEXT), next_cursor
|
||||
utility_providers — utility company: name, utility_type, usage_unit, default_account_id, category_id, billing_day
|
||||
utility_bills — one billing period: amount, period, due_date, is_paid, transaction_id, usage_amount, meter readings
|
||||
```
|
||||
|
||||
### Key Column Notes
|
||||
@@ -298,6 +324,9 @@ plaid_sync_previews — temporary preview data: item_id (UNIQUE), data_json
|
||||
- `users.totp_secret` — base32 TOTP secret (VARCHAR 64); NULL when 2FA disabled
|
||||
- `users.budget_alerts_enabled` — boolean; toggles budget threshold emails (Settings → Profile)
|
||||
- `budgets.alert_sent_80` / `alert_sent_100` — dedup flags so threshold emails/flashes fire once per month per category
|
||||
- `utility_bills.usage_amount` — mapped to the Python attribute `UtilityBill.usage`; the column is NOT named `usage` because that is a reserved word in MySQL
|
||||
- `utility_bills.transaction_id` — nullable FK to `transactions.id`; set by both mark-paid and link-existing. A transaction whose notes start with `Utility:<bill_id>` was generated by PFM and is deleted on reopen/bill-delete; anything else is left alone
|
||||
- `utility_providers.usage_unit` — blank/NULL means the provider has no consumption tracking
|
||||
|
||||
### Migration Scripts
|
||||
```
|
||||
@@ -305,8 +334,13 @@ scripts/add_investment_account.py — adds investments.account_id column (run
|
||||
scripts/add_security_columns.py — adds totp columns, audit_logs table, widens token columns to TEXT (run once)
|
||||
scripts/add_plaid_tables.py — creates plaid_items, plaid_accounts, plaid_sync_previews tables (run once)
|
||||
scripts/add_log_tables.py — creates audit_logs and app_logs tables (run once; safe to re-run)
|
||||
scripts/add_utility_tables.py — creates utility_providers and utility_bills tables (run once; safe to re-run)
|
||||
```
|
||||
|
||||
Prefer `flask db migrate` + `flask db upgrade` where the Alembic chain is healthy — these scripts are the
|
||||
fallback for schema managed outside the chain. Always read a generated migration before running it:
|
||||
autogenerate cannot see models missing from `app/models/__init__.py` and will propose dropping their tables.
|
||||
|
||||
---
|
||||
|
||||
## 4. Project File Structure (Actual)
|
||||
@@ -339,6 +373,7 @@ pfm/ # /home/pfm/web on server
|
||||
│ │ ├── net_worth_snapshot.py
|
||||
│ │ ├── ai_insight.py
|
||||
│ │ ├── fx_rate.py
|
||||
│ │ ├── utility.py # UtilityProvider, UtilityBill (+ UTILITY_TYPE_META)
|
||||
│ │ ├── audit_log.py # AuditLog model (action, description, ip_address, timestamp)
|
||||
│ │ ├── app_log.py # AppLog model (timestamp, level, module, message TEXT)
|
||||
│ │ ├── teller_enrollment.py # access_token now EncryptedText (TEXT column)
|
||||
@@ -361,7 +396,8 @@ pfm/ # /home/pfm/web on server
|
||||
│ │ ├── schwab.py # OAuth, mapping, sync, snapshot; audit calls; fallback type 'other'
|
||||
│ │ ├── plaid.py # Link flow, exchange, map, sync preview/confirm, balance, liabilities, resync, disconnect
|
||||
│ │ ├── bank_import.py
|
||||
│ │ └── logs.py # DB-backed API; purge endpoint; clear truncates DB + file
|
||||
│ │ ├── logs.py # DB-backed API; purge endpoint; clear truncates DB + file
|
||||
│ │ └── utilities.py # providers + bills CRUD, mark-paid, link payment, usage API
|
||||
│ │
|
||||
│ ├── services/
|
||||
│ │ ├── account_service.py
|
||||
@@ -380,6 +416,7 @@ pfm/ # /home/pfm/web on server
|
||||
│ │ ├── teller_service.py # auto_categorize; correct sign convention; live balance after sync
|
||||
│ │ ├── schwab_service.py # + expanded ACCOUNT_TYPE_MAP; refresh_token_expires_at always reset
|
||||
│ │ ├── plaid_service.py # Link token, exchange, accounts, balances, liabilities, cursor sync, parse, import
|
||||
│ │ ├── utility_service.py # bill roll-ups, usage/rate trends, payment matching
|
||||
│ │ └── bank_import_service.py
|
||||
│ │
|
||||
│ ├── templates/
|
||||
@@ -402,6 +439,8 @@ pfm/ # /home/pfm/web on server
|
||||
│ │ ├── schwab/ # index.html (+ ← Settings back btn), map_accounts.html, preview.html
|
||||
│ │ ├── plaid/ # index.html (+ ← Settings back btn), map_accounts.html, preview.html
|
||||
│ │ ├── logs/index.html # DB-backed viewer; Purge dropdown; ← Settings back btn moved to topbar
|
||||
│ │ ├── utilities/ # index.html, providers.html, provider_form.html, detail.html,
|
||||
│ │ │ # bills.html, bill_form.html, pay.html, link.html
|
||||
│ │ └── ... (other templates unchanged)
|
||||
│ │
|
||||
│ └── utils/
|
||||
@@ -422,6 +461,7 @@ pfm/ # /home/pfm/web on server
|
||||
│ ├── add_security_columns.py # adds TOTP cols, audit_logs table, widens token cols to TEXT
|
||||
│ ├── add_plaid_tables.py # creates plaid_items, plaid_accounts, plaid_sync_previews
|
||||
│ ├── add_log_tables.py # creates audit_logs + app_logs tables (safe to re-run)
|
||||
│ ├── add_utility_tables.py # creates utility_providers + utility_bills (safe to re-run)
|
||||
│ └── sync_schwab.py # daily Schwab auto-sync (balance + positions + transactions)
|
||||
│
|
||||
└── tests/
|
||||
@@ -549,7 +589,7 @@ sentry-sdk[flask]==2.7.0
|
||||
|
||||
## 12. UI/UX
|
||||
|
||||
- **Sidebar**: collapsible (desktop state saved in localStorage), mobile overlay; Teller Sync and Schwab Sync **removed** — accessible via Settings only
|
||||
- **Sidebar**: collapsible (desktop state saved in localStorage), mobile overlay; Teller Sync and Schwab Sync **removed** — accessible via Settings only; Utilities sits under Money with an amber badge counting bills due within 7 days (`utility_due_count`, set in the `inject_globals` context processor)
|
||||
- **Mobile responsive**: sidebar goes off-canvas with a dimmed overlay under 769px (topbar toggle button opens/closes it); topbar and main content collapse to full width; tables scroll horizontally via `.table-wrap` / `.pcard.p-0` wrapper classes + `.pfm-table` min-widths, with `.d-mob-none` hiding low-priority columns first; under 576px, button labels hide to icon-only (`.btn-label`) and chart/chat heights are capped (base.html)
|
||||
- **Dark mode**: toggle button in topbar (moon/sun icon); persisted via `localStorage['pfm_dark']`; applied pre-paint via a `data-pfm-dark` attribute to avoid flash-of-light-mode; CSS variable overrides plus targeted `[style*="..."]` overrides for hardcoded inline colors in templates (base.html)
|
||||
- **Charts**: Chart.js 4.x (CDN)
|
||||
@@ -640,7 +680,7 @@ APP_URL=https://pfm.ngodanguyen.tech # used to build links in alert emails
|
||||
|
||||
---
|
||||
|
||||
## 16. Blueprints Registered (17 total)
|
||||
## 16. Blueprints Registered (18 total)
|
||||
|
||||
| Blueprint | Prefix | Key routes |
|
||||
|-----------|--------|------------|
|
||||
@@ -661,6 +701,7 @@ APP_URL=https://pfm.ngodanguyen.tech # used to build links in alert emails
|
||||
| plaid | /plaid | index, create-link-token, exchange-token, map/`<id>`, sync/`<id>`, sync/confirm, balance/`<pa_id>`, liabilities/`<id>`, resync/`<id>`, disconnect/`<id>`, webhook, update-webhook |
|
||||
| bank_import | /bank-import | index, parse (AJAX), import (AJAX) |
|
||||
| logs | /logs | index, api (AJAX), clear (AJAX), download, purge (AJAX) |
|
||||
| utilities | /utilities | index, providers, providers/new, providers/`<id>`, providers/`<id>`/edit, providers/`<id>`/toggle, providers/`<id>`/delete, bills, bills/new, bills/`<id>`/edit, bills/`<id>`/delete, bills/`<id>`/pay, bills/`<id>`/link, bills/`<id>`/unpay, api/usage/`<id>` |
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in New Issue
Block a user