06/04 Improve system logs UI/UX

CLAUDE.md

@@ -21,6 +21,8 @@ Self-hosted personal finance web app. Tracks income, expenses, investments. AI a
 - Budget utilization per category
 - Recent transactions feed (last 8)
 - AI daily insight card (Groq-generated, stored in DB)
+- **Savings Rate** stat card — net cash flow ÷ income for the selected period; green ≥ 20%, blue > 0%, red negative
+- **Schwab expiry warning** — banner shown when Schwab refresh token expires within 2 days
 - USD → VND exchange rate widget — reference only, independent of app currency
   - Click to expand 30-day history chart
   - ↻ refresh button (force-fetches fresh rate without page reload)
@@ -89,6 +91,7 @@ Self-hosted personal finance web app. Tracks income, expenses, investments. AI a
 - **Per-account sections** — when investments span multiple accounts (e.g. Individual + Roth IRA), holdings are grouped into one card per account, each showing account name, total value, and holdings table. Allocation sidebar also shows "By Account" breakdown
 - `investments.account_id` FK — each Schwab-synced holding is stamped with its source account; same ticker in different accounts (AAPL in Individual vs Roth IRA) stays as separate rows
 - Holdings table uses a Jinja2 `{% macro %}` (reused across single and multi-account views)
+- **Price history chart** on investment detail page — 1W / 1M / 3M / 6M / 1Y timeframe buttons; fetches from `/investments/api/price-history/<ticker>`
 
 ### 2.8 AI Financial Assistant
 - Chat UI with SSE streaming (Groq API, word-by-word response)
@@ -124,6 +127,8 @@ Self-hosted personal finance web app. Tracks income, expenses, investments. AI a
 - Profile: name, email, timezone, currency, Groq model
 - 8 currency options (USD/VND/EUR/GBP/JPY/AUD/CAD/SGD) — auto-updates symbol
 - Password change (requires current password)
+- **Two-Factor Authentication (TOTP)** — enable/disable TOTP 2FA; setup shows QR code + manual key entry; disable requires password confirmation
+- **Audit Log** (`/settings/audit`) — paginated log of login, 2FA, password, and bank-connection events with IP address; filterable by event type
 - Recurring rules: CRUD, pause/enable, frequency (daily/weekly/biweekly/monthly/quarterly/yearly)
 - "Run Now" button to process due rules immediately
 - CSV import: upload → preview with ⚠ warnings → confirm
@@ -175,8 +180,10 @@ Self-hosted personal finance web app. Tracks income, expenses, investments. AI a
   - Position zero-quantity and empty-symbol positions skipped; `null` positions array guarded
 - **Investments sync**: topbar "Sync Schwab" button on investments page → `POST /investments/sync-schwab` → runs snapshot for all mapped accounts (ignores stale `connection_id` — always uses active connection)
 - Token auto-refresh: access token expires 30 min; refreshed automatically before API calls
+- **Refresh token expiry**: `refresh_token_expires_at` tracked in DB; reset on every successful token exchange; dashboard warns when ≤ 2 days remain
+- **Account type map**: CASH→checking, MARGIN/IRA/ROTH_IRA/ROLLOVER_IRA/TRADITIONAL_IRA/401K/ROTH_401K/BROKERAGE→investment; unknown types fall back to `'other'`
 - Config: `SCHWAB_CLIENT_ID`, `SCHWAB_CLIENT_SECRET`, `SCHWAB_REDIRECT_URI`
-- Models: `schwab_connections`, `schwab_accounts` (2 new tables)
+- Models: `schwab_connections`, `schwab_accounts` (2 tables)
 
 ### 2.15 Bank Statement Import
 - Sidebar link: "Import Statement" under Money section
@@ -195,9 +202,9 @@ Self-hosted personal finance web app. Tracks income, expenses, investments. AI a
 
 ## 3. Database Schema (MySQL)
 
-### All 18 Tables
+### All 19 Tables
 ```
-users                   — single user, hashed password, currency/timezone prefs
+users                   — single user, hashed password, currency/timezone prefs, totp_secret, totp_enabled
 accounts                — bank/wallet accounts (balance managed per provider rules)
 categories              — expense/income categories with color/icon
 transactions            — income/expense/transfer, receipt_id, recurring_rule_id
@@ -211,9 +218,10 @@ investment_transactions — buy/sell/dividend/split log
 net_worth_snapshots     — monthly snapshots: assets, liabilities, net_worth (JSON)
 ai_insights             — stored AI responses: daily_summary / chat_response
 fx_rates                — daily USD/VND rate cache (date UNIQUE, source)
-teller_enrollments      — Teller enrollment: enrollment_id, access_token, institution_name
+audit_logs              — security event log: action, description, ip_address, timestamp
+teller_enrollments      — Teller enrollment: enrollment_id, access_token (TEXT, encrypted), institution_name
 teller_accounts         — Teller account ↔ PFM account mapping, last_sync_date
-schwab_connections      — Schwab OAuth tokens: access_token, refresh_token, token_expires_at
+schwab_connections      — Schwab OAuth tokens (TEXT, encrypted), token_expires_at, refresh_token_expires_at
 schwab_accounts         — Schwab account ↔ PFM account mapping, account_hash (hashValue)
 ```
 
@@ -223,10 +231,14 @@ schwab_accounts         — Schwab account ↔ PFM account mapping, account_hash
 - `investments.shares` / `avg_cost_basis` — recalculated from `investment_transactions` (FIFO) for manual holdings; overwritten directly by Schwab snapshot for synced holdings
 - `transactions.notes` — used to store import source IDs: `Teller:<id>`, `Schwab:<activityId>`, or `import:<fitid>`
 - `schwab_accounts.account_hash` — Schwab `hashValue` (encrypted account number), required in all API paths
+- `teller_enrollments.access_token` — stored as `TEXT` (widened from VARCHAR(128)); encrypted at rest via `EncryptedText` TypeDecorator
+- `schwab_connections.access_token` / `refresh_token` — `TEXT`, encrypted at rest; `refresh_token_expires_at` reset on every token exchange
+- `users.totp_secret` — base32 TOTP secret (VARCHAR 64); NULL when 2FA disabled
 
 ### Migration Scripts
 ```
-scripts/add_investment_account.py   — adds investments.account_id column (run once after deploy)
+scripts/add_investment_account.py   — adds investments.account_id column (run once)
+scripts/add_security_columns.py     — adds totp columns, audit_logs table, widens token columns to TEXT (run once)
 ```
 
 ---
@@ -244,12 +256,12 @@ pfm/                             # /home/pfm/web on server
 │   └── app.log
 │
 ├── app/
-│   ├── __init__.py
-│   ├── config.py                # SCHWAB_* vars added
-│   ├── extensions.py
+│   ├── __init__.py              # session idle timeout hook; Sentry init; limiter init
+│   ├── config.py                # SENTRY_DSN, SESSION_IDLE_MINUTES, RATELIMIT_STORAGE_URI added
+│   ├── extensions.py            # + limiter (flask-limiter, storage via RATELIMIT_STORAGE_URI)
 │   │
 │   ├── models/
-│   │   ├── user.py
+│   │   ├── user.py              # + totp_secret, totp_enabled columns
 │   │   ├── account.py
 │   │   ├── category.py
 │   │   ├── transaction.py
@@ -261,23 +273,24 @@ pfm/                             # /home/pfm/web on server
 │   │   ├── net_worth_snapshot.py
 │   │   ├── ai_insight.py
 │   │   ├── fx_rate.py
-│   │   ├── teller_enrollment.py # TellerEnrollment + TellerAccount
-│   │   └── schwab_connection.py # SchwabConnection + SchwabAccount (NEW)
+│   │   ├── audit_log.py         # AuditLog model (action, description, ip_address, timestamp)
+│   │   ├── teller_enrollment.py # access_token now EncryptedText (TEXT column)
+│   │   └── schwab_connection.py # tokens now EncryptedText; + refresh_token_expires_at
 │   │
 │   ├── routes/
-│   │   ├── auth.py
-│   │   ├── dashboard.py         # credit card "owed" display; no recalc_all
+│   │   ├── auth.py              # + TOTP verify/setup/disable routes; rate limits; audit calls
+│   │   ├── dashboard.py         # + savings_rate; Schwab expiry warning
 │   │   ├── accounts.py          # teller_map + schwab_map; skip calc_balance for providers
 │   │   ├── categories.py
-│   │   ├── transactions.py      # + set-category AJAX endpoint
+│   │   ├── transactions.py      # + set-category AJAX; fixed income-form-submits-as-expense bug
 │   │   ├── budgets.py
 │   │   ├── goals.py
-│   │   ├── investments.py       # + sync-schwab route
+│   │   ├── investments.py       # + sync-schwab route; price-history API
 │   │   ├── reports.py
 │   │   ├── ai.py
-│   │   ├── settings.py
-│   │   ├── teller.py            # balance uses ledger/available correctly; full_resync honours next param
-│   │   ├── schwab.py            # OAuth, mapping, sync, snapshot (NEW)
+│   │   ├── settings.py          # + audit_log route; audit calls on password change
+│   │   ├── teller.py            # balance uses ledger/available correctly
+│   │   ├── schwab.py            # OAuth, mapping, sync, snapshot; audit calls; fallback type 'other'
 │   │   ├── bank_import.py
 │   │   └── logs.py
 │   │
@@ -294,22 +307,29 @@ pfm/                             # /home/pfm/web on server
 │   │   ├── recurring_service.py
 │   │   ├── report_service.py
 │   │   ├── teller_service.py      # auto_categorize; correct sign convention; live balance after sync
-│   │   ├── schwab_service.py      # OAuth, account hash, snapshot sync, position upsert (NEW)
+│   │   ├── schwab_service.py      # + expanded ACCOUNT_TYPE_MAP; refresh_token_expires_at always reset
 │   │   └── bank_import_service.py
 │   │
 │   ├── templates/
-│   │   ├── base.html
-│   │   ├── dashboard/index.html   # credit card owed display
+│   │   ├── base.html              # mobile responsive tweaks
+│   │   ├── auth/totp_setup.html   # QR code + manual key entry for 2FA setup
+│   │   ├── auth/totp_verify.html  # 6-digit code entry on login
+│   │   ├── dashboard/index.html   # + savings_rate card; Schwab warning banner
 │   │   ├── accounts/index.html    # Teller/Schwab badges + action buttons
-│   │   ├── transactions/index.html # inline category <select> + AJAX
-│   │   ├── investments/index.html  # per-account sections; holdings_table macro; Sync Schwab btn
-│   │   ├── teller/index.html       # connect/disconnect only (sync buttons removed)
-│   │   ├── schwab/                 # index.html, map_accounts.html, preview.html (NEW)
+│   │   ├── transactions/index.html # inline category <select> + AJAX; pagination info
+│   │   ├── investments/index.html  # per-account sections; Sync Schwab btn
+│   │   ├── investments/detail.html # + price history chart (1W/1M/3M/6M/1Y)
+│   │   ├── settings/audit.html    # audit log viewer with event filter
+│   │   ├── settings/index.html    # + 2FA section; audit log nav card
+│   │   ├── teller/index.html      # connect/disconnect only
+│   │   ├── schwab/                # index.html, map_accounts.html, preview.html
 │   │   └── ... (other templates unchanged)
 │   │
 │   └── utils/
 │       ├── formatters.py
-│       └── decorators.py
+│       ├── decorators.py
+│       ├── audit.py               # audit() helper — writes AuditLog rows; swallows DB errors
+│       └── crypto.py              # EncryptedText SQLAlchemy TypeDecorator (Fernet, key=SHA256(SECRET_KEY))
 │
 ├── scripts/
 │   ├── init_db.py
@@ -318,7 +338,9 @@ pfm/                             # /home/pfm/web on server
 │   ├── fetch_prices.py
 │   ├── daily_snapshot.py
 │   ├── daily_ai_insight.py
-│   └── add_investment_account.py  # NEW — adds investments.account_id column
+│   ├── add_investment_account.py  # adds investments.account_id column
+│   ├── add_security_columns.py    # adds TOTP cols, audit_logs table, widens token cols to TEXT
+│   └── sync_schwab.py             # daily Schwab auto-sync (balance + positions + transactions)
 │
 └── tests/
 ```
@@ -345,6 +367,12 @@ requests==2.32.3
 cryptography==44.0.2
 python-dateutil==2.9.0
 pdfplumber==0.11.4
+# Security
+flask-limiter==3.5.0
+pyotp==2.9.0
+qrcode==7.4.2
+# Monitoring
+sentry-sdk[flask]==2.7.0
 ```
 
 ---
@@ -452,10 +480,16 @@ pdfplumber==0.11.4
 - Single-user, Flask-Login, session-based
 - Hashed password (Werkzeug `generate_password_hash`)
 - `SESSION_COOKIE_SECURE=True`, `SESSION_COOKIE_HTTPONLY=True`, `SESSION_COOKIE_SAMESITE='Lax'`
+- **Session idle timeout** — configurable via `SESSION_IDLE_MINUTES` (default 60); enforced in `before_request` hook
+- **TOTP 2FA** — optional TOTP second factor (pyotp); setup via QR code; verify endpoint rate-limited `10/min; 30/hr`; 5 failed attempts clears pending session and forces re-login
+- **Rate limiting** — flask-limiter on login (`10/min; 30/hr`), TOTP verify (`10/min; 30/hr`), TOTP setup (`10/min`); storage backend set via `RATELIMIT_STORAGE_URI` (use Redis in production to share limits across Gunicorn workers; defaults to `memory://` per-process if unset)
+- **At-rest encryption** — Teller and Schwab OAuth tokens encrypted in DB via `EncryptedText` SQLAlchemy TypeDecorator (Fernet symmetric, key = SHA-256(SECRET_KEY)); columns are `TEXT` not `VARCHAR`
+- **Audit log** — security events written to `audit_logs` table via `app/utils/audit.py`; events: `login_success`, `login_success_2fa`, `login_failed`, `login_failed_2fa`, `totp_enabled`, `totp_disabled`, `password_changed`, `schwab_connected`, `schwab_disconnected`
 - CSRF protection on all forms (Flask-WTF); meta tag in base.html for AJAX
 - SQLAlchemy ORM (no raw SQL)
 - Schwab OAuth state parameter validated on callback (CSRF protection)
 - `next` redirect params validated to start with `/` (no open redirect)
+- **Sentry** (optional) — error monitoring; enable via `SENTRY_DSN` env var; `send_default_pii=False`
 
 ---
 
@@ -468,6 +502,7 @@ pdfplumber==0.11.4
 | Fetch investment prices | Mon-Fri 4PM | `fetch_prices.py` | yfinance, all tickers |
 | Net worth snapshot | 1st of month 00:05 | `daily_snapshot.py` | Saves to net_worth_snapshots |
 | AI daily insight | Daily 00:01 | `daily_ai_insight.py` | Skips if already done today |
+| Schwab auto-sync | Daily 7AM | `sync_schwab.py` | Balance + positions + transactions; warns if refresh token expires soon |
 | DB backup | Daily 2AM | pfm-backup (systemd) | mysqldump → gzip, keep 30 days |
 
 ---
@@ -497,6 +532,10 @@ TELLER_WEBHOOK_SECRET=your-webhook-secret
 SCHWAB_CLIENT_ID=your-schwab-client-id
 SCHWAB_CLIENT_SECRET=your-schwab-client-secret
 SCHWAB_REDIRECT_URI=https://pfm.ngodanguyen.tech/schwab/callback
+# Security (optional)
+SENTRY_DSN=                          # leave blank to disable Sentry
+SESSION_IDLE_MINUTES=60              # session idle timeout in minutes
+RATELIMIT_STORAGE_URI=redis://localhost:6379  # use Redis to share rate limits across Gunicorn workers
 ```
 
 ---
@@ -505,7 +544,7 @@ SCHWAB_REDIRECT_URI=https://pfm.ngodanguyen.tech/schwab/callback
 
 | Blueprint | Prefix | Key routes |
 |-----------|--------|------------|
-| auth | /auth | login, logout |
+| auth | /auth | login, logout, totp/verify, totp/setup, totp/disable |
 | dashboard | / | index, api/fx-history, api/fx-refresh |
 | accounts | /accounts | CRUD, adjust |
 | categories | /categories | CRUD |
@@ -515,7 +554,7 @@ SCHWAB_REDIRECT_URI=https://pfm.ngodanguyen.tech/schwab/callback
 | investments | /investments | index, new, detail, edit, delete, add_transaction, refresh-prices, sync-schwab, api/price, api/daychange, api/price-history |
 | reports | /reports | monthly, quarterly, yearly, tax, export/csv\|excel\|pdf, snapshot |
 | ai | /ai | index, stream (SSE), history, generate-insight |
-| settings | /settings | index, profile, password, recurring, import, recalc-balances, upload_receipt, delete_receipt, view_receipt |
+| settings | /settings | index, profile, password, audit, recurring, import, recalc-balances, upload_receipt, delete_receipt, view_receipt |
 | teller | /teller | callback, map, index, sync, sync/confirm, sync/all, balance, balance/all, resync, disconnect, webhook |
 | schwab | /schwab | connect, callback, index, map, sync/`<id>`, sync/confirm, resync, snapshot/`<id>`, disconnect |
 | bank_import | /bank-import | index, parse (AJAX), import (AJAX) |
@@ -529,9 +568,13 @@ SCHWAB_REDIRECT_URI=https://pfm.ngodanguyen.tech/schwab/callback
 - FX rate widget: `open.er-api.com` may return stale values; yfinance is the reliable primary
 - WeasyPrint PDF: requires `libpango*` system libs on server
 - Bank statement PDF import: scanned/image PDFs have no text layer; must use digital download
-- Schwab: `investments.account_id` column requires migration — run `scripts/add_investment_account.py` once after deploy
-- Schwab: after first connect, run "Balance & Positions" to populate investments; then re-sync if holdings were already added manually (they will be updated to link to the account)
+- Schwab: run `scripts/add_investment_account.py` then `scripts/add_security_columns.py` once after fresh deploy
+- Schwab: after first connect, run "Balance & Positions" to populate investments; then re-sync if holdings were already added manually
+- Schwab refresh token: Schwab tokens last ~7 days; `refresh_token_expires_at` is reset on every token exchange (including access-only refreshes); dashboard warns at ≤ 2 days
+- Teller: `access_token` column is `TEXT` (widened from VARCHAR(128) to fit Fernet-encrypted values); run `scripts/add_security_columns.py` to apply
 - Teller: development environment only; requires cert/key from Teller Dashboard
+- Rate limiter: defaults to `memory://` per-process if `RATELIMIT_STORAGE_URI` is not set — effective limit is `stated_limit × num_workers`; set `RATELIMIT_STORAGE_URI=redis://localhost:6379` in production
+- `EncryptedText` TypeDecorator: key = SHA-256(SECRET_KEY); changing SECRET_KEY invalidates all stored tokens (requires reconnect)
 - MySQL does not support `NULLS LAST`; use `func.isnull(column)` for null-last ordering
 
 ---
@@ -553,27 +596,37 @@ SCHWAB_REDIRECT_URI=https://pfm.ngodanguyen.tech/schwab/callback
 | 2026-06 | Schwab uses `hashValue` (not raw account number) in API paths | schwab.py, schwab_service.py |
 | 2026-06 | `next` redirect params validated to start with `/` (no open redirect) | teller.py, schwab.py |
 | 2026-06 | Teller income/expense type corrected (positive = income) | teller_service.py |
+| 2026-06 | `EncryptedText.process_bind_param` removed silent plaintext fallback — raises on encrypt failure | crypto.py |
+| 2026-06 | Rate limiter storage moved to `RATELIMIT_STORAGE_URI` config (was hardcoded `memory://` per-worker) | extensions.py, config.py |
+| 2026-06 | TOTP verify: added hourly rate limit (`30/hr`) + per-session attempt counter (locks out after 5 failures) | auth.py |
+| 2026-06 | TOTP setup endpoint: added `@limiter.limit('10 per minute')` | auth.py |
+| 2026-06 | `refresh_token_expires_at` now reset on every token exchange, not only when Schwab rotates the token | schwab_service.py |
+| 2026-06 | Schwab unknown account type fallback changed back to `'other'` (was incorrectly changed to `'investment'`) | schwab.py |
+| 2026-06 | `teller_enrollments.access_token` widened VARCHAR(128) → TEXT to fit Fernet-encrypted values | add_security_columns.py |
+| 2026-06 | New income transaction submitted as expense — `form.transaction_type.data` not set on GET | transactions.py |
 
 ---
 
 ## 19. To-Do / Roadmap
 
 ### High priority
-- [ ] **Mobile responsiveness pass** — sidebar auto-collapses on mobile; tables scroll horizontally
+- [ ] **Mobile responsiveness pass** — sidebar auto-collapses on mobile; tables scroll horizontally *(base.html mobile CSS improved; further work needed)*
 - [ ] **Budget alerts** — email/Twilio SMS when category spending hits 80% / 100%
 
 ### Medium priority
-- [ ] **Pagination info** — show "Page X of Y" on AI history and other paginated pages
 - [ ] **PDF export memory** — stream CSV/Excel exports for users with large transaction history
 - [ ] **Receipt MIME validation** — validate file magic bytes server-side, not just extension
 - [ ] **OCR ownership check** — verify re-extracted filename belongs to current user's transaction
 - [ ] **Bank import progress** — show per-row import progress for large statement files
-- [ ] **Schwab IRA account type** — map Schwab IRA account type to `investment` in ACCOUNT_TYPE_MAP
 
 ### Low priority / future
 - [ ] iOS companion app
 - [ ] Shared household mode (2 users, row-level isolation)
 - [ ] Bank statement PDF: table-extraction fallback (pdfplumber tables API) before Groq call
-- [ ] Investment price history chart per holding
 - [ ] Dark mode toggle
-- [ ] Schwab auto-sync on schedule (currently manual only)
+
+### Completed (removed from backlog)
+- [x] **Pagination info** — "Page X of Y" added to transactions, AI history, accounts/payments, audit log
+- [x] **Schwab IRA account type** — IRA/ROTH_IRA/401K/BROKERAGE types added to ACCOUNT_TYPE_MAP
+- [x] **Investment price history chart** — 1W/1M/3M/6M/1Y chart on investment detail page
+- [x] **Schwab auto-sync on schedule** — `scripts/sync_schwab.py` (cron at 7AM daily)