diff --git a/CLAUDE.md b/CLAUDE.md index a88bf64..e89647b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -21,6 +21,8 @@ Self-hosted personal finance web app. Tracks income, expenses, investments. AI a - Budget utilization per category - Recent transactions feed (last 8) - AI daily insight card (Groq-generated, stored in DB) +- **Savings Rate** stat card — net cash flow ÷ income for the selected period; green ≥ 20%, blue > 0%, red negative +- **Schwab expiry warning** — banner shown when Schwab refresh token expires within 2 days - USD → VND exchange rate widget — reference only, independent of app currency - Click to expand 30-day history chart - ↻ refresh button (force-fetches fresh rate without page reload) @@ -89,6 +91,7 @@ Self-hosted personal finance web app. Tracks income, expenses, investments. AI a - **Per-account sections** — when investments span multiple accounts (e.g. Individual + Roth IRA), holdings are grouped into one card per account, each showing account name, total value, and holdings table. Allocation sidebar also shows "By Account" breakdown - `investments.account_id` FK — each Schwab-synced holding is stamped with its source account; same ticker in different accounts (AAPL in Individual vs Roth IRA) stays as separate rows - Holdings table uses a Jinja2 `{% macro %}` (reused across single and multi-account views) +- **Price history chart** on investment detail page — 1W / 1M / 3M / 6M / 1Y timeframe buttons; fetches from `/investments/api/price-history/` ### 2.8 AI Financial Assistant - Chat UI with SSE streaming (Groq API, word-by-word response) @@ -124,6 +127,8 @@ Self-hosted personal finance web app. Tracks income, expenses, investments. AI a - Profile: name, email, timezone, currency, Groq model - 8 currency options (USD/VND/EUR/GBP/JPY/AUD/CAD/SGD) — auto-updates symbol - Password change (requires current password) +- **Two-Factor Authentication (TOTP)** — enable/disable TOTP 2FA; setup shows QR code + manual key entry; disable requires password confirmation +- **Audit Log** (`/settings/audit`) — paginated log of login, 2FA, password, and bank-connection events with IP address; filterable by event type - Recurring rules: CRUD, pause/enable, frequency (daily/weekly/biweekly/monthly/quarterly/yearly) - "Run Now" button to process due rules immediately - CSV import: upload → preview with ⚠ warnings → confirm @@ -175,8 +180,10 @@ Self-hosted personal finance web app. Tracks income, expenses, investments. AI a - Position zero-quantity and empty-symbol positions skipped; `null` positions array guarded - **Investments sync**: topbar "Sync Schwab" button on investments page → `POST /investments/sync-schwab` → runs snapshot for all mapped accounts (ignores stale `connection_id` — always uses active connection) - Token auto-refresh: access token expires 30 min; refreshed automatically before API calls +- **Refresh token expiry**: `refresh_token_expires_at` tracked in DB; reset on every successful token exchange; dashboard warns when ≤ 2 days remain +- **Account type map**: CASH→checking, MARGIN/IRA/ROTH_IRA/ROLLOVER_IRA/TRADITIONAL_IRA/401K/ROTH_401K/BROKERAGE→investment; unknown types fall back to `'other'` - Config: `SCHWAB_CLIENT_ID`, `SCHWAB_CLIENT_SECRET`, `SCHWAB_REDIRECT_URI` -- Models: `schwab_connections`, `schwab_accounts` (2 new tables) +- Models: `schwab_connections`, `schwab_accounts` (2 tables) ### 2.15 Bank Statement Import - Sidebar link: "Import Statement" under Money section @@ -195,9 +202,9 @@ Self-hosted personal finance web app. Tracks income, expenses, investments. AI a ## 3. Database Schema (MySQL) -### All 18 Tables +### All 19 Tables ``` -users — single user, hashed password, currency/timezone prefs +users — single user, hashed password, currency/timezone prefs, totp_secret, totp_enabled accounts — bank/wallet accounts (balance managed per provider rules) categories — expense/income categories with color/icon transactions — income/expense/transfer, receipt_id, recurring_rule_id @@ -211,9 +218,10 @@ investment_transactions — buy/sell/dividend/split log net_worth_snapshots — monthly snapshots: assets, liabilities, net_worth (JSON) ai_insights — stored AI responses: daily_summary / chat_response fx_rates — daily USD/VND rate cache (date UNIQUE, source) -teller_enrollments — Teller enrollment: enrollment_id, access_token, institution_name +audit_logs — security event log: action, description, ip_address, timestamp +teller_enrollments — Teller enrollment: enrollment_id, access_token (TEXT, encrypted), institution_name teller_accounts — Teller account ↔ PFM account mapping, last_sync_date -schwab_connections — Schwab OAuth tokens: access_token, refresh_token, token_expires_at +schwab_connections — Schwab OAuth tokens (TEXT, encrypted), token_expires_at, refresh_token_expires_at schwab_accounts — Schwab account ↔ PFM account mapping, account_hash (hashValue) ``` @@ -223,10 +231,14 @@ schwab_accounts — Schwab account ↔ PFM account mapping, account_hash - `investments.shares` / `avg_cost_basis` — recalculated from `investment_transactions` (FIFO) for manual holdings; overwritten directly by Schwab snapshot for synced holdings - `transactions.notes` — used to store import source IDs: `Teller:`, `Schwab:`, or `import:` - `schwab_accounts.account_hash` — Schwab `hashValue` (encrypted account number), required in all API paths +- `teller_enrollments.access_token` — stored as `TEXT` (widened from VARCHAR(128)); encrypted at rest via `EncryptedText` TypeDecorator +- `schwab_connections.access_token` / `refresh_token` — `TEXT`, encrypted at rest; `refresh_token_expires_at` reset on every token exchange +- `users.totp_secret` — base32 TOTP secret (VARCHAR 64); NULL when 2FA disabled ### Migration Scripts ``` -scripts/add_investment_account.py — adds investments.account_id column (run once after deploy) +scripts/add_investment_account.py — adds investments.account_id column (run once) +scripts/add_security_columns.py — adds totp columns, audit_logs table, widens token columns to TEXT (run once) ``` --- @@ -244,12 +256,12 @@ pfm/ # /home/pfm/web on server │ └── app.log │ ├── app/ -│ ├── __init__.py -│ ├── config.py # SCHWAB_* vars added -│ ├── extensions.py +│ ├── __init__.py # session idle timeout hook; Sentry init; limiter init +│ ├── config.py # SENTRY_DSN, SESSION_IDLE_MINUTES, RATELIMIT_STORAGE_URI added +│ ├── extensions.py # + limiter (flask-limiter, storage via RATELIMIT_STORAGE_URI) │ │ │ ├── models/ -│ │ ├── user.py +│ │ ├── user.py # + totp_secret, totp_enabled columns │ │ ├── account.py │ │ ├── category.py │ │ ├── transaction.py @@ -261,23 +273,24 @@ pfm/ # /home/pfm/web on server │ │ ├── net_worth_snapshot.py │ │ ├── ai_insight.py │ │ ├── fx_rate.py -│ │ ├── teller_enrollment.py # TellerEnrollment + TellerAccount -│ │ └── schwab_connection.py # SchwabConnection + SchwabAccount (NEW) +│ │ ├── audit_log.py # AuditLog model (action, description, ip_address, timestamp) +│ │ ├── teller_enrollment.py # access_token now EncryptedText (TEXT column) +│ │ └── schwab_connection.py # tokens now EncryptedText; + refresh_token_expires_at │ │ │ ├── routes/ -│ │ ├── auth.py -│ │ ├── dashboard.py # credit card "owed" display; no recalc_all +│ │ ├── auth.py # + TOTP verify/setup/disable routes; rate limits; audit calls +│ │ ├── dashboard.py # + savings_rate; Schwab expiry warning │ │ ├── accounts.py # teller_map + schwab_map; skip calc_balance for providers │ │ ├── categories.py -│ │ ├── transactions.py # + set-category AJAX endpoint +│ │ ├── transactions.py # + set-category AJAX; fixed income-form-submits-as-expense bug │ │ ├── budgets.py │ │ ├── goals.py -│ │ ├── investments.py # + sync-schwab route +│ │ ├── investments.py # + sync-schwab route; price-history API │ │ ├── reports.py │ │ ├── ai.py -│ │ ├── settings.py -│ │ ├── teller.py # balance uses ledger/available correctly; full_resync honours next param -│ │ ├── schwab.py # OAuth, mapping, sync, snapshot (NEW) +│ │ ├── settings.py # + audit_log route; audit calls on password change +│ │ ├── teller.py # balance uses ledger/available correctly +│ │ ├── schwab.py # OAuth, mapping, sync, snapshot; audit calls; fallback type 'other' │ │ ├── bank_import.py │ │ └── logs.py │ │ @@ -294,22 +307,29 @@ pfm/ # /home/pfm/web on server │ │ ├── recurring_service.py │ │ ├── report_service.py │ │ ├── teller_service.py # auto_categorize; correct sign convention; live balance after sync -│ │ ├── schwab_service.py # OAuth, account hash, snapshot sync, position upsert (NEW) +│ │ ├── schwab_service.py # + expanded ACCOUNT_TYPE_MAP; refresh_token_expires_at always reset │ │ └── bank_import_service.py │ │ │ ├── templates/ -│ │ ├── base.html -│ │ ├── dashboard/index.html # credit card owed display +│ │ ├── base.html # mobile responsive tweaks +│ │ ├── auth/totp_setup.html # QR code + manual key entry for 2FA setup +│ │ ├── auth/totp_verify.html # 6-digit code entry on login +│ │ ├── dashboard/index.html # + savings_rate card; Schwab warning banner │ │ ├── accounts/index.html # Teller/Schwab badges + action buttons -│ │ ├── transactions/index.html # inline category + AJAX; pagination info +│ │ ├── investments/index.html # per-account sections; Sync Schwab btn +│ │ ├── investments/detail.html # + price history chart (1W/1M/3M/6M/1Y) +│ │ ├── settings/audit.html # audit log viewer with event filter +│ │ ├── settings/index.html # + 2FA section; audit log nav card +│ │ ├── teller/index.html # connect/disconnect only +│ │ ├── schwab/ # index.html, map_accounts.html, preview.html │ │ └── ... (other templates unchanged) │ │ │ └── utils/ │ ├── formatters.py -│ └── decorators.py +│ ├── decorators.py +│ ├── audit.py # audit() helper — writes AuditLog rows; swallows DB errors +│ └── crypto.py # EncryptedText SQLAlchemy TypeDecorator (Fernet, key=SHA256(SECRET_KEY)) │ ├── scripts/ │ ├── init_db.py @@ -318,7 +338,9 @@ pfm/ # /home/pfm/web on server │ ├── fetch_prices.py │ ├── daily_snapshot.py │ ├── daily_ai_insight.py -│ └── add_investment_account.py # NEW — adds investments.account_id column +│ ├── add_investment_account.py # adds investments.account_id column +│ ├── add_security_columns.py # adds TOTP cols, audit_logs table, widens token cols to TEXT +│ └── sync_schwab.py # daily Schwab auto-sync (balance + positions + transactions) │ └── tests/ ``` @@ -345,6 +367,12 @@ requests==2.32.3 cryptography==44.0.2 python-dateutil==2.9.0 pdfplumber==0.11.4 +# Security +flask-limiter==3.5.0 +pyotp==2.9.0 +qrcode==7.4.2 +# Monitoring +sentry-sdk[flask]==2.7.0 ``` --- @@ -452,10 +480,16 @@ pdfplumber==0.11.4 - Single-user, Flask-Login, session-based - Hashed password (Werkzeug `generate_password_hash`) - `SESSION_COOKIE_SECURE=True`, `SESSION_COOKIE_HTTPONLY=True`, `SESSION_COOKIE_SAMESITE='Lax'` +- **Session idle timeout** — configurable via `SESSION_IDLE_MINUTES` (default 60); enforced in `before_request` hook +- **TOTP 2FA** — optional TOTP second factor (pyotp); setup via QR code; verify endpoint rate-limited `10/min; 30/hr`; 5 failed attempts clears pending session and forces re-login +- **Rate limiting** — flask-limiter on login (`10/min; 30/hr`), TOTP verify (`10/min; 30/hr`), TOTP setup (`10/min`); storage backend set via `RATELIMIT_STORAGE_URI` (use Redis in production to share limits across Gunicorn workers; defaults to `memory://` per-process if unset) +- **At-rest encryption** — Teller and Schwab OAuth tokens encrypted in DB via `EncryptedText` SQLAlchemy TypeDecorator (Fernet symmetric, key = SHA-256(SECRET_KEY)); columns are `TEXT` not `VARCHAR` +- **Audit log** — security events written to `audit_logs` table via `app/utils/audit.py`; events: `login_success`, `login_success_2fa`, `login_failed`, `login_failed_2fa`, `totp_enabled`, `totp_disabled`, `password_changed`, `schwab_connected`, `schwab_disconnected` - CSRF protection on all forms (Flask-WTF); meta tag in base.html for AJAX - SQLAlchemy ORM (no raw SQL) - Schwab OAuth state parameter validated on callback (CSRF protection) - `next` redirect params validated to start with `/` (no open redirect) +- **Sentry** (optional) — error monitoring; enable via `SENTRY_DSN` env var; `send_default_pii=False` --- @@ -468,6 +502,7 @@ pdfplumber==0.11.4 | Fetch investment prices | Mon-Fri 4PM | `fetch_prices.py` | yfinance, all tickers | | Net worth snapshot | 1st of month 00:05 | `daily_snapshot.py` | Saves to net_worth_snapshots | | AI daily insight | Daily 00:01 | `daily_ai_insight.py` | Skips if already done today | +| Schwab auto-sync | Daily 7AM | `sync_schwab.py` | Balance + positions + transactions; warns if refresh token expires soon | | DB backup | Daily 2AM | pfm-backup (systemd) | mysqldump → gzip, keep 30 days | --- @@ -497,6 +532,10 @@ TELLER_WEBHOOK_SECRET=your-webhook-secret SCHWAB_CLIENT_ID=your-schwab-client-id SCHWAB_CLIENT_SECRET=your-schwab-client-secret SCHWAB_REDIRECT_URI=https://pfm.ngodanguyen.tech/schwab/callback +# Security (optional) +SENTRY_DSN= # leave blank to disable Sentry +SESSION_IDLE_MINUTES=60 # session idle timeout in minutes +RATELIMIT_STORAGE_URI=redis://localhost:6379 # use Redis to share rate limits across Gunicorn workers ``` --- @@ -505,7 +544,7 @@ SCHWAB_REDIRECT_URI=https://pfm.ngodanguyen.tech/schwab/callback | Blueprint | Prefix | Key routes | |-----------|--------|------------| -| auth | /auth | login, logout | +| auth | /auth | login, logout, totp/verify, totp/setup, totp/disable | | dashboard | / | index, api/fx-history, api/fx-refresh | | accounts | /accounts | CRUD, adjust | | categories | /categories | CRUD | @@ -515,7 +554,7 @@ SCHWAB_REDIRECT_URI=https://pfm.ngodanguyen.tech/schwab/callback | investments | /investments | index, new, detail, edit, delete, add_transaction, refresh-prices, sync-schwab, api/price, api/daychange, api/price-history | | reports | /reports | monthly, quarterly, yearly, tax, export/csv\|excel\|pdf, snapshot | | ai | /ai | index, stream (SSE), history, generate-insight | -| settings | /settings | index, profile, password, recurring, import, recalc-balances, upload_receipt, delete_receipt, view_receipt | +| settings | /settings | index, profile, password, audit, recurring, import, recalc-balances, upload_receipt, delete_receipt, view_receipt | | teller | /teller | callback, map, index, sync, sync/confirm, sync/all, balance, balance/all, resync, disconnect, webhook | | schwab | /schwab | connect, callback, index, map, sync/``, sync/confirm, resync, snapshot/``, disconnect | | bank_import | /bank-import | index, parse (AJAX), import (AJAX) | @@ -529,9 +568,13 @@ SCHWAB_REDIRECT_URI=https://pfm.ngodanguyen.tech/schwab/callback - FX rate widget: `open.er-api.com` may return stale values; yfinance is the reliable primary - WeasyPrint PDF: requires `libpango*` system libs on server - Bank statement PDF import: scanned/image PDFs have no text layer; must use digital download -- Schwab: `investments.account_id` column requires migration — run `scripts/add_investment_account.py` once after deploy -- Schwab: after first connect, run "Balance & Positions" to populate investments; then re-sync if holdings were already added manually (they will be updated to link to the account) +- Schwab: run `scripts/add_investment_account.py` then `scripts/add_security_columns.py` once after fresh deploy +- Schwab: after first connect, run "Balance & Positions" to populate investments; then re-sync if holdings were already added manually +- Schwab refresh token: Schwab tokens last ~7 days; `refresh_token_expires_at` is reset on every token exchange (including access-only refreshes); dashboard warns at ≤ 2 days +- Teller: `access_token` column is `TEXT` (widened from VARCHAR(128) to fit Fernet-encrypted values); run `scripts/add_security_columns.py` to apply - Teller: development environment only; requires cert/key from Teller Dashboard +- Rate limiter: defaults to `memory://` per-process if `RATELIMIT_STORAGE_URI` is not set — effective limit is `stated_limit × num_workers`; set `RATELIMIT_STORAGE_URI=redis://localhost:6379` in production +- `EncryptedText` TypeDecorator: key = SHA-256(SECRET_KEY); changing SECRET_KEY invalidates all stored tokens (requires reconnect) - MySQL does not support `NULLS LAST`; use `func.isnull(column)` for null-last ordering --- @@ -553,27 +596,37 @@ SCHWAB_REDIRECT_URI=https://pfm.ngodanguyen.tech/schwab/callback | 2026-06 | Schwab uses `hashValue` (not raw account number) in API paths | schwab.py, schwab_service.py | | 2026-06 | `next` redirect params validated to start with `/` (no open redirect) | teller.py, schwab.py | | 2026-06 | Teller income/expense type corrected (positive = income) | teller_service.py | +| 2026-06 | `EncryptedText.process_bind_param` removed silent plaintext fallback — raises on encrypt failure | crypto.py | +| 2026-06 | Rate limiter storage moved to `RATELIMIT_STORAGE_URI` config (was hardcoded `memory://` per-worker) | extensions.py, config.py | +| 2026-06 | TOTP verify: added hourly rate limit (`30/hr`) + per-session attempt counter (locks out after 5 failures) | auth.py | +| 2026-06 | TOTP setup endpoint: added `@limiter.limit('10 per minute')` | auth.py | +| 2026-06 | `refresh_token_expires_at` now reset on every token exchange, not only when Schwab rotates the token | schwab_service.py | +| 2026-06 | Schwab unknown account type fallback changed back to `'other'` (was incorrectly changed to `'investment'`) | schwab.py | +| 2026-06 | `teller_enrollments.access_token` widened VARCHAR(128) → TEXT to fit Fernet-encrypted values | add_security_columns.py | +| 2026-06 | New income transaction submitted as expense — `form.transaction_type.data` not set on GET | transactions.py | --- ## 19. To-Do / Roadmap ### High priority -- [ ] **Mobile responsiveness pass** — sidebar auto-collapses on mobile; tables scroll horizontally +- [ ] **Mobile responsiveness pass** — sidebar auto-collapses on mobile; tables scroll horizontally *(base.html mobile CSS improved; further work needed)* - [ ] **Budget alerts** — email/Twilio SMS when category spending hits 80% / 100% ### Medium priority -- [ ] **Pagination info** — show "Page X of Y" on AI history and other paginated pages - [ ] **PDF export memory** — stream CSV/Excel exports for users with large transaction history - [ ] **Receipt MIME validation** — validate file magic bytes server-side, not just extension - [ ] **OCR ownership check** — verify re-extracted filename belongs to current user's transaction - [ ] **Bank import progress** — show per-row import progress for large statement files -- [ ] **Schwab IRA account type** — map Schwab IRA account type to `investment` in ACCOUNT_TYPE_MAP ### Low priority / future - [ ] iOS companion app - [ ] Shared household mode (2 users, row-level isolation) - [ ] Bank statement PDF: table-extraction fallback (pdfplumber tables API) before Groq call -- [ ] Investment price history chart per holding - [ ] Dark mode toggle -- [ ] Schwab auto-sync on schedule (currently manual only) + +### Completed (removed from backlog) +- [x] **Pagination info** — "Page X of Y" added to transactions, AI history, accounts/payments, audit log +- [x] **Schwab IRA account type** — IRA/ROTH_IRA/401K/BROKERAGE types added to ACCOUNT_TYPE_MAP +- [x] **Investment price history chart** — 1W/1M/3M/6M/1Y chart on investment detail page +- [x] **Schwab auto-sync on schedule** — `scripts/sync_schwab.py` (cron at 7AM daily) diff --git a/app/routes/logs.py b/app/routes/logs.py index a1e232a..0c97470 100644 --- a/app/routes/logs.py +++ b/app/routes/logs.py @@ -49,13 +49,26 @@ def _parse_line(raw): @logs_bp.route('/') @login_required def index(): + import datetime log_file = current_app.config.get('LOG_FILE_PATH', '') file_exists = bool(log_file) and os.path.isfile(log_file) file_size = os.path.getsize(log_file) if file_exists else 0 + file_mtime = None + if file_exists: + ts = os.path.getmtime(log_file) + file_mtime = datetime.datetime.fromtimestamp(ts).strftime('%b %d, %H:%M') + + def fmt_size(b): + if b < 1024: return f'{b} B' + if b < 1024**2: return f'{b/1024:.1f} KB' + return f'{b/1024**2:.1f} MB' + return render_template('logs/index.html', log_file=log_file, file_exists=file_exists, - file_size=file_size) + file_size=file_size, + file_size_fmt=fmt_size(file_size), + file_mtime=file_mtime) @logs_bp.route('/api') @@ -87,17 +100,21 @@ def api(): # Most-recent first, capped at limit entries = list(reversed(entries))[:limit] - # Count per level across ALL unfiltered lines (for the stats bar) + # Count per level + unique modules across ALL unfiltered lines all_entries = [_parse_line(l) for l in raw_lines if l.strip()] counts = {} + modules_seen = set() for e in all_entries: counts[e['level']] = counts.get(e['level'], 0) + 1 + if e['name']: + modules_seen.add(e['name']) return jsonify({ 'entries': entries, 'counts': counts, 'total_raw': len(raw_lines), 'log_file': log_file, + 'modules': sorted(modules_seen), }) diff --git a/app/templates/base.html b/app/templates/base.html index 1cf7411..e45f653 100644 --- a/app/templates/base.html +++ b/app/templates/base.html @@ -303,15 +303,9 @@ > Categories - - System Logs - Settings diff --git a/app/templates/logs/index.html b/app/templates/logs/index.html index 48fde5e..98a1537 100644 --- a/app/templates/logs/index.html +++ b/app/templates/logs/index.html @@ -4,269 +4,499 @@ {% block extra_css %} {% endblock %} -{% block topbar_actions %} - - - -{% endblock %} - {% block content %} -
-
Application Logs
- - - {% if file_exists %}{{ log_file }}{% else %}Log file not found{% endif %} - + +
+ +
+ {% if file_exists %} + {{ file_size_fmt }} + {{ file_mtime }} + + {{ log_file }} + + {% else %} + Log file not found + {% endif %} +
+
+ +
- -
- All — - Error 0 - Warning 0 - Info 0 - Debug 0 + +
+ All — + Error 0 + Critical 0 + Warning 0 + Info 0 + Debug 0
- - - -
- - +
+ + +
+ + + + + + + +
+ +
+ + Auto-refresh + +
+ + + Download + + +
-
- - - - - - - - - - - - -
TimestampLevelModuleMessage
- - Loading… -
+
+ + + + + + + + + + + + +
TimestampLevelModuleMessage
+ Loading… +
-
- - + +
+ +
{% endblock %} {% block extra_js %} {% endblock %} diff --git a/app/templates/settings/index.html b/app/templates/settings/index.html index 1c71ef8..2df4956 100644 --- a/app/templates/settings/index.html +++ b/app/templates/settings/index.html @@ -61,7 +61,7 @@
- +
@@ -72,6 +72,15 @@
+
+ +
+ +
System Logs
+
Application log viewer
+
+ +