06/03 Update documents
This commit is contained in:
@@ -7,7 +7,7 @@
|
||||
|
||||
## 1. Project Overview
|
||||
|
||||
Self-hosted personal finance web app. Tracks income, expenses, investments. AI assistant powered by **Groq API** (free tier, fast inference, no local hardware). Receipt OCR via Groq vision model. Bank account sync via Teller API (mTLS). Bank statement import (CSV, OFX/QFX, PDF). Everything runs on Ubuntu server behind Nginx + Certbot SSL.
|
||||
Self-hosted personal finance web app. Tracks income, expenses, investments. AI assistant powered by **Groq API** (free tier, fast inference, no local hardware). Receipt OCR via Groq vision model. Bank account sync via Teller API (mTLS) and Schwab Developer API (OAuth 2.0). Bank statement import (CSV, OFX/QFX, PDF). Everything runs on Ubuntu server behind Nginx + Certbot SSL.
|
||||
|
||||
**Status: All 7 phases complete + all post-MVP features implemented.**
|
||||
|
||||
@@ -27,6 +27,7 @@ Self-hosted personal finance web app. Tracks income, expenses, investments. AI a
|
||||
- Source label shown (yfinance / exchangerate-api)
|
||||
- Stale indicator if rate > 1 day old
|
||||
- Period selector: This Month / Last Month / Custom date range
|
||||
- Credit card accounts display "owed" balance (positive Amount Owed) not raw negative
|
||||
|
||||
### 2.2 Transactions
|
||||
- Income + Expense entry with Income/Expense tabs
|
||||
@@ -36,14 +37,24 @@ Self-hosted personal finance web app. Tracks income, expenses, investments. AI a
|
||||
- Receipt upload (PNG/JPG/WEBP/GIF/PDF, max 10MB)
|
||||
- **AI Receipt OCR** — drag-drop receipt image → Groq vision extracts amount/date/merchant/category → auto-fills form
|
||||
- Re-extract from already-uploaded receipt (edit mode)
|
||||
- **Inline category change** — Category column is a `<select>` dropdown; change fires AJAX `POST /transactions/<id>/set-category` with no page reload
|
||||
- Export to CSV / Excel
|
||||
- Edit form: receipt sub-forms are outside `#txnForm` to prevent nested-form bug
|
||||
|
||||
### 2.3 Accounts
|
||||
- Types: checking, savings, cash, credit_card, crypto, investment, other
|
||||
- Balance auto-calculated from all transactions (not manually entered)
|
||||
- Color + icon picker
|
||||
- Soft delete
|
||||
- Balance source of truth:
|
||||
- **Teller-linked accounts**: balance comes from Teller API (live refresh or after sync); `calc_balance` is NOT called on page load for these
|
||||
- **Schwab-linked accounts**: balance comes from Schwab snapshot sync; `calc_balance` is NOT called on page load for these
|
||||
- **Unlinked accounts**: balance auto-calculated from all transactions via `calc_balance`
|
||||
- **Teller badge** (blue) shown on account cards linked to Teller
|
||||
- **Schwab badge** (green) shown on account cards linked to Schwab
|
||||
- Per-account action buttons for provider-linked accounts:
|
||||
- **Teller**: Refresh (live balance AJAX), Sync (transaction preview), Reset (90-day resync)
|
||||
- **Schwab**: Balance & Positions (snapshot sync POST), Transactions (preview link)
|
||||
- Color + icon picker; soft delete
|
||||
- Credit cards show "Amount Owed" (positive) and "This Month" charges
|
||||
- Opening balance field on account creation (negative for credit cards = starting debt)
|
||||
|
||||
### 2.4 Categories
|
||||
- Expense + Income categories with color/icon
|
||||
@@ -72,8 +83,12 @@ Self-hosted personal finance web app. Tracks income, expenses, investments. AI a
|
||||
- yfinance price auto-fetch (daily 4PM weekdays via cron)
|
||||
- Manual price refresh button (portfolio page)
|
||||
- Live ticker check on add form
|
||||
- Doughnut allocation chart
|
||||
- Doughnut allocation chart + per-asset-type breakdown
|
||||
- P&L per holding + portfolio total
|
||||
- **"Sync Schwab"** button (topbar POST) — runs snapshot sync for all mapped Schwab accounts, redirects back to investments page
|
||||
- **Per-account sections** — when investments span multiple accounts (e.g. Individual + Roth IRA), holdings are grouped into one card per account, each showing account name, total value, and holdings table. Allocation sidebar also shows "By Account" breakdown
|
||||
- `investments.account_id` FK — each Schwab-synced holding is stamped with its source account; same ticker in different accounts (AAPL in Individual vs Roth IRA) stays as separate rows
|
||||
- Holdings table uses a Jinja2 `{% macro %}` (reused across single and multi-account views)
|
||||
|
||||
### 2.8 AI Financial Assistant
|
||||
- Chat UI with SSE streaming (Groq API, word-by-word response)
|
||||
@@ -126,61 +141,64 @@ Self-hosted personal finance web app. Tracks income, expenses, investments. AI a
|
||||
|
||||
### 2.13 Teller Bank Sync
|
||||
- Connects US bank accounts via Teller API (mTLS + HTTP Basic Auth)
|
||||
- Enrollment via Teller Connect modal (JavaScript widget)
|
||||
- Account mapping: each Teller account → PFM account (or auto-create new)
|
||||
- Account ID validated against DB before saving (security fix)
|
||||
- Transaction sync: preview → confirm → import
|
||||
- Duplicate detection using Teller transaction ID (stored as `Teller:<id>` in notes)
|
||||
- Balance refresh (live from Teller API)
|
||||
- **Bank Connections page** (`/teller/`) — connect/disconnect only; no sync buttons here
|
||||
- Shows: institution name, connected date, last synced, account list with linked PFM account names
|
||||
- "Map Account" button for unmapped accounts
|
||||
- **Accounts page** — all Teller action buttons live here per account card (see §2.3)
|
||||
- Transaction sync: preview → confirm → import; after import, live balance fetched from Teller API (not recalculated from transactions)
|
||||
- Duplicate detection using Teller transaction ID (`Teller:<id>` in notes)
|
||||
- Balance refresh: uses `ledger` field for credit cards (amount owed), `available` for bank accounts
|
||||
- Credit card ledger forced negative (our debt convention: `-abs(ledger)`)
|
||||
- Auto-categorize: keyword match on description (`auto_categorize` from `bank_import_service`) first; Teller category field as fallback
|
||||
- Income/expense type: positive Teller `amount` = income (credit), negative = expense (debit)
|
||||
- Webhook: `transactions.processed` event with HMAC-SHA256 signature + 5-min replay protection
|
||||
- Disconnect enrollment
|
||||
- Config: `TELLER_APP_ID`, `TELLER_ENV`, `TELLER_CERT_PATH`, `TELLER_KEY_PATH`, `TELLER_WEBHOOK_SECRET`
|
||||
- Models: `teller_enrollments`, `teller_accounts` (2 new tables)
|
||||
- Models: `teller_enrollments`, `teller_accounts` (2 tables)
|
||||
|
||||
### 2.14 Bank Statement Import
|
||||
### 2.14 Schwab Bank Sync
|
||||
- Connects Schwab brokerage/IRA accounts via Schwab Developer API (OAuth 2.0)
|
||||
- **OAuth flow**: `GET /schwab/connect` → redirect to Schwab with PKCE state → `GET /schwab/callback` → exchange code → store tokens
|
||||
- State parameter included in auth URL (fix for "OAuth state mismatch" error)
|
||||
- `SCHWAB_REDIRECT_URI` must match exactly what's registered in Schwab developer portal
|
||||
- **Account hashes**: Schwab requires `hashValue` (encrypted account number) in all API paths
|
||||
- On connect: calls `GET /trader/v1/accounts/accountNumbers` to get `{accountNumber → hashValue}` map
|
||||
- `account_hash` stored in DB is always the `hashValue`, never the raw account number
|
||||
- On reconnect: existing `SchwabAccount` records updated in-place (by hash, raw number, or masked display) to preserve `pfm_account_id` mapping
|
||||
- **Account mapping**: each Schwab account → PFM account (or create new); stored in `schwab_accounts`
|
||||
- **Transaction sync**: preview → confirm → import; uses same duplicate-skipping as other providers
|
||||
- After import: live balance fetched from Schwab API (not recalculated from transactions)
|
||||
- **Balance + position snapshot** (`POST /schwab/snapshot/<id>` or "Balance & Positions" button):
|
||||
- Fetches `GET /trader/v1/accounts/{hash}?fields=positions`
|
||||
- Updates linked PFM account balance from `currentBalances.liquidationValue`
|
||||
- Upserts `Investment` records for each long position, matched on `(ticker, account_id)`
|
||||
- Asset type mapping: EQUITY→stock, ETF→etf, MUTUAL_FUND→etf, FIXED_INCOME→bond, CASH_EQUIVALENT→cash, unknown→other
|
||||
- Position zero-quantity and empty-symbol positions skipped; `null` positions array guarded
|
||||
- **Investments sync**: topbar "Sync Schwab" button on investments page → `POST /investments/sync-schwab` → runs snapshot for all mapped accounts (ignores stale `connection_id` — always uses active connection)
|
||||
- Token auto-refresh: access token expires 30 min; refreshed automatically before API calls
|
||||
- Config: `SCHWAB_CLIENT_ID`, `SCHWAB_CLIENT_SECRET`, `SCHWAB_REDIRECT_URI`
|
||||
- Models: `schwab_connections`, `schwab_accounts` (2 new tables)
|
||||
|
||||
### 2.15 Bank Statement Import
|
||||
- Sidebar link: "Import Statement" under Money section
|
||||
- Supported formats:
|
||||
- **PDF** — pdfplumber text extraction + Groq LLM parsing (digital PDFs only; not scanned)
|
||||
- **OFX / QFX** — both SGML and XML variants; handles all TRNTYPE codes
|
||||
- **Chase** CSV — `Transaction Date, Description, Amount`
|
||||
- **Bank of America** CSV — `Posted Date, Payee, Amount`
|
||||
- **Citi** CSV — `Date, Description, Debit, Credit`
|
||||
- **Capital One** CSV — `Transaction Date, Description, Debit, Credit`
|
||||
- **Discover** CSV — `Trans. Date, Description, Amount` (positive = expense)
|
||||
- **Amex** CSV — `Date, Description, Amount`
|
||||
- **USAA** CSV — `Date, Description, Original Description, Amount`
|
||||
- **Wells Fargo** CSV — `Date, Amount, Description`
|
||||
- **Generic CSV** — heuristic column detection
|
||||
- **Custom mapping** — UI to map columns when auto-detect fails
|
||||
- Supported formats: PDF, OFX/QFX, Chase/BofA/Citi/Capital One/Discover/Amex/USAA/Wells Fargo CSV, Generic CSV, Custom column mapping
|
||||
- Auto-categorizes using 200+ keyword rules across 14 categories
|
||||
- Preview table: per-row checkboxes, editable category dropdowns
|
||||
- Duplicate detection: OFX FITID (stored as `import:<id>` in notes) or date+amount+description+account
|
||||
- PDF notes: 30 K char limit per upload; scanned PDFs rejected with clear error
|
||||
- AJAX-based: no page reloads, no session storage for rows
|
||||
- File input lives outside drop zone (prevents overlay-blocking other controls)
|
||||
- Duplicate detection: OFX FITID (`import:<id>` in notes) or date+amount+description+account
|
||||
- AJAX-based: no page reloads
|
||||
|
||||
### 2.15 System Logs Viewer
|
||||
- Sidebar link: "System Logs" (`bi-terminal`) in footer section
|
||||
### 2.16 System Logs Viewer
|
||||
- Log file: `logs/app.log` (rotating, 10 MB, 5 backups)
|
||||
- Format: `YYYY-MM-DD HH:MM:SS|LEVEL|module.name|message` (pipe-delimited for parsing)
|
||||
- Viewer at `/logs/`:
|
||||
- Colour-coded level pills: ERROR / WARNING / INFO / DEBUG
|
||||
- Free-text search + module filter + row limit (100/200/500/1000)
|
||||
- Auto-refresh every 5 s (toggle with green pulse indicator)
|
||||
- Clear log file button (POST with CSRF)
|
||||
- Download raw log file
|
||||
- Logging wired to Gunicorn via `gunicorn.error` handlers; also writes to stderr
|
||||
- `app.*` namespace loggers all inherit from `logging.getLogger('app')` at INFO level
|
||||
- All Teller API calls log: status, URL, and response body on error
|
||||
- Format: `YYYY-MM-DD HH:MM:SS|LEVEL|module.name|message`
|
||||
- Viewer at `/logs/`: colour-coded pills, free-text search, module filter, auto-refresh, clear, download
|
||||
|
||||
---
|
||||
|
||||
## 3. Database Schema (MySQL)
|
||||
|
||||
### All 16 Tables
|
||||
### All 18 Tables
|
||||
```
|
||||
users — single user, hashed password, currency/timezone prefs
|
||||
accounts — bank/wallet accounts (balance auto-calc from txns)
|
||||
accounts — bank/wallet accounts (balance managed per provider rules)
|
||||
categories — expense/income categories with color/icon
|
||||
transactions — income/expense/transfer, receipt_id, recurring_rule_id
|
||||
receipts — receipt file metadata (filename, size, mime_type)
|
||||
@@ -188,21 +206,28 @@ recurring_rules — templates: frequency, next_run, start/end date
|
||||
budgets — monthly limits per category, rollover support
|
||||
goals — savings goals with target amount/date
|
||||
goal_contributions — individual deposits toward each goal
|
||||
investments — holdings: ticker, shares, avg_cost_basis, current_price
|
||||
investments — holdings: ticker, shares, avg_cost_basis, current_price, account_id
|
||||
investment_transactions — buy/sell/dividend/split log
|
||||
net_worth_snapshots — monthly snapshots: assets, liabilities, net_worth (JSON)
|
||||
ai_insights — stored AI responses: daily_summary / chat_response
|
||||
fx_rates — daily USD/VND rate cache (date UNIQUE, source)
|
||||
teller_enrollments — Teller enrollment: enrollment_id, access_token, institution_name
|
||||
teller_accounts — Teller account ↔ PFM account mapping, last_sync_date
|
||||
schwab_connections — Schwab OAuth tokens: access_token, refresh_token, token_expires_at
|
||||
schwab_accounts — Schwab account ↔ PFM account mapping, account_hash (hashValue)
|
||||
```
|
||||
|
||||
### Key Column Notes
|
||||
- `transactions.balance` — NOT stored; calculated on-demand via `account_service.calc_balance()`
|
||||
- `investments.shares` / `avg_cost_basis` — recalculated from `investment_transactions` (FIFO)
|
||||
- `goals.current_amount` — updated on each contribution add/delete
|
||||
- `net_worth_snapshots.account_balances` — JSON snapshot of each account balance at time of snapshot
|
||||
- `transactions.notes` — used to store import source IDs: `Teller:<id>` or `import:<fitid>`
|
||||
- `accounts.balance` — set by `calc_balance()` for unlinked accounts; set directly by Teller/Schwab sync for provider-linked accounts; never overwritten on page load for provider accounts
|
||||
- `investments.account_id` — nullable FK to `accounts.id`; NULL for manually-added holdings, set to PFM account ID for Schwab-synced holdings; enables per-account grouping on portfolio page
|
||||
- `investments.shares` / `avg_cost_basis` — recalculated from `investment_transactions` (FIFO) for manual holdings; overwritten directly by Schwab snapshot for synced holdings
|
||||
- `transactions.notes` — used to store import source IDs: `Teller:<id>`, `Schwab:<activityId>`, or `import:<fitid>`
|
||||
- `schwab_accounts.account_hash` — Schwab `hashValue` (encrypted account number), required in all API paths
|
||||
|
||||
### Migration Scripts
|
||||
```
|
||||
scripts/add_investment_account.py — adds investments.account_id column (run once after deploy)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
@@ -210,21 +235,20 @@ teller_accounts — Teller account ↔ PFM account mapping, last_sync_da
|
||||
|
||||
```
|
||||
pfm/ # /home/pfm/web on server
|
||||
├── wsgi.py # Gunicorn entry — sys.path fix included
|
||||
├── wsgi.py
|
||||
├── requirements.txt
|
||||
├── .env # Not committed
|
||||
├── .env.example
|
||||
├── .gitignore
|
||||
├── logs/
|
||||
│ └── app.log # Rotating application log (created at runtime)
|
||||
│ └── app.log
|
||||
│
|
||||
├── app/
|
||||
│ ├── __init__.py # Flask app factory; _setup_logging(); all blueprints registered
|
||||
│ ├── config.py # Dev/Prod configs; LOG_FILE_PATH; TELLER_* vars
|
||||
│ ├── extensions.py # db, login_manager, migrate, csrf
|
||||
│ ├── __init__.py
|
||||
│ ├── config.py # SCHWAB_* vars added
|
||||
│ ├── extensions.py
|
||||
│ │
|
||||
│ ├── models/
|
||||
│ │ ├── __init__.py
|
||||
│ │ ├── user.py
|
||||
│ │ ├── account.py
|
||||
│ │ ├── category.py
|
||||
@@ -233,27 +257,29 @@ pfm/ # /home/pfm/web on server
|
||||
│ │ ├── recurring_rule.py
|
||||
│ │ ├── budget.py
|
||||
│ │ ├── goal.py
|
||||
│ │ ├── investment.py
|
||||
│ │ ├── investment.py # + account_id FK, account relationship
|
||||
│ │ ├── net_worth_snapshot.py
|
||||
│ │ ├── ai_insight.py
|
||||
│ │ ├── fx_rate.py
|
||||
│ │ └── teller_enrollment.py # TellerEnrollment + TellerAccount models
|
||||
│ │ ├── teller_enrollment.py # TellerEnrollment + TellerAccount
|
||||
│ │ └── schwab_connection.py # SchwabConnection + SchwabAccount (NEW)
|
||||
│ │
|
||||
│ ├── routes/
|
||||
│ │ ├── auth.py
|
||||
│ │ ├── dashboard.py
|
||||
│ │ ├── accounts.py
|
||||
│ │ ├── dashboard.py # credit card "owed" display; no recalc_all
|
||||
│ │ ├── accounts.py # teller_map + schwab_map; skip calc_balance for providers
|
||||
│ │ ├── categories.py
|
||||
│ │ ├── transactions.py # /transactions/ocr, /ocr-file; safe filter int parsing
|
||||
│ │ ├── transactions.py # + set-category AJAX endpoint
|
||||
│ │ ├── budgets.py
|
||||
│ │ ├── goals.py
|
||||
│ │ ├── investments.py
|
||||
│ │ ├── investments.py # + sync-schwab route
|
||||
│ │ ├── reports.py
|
||||
│ │ ├── ai.py
|
||||
│ │ ├── settings.py # view_receipt: os.path.basename() path-traversal fix
|
||||
│ │ ├── teller.py # Teller sync, webhook, account mapping (validated IDs)
|
||||
│ │ ├── bank_import.py # /bank-import/; parse (AJAX); import (AJAX)
|
||||
│ │ └── logs.py # /logs/; /logs/api; /logs/clear; /logs/download
|
||||
│ │ ├── settings.py
|
||||
│ │ ├── teller.py # balance uses ledger/available correctly; full_resync honours next param
|
||||
│ │ ├── schwab.py # OAuth, mapping, sync, snapshot (NEW)
|
||||
│ │ ├── bank_import.py
|
||||
│ │ └── logs.py
|
||||
│ │
|
||||
│ ├── services/
|
||||
│ │ ├── account_service.py
|
||||
@@ -262,49 +288,37 @@ pfm/ # /home/pfm/web on server
|
||||
│ │ ├── export_service.py
|
||||
│ │ ├── fx_service.py
|
||||
│ │ ├── goal_service.py
|
||||
│ │ ├── import_service.py # duplicate check now scoped by account_id
|
||||
│ │ ├── investment_service.py
|
||||
│ │ ├── import_service.py
|
||||
│ │ ├── investment_service.py # get_portfolio_summary returns account_groups
|
||||
│ │ ├── ocr_service.py
|
||||
│ │ ├── recurring_service.py # 90-day catchup cap prevents runaway loops
|
||||
│ │ ├── recurring_service.py
|
||||
│ │ ├── report_service.py
|
||||
│ │ ├── teller_service.py # mTLS session; full response-body logging on errors
|
||||
│ │ └── bank_import_service.py # PDF+OFX+CSV parsing; Groq PDF parsing; auto-categorize
|
||||
│ │ ├── teller_service.py # auto_categorize; correct sign convention; live balance after sync
|
||||
│ │ ├── schwab_service.py # OAuth, account hash, snapshot sync, position upsert (NEW)
|
||||
│ │ └── bank_import_service.py
|
||||
│ │
|
||||
│ ├── templates/
|
||||
│ │ ├── base.html # Sidebar (all active states); csrf-token meta tag
|
||||
│ │ ├── auth/login.html
|
||||
│ │ ├── dashboard/index.html
|
||||
│ │ ├── accounts/
|
||||
│ │ ├── categories/
|
||||
│ │ ├── transactions/ # form.html: receipt forms outside #txnForm (nested-form fix)
|
||||
│ │ ├── budgets/
|
||||
│ │ ├── goals/
|
||||
│ │ ├── investments/
|
||||
│ │ ├── reports/
|
||||
│ │ ├── ai/
|
||||
│ │ ├── settings/
|
||||
│ │ ├── teller/ # index.html, map_accounts.html, preview.html
|
||||
│ │ ├── bank_import/ # index.html (drag-drop; AJAX parse+import; column mapper)
|
||||
│ │ └── logs/ # index.html (live viewer; filters; auto-refresh)
|
||||
│ │
|
||||
│ ├── static/
|
||||
│ │ ├── css/ # (empty — all CSS inline in templates)
|
||||
│ │ ├── js/ # (empty — all JS inline in templates)
|
||||
│ │ └── img/
|
||||
│ │ ├── base.html
|
||||
│ │ ├── dashboard/index.html # credit card owed display
|
||||
│ │ ├── accounts/index.html # Teller/Schwab badges + action buttons
|
||||
│ │ ├── transactions/index.html # inline category <select> + AJAX
|
||||
│ │ ├── investments/index.html # per-account sections; holdings_table macro; Sync Schwab btn
|
||||
│ │ ├── teller/index.html # connect/disconnect only (sync buttons removed)
|
||||
│ │ ├── schwab/ # index.html, map_accounts.html, preview.html (NEW)
|
||||
│ │ └── ... (other templates unchanged)
|
||||
│ │
|
||||
│ └── utils/
|
||||
│ ├── formatters.py
|
||||
│ └── decorators.py
|
||||
│
|
||||
├── migrations/
|
||||
│
|
||||
├── scripts/
|
||||
│ ├── init_db.py
|
||||
│ ├── process_recurring.py
|
||||
│ ├── fetch_fx_rate.py
|
||||
│ ├── fetch_prices.py
|
||||
│ ├── daily_snapshot.py
|
||||
│ └── daily_ai_insight.py
|
||||
│ ├── daily_ai_insight.py
|
||||
│ └── add_investment_account.py # NEW — adds investments.account_id column
|
||||
│
|
||||
└── tests/
|
||||
```
|
||||
@@ -330,7 +344,7 @@ apscheduler==3.10.4
|
||||
requests==2.32.3
|
||||
cryptography==44.0.2
|
||||
python-dateutil==2.9.0
|
||||
pdfplumber==0.11.4 # PDF text extraction for bank statement import
|
||||
pdfplumber==0.11.4
|
||||
```
|
||||
|
||||
---
|
||||
@@ -351,9 +365,8 @@ pdfplumber==0.11.4 # PDF text extraction for bank statement import
|
||||
|
||||
### Bank Statement PDF Parsing
|
||||
- Text extracted by `pdfplumber` then sent to `llama-3.3-70b-versatile`
|
||||
- Prompt requests JSON array of `{date, description, amount, transaction_type}`
|
||||
- Max 30 K chars sent per request (~6 months of typical statements)
|
||||
- Scanned PDFs (no text layer) are rejected with a clear error message
|
||||
- Max 30 K chars sent per request
|
||||
- Scanned PDFs rejected with clear error message
|
||||
|
||||
### Free Tier Limits
|
||||
| Metric | Limit |
|
||||
@@ -371,64 +384,82 @@ pdfplumber==0.11.4 # PDF text extraction for bank statement import
|
||||
- Endpoints: `GET /accounts`, `GET /accounts/:id/balances`, `GET /accounts/:id/transactions`
|
||||
- All API errors logged with status code + full response body
|
||||
- Webhook: HMAC-SHA256 `Teller-Signature` header; 5-minute replay window
|
||||
- **Balance convention**: credit cards use `ledger` (amount owed, stored as negative); bank accounts use `available`
|
||||
- **After transaction sync**: live balance re-fetched from Teller API instead of computing from transactions
|
||||
|
||||
---
|
||||
|
||||
## 8. Bank Statement Import
|
||||
## 8. Schwab Developer API
|
||||
|
||||
- OAuth 2.0: `SCHWAB_AUTH_URL` + `SCHWAB_TOKEN_URL`
|
||||
- State parameter sent in auth URL (CSRF protection)
|
||||
- Account identification: `hashValue` from `/trader/v1/accounts/accountNumbers` (NOT raw account number)
|
||||
- Token refresh: access tokens expire 30 min; auto-refreshed via `_ensure_fresh(connection)`
|
||||
- Endpoints:
|
||||
- `GET /trader/v1/accounts/accountNumbers` → `{accountNumber: hashValue}` map
|
||||
- `GET /trader/v1/accounts?fields=positions` → accounts list with balances + positions
|
||||
- `GET /trader/v1/accounts/{hash}?fields=positions` → single account
|
||||
- `GET /trader/v1/accounts/{hash}/transactions?startDate&endDate` → transactions
|
||||
|
||||
---
|
||||
|
||||
## 9. Bank Statement Import
|
||||
|
||||
- Route: `/bank-import/` (blueprint `bank_import_bp`)
|
||||
- Parse: `POST /bank-import/parse` (AJAX, multipart with `X-CSRFToken` header)
|
||||
- Import: `POST /bank-import/import` (AJAX, JSON with `X-CSRFToken` header)
|
||||
- File input is hidden and outside the drop zone (`fileInput.click()` on drop zone click)
|
||||
- Duplicate detection:
|
||||
- OFX: match on `import:<FITID>` in notes
|
||||
- CSV/PDF: match on date + amount + type + description scoped to same account_id
|
||||
- Duplicate detection: OFX `import:<FITID>` in notes; CSV/PDF: date+amount+type+description scoped to account_id
|
||||
|
||||
---
|
||||
|
||||
## 9. Logging System
|
||||
## 10. Account Balance Rules
|
||||
|
||||
| Account type | Balance source | When updated |
|
||||
|---|---|---|
|
||||
| Unlinked (no provider) | `calc_balance()` from transactions | After every txn add/edit/delete; on accounts page load |
|
||||
| Teller-linked | Teller API `available` (bank) or `ledger` (credit card) | After Teller sync; when Refresh button clicked |
|
||||
| Schwab-linked | Schwab API `liquidationValue` | After Schwab sync; when Balance & Positions clicked |
|
||||
|
||||
**Key rule**: accounts page load calls `calc_balance` ONLY for accounts NOT in `teller_map` or `schwab_map`. Dashboard does NOT call `calc_balance` (reads stored values).
|
||||
|
||||
---
|
||||
|
||||
## 11. Logging System
|
||||
|
||||
- Config key: `LOG_FILE_PATH` (default: `<project-root>/logs/app.log`)
|
||||
- Handler: `RotatingFileHandler` — 10 MB per file, 5 backups
|
||||
- Format: `YYYY-MM-DD HH:MM:SS|LEVEL|module.name|message`
|
||||
- Namespace: `logging.getLogger('app')` at INFO; `propagate=False`
|
||||
- Also writes to stderr (Gunicorn captures it)
|
||||
- Viewer: `/logs/` — real-time filtered display, per-level counts, auto-refresh, clear, download
|
||||
- Schwab snapshot sync logs: number of positions returned, per-position symbol/type/qty/mapped-type
|
||||
|
||||
---
|
||||
|
||||
## 10. UI/UX
|
||||
## 12. UI/UX
|
||||
|
||||
- **Sidebar**: collapsible (desktop state saved in localStorage), mobile overlay
|
||||
- Active states: `{% if request.blueprint == '...' %}active{% endif %}`
|
||||
- Links: Dashboard · Transactions · Add Income · Add Expense · Accounts · Import Statement · Budgets · Goals · Investments · Reports · AI Assistant · Categories · System Logs · Settings · Logout
|
||||
- **Charts**: Chart.js 4.x (CDN)
|
||||
- **Forms**: WTForms + Bootstrap 5.3
|
||||
- **Icons**: Bootstrap Icons 1.11
|
||||
- **Fonts**: DM Sans + DM Mono (Google Fonts CDN)
|
||||
- **Color scheme**: `#0f172a` sidebar, `#f1f5f9` body, `#10b981` income, `#ef4444` expense, `#3b82f6` invest
|
||||
- **CSS**: All inline in templates (no build step)
|
||||
- **SSE**: AI chat stream + FX refresh
|
||||
- **CSRF meta tag**: `<meta name="csrf-token">` in `base.html` for JS fetch calls
|
||||
|
||||
---
|
||||
|
||||
## 11. Authentication & Security
|
||||
## 13. Authentication & Security
|
||||
|
||||
- Single-user, Flask-Login, session-based
|
||||
- Hashed password (Werkzeug `generate_password_hash`)
|
||||
- `SESSION_COOKIE_SECURE=True` in production
|
||||
- `SESSION_COOKIE_HTTPONLY=True`, `SESSION_COOKIE_SAMESITE='Lax'`
|
||||
- `SESSION_COOKIE_SECURE=True`, `SESSION_COOKIE_HTTPONLY=True`, `SESSION_COOKIE_SAMESITE='Lax'`
|
||||
- CSRF protection on all forms (Flask-WTF); meta tag in base.html for AJAX
|
||||
- SQLAlchemy ORM (no raw SQL)
|
||||
- Receipt file path: `os.path.basename()` in both upload AND view_receipt (path-traversal fix)
|
||||
- Teller account IDs validated against DB before mapping
|
||||
- Transaction filter params safely cast with try/except (no crash on bad int input)
|
||||
- Groq receives anonymised transaction summaries (no account/personal names)
|
||||
- Schwab OAuth state parameter validated on callback (CSRF protection)
|
||||
- `next` redirect params validated to start with `/` (no open redirect)
|
||||
|
||||
---
|
||||
|
||||
## 12. Scheduled Jobs
|
||||
## 14. Scheduled Jobs
|
||||
|
||||
| Job | Schedule | Script | Notes |
|
||||
|-----|----------|--------|-------|
|
||||
@@ -441,7 +472,7 @@ pdfplumber==0.11.4 # PDF text extraction for bank statement import
|
||||
|
||||
---
|
||||
|
||||
## 13. Environment Variables (`.env`)
|
||||
## 15. Environment Variables (`.env`)
|
||||
|
||||
```
|
||||
SECRET_KEY=your-secret-key
|
||||
@@ -462,44 +493,50 @@ TELLER_ENV=development
|
||||
TELLER_CERT_PATH=/home/pfm/teller/certificate.pem
|
||||
TELLER_KEY_PATH=/home/pfm/teller/private_key.pem
|
||||
TELLER_WEBHOOK_SECRET=your-webhook-secret
|
||||
# Schwab
|
||||
SCHWAB_CLIENT_ID=your-schwab-client-id
|
||||
SCHWAB_CLIENT_SECRET=your-schwab-client-secret
|
||||
SCHWAB_REDIRECT_URI=https://pfm.ngodanguyen.tech/schwab/callback
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 14. Blueprints Registered (13 total)
|
||||
## 16. Blueprints Registered (15 total)
|
||||
|
||||
| Blueprint | Prefix | Key routes |
|
||||
|-----------|--------|------------|
|
||||
| auth | /auth | login, logout |
|
||||
| dashboard | / | index, api/fx-history, api/fx-refresh |
|
||||
| accounts | /accounts | CRUD |
|
||||
| accounts | /accounts | CRUD, adjust |
|
||||
| categories | /categories | CRUD |
|
||||
| transactions | /transactions | index, new, edit, delete, transfer, ocr, ocr-file |
|
||||
| transactions | /transactions | index, new, edit, delete, transfer, ocr, ocr-file, `<id>/set-category` |
|
||||
| budgets | /budgets | index, new, edit, delete, copy |
|
||||
| goals | /goals | index, new, edit, delete, contribute, contributions |
|
||||
| investments | /investments | index, new, detail, edit, delete, add_transaction, refresh-prices, api/price |
|
||||
| investments | /investments | index, new, detail, edit, delete, add_transaction, refresh-prices, sync-schwab, api/price, api/daychange, api/price-history |
|
||||
| reports | /reports | monthly, quarterly, yearly, tax, export/csv\|excel\|pdf, snapshot |
|
||||
| ai | /ai | index, stream (SSE), history, generate-insight |
|
||||
| settings | /settings | index, profile, password, recurring, import, upload_receipt, delete_receipt, view_receipt |
|
||||
| teller | /teller | callback, map, index, sync, sync/confirm, sync/all, balance, disconnect, webhook |
|
||||
| settings | /settings | index, profile, password, recurring, import, recalc-balances, upload_receipt, delete_receipt, view_receipt |
|
||||
| teller | /teller | callback, map, index, sync, sync/confirm, sync/all, balance, balance/all, resync, disconnect, webhook |
|
||||
| schwab | /schwab | connect, callback, index, map, sync/`<id>`, sync/confirm, resync, snapshot/`<id>`, disconnect |
|
||||
| bank_import | /bank-import | index, parse (AJAX), import (AJAX) |
|
||||
| logs | /logs | index, api (AJAX), clear (AJAX), download |
|
||||
|
||||
---
|
||||
|
||||
## 15. Known Issues / Notes
|
||||
## 17. Known Issues / Notes
|
||||
|
||||
- `wsgi.py` has `sys.path.insert(0, ...)` — required for Gunicorn at `/home/pfm/web/`
|
||||
- FX rate widget: `open.er-api.com` may return stale values; yfinance is the reliable primary
|
||||
- WeasyPrint PDF: requires `libpango*` system libs on server
|
||||
- Bank statement PDF import: scanned/image PDFs have no text layer; must use digital download
|
||||
- Bank statement PDF import: large PDFs (>30 K chars) are truncated; split into shorter date ranges
|
||||
- pdfplumber must be installed: `pip install pdfplumber==0.11.4`
|
||||
- Schwab: `investments.account_id` column requires migration — run `scripts/add_investment_account.py` once after deploy
|
||||
- Schwab: after first connect, run "Balance & Positions" to populate investments; then re-sync if holdings were already added manually (they will be updated to link to the account)
|
||||
- Teller: development environment only; requires cert/key from Teller Dashboard
|
||||
- MySQL does not support `NULLS LAST`; use `func.isnull(column)` for null-last ordering
|
||||
|
||||
---
|
||||
|
||||
## 16. Security Fixes Applied (session log)
|
||||
## 18. Security Fixes Applied (session log)
|
||||
|
||||
| Date | Fix | File |
|
||||
|------|-----|------|
|
||||
@@ -511,23 +548,27 @@ TELLER_WEBHOOK_SECRET=your-webhook-secret
|
||||
| 2026-06 | CSRF token added to bank import AJAX parse request | bank_import/index.html |
|
||||
| 2026-06 | Receipt sub-forms moved outside `#txnForm` (nested-form bug) | transactions/form.html |
|
||||
| 2026-06 | Drop zone file input moved outside overlay (blocked account select) | bank_import/index.html |
|
||||
| 2026-06 | Teller balance refresh uses `ledger` for credit cards (not `available`) | teller.py |
|
||||
| 2026-06 | Schwab OAuth `state` param added to auth URL (state mismatch fix) | schwab_service.py |
|
||||
| 2026-06 | Schwab uses `hashValue` (not raw account number) in API paths | schwab.py, schwab_service.py |
|
||||
| 2026-06 | `next` redirect params validated to start with `/` (no open redirect) | teller.py, schwab.py |
|
||||
| 2026-06 | Teller income/expense type corrected (positive = income) | teller_service.py |
|
||||
|
||||
---
|
||||
|
||||
## 17. To-Do / Roadmap
|
||||
## 19. To-Do / Roadmap
|
||||
|
||||
### High priority
|
||||
- [ ] **Mobile responsiveness pass** — sidebar auto-collapses on mobile; tables scroll horizontally
|
||||
- [ ] **Empty-state messages** — transactions, budgets, goals, investments pages when no data
|
||||
- [ ] **Budget alerts** — email/Twilio SMS when category spending hits 80% / 100%
|
||||
|
||||
### Medium priority
|
||||
- [ ] **Pagination info** — show "Page X of Y" on AI history and other paginated pages
|
||||
- [ ] **PDF export memory** — stream CSV/Excel exports for users with large transaction history
|
||||
- [ ] **Receipt MIME validation** — validate file magic bytes server-side, not just extension
|
||||
- [ ] **Teller multi-account sync** — sync all mapped accounts in sequence (currently syncs first only)
|
||||
- [ ] **OCR ownership check** — verify re-extracted filename belongs to current user's transaction
|
||||
- [ ] **Bank import progress** — show per-row import progress for large statement files
|
||||
- [ ] **Schwab IRA account type** — map Schwab IRA account type to `investment` in ACCOUNT_TYPE_MAP
|
||||
|
||||
### Low priority / future
|
||||
- [ ] iOS companion app
|
||||
@@ -535,3 +576,4 @@ TELLER_WEBHOOK_SECRET=your-webhook-secret
|
||||
- [ ] Bank statement PDF: table-extraction fallback (pdfplumber tables API) before Groq call
|
||||
- [ ] Investment price history chart per holding
|
||||
- [ ] Dark mode toggle
|
||||
- [ ] Schwab auto-sync on schedule (currently manual only)
|
||||
|
||||
Reference in New Issue
Block a user