# Personal Finance Management System (PFM)
> Stack: Python Flask · MySQL · Ubuntu Server · Nginx · Gunicorn · Groq API (free AI)
> App URL: https://pfm.ngodanguyen.tech
> Code: /home/pfm/web · User: pfm · Gitea: gitea.ngodanguyen.tech

---

## 1. Project Overview

Self-hosted personal finance web app. Tracks income, expenses, investments. AI assistant powered by **Groq API** (free tier, fast inference, no local hardware). Receipt OCR via Groq vision model. Bank account sync via Teller API (mTLS). Bank statement import (CSV, OFX/QFX, PDF). Everything runs on Ubuntu server behind Nginx + Certbot SSL.

**Status: All 7 phases complete + all post-MVP features implemented.**

---

## 2. Core Features (Implemented)

### 2.1 Dashboard
- Net worth snapshot (assets − liabilities)
- Monthly cash flow bar chart (6 months)
- Budget utilization per category
- Recent transactions feed (last 8)
- AI daily insight card (Groq-generated, stored in DB)
- USD → VND exchange rate widget — reference only, independent of app currency
  - Click to expand 30-day history chart
  - ↻ refresh button (force-fetches fresh rate without page reload)
  - Source label shown (yfinance / exchangerate-api)
  - Stale indicator if rate > 1 day old
- Period selector: This Month / Last Month / Custom date range

### 2.2 Transactions
- Income + Expense entry with Income/Expense tabs
- Transfer between accounts
- Filter: search, category, account, date range (safe int parsing — no crash on bad params)
- Pagination (30/page)
- Receipt upload (PNG/JPG/WEBP/GIF/PDF, max 10MB)
- **AI Receipt OCR** — drag-drop receipt image → Groq vision extracts amount/date/merchant/category → auto-fills form
- Re-extract from already-uploaded receipt (edit mode)
- Export to CSV / Excel
- Edit form: receipt sub-forms are outside `#txnForm` to prevent nested-form bug

### 2.3 Accounts
- Types: checking, savings, cash, credit_card, crypto, investment, other
- Balance auto-calculated from all transactions (not manually entered)
- Color + icon picker
- Soft delete

### 2.4 Categories
- Expense + Income categories with color/icon
- System categories (protected from delete)
- Custom categories (user-created)
- 14 expense + 7 income defaults seeded on init

### 2.5 Budget Planner
- Monthly limits per expense category
- Progress bars: green → amber (80%) → red (100%+)
- Rollover unused budget to next month (toggle)
- Copy previous month's budgets in one click
- Unbudgeted spending shown with "Set Budget" prompt

### 2.6 Goals & Savings
- Goals with target amount, target date, color, icon
- Contribution tracking + history
- Progress bar + projected completion date (based on avg monthly contrib)
- Auto-complete on 100%
- Emergency fund tracker (liquid assets vs 3-month / 6-month expense targets)

### 2.7 Investments
- Asset types: stock, ETF, crypto, real_estate, bond, cash, other
- Buy/sell/dividend/split transaction log
- FIFO cost basis auto-recalculated from transaction log
- yfinance price auto-fetch (daily 4PM weekdays via cron)
- Manual price refresh button (portfolio page)
- Live ticker check on add form
- Doughnut allocation chart
- P&L per holding + portfolio total

### 2.8 AI Financial Assistant
- Chat UI with SSE streaming (Groq API, word-by-word response)
- Context: last 90 days transactions + budget status + goals + investments
- 8 suggested question buttons
- Daily auto-insight generated at midnight (stored in `ai_insights` table)
- Manual "Generate Now" button
- Chat history page
- Model: `llama-3.3-70b-versatile` (default) or `llama-3.1-8b-instant` (fast)
- Fallback messages for rate limit / invalid key / unavailable

### 2.9 Receipt OCR (Groq Vision)
- Model: `meta-llama/llama-4-scout-17b-16e-instruct`
- Drag-drop or click-to-upload on new expense form
- Extracts: amount, date, merchant name, category suggestion, notes
- Maps category suggestion → system category ID
- Auto-fills form fields with green flash animation
- Re-extract button on existing receipt (edit mode)
- Auto-triggers OCR when image file selected in edit mode
- Handles fenced markdown JSON output from LLM
- Sanity check: rejects amounts < 1000 (catches garbage values)
- Full error handling: 400/401/429/timeout/bad JSON

### 2.10 Reports & Export
- Monthly / Quarterly / Yearly summary reports
- Tax year summary (all income by source, all expenses by category)
- Net worth history line chart (from monthly snapshots)
- Category spending trends (top 6 categories, 6-month line chart)
- "Snapshot Now" manual button
- Export: CSV, Excel (color-coded, formatted), PDF (WeasyPrint)

### 2.11 Settings
- Profile: name, email, timezone, currency, Groq model
- 8 currency options (USD/VND/EUR/GBP/JPY/AUD/CAD/SGD) — auto-updates symbol
- Password change (requires current password)
- Recurring rules: CRUD, pause/enable, frequency (daily/weekly/biweekly/monthly/quarterly/yearly)
- "Run Now" button to process due rules immediately
- CSV import: upload → preview with ⚠ warnings → confirm
- Upcoming recurring transactions (30-day view)

### 2.12 USD → VND Exchange Rate
- **Reference widget only** — not used in transaction calculations
- Primary source: yfinance `USDVND=X` (Yahoo Finance forex)
- Fallback: `open.er-api.com` (free, no key)
- Sanity check: rate must be > 1000 (rejects garbage values)
- `force_refresh()` — always fetches fresh on cron, bypasses cache
- DB caches one record per day (`fx_rates` table)
- Dashboard: shows rate, date, source, stale warning, ↻ refresh button
- 30-day history chart (click widget to expand)

### 2.13 Teller Bank Sync
- Connects US bank accounts via Teller API (mTLS + HTTP Basic Auth)
- Enrollment via Teller Connect modal (JavaScript widget)
- Account mapping: each Teller account → PFM account (or auto-create new)
  - Account ID validated against DB before saving (security fix)
- Transaction sync: preview → confirm → import
- Duplicate detection using Teller transaction ID (stored as `Teller:<id>` in notes)
- Balance refresh (live from Teller API)
- Webhook: `transactions.processed` event with HMAC-SHA256 signature + 5-min replay protection
- Disconnect enrollment
- Config: `TELLER_APP_ID`, `TELLER_ENV`, `TELLER_CERT_PATH`, `TELLER_KEY_PATH`, `TELLER_WEBHOOK_SECRET`
- Models: `teller_enrollments`, `teller_accounts` (2 new tables)

### 2.14 Bank Statement Import
- Sidebar link: "Import Statement" under Money section
- Supported formats:
  - **PDF** — pdfplumber text extraction + Groq LLM parsing (digital PDFs only; not scanned)
  - **OFX / QFX** — both SGML and XML variants; handles all TRNTYPE codes
  - **Chase** CSV — `Transaction Date, Description, Amount`
  - **Bank of America** CSV — `Posted Date, Payee, Amount`
  - **Citi** CSV — `Date, Description, Debit, Credit`
  - **Capital One** CSV — `Transaction Date, Description, Debit, Credit`
  - **Discover** CSV — `Trans. Date, Description, Amount` (positive = expense)
  - **Amex** CSV — `Date, Description, Amount`
  - **USAA** CSV — `Date, Description, Original Description, Amount`
  - **Wells Fargo** CSV — `Date, Amount, Description`
  - **Generic CSV** — heuristic column detection
  - **Custom mapping** — UI to map columns when auto-detect fails
- Auto-categorizes using 200+ keyword rules across 14 categories
- Preview table: per-row checkboxes, editable category dropdowns
- Duplicate detection: OFX FITID (stored as `import:<id>` in notes) or date+amount+description+account
- PDF notes: 30 K char limit per upload; scanned PDFs rejected with clear error
- AJAX-based: no page reloads, no session storage for rows
- File input lives outside drop zone (prevents overlay-blocking other controls)

### 2.15 System Logs Viewer
- Sidebar link: "System Logs" (`bi-terminal`) in footer section
- Log file: `logs/app.log` (rotating, 10 MB, 5 backups)
- Format: `YYYY-MM-DD HH:MM:SS|LEVEL|module.name|message` (pipe-delimited for parsing)
- Viewer at `/logs/`:
  - Colour-coded level pills: ERROR / WARNING / INFO / DEBUG
  - Free-text search + module filter + row limit (100/200/500/1000)
  - Auto-refresh every 5 s (toggle with green pulse indicator)
  - Clear log file button (POST with CSRF)
  - Download raw log file
- Logging wired to Gunicorn via `gunicorn.error` handlers; also writes to stderr
- `app.*` namespace loggers all inherit from `logging.getLogger('app')` at INFO level
- All Teller API calls log: status, URL, and response body on error

---

## 3. Database Schema (MySQL)

### All 16 Tables
```
users                   — single user, hashed password, currency/timezone prefs
accounts                — bank/wallet accounts (balance auto-calc from txns)
categories              — expense/income categories with color/icon
transactions            — income/expense/transfer, receipt_id, recurring_rule_id
receipts                — receipt file metadata (filename, size, mime_type)
recurring_rules         — templates: frequency, next_run, start/end date
budgets                 — monthly limits per category, rollover support
goals                   — savings goals with target amount/date
goal_contributions      — individual deposits toward each goal
investments             — holdings: ticker, shares, avg_cost_basis, current_price
investment_transactions — buy/sell/dividend/split log
net_worth_snapshots     — monthly snapshots: assets, liabilities, net_worth (JSON)
ai_insights             — stored AI responses: daily_summary / chat_response
fx_rates                — daily USD/VND rate cache (date UNIQUE, source)
teller_enrollments      — Teller enrollment: enrollment_id, access_token, institution_name
teller_accounts         — Teller account ↔ PFM account mapping, last_sync_date
```

### Key Column Notes
- `transactions.balance` — NOT stored; calculated on-demand via `account_service.calc_balance()`
- `investments.shares` / `avg_cost_basis` — recalculated from `investment_transactions` (FIFO)
- `goals.current_amount` — updated on each contribution add/delete
- `net_worth_snapshots.account_balances` — JSON snapshot of each account balance at time of snapshot
- `transactions.notes` — used to store import source IDs: `Teller:<id>` or `import:<fitid>`

---

## 4. Project File Structure (Actual)

```
pfm/                             # /home/pfm/web on server
├── wsgi.py                      # Gunicorn entry — sys.path fix included
├── requirements.txt
├── .env                         # Not committed
├── .env.example
├── .gitignore
├── logs/
│   └── app.log                  # Rotating application log (created at runtime)
│
├── app/
│   ├── __init__.py              # Flask app factory; _setup_logging(); all blueprints registered
│   ├── config.py                # Dev/Prod configs; LOG_FILE_PATH; TELLER_* vars
│   ├── extensions.py            # db, login_manager, migrate, csrf
│   │
│   ├── models/
│   │   ├── __init__.py
│   │   ├── user.py
│   │   ├── account.py
│   │   ├── category.py
│   │   ├── transaction.py
│   │   ├── receipt.py
│   │   ├── recurring_rule.py
│   │   ├── budget.py
│   │   ├── goal.py
│   │   ├── investment.py
│   │   ├── net_worth_snapshot.py
│   │   ├── ai_insight.py
│   │   ├── fx_rate.py
│   │   └── teller_enrollment.py # TellerEnrollment + TellerAccount models
│   │
│   ├── routes/
│   │   ├── auth.py
│   │   ├── dashboard.py
│   │   ├── accounts.py
│   │   ├── categories.py
│   │   ├── transactions.py      # /transactions/ocr, /ocr-file; safe filter int parsing
│   │   ├── budgets.py
│   │   ├── goals.py
│   │   ├── investments.py
│   │   ├── reports.py
│   │   ├── ai.py
│   │   ├── settings.py          # view_receipt: os.path.basename() path-traversal fix
│   │   ├── teller.py            # Teller sync, webhook, account mapping (validated IDs)
│   │   ├── bank_import.py       # /bank-import/; parse (AJAX); import (AJAX)
│   │   └── logs.py              # /logs/; /logs/api; /logs/clear; /logs/download
│   │
│   ├── services/
│   │   ├── account_service.py
│   │   ├── ai_service.py
│   │   ├── budget_service.py
│   │   ├── export_service.py
│   │   ├── fx_service.py
│   │   ├── goal_service.py
│   │   ├── import_service.py    # duplicate check now scoped by account_id
│   │   ├── investment_service.py
│   │   ├── ocr_service.py
│   │   ├── recurring_service.py # 90-day catchup cap prevents runaway loops
│   │   ├── report_service.py
│   │   ├── teller_service.py    # mTLS session; full response-body logging on errors
│   │   └── bank_import_service.py # PDF+OFX+CSV parsing; Groq PDF parsing; auto-categorize
│   │
│   ├── templates/
│   │   ├── base.html            # Sidebar (all active states); csrf-token meta tag
│   │   ├── auth/login.html
│   │   ├── dashboard/index.html
│   │   ├── accounts/
│   │   ├── categories/
│   │   ├── transactions/        # form.html: receipt forms outside #txnForm (nested-form fix)
│   │   ├── budgets/
│   │   ├── goals/
│   │   ├── investments/
│   │   ├── reports/
│   │   ├── ai/
│   │   ├── settings/
│   │   ├── teller/              # index.html, map_accounts.html, preview.html
│   │   ├── bank_import/         # index.html (drag-drop; AJAX parse+import; column mapper)
│   │   └── logs/                # index.html (live viewer; filters; auto-refresh)
│   │
│   ├── static/
│   │   ├── css/                 # (empty — all CSS inline in templates)
│   │   ├── js/                  # (empty — all JS inline in templates)
│   │   └── img/
│   │
│   └── utils/
│       ├── formatters.py
│       └── decorators.py
│
├── migrations/
│
├── scripts/
│   ├── init_db.py
│   ├── process_recurring.py
│   ├── fetch_fx_rate.py
│   ├── fetch_prices.py
│   ├── daily_snapshot.py
│   └── daily_ai_insight.py
│
└── tests/
```

---

## 5. Python Dependencies (`requirements.txt`)

```
flask==3.1.0
flask-sqlalchemy==3.1.1
flask-login==0.6.3
flask-migrate==4.1.0
flask-wtf==1.2.2
pymysql==1.1.1
python-dotenv==1.0.1
gunicorn==23.0.0
groq==0.13.1
weasyprint==63.1
openpyxl==3.1.5
Pillow==11.1.0
apscheduler==3.10.4
requests==2.32.3
cryptography==44.0.2
python-dateutil==2.9.0
pdfplumber==0.11.4          # PDF text extraction for bank statement import
```

---

## 6. AI Integration — Groq API

### Chat + Daily Insights
- Model: `llama-3.3-70b-versatile` (default) / `llama-3.1-8b-instant` (fast)
- Context: last 90 days transactions, budget status, goals, investments (anonymised)
- SSE streaming: `stream_chat()` yields `data: <chunk>\n\n`
- Daily insight: non-streaming, stored in `ai_insights` table, max 400 tokens

### Receipt OCR (Vision)
- Model: `meta-llama/llama-4-scout-17b-16e-instruct`
- Input: base64-encoded image (JPEG/PNG/GIF/WEBP)
- Prompt: structured JSON extraction (amount, date, merchant, category, notes)
- Temperature: 0.1

### Bank Statement PDF Parsing
- Text extracted by `pdfplumber` then sent to `llama-3.3-70b-versatile`
- Prompt requests JSON array of `{date, description, amount, transaction_type}`
- Max 30 K chars sent per request (~6 months of typical statements)
- Scanned PDFs (no text layer) are rejected with a clear error message

### Free Tier Limits
| Metric | Limit |
|--------|-------|
| Requests/day | 14,400 |
| Tokens/minute | 500,000 |
| Cost | Free |

---

## 7. Teller Bank Sync

- mTLS: client cert + key from `TELLER_CERT_PATH` / `TELLER_KEY_PATH`
- HTTP Basic Auth: `access_token` as username, empty password
- Endpoints: `GET /accounts`, `GET /accounts/:id/balances`, `GET /accounts/:id/transactions`
- All API errors logged with status code + full response body
- Webhook: HMAC-SHA256 `Teller-Signature` header; 5-minute replay window

---

## 8. Bank Statement Import

- Route: `/bank-import/` (blueprint `bank_import_bp`)
- Parse: `POST /bank-import/parse` (AJAX, multipart with `X-CSRFToken` header)
- Import: `POST /bank-import/import` (AJAX, JSON with `X-CSRFToken` header)
- File input is hidden and outside the drop zone (`fileInput.click()` on drop zone click)
- Duplicate detection:
  - OFX: match on `import:<FITID>` in notes
  - CSV/PDF: match on date + amount + type + description scoped to same account_id

---

## 9. Logging System

- Config key: `LOG_FILE_PATH` (default: `<project-root>/logs/app.log`)
- Handler: `RotatingFileHandler` — 10 MB per file, 5 backups
- Format: `YYYY-MM-DD HH:MM:SS|LEVEL|module.name|message`
- Namespace: `logging.getLogger('app')` at INFO; `propagate=False`
- Also writes to stderr (Gunicorn captures it)
- Viewer: `/logs/` — real-time filtered display, per-level counts, auto-refresh, clear, download

---

## 10. UI/UX

- **Sidebar**: collapsible (desktop state saved in localStorage), mobile overlay
  - Active states: `{% if request.blueprint == '...' %}active{% endif %}`
  - Links: Dashboard · Transactions · Add Income · Add Expense · Accounts · Import Statement · Budgets · Goals · Investments · Reports · AI Assistant · Categories · System Logs · Settings · Logout
- **Charts**: Chart.js 4.x (CDN)
- **Forms**: WTForms + Bootstrap 5.3
- **Icons**: Bootstrap Icons 1.11
- **Fonts**: DM Sans + DM Mono (Google Fonts CDN)
- **Color scheme**: `#0f172a` sidebar, `#f1f5f9` body, `#10b981` income, `#ef4444` expense, `#3b82f6` invest
- **CSS**: All inline in templates (no build step)
- **SSE**: AI chat stream + FX refresh
- **CSRF meta tag**: `<meta name="csrf-token">` in `base.html` for JS fetch calls

---

## 11. Authentication & Security

- Single-user, Flask-Login, session-based
- Hashed password (Werkzeug `generate_password_hash`)
- `SESSION_COOKIE_SECURE=True` in production
- `SESSION_COOKIE_HTTPONLY=True`, `SESSION_COOKIE_SAMESITE='Lax'`
- CSRF protection on all forms (Flask-WTF); meta tag in base.html for AJAX
- SQLAlchemy ORM (no raw SQL)
- Receipt file path: `os.path.basename()` in both upload AND view_receipt (path-traversal fix)
- Teller account IDs validated against DB before mapping
- Transaction filter params safely cast with try/except (no crash on bad int input)
- Groq receives anonymised transaction summaries (no account/personal names)

---

## 12. Scheduled Jobs

| Job | Schedule | Script | Notes |
|-----|----------|--------|-------|
| Process recurring transactions | Daily 6AM | `process_recurring.py` | 90-day catchup cap |
| Fetch USD/VND rate | Daily 8AM | `fetch_fx_rate.py` | force_refresh(), yfinance primary |
| Fetch investment prices | Mon-Fri 4PM | `fetch_prices.py` | yfinance, all tickers |
| Net worth snapshot | 1st of month 00:05 | `daily_snapshot.py` | Saves to net_worth_snapshots |
| AI daily insight | Daily 00:01 | `daily_ai_insight.py` | Skips if already done today |
| DB backup | Daily 2AM | pfm-backup (systemd) | mysqldump → gzip, keep 30 days |

---

## 13. Environment Variables (`.env`)

```
SECRET_KEY=your-secret-key
DATABASE_URL=mysql+pymysql://pfm_user:password@localhost/pfm_db
GROQ_API_KEY=your-groq-api-key-here
GROQ_MODEL=llama-3.3-70b-versatile
UPLOAD_FOLDER=/home/pfm/web/uploads
MAX_CONTENT_LENGTH=10485760
FLASK_ENV=production
FLASK_APP=wsgi:app
APP_CURRENCY=USD
APP_CURRENCY_SYMBOL=$
APP_TIMEZONE=Asia/Ho_Chi_Minh
LOG_FILE_PATH=/home/pfm/web/logs/app.log
# Teller
TELLER_APP_ID=your-teller-app-id
TELLER_ENV=development
TELLER_CERT_PATH=/home/pfm/teller/certificate.pem
TELLER_KEY_PATH=/home/pfm/teller/private_key.pem
TELLER_WEBHOOK_SECRET=your-webhook-secret
```

---

## 14. Blueprints Registered (13 total)

| Blueprint | Prefix | Key routes |
|-----------|--------|------------|
| auth | /auth | login, logout |
| dashboard | / | index, api/fx-history, api/fx-refresh |
| accounts | /accounts | CRUD |
| categories | /categories | CRUD |
| transactions | /transactions | index, new, edit, delete, transfer, ocr, ocr-file |
| budgets | /budgets | index, new, edit, delete, copy |
| goals | /goals | index, new, edit, delete, contribute, contributions |
| investments | /investments | index, new, detail, edit, delete, add_transaction, refresh-prices, api/price |
| reports | /reports | monthly, quarterly, yearly, tax, export/csv\|excel\|pdf, snapshot |
| ai | /ai | index, stream (SSE), history, generate-insight |
| settings | /settings | index, profile, password, recurring, import, upload_receipt, delete_receipt, view_receipt |
| teller | /teller | callback, map, index, sync, sync/confirm, sync/all, balance, disconnect, webhook |
| bank_import | /bank-import | index, parse (AJAX), import (AJAX) |
| logs | /logs | index, api (AJAX), clear (AJAX), download |

---

## 15. Known Issues / Notes

- `wsgi.py` has `sys.path.insert(0, ...)` — required for Gunicorn at `/home/pfm/web/`
- FX rate widget: `open.er-api.com` may return stale values; yfinance is the reliable primary
- WeasyPrint PDF: requires `libpango*` system libs on server
- Bank statement PDF import: scanned/image PDFs have no text layer; must use digital download
- Bank statement PDF import: large PDFs (>30 K chars) are truncated; split into shorter date ranges
- pdfplumber must be installed: `pip install pdfplumber==0.11.4`
- Teller: development environment only; requires cert/key from Teller Dashboard

---

## 16. Security Fixes Applied (session log)

| Date | Fix | File |
|------|-----|------|
| 2026-06 | Path traversal in `view_receipt` — added `os.path.basename()` | settings.py |
| 2026-06 | `int()` crash on bad filter params in transactions index | transactions.py |
| 2026-06 | Teller account mapping validates ID exists in DB | teller.py |
| 2026-06 | Recurring catchup capped at 90 days (prevents runaway loops) | recurring_service.py |
| 2026-06 | CSV/bank import duplicate check scoped by `account_id` | import_service.py |
| 2026-06 | CSRF token added to bank import AJAX parse request | bank_import/index.html |
| 2026-06 | Receipt sub-forms moved outside `#txnForm` (nested-form bug) | transactions/form.html |
| 2026-06 | Drop zone file input moved outside overlay (blocked account select) | bank_import/index.html |

---

## 17. To-Do / Roadmap

### High priority
- [ ] **Mobile responsiveness pass** — sidebar auto-collapses on mobile; tables scroll horizontally
- [ ] **Empty-state messages** — transactions, budgets, goals, investments pages when no data
- [ ] **Budget alerts** — email/Twilio SMS when category spending hits 80% / 100%

### Medium priority
- [ ] **Pagination info** — show "Page X of Y" on AI history and other paginated pages
- [ ] **PDF export memory** — stream CSV/Excel exports for users with large transaction history
- [ ] **Receipt MIME validation** — validate file magic bytes server-side, not just extension
- [ ] **Teller multi-account sync** — sync all mapped accounts in sequence (currently syncs first only)
- [ ] **OCR ownership check** — verify re-extracted filename belongs to current user's transaction
- [ ] **Bank import progress** — show per-row import progress for large statement files

### Low priority / future
- [ ] iOS companion app
- [ ] Shared household mode (2 users, row-level isolation)
- [ ] Bank statement PDF: table-extraction fallback (pdfplumber tables API) before Groq call
- [ ] Investment price history chart per holding
- [ ] Dark mode toggle