Personal-Finance-Management/CLAUDE.md

# Personal Finance Management System (PFM)
> Stack: Python Flask · MySQL · Ubuntu Server · Nginx · Gunicorn · Groq API (free AI)
> App URL: https://pfm.ngodanguyen.tech
> Code: /home/pfm/web · User: pfm · Gitea: gitea.ngodanguyen.tech

---

## 1. Project Overview

Self-hosted personal finance web app. Tracks income, expenses, investments. AI assistant powered by **Groq API** (free tier, fast inference, no local hardware). Receipt OCR via Groq vision model. Everything runs on Ubuntu server behind Nginx + Certbot SSL.

**Status: All 7 phases complete + Receipt OCR post-MVP feature.**

---

## 2. Core Features (Implemented)

### 2.1 Dashboard
- Net worth snapshot (assets − liabilities)
- Monthly cash flow bar chart (6 months)
- Budget utilization per category
- Recent transactions feed (last 8)
- AI daily insight card (Groq-generated, stored in DB)
- USD → VND exchange rate widget — reference only, independent of app currency
  - Click to expand 30-day history chart
  - ↻ refresh button (force-fetches fresh rate without page reload)
  - Source label shown (yfinance / exchangerate-api)
  - Stale indicator if rate > 1 day old
- Period selector: This Month / Last Month / Custom date range

### 2.2 Transactions
- Income + Expense entry with Income/Expense tabs
- Transfer between accounts
- Filter: search, category, account, date range
- Pagination (30/page)
- Receipt upload (PNG/JPG/WEBP/GIF/PDF, max 10MB)
- **AI Receipt OCR** — drag-drop receipt image → Groq vision extracts amount/date/merchant/category → auto-fills form
- Re-extract from already-uploaded receipt (edit mode)
- Export to CSV / Excel

### 2.3 Accounts
- Types: checking, savings, cash, credit_card, crypto, investment, other
- Balance auto-calculated from all transactions (not manually entered)
- Color + icon picker
- Soft delete

### 2.4 Categories
- Expense + Income categories with color/icon
- System categories (protected from delete)
- Custom categories (user-created)
- 14 expense + 7 income defaults seeded on init

### 2.5 Budget Planner
- Monthly limits per expense category
- Progress bars: green → amber (80%) → red (100%+)
- Rollover unused budget to next month (toggle)
- Copy previous month's budgets in one click
- Unbudgeted spending shown with "Set Budget" prompt

### 2.6 Goals & Savings
- Goals with target amount, target date, color, icon
- Contribution tracking + history
- Progress bar + projected completion date (based on avg monthly contrib)
- Auto-complete on 100%
- Emergency fund tracker (liquid assets vs 3-month / 6-month expense targets)

### 2.7 Investments
- Asset types: stock, ETF, crypto, real_estate, bond, cash, other
- Buy/sell/dividend/split transaction log
- FIFO cost basis auto-recalculated from transaction log
- yfinance price auto-fetch (daily 4PM weekdays via cron)
- Manual price refresh button (portfolio page)
- Live ticker check on add form
- Doughnut allocation chart
- P&L per holding + portfolio total

### 2.8 AI Financial Assistant
- Chat UI with SSE streaming (Groq API, word-by-word response)
- Context: last 90 days transactions + budget status + goals + investments
- 8 suggested question buttons
- Daily auto-insight generated at midnight (stored in `ai_insights` table)
- Manual "Generate Now" button
- Chat history page
- Model: `llama-3.3-70b-versatile` (default) or `llama-3.1-8b-instant` (fast)
- Fallback messages for rate limit / invalid key / unavailable

### 2.9 Receipt OCR (Groq Vision)
- Model: `meta-llama/llama-4-scout-17b-16e-instruct`
- Drag-drop or click-to-upload on new expense form
- Extracts: amount, date, merchant name, category suggestion, notes
- Maps category suggestion → system category ID
- Auto-fills form fields with green flash animation
- Re-extract button on existing receipt (edit mode)
- Auto-triggers OCR when image file selected in edit mode
- Handles fenced markdown JSON output from LLM
- Sanity check: rejects amounts < 1000 (catches garbage values)
- Full error handling: 400/401/429/timeout/bad JSON

### 2.10 Reports & Export
- Monthly / Quarterly / Yearly summary reports
- Tax year summary (all income by source, all expenses by category)
- Net worth history line chart (from monthly snapshots)
- Category spending trends (top 6 categories, 6-month line chart)
- "Snapshot Now" manual button
- Export: CSV, Excel (color-coded, formatted), PDF (WeasyPrint)

### 2.11 Settings
- Profile: name, email, timezone, currency, Groq model
- 8 currency options (USD/VND/EUR/GBP/JPY/AUD/CAD/SGD) — auto-updates symbol
- Password change (requires current password)
- Recurring rules: CRUD, pause/enable, frequency (daily/weekly/biweekly/monthly/quarterly/yearly)
- "Run Now" button to process due rules immediately
- CSV import: upload → preview with ⚠ warnings → confirm
- Upcoming recurring transactions (30-day view)

### 2.12 USD → VND Exchange Rate
- **Reference widget only** — not used in transaction calculations
- Primary source: yfinance `USDVND=X` (Yahoo Finance forex)
- Fallback: `open.er-api.com` (free, no key)
- Sanity check: rate must be > 1000 (rejects garbage values)
- `force_refresh()` — always fetches fresh on cron, bypasses cache
- DB caches one record per day (`fx_rates` table)
- Dashboard: shows rate, date, source, stale warning, ↻ refresh button
- 30-day history chart (click widget to expand)

---

## 3. Database Schema (MySQL)

### All 14 Tables
```
users                   — single user, hashed password, currency/timezone prefs
accounts                — bank/wallet accounts (balance auto-calc from txns)
categories              — expense/income categories with color/icon
transactions            — income/expense/transfer, receipt_id, recurring_rule_id
receipts                — receipt file metadata (filename, size, mime_type)
recurring_rules         — templates: frequency, next_run, start/end date
budgets                 — monthly limits per category, rollover support
goals                   — savings goals with target amount/date
goal_contributions      — individual deposits toward each goal
investments             — holdings: ticker, shares, avg_cost_basis, current_price
investment_transactions — buy/sell/dividend/split log
net_worth_snapshots     — monthly snapshots: assets, liabilities, net_worth (JSON)
ai_insights             — stored AI responses: daily_summary / chat_response
fx_rates                — daily USD/VND rate cache (date UNIQUE, source)
```

### Key Column Notes
- `transactions.balance` — NOT stored; calculated on-demand via `account_service.calc_balance()`
- `investments.shares` / `avg_cost_basis` — recalculated from `investment_transactions` (FIFO)
- `goals.current_amount` — updated on each contribution add/delete
- `net_worth_snapshots.account_balances` — JSON snapshot of each account balance at time of snapshot

---

## 4. Project File Structure (Actual)

```
pfm/                             # /home/pfm/web on server
├── wsgi.py                      # Gunicorn entry — sys.path fix included
├── requirements.txt
├── .env                         # Not committed
├── .env.example
├── .gitignore
│
├── app/
│   ├── __init__.py              # Flask app factory, all blueprints registered
│   ├── config.py                # Dev/Prod configs, SESSION_COOKIE_SECURE in prod
│   ├── extensions.py            # db, login_manager, migrate, csrf
│   │
│   ├── models/
│   │   ├── __init__.py          # Imports all models (required for Flask-Migrate)
│   │   ├── user.py              # UserMixin, set/check password, load_user hook
│   │   ├── account.py           # account_type enum, color, icon
│   │   ├── category.py          # Self-referential (subcategories), is_system flag
│   │   ├── transaction.py       # Dual FK to accounts (account_id + to_account_id)
│   │   ├── receipt.py           # filename, original_filename, mime_type
│   │   ├── recurring_rule.py    # frequency enum, next_run date
│   │   ├── budget.py            # UniqueConstraint(category_id, month)
│   │   ├── goal.py              # progress_percent property
│   │   ├── investment.py        # total_cost/current_value/unrealized_gain properties
│   │   ├── net_worth_snapshot.py # account_balances JSON field
│   │   ├── ai_insight.py        # insight_type enum
│   │   └── fx_rate.py           # date UNIQUE index
│   │
│   ├── routes/
│   │   ├── auth.py              # /auth/login, /auth/logout
│   │   ├── dashboard.py         # /, /api/fx-history, /api/fx-refresh (POST)
│   │   ├── accounts.py          # /accounts/
│   │   ├── categories.py        # /categories/
│   │   ├── transactions.py      # /transactions/, /transactions/ocr (POST),
│   │   │                        # /transactions/ocr-file (POST)
│   │   ├── budgets.py           # /budgets/, /budgets/copy (POST)
│   │   ├── goals.py             # /goals/, contribute, contributions, delete_contribution
│   │   ├── investments.py       # /investments/, detail, add_transaction,
│   │   │                        # refresh-prices, /api/price/<ticker>
│   │   ├── reports.py           # /reports/monthly|quarterly|yearly|tax
│   │   │                        # /reports/export/csv|excel|pdf
│   │   ├── ai.py                # /ai/, /ai/stream (SSE), /ai/history,
│   │   │                        # /ai/generate-insight (POST)
│   │   └── settings.py          # /settings/, profile, password, recurring,
│   │                            # import, upload_receipt, delete_receipt, view_receipt
│   │
│   ├── services/
│   │   ├── account_service.py   # calc_balance(), recalc_all(), get_total_assets/liabilities()
│   │   ├── ai_service.py        # build_context(), stream_chat() SSE gen, generate_daily_insight()
│   │   ├── budget_service.py    # get_budget_summary(), apply_rollovers()
│   │   ├── export_service.py    # transactions_to_csv/excel(), report_to_pdf(), build_report_html()
│   │   ├── fx_service.py        # get_today_rate(), force_refresh(), _fetch_yfinance(),
│   │   │                        # _fetch_er_api(), get_rate_history()
│   │   ├── goal_service.py      # get_projected_completion(), get_emergency_fund_status()
│   │   ├── import_service.py    # parse_csv(), import_rows(), duplicate detection
│   │   ├── investment_service.py # fetch_price(), update_prices(), get_portfolio_summary()
│   │   ├── ocr_service.py       # extract_from_file(), extract_from_bytes(),
│   │   │                        # Groq vision model, JSON parse + sanitise
│   │   ├── recurring_service.py # process_due_rules(), next_occurrence(), get_upcoming()
│   │   └── report_service.py    # monthly/quarterly/yearly/tax reports,
│   │                            # net_worth_history(), category_trends(), take_net_worth_snapshot()
│   │
│   ├── templates/
│   │   ├── base.html            # Collapsible sidebar, topbar, flash messages,
│   │   │                        # @keyframes spin, Bootstrap 5 + Bootstrap Icons
│   │   ├── auth/login.html      # Dark themed, password toggle
│   │   ├── dashboard/index.html # All widgets, FX refresh JS, SSE-compatible
│   │   ├── accounts/            # index.html, form.html (color/icon picker)
│   │   ├── categories/          # index.html, form.html
│   │   ├── transactions/        # index.html (tabs+filter+pagination)
│   │   │                        # form.html (OCR panel + drag-drop + field flash)
│   │   │                        # transfer.html
│   │   ├── budgets/             # index.html (progress bars), form.html
│   │   ├── goals/               # index.html (emergency fund + cards), form.html,
│   │   │                        # contribute.html, contributions.html
│   │   ├── investments/         # index.html (doughnut chart), detail.html,
│   │   │                        # form.html (ticker check), transaction_form.html
│   │   ├── reports/             # index.html (4 chart types), tax.html
│   │   ├── ai/                  # index.html (SSE chat + suggestions), history.html
│   │   └── settings/            # index.html, profile.html, password.html,
│   │                            # recurring.html, recurring_form.html, import.html
│   │
│   ├── static/
│   │   ├── css/                 # (empty — all CSS inline in templates)
│   │   ├── js/                  # (empty — all JS inline in templates)
│   │   └── img/
│   │
│   └── utils/
│       ├── formatters.py        # format_currency(), format_percent(), format_large_number()
│       └── decorators.py        # login_required_custom (unused — Flask-Login handles it)
│
├── migrations/                  # Flask-Migrate / Alembic
│
├── scripts/                     # All cron scripts — sys.path fix at top of each
│   ├── init_db.py               # Seeds 21 default categories, creates admin user interactively
│   ├── process_recurring.py     # Processes due recurring rules, creates transactions
│   ├── fetch_fx_rate.py         # force_refresh() — always fetches fresh USD/VND
│   ├── fetch_prices.py          # yfinance price update for all investment tickers
│   ├── daily_snapshot.py        # Saves net worth snapshot (1st of month)
│   └── daily_ai_insight.py      # Generates daily AI summary via Groq
│
└── tests/
```

---

## 5. Python Dependencies (`requirements.txt`)

```
flask==3.1.0
flask-sqlalchemy==3.1.1
flask-login==0.6.3
flask-migrate==4.1.0
flask-wtf==1.2.2
pymysql==1.1.1
python-dotenv==1.0.1
gunicorn==23.0.0
groq==0.13.1
yfinance==0.2.54
weasyprint==63.1
openpyxl==3.1.5
Pillow==11.1.0
apscheduler==3.10.4
requests==2.32.3
cryptography==44.0.2
python-dateutil==2.9.0
```

---

## 6. AI Integration — Groq API

### Chat + Daily Insights
- Model: `llama-3.3-70b-versatile` (default) / `llama-3.1-8b-instant` (fast)
- Context: last 90 days transactions, budget status, goals, investments (anonymised)
- SSE streaming: `stream_chat()` yields `data: <chunk>\n\n`
- Daily insight: non-streaming, stored in `ai_insights` table, max 400 tokens

### Receipt OCR (Vision)
- Model: `meta-llama/llama-4-scout-17b-16e-instruct`
- Input: base64-encoded image (JPEG/PNG/GIF/WEBP)
- Prompt: structured JSON extraction (amount, date, merchant, category, notes)
- Temperature: 0.1 (low, for consistent output)
- Post-processing: strips markdown fences, extracts JSON via regex fallback,
  normalises amount/date, maps category to system names
- Endpoints: `POST /transactions/ocr` (upload bytes), `POST /transactions/ocr-file` (stored file)

### Free Tier Limits
| Metric | Limit |
|--------|-------|
| Requests/day | 14,400 |
| Tokens/minute | 500,000 |
| Cost | Free |

---

## 7. USD → VND Exchange Rate

> **Reference widget only.** All transactions use the single configured app currency.

### Fetch Priority
1. DB cache — same day record in `fx_rates`
2. **yfinance** `USDVND=X` forex ticker (primary — most reliable)
3. `open.er-api.com` free REST API (fallback)
4. Last known DB record (stale fallback, shows ⚠ indicator)

### Sanity Check
Rate must be `> 1000` — rejects garbage values (e.g. 1.0, 0.0) that some APIs return.

### Dashboard Widget
- Shows rate, date, source
- ↻ button → `POST /api/fx-refresh` → updates rate in-place without page reload
- Click widget → toggles 30-day Chart.js line chart
- `force_refresh()` used by daily cron — always bypasses cache

---

## 8. UI/UX

- **Sidebar**: collapsible (desktop state saved in localStorage), mobile overlay
- **Charts**: Chart.js 4.x (CDN)
- **Forms**: WTForms + Bootstrap 5.3
- **Icons**: Bootstrap Icons 1.11
- **Fonts**: DM Sans + DM Mono (Google Fonts CDN)
- **Color scheme**: `#0f172a` sidebar, `#f1f5f9` body, `#10b981` income, `#ef4444` expense, `#3b82f6` invest
- **CSS**: All inline in templates (no build step)
- **SSE**: used for AI chat stream + FX refresh

---

## 9. Authentication

- Single-user, Flask-Login, session-based
- Hashed password (Werkzeug `generate_password_hash`)
- `SESSION_COOKIE_SECURE=True` in production
- `SESSION_COOKIE_HTTPONLY=True`, `SESSION_COOKIE_SAMESITE='Lax'`
- CSRF protection on all forms (Flask-WTF)

---

## 10. Scheduled Jobs

| Job | Schedule | Script | Notes |
|-----|----------|--------|-------|
| Process recurring transactions | Daily 6AM | `process_recurring.py` | Creates missed occurrences |
| Fetch USD/VND rate | Daily 8AM | `fetch_fx_rate.py` | force_refresh(), yfinance primary |
| Fetch investment prices | Mon-Fri 4PM | `fetch_prices.py` | yfinance, all tickers |
| Net worth snapshot | 1st of month 00:05 | `daily_snapshot.py` | Saves to net_worth_snapshots |
| AI daily insight | Daily 00:01 | `daily_ai_insight.py` | Skips if already done today |
| DB backup | Daily 2AM | pfm-backup (systemd) | mysqldump → gzip, keep 30 days |

---

## 11. Environment Variables (`.env`)

```
SECRET_KEY=your-secret-key
DATABASE_URL=mysql+pymysql://pfm_user:password@localhost/pfm_db
GROQ_API_KEY=your-groq-api-key-here
GROQ_MODEL=llama-3.3-70b-versatile
UPLOAD_FOLDER=/home/pfm/web/uploads
MAX_CONTENT_LENGTH=10485760
FLASK_ENV=production
FLASK_APP=wsgi:app
APP_CURRENCY=USD
APP_CURRENCY_SYMBOL=$
APP_TIMEZONE=Asia/Ho_Chi_Minh
```

---

## 12. Blueprints Registered (11 total)

| Blueprint | Prefix | Key routes |
|-----------|--------|------------|
| auth | /auth | login, logout |
| dashboard | / | index, api/fx-history, api/fx-refresh |
| accounts | /accounts | CRUD |
| categories | /categories | CRUD |
| transactions | /transactions | index, new, edit, delete, transfer, ocr, ocr-file |
| budgets | /budgets | index, new, edit, delete, copy |
| goals | /goals | index, new, edit, delete, contribute, contributions |
| investments | /investments | index, new, detail, edit, delete, add_transaction, refresh-prices, api/price |
| reports | /reports | monthly, quarterly, yearly, tax, export/csv|excel|pdf, snapshot |
| ai | /ai | index, stream (SSE), history, generate-insight |
| settings | /settings | index, profile, password, recurring, import, upload_receipt, delete_receipt, view_receipt |

---

## 13. Security Notes

- All routes `@login_required`
- CSRF on all POST forms
- SQLAlchemy ORM (no raw SQL)
- Receipt file path: `os.path.basename()` prevents path traversal
- HTTPS via Let's Encrypt (Certbot) — `pfm.ngodanguyen.tech`
- Groq receives anonymised transaction summaries (no account/personal names)
- `GROQ_API_KEY` in `.env` (chmod 600), never in frontend

---

## 14. Known Issues / Notes

- `wsgi.py` has `sys.path.insert(0, ...)` fix — required because app deploys at `/home/pfm/web/` which would otherwise be treated as a Python package
- FX rate widget: `open.er-api.com` may return stale values; yfinance is the reliable primary
- WeasyPrint PDF: requires `libpango*` system libs (included in deploy.md apt install)
- Import preview uses Flask session to pass rows to confirm step — requires `SECRET_KEY` to be set

---

## 15. Post-MVP Roadmap

- [ ] iOS companion app
- [ ] Bank statement PDF auto-import (parse PDF → extract transactions)
- [ ] Budget alerts + Twilio SMS/email notifications
- [ ] Shared household mode (2 users, row-level isolation)
- [ ] Mobile responsiveness pass