nngo/Personal-Finance-Management
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
4e47eddf70842d38769c2100483dc5a8d602ac52
Personal-Finance-Management/CLAUDE.md
T

23 KiB
Raw Blame History

Personal Finance Management System (PFM)

Stack: Python Flask · MySQL · Ubuntu Server · Nginx · Gunicorn · Groq API (free AI) App URL: https://pfm.ngodanguyen.tech Code: /home/pfm/web · User: pfm · Gitea: gitea.ngodanguyen.tech

1. Project Overview

Self-hosted personal finance web app. Tracks income, expenses, investments. AI assistant powered by Groq API (free tier, fast inference, no local hardware). Receipt OCR via Groq vision model. Bank account sync via Teller API (mTLS). Bank statement import (CSV, OFX/QFX, PDF). Everything runs on Ubuntu server behind Nginx + Certbot SSL.

Status: All 7 phases complete + all post-MVP features implemented.

2. Core Features (Implemented)

2.1 Dashboard

  • Net worth snapshot (assets − liabilities)
  • Monthly cash flow bar chart (6 months)
  • Budget utilization per category
  • Recent transactions feed (last 8)
  • AI daily insight card (Groq-generated, stored in DB)
  • USD → VND exchange rate widget — reference only, independent of app currency
    • Click to expand 30-day history chart
    • ↻ refresh button (force-fetches fresh rate without page reload)
    • Source label shown (yfinance / exchangerate-api)
    • Stale indicator if rate > 1 day old
  • Period selector: This Month / Last Month / Custom date range

2.2 Transactions

  • Income + Expense entry with Income/Expense tabs
  • Transfer between accounts
  • Filter: search, category, account, date range (safe int parsing — no crash on bad params)
  • Pagination (30/page)
  • Receipt upload (PNG/JPG/WEBP/GIF/PDF, max 10MB)
  • AI Receipt OCR — drag-drop receipt image → Groq vision extracts amount/date/merchant/category → auto-fills form
  • Re-extract from already-uploaded receipt (edit mode)
  • Export to CSV / Excel
  • Edit form: receipt sub-forms are outside #txnForm to prevent nested-form bug

2.3 Accounts

  • Types: checking, savings, cash, credit_card, crypto, investment, other
  • Balance auto-calculated from all transactions (not manually entered)
  • Color + icon picker
  • Soft delete

2.4 Categories

  • Expense + Income categories with color/icon
  • System categories (protected from delete)
  • Custom categories (user-created)
  • 14 expense + 7 income defaults seeded on init

2.5 Budget Planner

  • Monthly limits per expense category
  • Progress bars: green → amber (80%) → red (100%+)
  • Rollover unused budget to next month (toggle)
  • Copy previous month's budgets in one click
  • Unbudgeted spending shown with "Set Budget" prompt

2.6 Goals & Savings

  • Goals with target amount, target date, color, icon
  • Contribution tracking + history
  • Progress bar + projected completion date (based on avg monthly contrib)
  • Auto-complete on 100%
  • Emergency fund tracker (liquid assets vs 3-month / 6-month expense targets)

2.7 Investments

  • Asset types: stock, ETF, crypto, real_estate, bond, cash, other
  • Buy/sell/dividend/split transaction log
  • FIFO cost basis auto-recalculated from transaction log
  • yfinance price auto-fetch (daily 4PM weekdays via cron)
  • Manual price refresh button (portfolio page)
  • Live ticker check on add form
  • Doughnut allocation chart
  • P&L per holding + portfolio total

2.8 AI Financial Assistant

  • Chat UI with SSE streaming (Groq API, word-by-word response)
  • Context: last 90 days transactions + budget status + goals + investments
  • 8 suggested question buttons
  • Daily auto-insight generated at midnight (stored in ai_insights table)
  • Manual "Generate Now" button
  • Chat history page
  • Model: llama-3.3-70b-versatile (default) or llama-3.1-8b-instant (fast)
  • Fallback messages for rate limit / invalid key / unavailable

2.9 Receipt OCR (Groq Vision)

  • Model: meta-llama/llama-4-scout-17b-16e-instruct
  • Drag-drop or click-to-upload on new expense form
  • Extracts: amount, date, merchant name, category suggestion, notes
  • Maps category suggestion → system category ID
  • Auto-fills form fields with green flash animation
  • Re-extract button on existing receipt (edit mode)
  • Auto-triggers OCR when image file selected in edit mode
  • Handles fenced markdown JSON output from LLM
  • Sanity check: rejects amounts < 1000 (catches garbage values)
  • Full error handling: 400/401/429/timeout/bad JSON

2.10 Reports & Export

  • Monthly / Quarterly / Yearly summary reports
  • Tax year summary (all income by source, all expenses by category)
  • Net worth history line chart (from monthly snapshots)
  • Category spending trends (top 6 categories, 6-month line chart)
  • "Snapshot Now" manual button
  • Export: CSV, Excel (color-coded, formatted), PDF (WeasyPrint)

2.11 Settings

  • Profile: name, email, timezone, currency, Groq model
  • 8 currency options (USD/VND/EUR/GBP/JPY/AUD/CAD/SGD) — auto-updates symbol
  • Password change (requires current password)
  • Recurring rules: CRUD, pause/enable, frequency (daily/weekly/biweekly/monthly/quarterly/yearly)
  • "Run Now" button to process due rules immediately
  • CSV import: upload → preview with ⚠ warnings → confirm
  • Upcoming recurring transactions (30-day view)

2.12 USD → VND Exchange Rate

  • Reference widget only — not used in transaction calculations
  • Primary source: yfinance USDVND=X (Yahoo Finance forex)
  • Fallback: open.er-api.com (free, no key)
  • Sanity check: rate must be > 1000 (rejects garbage values)
  • force_refresh() — always fetches fresh on cron, bypasses cache
  • DB caches one record per day (fx_rates table)
  • Dashboard: shows rate, date, source, stale warning, ↻ refresh button
  • 30-day history chart (click widget to expand)

2.13 Teller Bank Sync

  • Connects US bank accounts via Teller API (mTLS + HTTP Basic Auth)
  • Enrollment via Teller Connect modal (JavaScript widget)
  • Account mapping: each Teller account → PFM account (or auto-create new)
    • Account ID validated against DB before saving (security fix)
  • Transaction sync: preview → confirm → import
  • Duplicate detection using Teller transaction ID (stored as Teller:<id> in notes)
  • Balance refresh (live from Teller API)
  • Webhook: transactions.processed event with HMAC-SHA256 signature + 5-min replay protection
  • Disconnect enrollment
  • Config: TELLER_APP_ID, TELLER_ENV, TELLER_CERT_PATH, TELLER_KEY_PATH, TELLER_WEBHOOK_SECRET
  • Models: teller_enrollments, teller_accounts (2 new tables)

2.14 Bank Statement Import

  • Sidebar link: "Import Statement" under Money section
  • Supported formats:
    • PDF — pdfplumber text extraction + Groq LLM parsing (digital PDFs only; not scanned)
    • OFX / QFX — both SGML and XML variants; handles all TRNTYPE codes
    • Chase CSV — Transaction Date, Description, Amount
    • Bank of America CSV — Posted Date, Payee, Amount
    • Citi CSV — Date, Description, Debit, Credit
    • Capital One CSV — Transaction Date, Description, Debit, Credit
    • Discover CSV — Trans. Date, Description, Amount (positive = expense)
    • Amex CSV — Date, Description, Amount
    • USAA CSV — Date, Description, Original Description, Amount
    • Wells Fargo CSV — Date, Amount, Description
    • Generic CSV — heuristic column detection
    • Custom mapping — UI to map columns when auto-detect fails
  • Auto-categorizes using 200+ keyword rules across 14 categories
  • Preview table: per-row checkboxes, editable category dropdowns
  • Duplicate detection: OFX FITID (stored as import:<id> in notes) or date+amount+description+account
  • PDF notes: 30 K char limit per upload; scanned PDFs rejected with clear error
  • AJAX-based: no page reloads, no session storage for rows
  • File input lives outside drop zone (prevents overlay-blocking other controls)

2.15 System Logs Viewer

  • Sidebar link: "System Logs" (bi-terminal) in footer section
  • Log file: logs/app.log (rotating, 10 MB, 5 backups)
  • Format: YYYY-MM-DD HH:MM:SS|LEVEL|module.name|message (pipe-delimited for parsing)
  • Viewer at /logs/:
    • Colour-coded level pills: ERROR / WARNING / INFO / DEBUG
    • Free-text search + module filter + row limit (100/200/500/1000)
    • Auto-refresh every 5 s (toggle with green pulse indicator)
    • Clear log file button (POST with CSRF)
    • Download raw log file
  • Logging wired to Gunicorn via gunicorn.error handlers; also writes to stderr
  • app.* namespace loggers all inherit from logging.getLogger('app') at INFO level
  • All Teller API calls log: status, URL, and response body on error

3. Database Schema (MySQL)

All 16 Tables

users                   — single user, hashed password, currency/timezone prefs
accounts                — bank/wallet accounts (balance auto-calc from txns)
categories              — expense/income categories with color/icon
transactions            — income/expense/transfer, receipt_id, recurring_rule_id
receipts                — receipt file metadata (filename, size, mime_type)
recurring_rules         — templates: frequency, next_run, start/end date
budgets                 — monthly limits per category, rollover support
goals                   — savings goals with target amount/date
goal_contributions      — individual deposits toward each goal
investments             — holdings: ticker, shares, avg_cost_basis, current_price
investment_transactions — buy/sell/dividend/split log
net_worth_snapshots     — monthly snapshots: assets, liabilities, net_worth (JSON)
ai_insights             — stored AI responses: daily_summary / chat_response
fx_rates                — daily USD/VND rate cache (date UNIQUE, source)
teller_enrollments      — Teller enrollment: enrollment_id, access_token, institution_name
teller_accounts         — Teller account ↔ PFM account mapping, last_sync_date

Key Column Notes

  • transactions.balance — NOT stored; calculated on-demand via account_service.calc_balance()
  • investments.shares / avg_cost_basis — recalculated from investment_transactions (FIFO)
  • goals.current_amount — updated on each contribution add/delete
  • net_worth_snapshots.account_balances — JSON snapshot of each account balance at time of snapshot
  • transactions.notes — used to store import source IDs: Teller:<id> or import:<fitid>

4. Project File Structure (Actual)

pfm/                             # /home/pfm/web on server
├── wsgi.py                      # Gunicorn entry — sys.path fix included
├── requirements.txt
├── .env                         # Not committed
├── .env.example
├── .gitignore
├── logs/
│   └── app.log                  # Rotating application log (created at runtime)
│
├── app/
│   ├── __init__.py              # Flask app factory; _setup_logging(); all blueprints registered
│   ├── config.py                # Dev/Prod configs; LOG_FILE_PATH; TELLER_* vars
│   ├── extensions.py            # db, login_manager, migrate, csrf
│   │
│   ├── models/
│   │   ├── __init__.py
│   │   ├── user.py
│   │   ├── account.py
│   │   ├── category.py
│   │   ├── transaction.py
│   │   ├── receipt.py
│   │   ├── recurring_rule.py
│   │   ├── budget.py
│   │   ├── goal.py
│   │   ├── investment.py
│   │   ├── net_worth_snapshot.py
│   │   ├── ai_insight.py
│   │   ├── fx_rate.py
│   │   └── teller_enrollment.py # TellerEnrollment + TellerAccount models
│   │
│   ├── routes/
│   │   ├── auth.py
│   │   ├── dashboard.py
│   │   ├── accounts.py
│   │   ├── categories.py
│   │   ├── transactions.py      # /transactions/ocr, /ocr-file; safe filter int parsing
│   │   ├── budgets.py
│   │   ├── goals.py
│   │   ├── investments.py
│   │   ├── reports.py
│   │   ├── ai.py
│   │   ├── settings.py          # view_receipt: os.path.basename() path-traversal fix
│   │   ├── teller.py            # Teller sync, webhook, account mapping (validated IDs)
│   │   ├── bank_import.py       # /bank-import/; parse (AJAX); import (AJAX)
│   │   └── logs.py              # /logs/; /logs/api; /logs/clear; /logs/download
│   │
│   ├── services/
│   │   ├── account_service.py
│   │   ├── ai_service.py
│   │   ├── budget_service.py
│   │   ├── export_service.py
│   │   ├── fx_service.py
│   │   ├── goal_service.py
│   │   ├── import_service.py    # duplicate check now scoped by account_id
│   │   ├── investment_service.py
│   │   ├── ocr_service.py
│   │   ├── recurring_service.py # 90-day catchup cap prevents runaway loops
│   │   ├── report_service.py
│   │   ├── teller_service.py    # mTLS session; full response-body logging on errors
│   │   └── bank_import_service.py # PDF+OFX+CSV parsing; Groq PDF parsing; auto-categorize
│   │
│   ├── templates/
│   │   ├── base.html            # Sidebar (all active states); csrf-token meta tag
│   │   ├── auth/login.html
│   │   ├── dashboard/index.html
│   │   ├── accounts/
│   │   ├── categories/
│   │   ├── transactions/        # form.html: receipt forms outside #txnForm (nested-form fix)
│   │   ├── budgets/
│   │   ├── goals/
│   │   ├── investments/
│   │   ├── reports/
│   │   ├── ai/
│   │   ├── settings/
│   │   ├── teller/              # index.html, map_accounts.html, preview.html
│   │   ├── bank_import/         # index.html (drag-drop; AJAX parse+import; column mapper)
│   │   └── logs/                # index.html (live viewer; filters; auto-refresh)
│   │
│   ├── static/
│   │   ├── css/                 # (empty — all CSS inline in templates)
│   │   ├── js/                  # (empty — all JS inline in templates)
│   │   └── img/
│   │
│   └── utils/
│       ├── formatters.py
│       └── decorators.py
│
├── migrations/
│
├── scripts/
│   ├── init_db.py
│   ├── process_recurring.py
│   ├── fetch_fx_rate.py
│   ├── fetch_prices.py
│   ├── daily_snapshot.py
│   └── daily_ai_insight.py
│
└── tests/

5. Python Dependencies (requirements.txt)

flask==3.1.0
flask-sqlalchemy==3.1.1
flask-login==0.6.3
flask-migrate==4.1.0
flask-wtf==1.2.2
pymysql==1.1.1
python-dotenv==1.0.1
gunicorn==23.0.0
groq==0.13.1
weasyprint==63.1
openpyxl==3.1.5
Pillow==11.1.0
apscheduler==3.10.4
requests==2.32.3
cryptography==44.0.2
python-dateutil==2.9.0
pdfplumber==0.11.4          # PDF text extraction for bank statement import

6. AI Integration — Groq API

Chat + Daily Insights

  • Model: llama-3.3-70b-versatile (default) / llama-3.1-8b-instant (fast)
  • Context: last 90 days transactions, budget status, goals, investments (anonymised)
  • SSE streaming: stream_chat() yields data: <chunk>\n\n
  • Daily insight: non-streaming, stored in ai_insights table, max 400 tokens

Receipt OCR (Vision)

  • Model: meta-llama/llama-4-scout-17b-16e-instruct
  • Input: base64-encoded image (JPEG/PNG/GIF/WEBP)
  • Prompt: structured JSON extraction (amount, date, merchant, category, notes)
  • Temperature: 0.1

Bank Statement PDF Parsing

  • Text extracted by pdfplumber then sent to llama-3.3-70b-versatile
  • Prompt requests JSON array of {date, description, amount, transaction_type}
  • Max 30 K chars sent per request (~6 months of typical statements)
  • Scanned PDFs (no text layer) are rejected with a clear error message

Free Tier Limits

Metric Limit
Requests/day 14,400
Tokens/minute 500,000
Cost Free

7. Teller Bank Sync

  • mTLS: client cert + key from TELLER_CERT_PATH / TELLER_KEY_PATH
  • HTTP Basic Auth: access_token as username, empty password
  • Endpoints: GET /accounts, GET /accounts/:id/balances, GET /accounts/:id/transactions
  • All API errors logged with status code + full response body
  • Webhook: HMAC-SHA256 Teller-Signature header; 5-minute replay window

8. Bank Statement Import

  • Route: /bank-import/ (blueprint bank_import_bp)
  • Parse: POST /bank-import/parse (AJAX, multipart with X-CSRFToken header)
  • Import: POST /bank-import/import (AJAX, JSON with X-CSRFToken header)
  • File input is hidden and outside the drop zone (fileInput.click() on drop zone click)
  • Duplicate detection:
    • OFX: match on import:<FITID> in notes
    • CSV/PDF: match on date + amount + type + description scoped to same account_id

9. Logging System

  • Config key: LOG_FILE_PATH (default: <project-root>/logs/app.log)
  • Handler: RotatingFileHandler — 10 MB per file, 5 backups
  • Format: YYYY-MM-DD HH:MM:SS|LEVEL|module.name|message
  • Namespace: logging.getLogger('app') at INFO; propagate=False
  • Also writes to stderr (Gunicorn captures it)
  • Viewer: /logs/ — real-time filtered display, per-level counts, auto-refresh, clear, download

10. UI/UX

  • Sidebar: collapsible (desktop state saved in localStorage), mobile overlay
    • Active states: {% if request.blueprint == '...' %}active{% endif %}
    • Links: Dashboard · Transactions · Add Income · Add Expense · Accounts · Import Statement · Budgets · Goals · Investments · Reports · AI Assistant · Categories · System Logs · Settings · Logout
  • Charts: Chart.js 4.x (CDN)
  • Forms: WTForms + Bootstrap 5.3
  • Icons: Bootstrap Icons 1.11
  • Fonts: DM Sans + DM Mono (Google Fonts CDN)
  • Color scheme: #0f172a sidebar, #f1f5f9 body, #10b981 income, #ef4444 expense, #3b82f6 invest
  • CSS: All inline in templates (no build step)
  • SSE: AI chat stream + FX refresh
  • CSRF meta tag: <meta name="csrf-token"> in base.html for JS fetch calls

11. Authentication & Security

  • Single-user, Flask-Login, session-based
  • Hashed password (Werkzeug generate_password_hash)
  • SESSION_COOKIE_SECURE=True in production
  • SESSION_COOKIE_HTTPONLY=True, SESSION_COOKIE_SAMESITE='Lax'
  • CSRF protection on all forms (Flask-WTF); meta tag in base.html for AJAX
  • SQLAlchemy ORM (no raw SQL)
  • Receipt file path: os.path.basename() in both upload AND view_receipt (path-traversal fix)
  • Teller account IDs validated against DB before mapping
  • Transaction filter params safely cast with try/except (no crash on bad int input)
  • Groq receives anonymised transaction summaries (no account/personal names)

12. Scheduled Jobs

Job Schedule Script Notes
Process recurring transactions Daily 6AM process_recurring.py 90-day catchup cap
Fetch USD/VND rate Daily 8AM fetch_fx_rate.py force_refresh(), yfinance primary
Fetch investment prices Mon-Fri 4PM fetch_prices.py yfinance, all tickers
Net worth snapshot 1st of month 00:05 daily_snapshot.py Saves to net_worth_snapshots
AI daily insight Daily 00:01 daily_ai_insight.py Skips if already done today
DB backup Daily 2AM pfm-backup (systemd) mysqldump → gzip, keep 30 days

13. Environment Variables (.env)

SECRET_KEY=your-secret-key
DATABASE_URL=mysql+pymysql://pfm_user:password@localhost/pfm_db
GROQ_API_KEY=your-groq-api-key-here
GROQ_MODEL=llama-3.3-70b-versatile
UPLOAD_FOLDER=/home/pfm/web/uploads
MAX_CONTENT_LENGTH=10485760
FLASK_ENV=production
FLASK_APP=wsgi:app
APP_CURRENCY=USD
APP_CURRENCY_SYMBOL=$
APP_TIMEZONE=Asia/Ho_Chi_Minh
LOG_FILE_PATH=/home/pfm/web/logs/app.log
# Teller
TELLER_APP_ID=your-teller-app-id
TELLER_ENV=development
TELLER_CERT_PATH=/home/pfm/teller/certificate.pem
TELLER_KEY_PATH=/home/pfm/teller/private_key.pem
TELLER_WEBHOOK_SECRET=your-webhook-secret

14. Blueprints Registered (13 total)

Blueprint Prefix Key routes
auth /auth login, logout
dashboard / index, api/fx-history, api/fx-refresh
accounts /accounts CRUD
categories /categories CRUD
transactions /transactions index, new, edit, delete, transfer, ocr, ocr-file
budgets /budgets index, new, edit, delete, copy
goals /goals index, new, edit, delete, contribute, contributions
investments /investments index, new, detail, edit, delete, add_transaction, refresh-prices, api/price
reports /reports monthly, quarterly, yearly, tax, export/csv|excel|pdf, snapshot
ai /ai index, stream (SSE), history, generate-insight
settings /settings index, profile, password, recurring, import, upload_receipt, delete_receipt, view_receipt
teller /teller callback, map, index, sync, sync/confirm, sync/all, balance, disconnect, webhook
bank_import /bank-import index, parse (AJAX), import (AJAX)
logs /logs index, api (AJAX), clear (AJAX), download

15. Known Issues / Notes

  • wsgi.py has sys.path.insert(0, ...) — required for Gunicorn at /home/pfm/web/
  • FX rate widget: open.er-api.com may return stale values; yfinance is the reliable primary
  • WeasyPrint PDF: requires libpango* system libs on server
  • Bank statement PDF import: scanned/image PDFs have no text layer; must use digital download
  • Bank statement PDF import: large PDFs (>30 K chars) are truncated; split into shorter date ranges
  • pdfplumber must be installed: pip install pdfplumber==0.11.4
  • Teller: development environment only; requires cert/key from Teller Dashboard

16. Security Fixes Applied (session log)

Date Fix File
2026-06 Path traversal in view_receipt — added os.path.basename() settings.py
2026-06 int() crash on bad filter params in transactions index transactions.py
2026-06 Teller account mapping validates ID exists in DB teller.py
2026-06 Recurring catchup capped at 90 days (prevents runaway loops) recurring_service.py
2026-06 CSV/bank import duplicate check scoped by account_id import_service.py
2026-06 CSRF token added to bank import AJAX parse request bank_import/index.html
2026-06 Receipt sub-forms moved outside #txnForm (nested-form bug) transactions/form.html
2026-06 Drop zone file input moved outside overlay (blocked account select) bank_import/index.html

17. To-Do / Roadmap

High priority

  • Mobile responsiveness pass — sidebar auto-collapses on mobile; tables scroll horizontally
  • Empty-state messages — transactions, budgets, goals, investments pages when no data
  • Budget alerts — email/Twilio SMS when category spending hits 80% / 100%

Medium priority

  • Pagination info — show "Page X of Y" on AI history and other paginated pages
  • PDF export memory — stream CSV/Excel exports for users with large transaction history
  • Receipt MIME validation — validate file magic bytes server-side, not just extension
  • Teller multi-account sync — sync all mapped accounts in sequence (currently syncs first only)
  • OCR ownership check — verify re-extracted filename belongs to current user's transaction
  • Bank import progress — show per-row import progress for large statement files

Low priority / future

  • iOS companion app
  • Shared household mode (2 users, row-level isolation)
  • Bank statement PDF: table-extraction fallback (pdfplumber tables API) before Groq call
  • Investment price history chart per holding
  • Dark mode toggle
Reference in New Issue View Git Blame Copy Permalink