|
|
|
@@ -7,9 +7,9 @@
|
|
|
|
|
|
|
|
|
|
## 1. Project Overview
|
|
|
|
|
|
|
|
|
|
Self-hosted personal finance web app. Tracks income, expenses, investments. AI assistant powered by **Groq API** (free tier, fast inference, no local hardware). Receipt OCR via Groq vision model. Everything runs on Ubuntu server behind Nginx + Certbot SSL.
|
|
|
|
|
Self-hosted personal finance web app. Tracks income, expenses, investments. AI assistant powered by **Groq API** (free tier, fast inference, no local hardware). Receipt OCR via Groq vision model. Bank account sync via Teller API (mTLS). Bank statement import (CSV, OFX/QFX, PDF). Everything runs on Ubuntu server behind Nginx + Certbot SSL.
|
|
|
|
|
|
|
|
|
|
**Status: All 7 phases complete + Receipt OCR post-MVP feature.**
|
|
|
|
|
**Status: All 7 phases complete + all post-MVP features implemented.**
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
@@ -31,12 +31,13 @@ Self-hosted personal finance web app. Tracks income, expenses, investments. AI a
|
|
|
|
|
### 2.2 Transactions
|
|
|
|
|
- Income + Expense entry with Income/Expense tabs
|
|
|
|
|
- Transfer between accounts
|
|
|
|
|
- Filter: search, category, account, date range
|
|
|
|
|
- Filter: search, category, account, date range (safe int parsing — no crash on bad params)
|
|
|
|
|
- Pagination (30/page)
|
|
|
|
|
- Receipt upload (PNG/JPG/WEBP/GIF/PDF, max 10MB)
|
|
|
|
|
- **AI Receipt OCR** — drag-drop receipt image → Groq vision extracts amount/date/merchant/category → auto-fills form
|
|
|
|
|
- Re-extract from already-uploaded receipt (edit mode)
|
|
|
|
|
- Export to CSV / Excel
|
|
|
|
|
- Edit form: receipt sub-forms are outside `#txnForm` to prevent nested-form bug
|
|
|
|
|
|
|
|
|
|
### 2.3 Accounts
|
|
|
|
|
- Types: checking, savings, cash, credit_card, crypto, investment, other
|
|
|
|
@@ -123,11 +124,60 @@ Self-hosted personal finance web app. Tracks income, expenses, investments. AI a
|
|
|
|
|
- Dashboard: shows rate, date, source, stale warning, ↻ refresh button
|
|
|
|
|
- 30-day history chart (click widget to expand)
|
|
|
|
|
|
|
|
|
|
### 2.13 Teller Bank Sync
|
|
|
|
|
- Connects US bank accounts via Teller API (mTLS + HTTP Basic Auth)
|
|
|
|
|
- Enrollment via Teller Connect modal (JavaScript widget)
|
|
|
|
|
- Account mapping: each Teller account → PFM account (or auto-create new)
|
|
|
|
|
- Account ID validated against DB before saving (security fix)
|
|
|
|
|
- Transaction sync: preview → confirm → import
|
|
|
|
|
- Duplicate detection using Teller transaction ID (stored as `Teller:<id>` in notes)
|
|
|
|
|
- Balance refresh (live from Teller API)
|
|
|
|
|
- Webhook: `transactions.processed` event with HMAC-SHA256 signature + 5-min replay protection
|
|
|
|
|
- Disconnect enrollment
|
|
|
|
|
- Config: `TELLER_APP_ID`, `TELLER_ENV`, `TELLER_CERT_PATH`, `TELLER_KEY_PATH`, `TELLER_WEBHOOK_SECRET`
|
|
|
|
|
- Models: `teller_enrollments`, `teller_accounts` (2 new tables)
|
|
|
|
|
|
|
|
|
|
### 2.14 Bank Statement Import
|
|
|
|
|
- Sidebar link: "Import Statement" under Money section
|
|
|
|
|
- Supported formats:
|
|
|
|
|
- **PDF** — pdfplumber text extraction + Groq LLM parsing (digital PDFs only; not scanned)
|
|
|
|
|
- **OFX / QFX** — both SGML and XML variants; handles all TRNTYPE codes
|
|
|
|
|
- **Chase** CSV — `Transaction Date, Description, Amount`
|
|
|
|
|
- **Bank of America** CSV — `Posted Date, Payee, Amount`
|
|
|
|
|
- **Citi** CSV — `Date, Description, Debit, Credit`
|
|
|
|
|
- **Capital One** CSV — `Transaction Date, Description, Debit, Credit`
|
|
|
|
|
- **Discover** CSV — `Trans. Date, Description, Amount` (positive = expense)
|
|
|
|
|
- **Amex** CSV — `Date, Description, Amount`
|
|
|
|
|
- **USAA** CSV — `Date, Description, Original Description, Amount`
|
|
|
|
|
- **Wells Fargo** CSV — `Date, Amount, Description`
|
|
|
|
|
- **Generic CSV** — heuristic column detection
|
|
|
|
|
- **Custom mapping** — UI to map columns when auto-detect fails
|
|
|
|
|
- Auto-categorizes using 200+ keyword rules across 14 categories
|
|
|
|
|
- Preview table: per-row checkboxes, editable category dropdowns
|
|
|
|
|
- Duplicate detection: OFX FITID (stored as `import:<id>` in notes) or date+amount+description+account
|
|
|
|
|
- PDF notes: 30 K char limit per upload; scanned PDFs rejected with clear error
|
|
|
|
|
- AJAX-based: no page reloads, no session storage for rows
|
|
|
|
|
- File input lives outside drop zone (prevents overlay-blocking other controls)
|
|
|
|
|
|
|
|
|
|
### 2.15 System Logs Viewer
|
|
|
|
|
- Sidebar link: "System Logs" (`bi-terminal`) in footer section
|
|
|
|
|
- Log file: `logs/app.log` (rotating, 10 MB, 5 backups)
|
|
|
|
|
- Format: `YYYY-MM-DD HH:MM:SS|LEVEL|module.name|message` (pipe-delimited for parsing)
|
|
|
|
|
- Viewer at `/logs/`:
|
|
|
|
|
- Colour-coded level pills: ERROR / WARNING / INFO / DEBUG
|
|
|
|
|
- Free-text search + module filter + row limit (100/200/500/1000)
|
|
|
|
|
- Auto-refresh every 5 s (toggle with green pulse indicator)
|
|
|
|
|
- Clear log file button (POST with CSRF)
|
|
|
|
|
- Download raw log file
|
|
|
|
|
- Logging wired to Gunicorn via `gunicorn.error` handlers; also writes to stderr
|
|
|
|
|
- `app.*` namespace loggers all inherit from `logging.getLogger('app')` at INFO level
|
|
|
|
|
- All Teller API calls log: status, URL, and response body on error
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## 3. Database Schema (MySQL)
|
|
|
|
|
|
|
|
|
|
### All 14 Tables
|
|
|
|
|
### All 16 Tables
|
|
|
|
|
```
|
|
|
|
|
users — single user, hashed password, currency/timezone prefs
|
|
|
|
|
accounts — bank/wallet accounts (balance auto-calc from txns)
|
|
|
|
@@ -143,6 +193,8 @@ investment_transactions — buy/sell/dividend/split log
|
|
|
|
|
net_worth_snapshots — monthly snapshots: assets, liabilities, net_worth (JSON)
|
|
|
|
|
ai_insights — stored AI responses: daily_summary / chat_response
|
|
|
|
|
fx_rates — daily USD/VND rate cache (date UNIQUE, source)
|
|
|
|
|
teller_enrollments — Teller enrollment: enrollment_id, access_token, institution_name
|
|
|
|
|
teller_accounts — Teller account ↔ PFM account mapping, last_sync_date
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
### Key Column Notes
|
|
|
|
@@ -150,6 +202,7 @@ fx_rates — daily USD/VND rate cache (date UNIQUE, source)
|
|
|
|
|
- `investments.shares` / `avg_cost_basis` — recalculated from `investment_transactions` (FIFO)
|
|
|
|
|
- `goals.current_amount` — updated on each contribution add/delete
|
|
|
|
|
- `net_worth_snapshots.account_balances` — JSON snapshot of each account balance at time of snapshot
|
|
|
|
|
- `transactions.notes` — used to store import source IDs: `Teller:<id>` or `import:<fitid>`
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
@@ -162,80 +215,77 @@ pfm/ # /home/pfm/web on server
|
|
|
|
|
├── .env # Not committed
|
|
|
|
|
├── .env.example
|
|
|
|
|
├── .gitignore
|
|
|
|
|
├── logs/
|
|
|
|
|
│ └── app.log # Rotating application log (created at runtime)
|
|
|
|
|
│
|
|
|
|
|
├── app/
|
|
|
|
|
│ ├── __init__.py # Flask app factory, all blueprints registered
|
|
|
|
|
│ ├── config.py # Dev/Prod configs, SESSION_COOKIE_SECURE in prod
|
|
|
|
|
│ ├── __init__.py # Flask app factory; _setup_logging(); all blueprints registered
|
|
|
|
|
│ ├── config.py # Dev/Prod configs; LOG_FILE_PATH; TELLER_* vars
|
|
|
|
|
│ ├── extensions.py # db, login_manager, migrate, csrf
|
|
|
|
|
│ │
|
|
|
|
|
│ ├── models/
|
|
|
|
|
│ │ ├── __init__.py # Imports all models (required for Flask-Migrate)
|
|
|
|
|
│ │ ├── user.py # UserMixin, set/check password, load_user hook
|
|
|
|
|
│ │ ├── account.py # account_type enum, color, icon
|
|
|
|
|
│ │ ├── category.py # Self-referential (subcategories), is_system flag
|
|
|
|
|
│ │ ├── transaction.py # Dual FK to accounts (account_id + to_account_id)
|
|
|
|
|
│ │ ├── receipt.py # filename, original_filename, mime_type
|
|
|
|
|
│ │ ├── recurring_rule.py # frequency enum, next_run date
|
|
|
|
|
│ │ ├── budget.py # UniqueConstraint(category_id, month)
|
|
|
|
|
│ │ ├── goal.py # progress_percent property
|
|
|
|
|
│ │ ├── investment.py # total_cost/current_value/unrealized_gain properties
|
|
|
|
|
│ │ ├── net_worth_snapshot.py # account_balances JSON field
|
|
|
|
|
│ │ ├── ai_insight.py # insight_type enum
|
|
|
|
|
│ │ └── fx_rate.py # date UNIQUE index
|
|
|
|
|
│ │ ├── __init__.py
|
|
|
|
|
│ │ ├── user.py
|
|
|
|
|
│ │ ├── account.py
|
|
|
|
|
│ │ ├── category.py
|
|
|
|
|
│ │ ├── transaction.py
|
|
|
|
|
│ │ ├── receipt.py
|
|
|
|
|
│ │ ├── recurring_rule.py
|
|
|
|
|
│ │ ├── budget.py
|
|
|
|
|
│ │ ├── goal.py
|
|
|
|
|
│ │ ├── investment.py
|
|
|
|
|
│ │ ├── net_worth_snapshot.py
|
|
|
|
|
│ │ ├── ai_insight.py
|
|
|
|
|
│ │ ├── fx_rate.py
|
|
|
|
|
│ │ └── teller_enrollment.py # TellerEnrollment + TellerAccount models
|
|
|
|
|
│ │
|
|
|
|
|
│ ├── routes/
|
|
|
|
|
│ │ ├── auth.py # /auth/login, /auth/logout
|
|
|
|
|
│ │ ├── dashboard.py # /, /api/fx-history, /api/fx-refresh (POST)
|
|
|
|
|
│ │ ├── accounts.py # /accounts/
|
|
|
|
|
│ │ ├── categories.py # /categories/
|
|
|
|
|
│ │ ├── transactions.py # /transactions/, /transactions/ocr (POST),
|
|
|
|
|
│ │ │ # /transactions/ocr-file (POST)
|
|
|
|
|
│ │ ├── budgets.py # /budgets/, /budgets/copy (POST)
|
|
|
|
|
│ │ ├── goals.py # /goals/, contribute, contributions, delete_contribution
|
|
|
|
|
│ │ ├── investments.py # /investments/, detail, add_transaction,
|
|
|
|
|
│ │ │ # refresh-prices, /api/price/<ticker>
|
|
|
|
|
│ │ ├── reports.py # /reports/monthly|quarterly|yearly|tax
|
|
|
|
|
│ │ │ # /reports/export/csv|excel|pdf
|
|
|
|
|
│ │ ├── ai.py # /ai/, /ai/stream (SSE), /ai/history,
|
|
|
|
|
│ │ │ # /ai/generate-insight (POST)
|
|
|
|
|
│ │ └── settings.py # /settings/, profile, password, recurring,
|
|
|
|
|
│ │ # import, upload_receipt, delete_receipt, view_receipt
|
|
|
|
|
│ │ ├── auth.py
|
|
|
|
|
│ │ ├── dashboard.py
|
|
|
|
|
│ │ ├── accounts.py
|
|
|
|
|
│ │ ├── categories.py
|
|
|
|
|
│ │ ├── transactions.py # /transactions/ocr, /ocr-file; safe filter int parsing
|
|
|
|
|
│ │ ├── budgets.py
|
|
|
|
|
│ │ ├── goals.py
|
|
|
|
|
│ │ ├── investments.py
|
|
|
|
|
│ │ ├── reports.py
|
|
|
|
|
│ │ ├── ai.py
|
|
|
|
|
│ │ ├── settings.py # view_receipt: os.path.basename() path-traversal fix
|
|
|
|
|
│ │ ├── teller.py # Teller sync, webhook, account mapping (validated IDs)
|
|
|
|
|
│ │ ├── bank_import.py # /bank-import/; parse (AJAX); import (AJAX)
|
|
|
|
|
│ │ └── logs.py # /logs/; /logs/api; /logs/clear; /logs/download
|
|
|
|
|
│ │
|
|
|
|
|
│ ├── services/
|
|
|
|
|
│ │ ├── account_service.py # calc_balance(), recalc_all(), get_total_assets/liabilities()
|
|
|
|
|
│ │ ├── ai_service.py # build_context(), stream_chat() SSE gen, generate_daily_insight()
|
|
|
|
|
│ │ ├── budget_service.py # get_budget_summary(), apply_rollovers()
|
|
|
|
|
│ │ ├── export_service.py # transactions_to_csv/excel(), report_to_pdf(), build_report_html()
|
|
|
|
|
│ │ ├── fx_service.py # get_today_rate(), force_refresh(), _fetch_yfinance(),
|
|
|
|
|
│ │ │ # _fetch_er_api(), get_rate_history()
|
|
|
|
|
│ │ ├── goal_service.py # get_projected_completion(), get_emergency_fund_status()
|
|
|
|
|
│ │ ├── import_service.py # parse_csv(), import_rows(), duplicate detection
|
|
|
|
|
│ │ ├── investment_service.py # fetch_price(), update_prices(), get_portfolio_summary()
|
|
|
|
|
│ │ ├── ocr_service.py # extract_from_file(), extract_from_bytes(),
|
|
|
|
|
│ │ │ # Groq vision model, JSON parse + sanitise
|
|
|
|
|
│ │ ├── recurring_service.py # process_due_rules(), next_occurrence(), get_upcoming()
|
|
|
|
|
│ │ └── report_service.py # monthly/quarterly/yearly/tax reports,
|
|
|
|
|
│ │ # net_worth_history(), category_trends(), take_net_worth_snapshot()
|
|
|
|
|
│ │ ├── account_service.py
|
|
|
|
|
│ │ ├── ai_service.py
|
|
|
|
|
│ │ ├── budget_service.py
|
|
|
|
|
│ │ ├── export_service.py
|
|
|
|
|
│ │ ├── fx_service.py
|
|
|
|
|
│ │ ├── goal_service.py
|
|
|
|
|
│ │ ├── import_service.py # duplicate check now scoped by account_id
|
|
|
|
|
│ │ ├── investment_service.py
|
|
|
|
|
│ │ ├── ocr_service.py
|
|
|
|
|
│ │ ├── recurring_service.py # 90-day catchup cap prevents runaway loops
|
|
|
|
|
│ │ ├── report_service.py
|
|
|
|
|
│ │ ├── teller_service.py # mTLS session; full response-body logging on errors
|
|
|
|
|
│ │ └── bank_import_service.py # PDF+OFX+CSV parsing; Groq PDF parsing; auto-categorize
|
|
|
|
|
│ │
|
|
|
|
|
│ ├── templates/
|
|
|
|
|
│ │ ├── base.html # Collapsible sidebar, topbar, flash messages,
|
|
|
|
|
│ │ │ # @keyframes spin, Bootstrap 5 + Bootstrap Icons
|
|
|
|
|
│ │ ├── auth/login.html # Dark themed, password toggle
|
|
|
|
|
│ │ ├── dashboard/index.html # All widgets, FX refresh JS, SSE-compatible
|
|
|
|
|
│ │ ├── accounts/ # index.html, form.html (color/icon picker)
|
|
|
|
|
│ │ ├── categories/ # index.html, form.html
|
|
|
|
|
│ │ ├── transactions/ # index.html (tabs+filter+pagination)
|
|
|
|
|
│ │ │ # form.html (OCR panel + drag-drop + field flash)
|
|
|
|
|
│ │ │ # transfer.html
|
|
|
|
|
│ │ ├── budgets/ # index.html (progress bars), form.html
|
|
|
|
|
│ │ ├── goals/ # index.html (emergency fund + cards), form.html,
|
|
|
|
|
│ │ │ # contribute.html, contributions.html
|
|
|
|
|
│ │ ├── investments/ # index.html (doughnut chart), detail.html,
|
|
|
|
|
│ │ │ # form.html (ticker check), transaction_form.html
|
|
|
|
|
│ │ ├── reports/ # index.html (4 chart types), tax.html
|
|
|
|
|
│ │ ├── ai/ # index.html (SSE chat + suggestions), history.html
|
|
|
|
|
│ │ └── settings/ # index.html, profile.html, password.html,
|
|
|
|
|
│ │ # recurring.html, recurring_form.html, import.html
|
|
|
|
|
│ │ ├── base.html # Sidebar (all active states); csrf-token meta tag
|
|
|
|
|
│ │ ├── auth/login.html
|
|
|
|
|
│ │ ├── dashboard/index.html
|
|
|
|
|
│ │ ├── accounts/
|
|
|
|
|
│ │ ├── categories/
|
|
|
|
|
│ │ ├── transactions/ # form.html: receipt forms outside #txnForm (nested-form fix)
|
|
|
|
|
│ │ ├── budgets/
|
|
|
|
|
│ │ ├── goals/
|
|
|
|
|
│ │ ├── investments/
|
|
|
|
|
│ │ ├── reports/
|
|
|
|
|
│ │ ├── ai/
|
|
|
|
|
│ │ ├── settings/
|
|
|
|
|
│ │ ├── teller/ # index.html, map_accounts.html, preview.html
|
|
|
|
|
│ │ ├── bank_import/ # index.html (drag-drop; AJAX parse+import; column mapper)
|
|
|
|
|
│ │ └── logs/ # index.html (live viewer; filters; auto-refresh)
|
|
|
|
|
│ │
|
|
|
|
|
│ ├── static/
|
|
|
|
|
│ │ ├── css/ # (empty — all CSS inline in templates)
|
|
|
|
@@ -243,18 +293,18 @@ pfm/ # /home/pfm/web on server
|
|
|
|
|
│ │ └── img/
|
|
|
|
|
│ │
|
|
|
|
|
│ └── utils/
|
|
|
|
|
│ ├── formatters.py # format_currency(), format_percent(), format_large_number()
|
|
|
|
|
│ └── decorators.py # login_required_custom (unused — Flask-Login handles it)
|
|
|
|
|
│ ├── formatters.py
|
|
|
|
|
│ └── decorators.py
|
|
|
|
|
│
|
|
|
|
|
├── migrations/ # Flask-Migrate / Alembic
|
|
|
|
|
├── migrations/
|
|
|
|
|
│
|
|
|
|
|
├── scripts/ # All cron scripts — sys.path fix at top of each
|
|
|
|
|
│ ├── init_db.py # Seeds 21 default categories, creates admin user interactively
|
|
|
|
|
│ ├── process_recurring.py # Processes due recurring rules, creates transactions
|
|
|
|
|
│ ├── fetch_fx_rate.py # force_refresh() — always fetches fresh USD/VND
|
|
|
|
|
│ ├── fetch_prices.py # yfinance price update for all investment tickers
|
|
|
|
|
│ ├── daily_snapshot.py # Saves net worth snapshot (1st of month)
|
|
|
|
|
│ └── daily_ai_insight.py # Generates daily AI summary via Groq
|
|
|
|
|
├── scripts/
|
|
|
|
|
│ ├── init_db.py
|
|
|
|
|
│ ├── process_recurring.py
|
|
|
|
|
│ ├── fetch_fx_rate.py
|
|
|
|
|
│ ├── fetch_prices.py
|
|
|
|
|
│ ├── daily_snapshot.py
|
|
|
|
|
│ └── daily_ai_insight.py
|
|
|
|
|
│
|
|
|
|
|
└── tests/
|
|
|
|
|
```
|
|
|
|
@@ -273,7 +323,6 @@ pymysql==1.1.1
|
|
|
|
|
python-dotenv==1.0.1
|
|
|
|
|
gunicorn==23.0.0
|
|
|
|
|
groq==0.13.1
|
|
|
|
|
yfinance==0.2.54
|
|
|
|
|
weasyprint==63.1
|
|
|
|
|
openpyxl==3.1.5
|
|
|
|
|
Pillow==11.1.0
|
|
|
|
@@ -281,6 +330,7 @@ apscheduler==3.10.4
|
|
|
|
|
requests==2.32.3
|
|
|
|
|
cryptography==44.0.2
|
|
|
|
|
python-dateutil==2.9.0
|
|
|
|
|
pdfplumber==0.11.4 # PDF text extraction for bank statement import
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
@@ -297,10 +347,13 @@ python-dateutil==2.9.0
|
|
|
|
|
- Model: `meta-llama/llama-4-scout-17b-16e-instruct`
|
|
|
|
|
- Input: base64-encoded image (JPEG/PNG/GIF/WEBP)
|
|
|
|
|
- Prompt: structured JSON extraction (amount, date, merchant, category, notes)
|
|
|
|
|
- Temperature: 0.1 (low, for consistent output)
|
|
|
|
|
- Post-processing: strips markdown fences, extracts JSON via regex fallback,
|
|
|
|
|
normalises amount/date, maps category to system names
|
|
|
|
|
- Endpoints: `POST /transactions/ocr` (upload bytes), `POST /transactions/ocr-file` (stored file)
|
|
|
|
|
- Temperature: 0.1
|
|
|
|
|
|
|
|
|
|
### Bank Statement PDF Parsing
|
|
|
|
|
- Text extracted by `pdfplumber` then sent to `llama-3.3-70b-versatile`
|
|
|
|
|
- Prompt requests JSON array of `{date, description, amount, transaction_type}`
|
|
|
|
|
- Max 30 K chars sent per request (~6 months of typical statements)
|
|
|
|
|
- Scanned PDFs (no text layer) are rejected with a clear error message
|
|
|
|
|
|
|
|
|
|
### Free Tier Limits
|
|
|
|
|
| Metric | Limit |
|
|
|
|
@@ -311,55 +364,75 @@ python-dateutil==2.9.0
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## 7. USD → VND Exchange Rate
|
|
|
|
|
## 7. Teller Bank Sync
|
|
|
|
|
|
|
|
|
|
> **Reference widget only.** All transactions use the single configured app currency.
|
|
|
|
|
|
|
|
|
|
### Fetch Priority
|
|
|
|
|
1. DB cache — same day record in `fx_rates`
|
|
|
|
|
2. **yfinance** `USDVND=X` forex ticker (primary — most reliable)
|
|
|
|
|
3. `open.er-api.com` free REST API (fallback)
|
|
|
|
|
4. Last known DB record (stale fallback, shows ⚠ indicator)
|
|
|
|
|
|
|
|
|
|
### Sanity Check
|
|
|
|
|
Rate must be `> 1000` — rejects garbage values (e.g. 1.0, 0.0) that some APIs return.
|
|
|
|
|
|
|
|
|
|
### Dashboard Widget
|
|
|
|
|
- Shows rate, date, source
|
|
|
|
|
- ↻ button → `POST /api/fx-refresh` → updates rate in-place without page reload
|
|
|
|
|
- Click widget → toggles 30-day Chart.js line chart
|
|
|
|
|
- `force_refresh()` used by daily cron — always bypasses cache
|
|
|
|
|
- mTLS: client cert + key from `TELLER_CERT_PATH` / `TELLER_KEY_PATH`
|
|
|
|
|
- HTTP Basic Auth: `access_token` as username, empty password
|
|
|
|
|
- Endpoints: `GET /accounts`, `GET /accounts/:id/balances`, `GET /accounts/:id/transactions`
|
|
|
|
|
- All API errors logged with status code + full response body
|
|
|
|
|
- Webhook: HMAC-SHA256 `Teller-Signature` header; 5-minute replay window
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## 8. UI/UX
|
|
|
|
|
## 8. Bank Statement Import
|
|
|
|
|
|
|
|
|
|
- Route: `/bank-import/` (blueprint `bank_import_bp`)
|
|
|
|
|
- Parse: `POST /bank-import/parse` (AJAX, multipart with `X-CSRFToken` header)
|
|
|
|
|
- Import: `POST /bank-import/import` (AJAX, JSON with `X-CSRFToken` header)
|
|
|
|
|
- File input is hidden and outside the drop zone (`fileInput.click()` on drop zone click)
|
|
|
|
|
- Duplicate detection:
|
|
|
|
|
- OFX: match on `import:<FITID>` in notes
|
|
|
|
|
- CSV/PDF: match on date + amount + type + description scoped to same account_id
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## 9. Logging System
|
|
|
|
|
|
|
|
|
|
- Config key: `LOG_FILE_PATH` (default: `<project-root>/logs/app.log`)
|
|
|
|
|
- Handler: `RotatingFileHandler` — 10 MB per file, 5 backups
|
|
|
|
|
- Format: `YYYY-MM-DD HH:MM:SS|LEVEL|module.name|message`
|
|
|
|
|
- Namespace: `logging.getLogger('app')` at INFO; `propagate=False`
|
|
|
|
|
- Also writes to stderr (Gunicorn captures it)
|
|
|
|
|
- Viewer: `/logs/` — real-time filtered display, per-level counts, auto-refresh, clear, download
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## 10. UI/UX
|
|
|
|
|
|
|
|
|
|
- **Sidebar**: collapsible (desktop state saved in localStorage), mobile overlay
|
|
|
|
|
- Active states: `{% if request.blueprint == '...' %}active{% endif %}`
|
|
|
|
|
- Links: Dashboard · Transactions · Add Income · Add Expense · Accounts · Import Statement · Budgets · Goals · Investments · Reports · AI Assistant · Categories · System Logs · Settings · Logout
|
|
|
|
|
- **Charts**: Chart.js 4.x (CDN)
|
|
|
|
|
- **Forms**: WTForms + Bootstrap 5.3
|
|
|
|
|
- **Icons**: Bootstrap Icons 1.11
|
|
|
|
|
- **Fonts**: DM Sans + DM Mono (Google Fonts CDN)
|
|
|
|
|
- **Color scheme**: `#0f172a` sidebar, `#f1f5f9` body, `#10b981` income, `#ef4444` expense, `#3b82f6` invest
|
|
|
|
|
- **CSS**: All inline in templates (no build step)
|
|
|
|
|
- **SSE**: used for AI chat stream + FX refresh
|
|
|
|
|
- **SSE**: AI chat stream + FX refresh
|
|
|
|
|
- **CSRF meta tag**: `<meta name="csrf-token">` in `base.html` for JS fetch calls
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## 9. Authentication
|
|
|
|
|
## 11. Authentication & Security
|
|
|
|
|
|
|
|
|
|
- Single-user, Flask-Login, session-based
|
|
|
|
|
- Hashed password (Werkzeug `generate_password_hash`)
|
|
|
|
|
- `SESSION_COOKIE_SECURE=True` in production
|
|
|
|
|
- `SESSION_COOKIE_HTTPONLY=True`, `SESSION_COOKIE_SAMESITE='Lax'`
|
|
|
|
|
- CSRF protection on all forms (Flask-WTF)
|
|
|
|
|
- CSRF protection on all forms (Flask-WTF); meta tag in base.html for AJAX
|
|
|
|
|
- SQLAlchemy ORM (no raw SQL)
|
|
|
|
|
- Receipt file path: `os.path.basename()` in both upload AND view_receipt (path-traversal fix)
|
|
|
|
|
- Teller account IDs validated against DB before mapping
|
|
|
|
|
- Transaction filter params safely cast with try/except (no crash on bad int input)
|
|
|
|
|
- Groq receives anonymised transaction summaries (no account/personal names)
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## 10. Scheduled Jobs
|
|
|
|
|
## 12. Scheduled Jobs
|
|
|
|
|
|
|
|
|
|
| Job | Schedule | Script | Notes |
|
|
|
|
|
|-----|----------|--------|-------|
|
|
|
|
|
| Process recurring transactions | Daily 6AM | `process_recurring.py` | Creates missed occurrences |
|
|
|
|
|
| Process recurring transactions | Daily 6AM | `process_recurring.py` | 90-day catchup cap |
|
|
|
|
|
| Fetch USD/VND rate | Daily 8AM | `fetch_fx_rate.py` | force_refresh(), yfinance primary |
|
|
|
|
|
| Fetch investment prices | Mon-Fri 4PM | `fetch_prices.py` | yfinance, all tickers |
|
|
|
|
|
| Net worth snapshot | 1st of month 00:05 | `daily_snapshot.py` | Saves to net_worth_snapshots |
|
|
|
|
@@ -368,7 +441,7 @@ Rate must be `> 1000` — rejects garbage values (e.g. 1.0, 0.0) that some APIs
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## 11. Environment Variables (`.env`)
|
|
|
|
|
## 13. Environment Variables (`.env`)
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
SECRET_KEY=your-secret-key
|
|
|
|
@@ -382,11 +455,18 @@ FLASK_APP=wsgi:app
|
|
|
|
|
APP_CURRENCY=USD
|
|
|
|
|
APP_CURRENCY_SYMBOL=$
|
|
|
|
|
APP_TIMEZONE=Asia/Ho_Chi_Minh
|
|
|
|
|
LOG_FILE_PATH=/home/pfm/web/logs/app.log
|
|
|
|
|
# Teller
|
|
|
|
|
TELLER_APP_ID=your-teller-app-id
|
|
|
|
|
TELLER_ENV=development
|
|
|
|
|
TELLER_CERT_PATH=/home/pfm/teller/certificate.pem
|
|
|
|
|
TELLER_KEY_PATH=/home/pfm/teller/private_key.pem
|
|
|
|
|
TELLER_WEBHOOK_SECRET=your-webhook-secret
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## 12. Blueprints Registered (11 total)
|
|
|
|
|
## 14. Blueprints Registered (13 total)
|
|
|
|
|
|
|
|
|
|
| Blueprint | Prefix | Key routes |
|
|
|
|
|
|-----------|--------|------------|
|
|
|
|
@@ -398,37 +478,60 @@ APP_TIMEZONE=Asia/Ho_Chi_Minh
|
|
|
|
|
| budgets | /budgets | index, new, edit, delete, copy |
|
|
|
|
|
| goals | /goals | index, new, edit, delete, contribute, contributions |
|
|
|
|
|
| investments | /investments | index, new, detail, edit, delete, add_transaction, refresh-prices, api/price |
|
|
|
|
|
| reports | /reports | monthly, quarterly, yearly, tax, export/csv|excel|pdf, snapshot |
|
|
|
|
|
| reports | /reports | monthly, quarterly, yearly, tax, export/csv\|excel\|pdf, snapshot |
|
|
|
|
|
| ai | /ai | index, stream (SSE), history, generate-insight |
|
|
|
|
|
| settings | /settings | index, profile, password, recurring, import, upload_receipt, delete_receipt, view_receipt |
|
|
|
|
|
| teller | /teller | callback, map, index, sync, sync/confirm, sync/all, balance, disconnect, webhook |
|
|
|
|
|
| bank_import | /bank-import | index, parse (AJAX), import (AJAX) |
|
|
|
|
|
| logs | /logs | index, api (AJAX), clear (AJAX), download |
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## 13. Security Notes
|
|
|
|
|
## 15. Known Issues / Notes
|
|
|
|
|
|
|
|
|
|
- All routes `@login_required`
|
|
|
|
|
- CSRF on all POST forms
|
|
|
|
|
- SQLAlchemy ORM (no raw SQL)
|
|
|
|
|
- Receipt file path: `os.path.basename()` prevents path traversal
|
|
|
|
|
- HTTPS via Let's Encrypt (Certbot) — `pfm.ngodanguyen.tech`
|
|
|
|
|
- Groq receives anonymised transaction summaries (no account/personal names)
|
|
|
|
|
- `GROQ_API_KEY` in `.env` (chmod 600), never in frontend
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## 14. Known Issues / Notes
|
|
|
|
|
|
|
|
|
|
- `wsgi.py` has `sys.path.insert(0, ...)` fix — required because app deploys at `/home/pfm/web/` which would otherwise be treated as a Python package
|
|
|
|
|
- `wsgi.py` has `sys.path.insert(0, ...)` — required for Gunicorn at `/home/pfm/web/`
|
|
|
|
|
- FX rate widget: `open.er-api.com` may return stale values; yfinance is the reliable primary
|
|
|
|
|
- WeasyPrint PDF: requires `libpango*` system libs (included in deploy.md apt install)
|
|
|
|
|
- Import preview uses Flask session to pass rows to confirm step — requires `SECRET_KEY` to be set
|
|
|
|
|
- WeasyPrint PDF: requires `libpango*` system libs on server
|
|
|
|
|
- Bank statement PDF import: scanned/image PDFs have no text layer; must use digital download
|
|
|
|
|
- Bank statement PDF import: large PDFs (>30 K chars) are truncated; split into shorter date ranges
|
|
|
|
|
- pdfplumber must be installed: `pip install pdfplumber==0.11.4`
|
|
|
|
|
- Teller: development environment only; requires cert/key from Teller Dashboard
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## 15. Post-MVP Roadmap
|
|
|
|
|
## 16. Security Fixes Applied (session log)
|
|
|
|
|
|
|
|
|
|
| Date | Fix | File |
|
|
|
|
|
|------|-----|------|
|
|
|
|
|
| 2026-06 | Path traversal in `view_receipt` — added `os.path.basename()` | settings.py |
|
|
|
|
|
| 2026-06 | `int()` crash on bad filter params in transactions index | transactions.py |
|
|
|
|
|
| 2026-06 | Teller account mapping validates ID exists in DB | teller.py |
|
|
|
|
|
| 2026-06 | Recurring catchup capped at 90 days (prevents runaway loops) | recurring_service.py |
|
|
|
|
|
| 2026-06 | CSV/bank import duplicate check scoped by `account_id` | import_service.py |
|
|
|
|
|
| 2026-06 | CSRF token added to bank import AJAX parse request | bank_import/index.html |
|
|
|
|
|
| 2026-06 | Receipt sub-forms moved outside `#txnForm` (nested-form bug) | transactions/form.html |
|
|
|
|
|
| 2026-06 | Drop zone file input moved outside overlay (blocked account select) | bank_import/index.html |
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## 17. To-Do / Roadmap
|
|
|
|
|
|
|
|
|
|
### High priority
|
|
|
|
|
- [ ] **Mobile responsiveness pass** — sidebar auto-collapses on mobile; tables scroll horizontally
|
|
|
|
|
- [ ] **Empty-state messages** — transactions, budgets, goals, investments pages when no data
|
|
|
|
|
- [ ] **Budget alerts** — email/Twilio SMS when category spending hits 80% / 100%
|
|
|
|
|
|
|
|
|
|
### Medium priority
|
|
|
|
|
- [ ] **Pagination info** — show "Page X of Y" on AI history and other paginated pages
|
|
|
|
|
- [ ] **PDF export memory** — stream CSV/Excel exports for users with large transaction history
|
|
|
|
|
- [ ] **Receipt MIME validation** — validate file magic bytes server-side, not just extension
|
|
|
|
|
- [ ] **Teller multi-account sync** — sync all mapped accounts in sequence (currently syncs first only)
|
|
|
|
|
- [ ] **OCR ownership check** — verify re-extracted filename belongs to current user's transaction
|
|
|
|
|
- [ ] **Bank import progress** — show per-row import progress for large statement files
|
|
|
|
|
|
|
|
|
|
### Low priority / future
|
|
|
|
|
- [ ] iOS companion app
|
|
|
|
|
- [ ] Bank statement PDF auto-import (parse PDF → extract transactions)
|
|
|
|
|
- [ ] Budget alerts + Twilio SMS/email notifications
|
|
|
|
|
- [ ] Shared household mode (2 users, row-level isolation)
|
|
|
|
|
- [ ] Mobile responsiveness pass
|
|
|
|
|
- [ ] Bank statement PDF: table-extraction fallback (pdfplumber tables API) before Groq call
|
|
|
|
|
- [ ] Investment price history chart per holding
|
|
|
|
|
- [ ] Dark mode toggle
|
|
|
|
|