05/31 Update documents

This commit is contained in:
2026-05-31 17:28:50 -04:00
parent 5983794217
commit 8b1594da5a
3 changed files with 1249 additions and 626 deletions
+325 -376
View File
@@ -1,236 +1,262 @@
# Personal Finance Management System (PFMS) # Personal Finance Management System (PFM)
> Stack: Python Flask · MySQL · Ubuntu Server · Nginx · Gunicorn · Groq API (free AI) > Stack: Python Flask · MySQL · Ubuntu Server · Nginx · Gunicorn · Groq API (free AI)
> App URL: https://pfm.ngodanguyen.tech
> Code: /home/pfm/web · User: pfm · Gitea: gitea.ngodanguyen.tech
--- ---
## 1. Project Overview ## 1. Project Overview
Self-hosted personal finance web app. Tracks income, expenses, investments. AI assistant powered by **Groq API** (free tier, extremely fast inference, no local hardware needed). Everything runs on your Ubuntu server behind Nginx. Self-hosted personal finance web app. Tracks income, expenses, investments. AI assistant powered by **Groq API** (free tier, fast inference, no local hardware). Receipt OCR via Groq vision model. Everything runs on Ubuntu server behind Nginx + Certbot SSL.
**Status: All 7 phases complete + Receipt OCR post-MVP feature.**
--- ---
## 2. Core Features ## 2. Core Features (Implemented)
### 2.1 Dashboard ### 2.1 Dashboard
- Net worth snapshot (assets liabilities) - Net worth snapshot (assets liabilities)
- Monthly cash flow chart (income vs expenses) - Monthly cash flow bar chart (6 months)
- Budget utilization gauges per category - Budget utilization per category
- Recent transactions feed - Recent transactions feed (last 8)
- AI insight card (auto-generated daily summary) - AI daily insight card (Groq-generated, stored in DB)
- Investment portfolio mini-widget - USD → VND exchange rate widget — reference only, independent of app currency
- **USD → VND exchange rate widget** (daily rate, fetched once/day, cached in DB — reference only, independent of app currency) - Click to expand 30-day history chart
- ↻ refresh button (force-fetches fresh rate without page reload)
- Source label shown (yfinance / exchangerate-api)
- Stale indicator if rate > 1 day old
- Period selector: This Month / Last Month / Custom date range
### 2.2 Income Management ### 2.2 Transactions
- Log income entries (salary, freelance, passive, other) - Income + Expense entry with Income/Expense tabs
- Recurring income templates (auto-create entries on schedule) - Transfer between accounts
- Income source breakdown (chart by source) - Filter: search, category, account, date range
- Month-over-month comparison - Pagination (30/page)
- Export to CSV/Excel - Receipt upload (PNG/JPG/WEBP/GIF/PDF, max 10MB)
- **AI Receipt OCR** — drag-drop receipt image → Groq vision extracts amount/date/merchant/category → auto-fills form
- Re-extract from already-uploaded receipt (edit mode)
- Export to CSV / Excel
### 2.3 Expense Management ### 2.3 Accounts
- Manual expense entry - Types: checking, savings, cash, credit_card, crypto, investment, other
- Category tagging (custom + predefined: Food, Rent, Utilities, Transport, Health, Entertainment, etc.) - Balance auto-calculated from all transactions (not manually entered)
- Subcategory support - Color + icon picker
- Receipt photo upload (stored locally) - Soft delete
- Recurring expense detection
- Budget limits per category with alert thresholds
- Expense search + filter (date range, category, amount range, keyword)
- Export to CSV/Excel
### 2.4 Investment Portfolio ### 2.4 Categories
- Asset types: Stocks, ETF, Crypto, Real Estate, Bonds, Cash, Other - Expense + Income categories with color/icon
- Holdings tracker (ticker, shares/units, buy price, current price) - System categories (protected from delete)
- Manual price update OR auto-fetch via free API (Yahoo Finance via `yfinance`) - Custom categories (user-created)
- P&L per holding (unrealized gain/loss) - 14 expense + 7 income defaults seeded on init
- Portfolio allocation pie chart
- Transaction log (buy/sell history per asset)
- Cost basis tracking (FIFO)
### 2.5 Budget Planner ### 2.5 Budget Planner
- Monthly budget templates - Monthly limits per expense category
- Set budget limits per category - Progress bars: green → amber (80%) → red (100%+)
- Real-time spending vs budget comparison - Rollover unused budget to next month (toggle)
- Rollover unused budget (optional toggle) - Copy previous month's budgets in one click
- Budget history archive - Unbudgeted spending shown with "Set Budget" prompt
### 2.6 Goals & Savings ### 2.6 Goals & Savings
- Create savings goals (name, target amount, target date, linked account) - Goals with target amount, target date, color, icon
- Track contributions toward each goal - Contribution tracking + history
- Progress bar + projected completion date - Progress bar + projected completion date (based on avg monthly contrib)
- Emergency fund tracker (X months of expenses) - Auto-complete on 100%
- Emergency fund tracker (liquid assets vs 3-month / 6-month expense targets)
### 2.7 Reports & Analytics ### 2.7 Investments
- Monthly/quarterly/yearly summary reports - Asset types: stock, ETF, crypto, real_estate, bond, cash, other
- Category spending trends (line chart over time) - Buy/sell/dividend/split transaction log
- Income growth chart - FIFO cost basis auto-recalculated from transaction log
- Net worth over time (historical snapshots, monthly auto-saved) - yfinance price auto-fetch (daily 4PM weekdays via cron)
- Tax year summary (income + deductible expenses) - Manual price refresh button (portfolio page)
- Printable PDF report (via WeasyPrint) - Live ticker check on add form
- Doughnut allocation chart
- P&L per holding + portfolio total
### 2.8 Accounts & Wallets ### 2.8 AI Financial Assistant
- Multiple accounts (bank checking, savings, cash, credit card, crypto wallet) - Chat UI with SSE streaming (Groq API, word-by-word response)
- Account balances tracked manually - Context: last 90 days transactions + budget status + goals + investments
- Transfer between accounts (internal transaction) - 8 suggested question buttons
- Credit card balance + due date tracking - Daily auto-insight generated at midnight (stored in `ai_insights` table)
- Manual "Generate Now" button
- Chat history page
- Model: `llama-3.3-70b-versatile` (default) or `llama-3.1-8b-instant` (fast)
- Fallback messages for rate limit / invalid key / unavailable
### 2.9 AI Financial Assistant (Groq API — Free Tier) ### 2.9 Receipt OCR (Groq Vision)
- Chat interface (ask questions about your finances) - Model: `meta-llama/llama-4-scout-17b-16e-instruct`
- Context: last 90 days of transactions injected into prompt - Drag-drop or click-to-upload on new expense form
- Example queries: - Extracts: amount, date, merchant name, category suggestion, notes
- "Where did I overspend this month?" - Maps category suggestion → system category ID
- "Am I on track for my vacation goal?" - Auto-fills form fields with green flash animation
- "Summarize my Q1 spending" - Re-extract button on existing receipt (edit mode)
- "What categories can I cut to save $500/month?" - Auto-triggers OCR when image file selected in edit mode
- Auto-insight: daily AI summary generated at midnight via cron - Handles fenced markdown JSON output from LLM
- Model: `llama-3.3-70b-versatile` or `llama-3.1-8b-instant` via Groq (configurable in `.env`) - Sanity check: rejects amounts < 1000 (catches garbage values)
- Streaming response (SSE) for real-time chat feel - Full error handling: 400/401/429/timeout/bad JSON
### 2.10 Notifications & Alerts ### 2.10 Reports & Export
- Browser notifications (via Web Push or in-app toast) - Monthly / Quarterly / Yearly summary reports
- Budget threshold alerts (e.g., 80% of category budget used) - Tax year summary (all income by source, all expenses by category)
- Bill/recurring expense due reminders - Net worth history line chart (from monthly snapshots)
- Goal milestone celebrations - Category spending trends (top 6 categories, 6-month line chart)
- "Snapshot Now" manual button
- Export: CSV, Excel (color-coded, formatted), PDF (WeasyPrint)
### 2.11 Settings & Config ### 2.11 Settings
- Profile (name, timezone) - Profile: name, email, timezone, currency, Groq model
- **App currency** — single configurable currency (e.g. USD, VND, EUR — set once, used everywhere for all transactions/display) - 8 currency options (USD/VND/EUR/GBP/JPY/AUD/CAD/SGD) — auto-updates symbol
- USD→VND rate source preference (ExchangeRate-API free or VCB scrape fallback) - Password change (requires current password)
- Category management (add/edit/delete custom categories) - Recurring rules: CRUD, pause/enable, frequency (daily/weekly/biweekly/monthly/quarterly/yearly)
- Data backup (export full MySQL dump) - "Run Now" button to process due rules immediately
- Data import (CSV import for bulk transactions) - CSV import: upload → preview with ⚠ warnings → confirm
- Groq model selector (choose speed vs quality) - Upcoming recurring transactions (30-day view)
### 2.12 USD → VND Exchange Rate
- **Reference widget only** — not used in transaction calculations
- Primary source: yfinance `USDVND=X` (Yahoo Finance forex)
- Fallback: `open.er-api.com` (free, no key)
- Sanity check: rate must be > 1000 (rejects garbage values)
- `force_refresh()` — always fetches fresh on cron, bypasses cache
- DB caches one record per day (`fx_rates` table)
- Dashboard: shows rate, date, source, stale warning, ↻ refresh button
- 30-day history chart (click widget to expand)
--- ---
## 3. Database Schema (MySQL) ## 3. Database Schema (MySQL)
### Tables ### All 14 Tables
``` ```
users — single user (self-hosted, no multi-tenant) users — single user, hashed password, currency/timezone prefs
accounts — bank/wallet accounts accounts — bank/wallet accounts (balance auto-calc from txns)
categories — expense/income categories categories — expense/income categories with color/icon
transactions — all money movements (income/expense/transfer) transactions — income/expense/transfer, receipt_id, recurring_rule_id
investments — holdings/portfolio positions receipts — receipt file metadata (filename, size, mime_type)
investment_transactions — buy/sell log recurring_rules — templates: frequency, next_run, start/end date
budgets — monthly budget limits per category budgets — monthly limits per category, rollover support
goals — savings goals goals — savings goals with target amount/date
goal_contributions deposits toward each goal goal_contributions — individual deposits toward each goal
net_worth_snapshots — monthly net worth history investments — holdings: ticker, shares, avg_cost_basis, current_price
ai_insights — stored daily AI summaries investment_transactions — buy/sell/dividend/split log
recurring_rules — templates for recurring income/expenses net_worth_snapshots — monthly snapshots: assets, liabilities, net_worth (JSON)
receipts — receipt image metadata ai_insights — stored AI responses: daily_summary / chat_response
fx_rates — daily USD/VND rate cache (date, rate, source) fx_rates — daily USD/VND rate cache (date UNIQUE, source)
``` ```
### Key Table: `transactions` ### Key Column Notes
```sql - `transactions.balance` — NOT stored; calculated on-demand via `account_service.calc_balance()`
id, account_id, category_id, type (income/expense/transfer), - `investments.shares` / `avg_cost_basis` — recalculated from `investment_transactions` (FIFO)
amount, currency, description, date, notes, - `goals.current_amount` — updated on each contribution add/delete
is_recurring, recurring_rule_id, receipt_id, - `net_worth_snapshots.account_balances` — JSON snapshot of each account balance at time of snapshot
created_at, updated_at
```
### Key Table: `investments`
```sql
id, asset_name, ticker, asset_type (stock/etf/crypto/real_estate/bond/cash/other),
shares, avg_cost_basis, current_price, last_price_update,
currency, notes, created_at
```
### Key Table: `budgets`
```sql
id, category_id, month (YYYY-MM), limit_amount,
rollover_enabled, rollover_amount, created_at
```
--- ---
## 4. Project File Structure ## 4. Project File Structure (Actual)
``` ```
pfm/ pfm/ # /home/pfm/web on server
├── wsgi.py # Gunicorn entry — sys.path fix included
├── requirements.txt
├── .env # Not committed
├── .env.example
├── .gitignore
├── app/ ├── app/
│ ├── __init__.py # Flask app factory │ ├── __init__.py # Flask app factory, all blueprints registered
│ ├── config.py # Config classes (dev/prod) │ ├── config.py # Dev/Prod configs, SESSION_COOKIE_SECURE in prod
│ ├── extensions.py # db, login_manager, etc. │ ├── extensions.py # db, login_manager, migrate, csrf
│ │ │ │
│ ├── models/ │ ├── models/
│ │ ├── user.py │ │ ├── __init__.py # Imports all models (required for Flask-Migrate)
│ │ ├── account.py │ │ ├── user.py # UserMixin, set/check password, load_user hook
│ │ ├── transaction.py │ │ ├── account.py # account_type enum, color, icon
│ │ ├── category.py │ │ ├── category.py # Self-referential (subcategories), is_system flag
│ │ ├── investment.py │ │ ├── transaction.py # Dual FK to accounts (account_id + to_account_id)
│ │ ├── budget.py │ │ ├── receipt.py # filename, original_filename, mime_type
│ │ ├── goal.py │ │ ├── recurring_rule.py # frequency enum, next_run date
│ │ ├── ai_insight.py │ │ ├── budget.py # UniqueConstraint(category_id, month)
│ │ ├── fx_rate.py │ │ ├── goal.py # progress_percent property
│ │ ── recurring_rule.py │ │ ── investment.py # total_cost/current_value/unrealized_gain properties
│ │ ├── net_worth_snapshot.py # account_balances JSON field
│ │ ├── ai_insight.py # insight_type enum
│ │ └── fx_rate.py # date UNIQUE index
│ │ │ │
│ ├── routes/ │ ├── routes/
│ │ ├── auth.py # Login/logout (single user) │ │ ├── auth.py # /auth/login, /auth/logout
│ │ ├── dashboard.py │ │ ├── dashboard.py # /, /api/fx-history, /api/fx-refresh (POST)
│ │ ├── transactions.py │ │ ├── accounts.py # /accounts/
│ │ ├── income.py │ │ ├── categories.py # /categories/
│ │ ├── expenses.py │ │ ├── transactions.py # /transactions/, /transactions/ocr (POST),
│ │ ├── investments.py │ │ │ # /transactions/ocr-file (POST)
│ │ ├── budgets.py │ │ ├── budgets.py # /budgets/, /budgets/copy (POST)
│ │ ├── goals.py │ │ ├── goals.py # /goals/, contribute, contributions, delete_contribution
│ │ ├── accounts.py │ │ ├── investments.py # /investments/, detail, add_transaction,
│ │ ├── reports.py │ │ │ # refresh-prices, /api/price/<ticker>
│ │ ├── ai.py # AI chat + insights (SSE) │ │ ├── reports.py # /reports/monthly|quarterly|yearly|tax
│ │ ├── settings.py │ │ │ # /reports/export/csv|excel|pdf
│ │ ── api.py # Internal JSON API endpoints │ │ ── ai.py # /ai/, /ai/stream (SSE), /ai/history,
│ │ │ # /ai/generate-insight (POST)
│ │ └── settings.py # /settings/, profile, password, recurring,
│ │ # import, upload_receipt, delete_receipt, view_receipt
│ │ │ │
│ ├── services/ │ ├── services/
│ │ ├── ai_service.py # Groq API integration + context builder │ │ ├── account_service.py # calc_balance(), recalc_all(), get_total_assets/liabilities()
│ │ ├── budget_service.py # Budget calc + alert logic │ │ ├── ai_service.py # build_context(), stream_chat() SSE gen, generate_daily_insight()
│ │ ├── investment_service.py # yfinance price fetcher │ │ ├── budget_service.py # get_budget_summary(), apply_rollovers()
│ │ ├── fx_service.py # USD/VND rate fetch + cache logic │ │ ├── export_service.py # transactions_to_csv/excel(), report_to_pdf(), build_report_html()
│ │ ├── report_service.py # PDF generation (WeasyPrint) │ │ ├── fx_service.py # get_today_rate(), force_refresh(), _fetch_yfinance(),
│ │ ├── recurring_service.py # Recurring rule processor │ │ │ # _fetch_er_api(), get_rate_history()
│ │ ├── import_service.py # CSV import parser │ │ ├── goal_service.py # get_projected_completion(), get_emergency_fund_status()
│ │ ── snapshot_service.py # Net worth snapshot scheduler │ │ ── import_service.py # parse_csv(), import_rows(), duplicate detection
│ │ ├── investment_service.py # fetch_price(), update_prices(), get_portfolio_summary()
│ │ ├── ocr_service.py # extract_from_file(), extract_from_bytes(),
│ │ │ # Groq vision model, JSON parse + sanitise
│ │ ├── recurring_service.py # process_due_rules(), next_occurrence(), get_upcoming()
│ │ └── report_service.py # monthly/quarterly/yearly/tax reports,
│ │ # net_worth_history(), category_trends(), take_net_worth_snapshot()
│ │ │ │
│ ├── templates/ │ ├── templates/
│ │ ├── base.html │ │ ├── base.html # Collapsible sidebar, topbar, flash messages,
│ │ ├── auth/ │ │ │ # @keyframes spin, Bootstrap 5 + Bootstrap Icons
│ │ ├── dashboard/ │ │ ├── auth/login.html # Dark themed, password toggle
│ │ ├── transactions/ │ │ ├── dashboard/index.html # All widgets, FX refresh JS, SSE-compatible
│ │ ├── investments/ │ │ ├── accounts/ # index.html, form.html (color/icon picker)
│ │ ├── budgets/ │ │ ├── categories/ # index.html, form.html
│ │ ├── goals/ │ │ ├── transactions/ # index.html (tabs+filter+pagination)
│ │ ├── accounts/ │ │ │ # form.html (OCR panel + drag-drop + field flash)
│ │ ├── reports/ │ │ │ # transfer.html
│ │ ├── ai/ │ │ ├── budgets/ # index.html (progress bars), form.html
│ │ ── settings/ │ │ ── goals/ # index.html (emergency fund + cards), form.html,
│ │ │ # contribute.html, contributions.html
│ │ ├── investments/ # index.html (doughnut chart), detail.html,
│ │ │ # form.html (ticker check), transaction_form.html
│ │ ├── reports/ # index.html (4 chart types), tax.html
│ │ ├── ai/ # index.html (SSE chat + suggestions), history.html
│ │ └── settings/ # index.html, profile.html, password.html,
│ │ # recurring.html, recurring_form.html, import.html
│ │ │ │
│ ├── static/ │ ├── static/
│ │ ├── css/ │ │ ├── css/ # (empty — all CSS inline in templates)
│ │ ├── js/ │ │ ├── js/ # (empty — all JS inline in templates)
│ │ └── img/ # Static images │ │ └── img/
│ │ │ │
│ └── utils/ │ └── utils/
│ ├── decorators.py # Auth required, etc. │ ├── formatters.py # format_currency(), format_percent(), format_large_number()
── formatters.py # Currency, date formatting ── decorators.py # login_required_custom (unused — Flask-Login handles it)
│ └── validators.py
├── migrations/ # Flask-Migrate (Alembic) ├── migrations/ # Flask-Migrate / Alembic
├── scripts/
│ ├── init_db.py # First-run DB setup + seed categories
│ ├── daily_snapshot.py # Cron: net worth snapshot
│ ├── daily_ai_insight.py # Cron: generate AI summary
│ ├── process_recurring.py # Cron: create recurring transactions
│ ├── fetch_prices.py # Cron: update investment prices
│ └── fetch_fx_rate.py # Cron: fetch daily USD/VND rate
├── tests/ ├── scripts/ # All cron scripts — sys.path fix at top of each
├── .env # Secrets (not committed) │ ├── init_db.py # Seeds 21 default categories, creates admin user interactively
├── .env.example │ ├── process_recurring.py # Processes due recurring rules, creates transactions
├── requirements.txt │ ├── fetch_fx_rate.py # force_refresh() — always fetches fresh USD/VND
├── wsgi.py │ ├── fetch_prices.py # yfinance price update for all investment tickers
├── CLAUDE.md # This file │ ├── daily_snapshot.py # Saves net worth snapshot (1st of month)
└── deploy.md # Server setup guide └── daily_ai_insight.py # Generates daily AI summary via Groq
└── tests/
``` ```
--- ---
@@ -238,248 +264,171 @@ pfm/
## 5. Python Dependencies (`requirements.txt`) ## 5. Python Dependencies (`requirements.txt`)
``` ```
flask flask==3.1.0
flask-sqlalchemy flask-sqlalchemy==3.1.1
flask-login flask-login==0.6.3
flask-migrate flask-migrate==4.1.0
flask-wtf flask-wtf==1.2.2
pymysql pymysql==1.1.1
python-dotenv python-dotenv==1.0.1
gunicorn gunicorn==23.0.0
groq # Groq official Python SDK groq==0.13.1
yfinance # Investment price fetching yfinance==0.2.54
weasyprint # PDF report generation weasyprint==63.1
openpyxl # Excel export openpyxl==3.1.5
Pillow # Receipt image processing Pillow==11.1.0
apscheduler # In-process scheduler (alternative to cron) apscheduler==3.10.4
requests==2.32.3
cryptography==44.0.2
python-dateutil==2.9.0
``` ```
--- ---
## 6. AI Integration — Groq API (Free Tier) ## 6. AI Integration — Groq API
### Why Groq ### Chat + Daily Insights
- Free tier: 14,400 requests/day, 500,000 tokens/minute - Model: `llama-3.3-70b-versatile` (default) / `llama-3.1-8b-instant` (fast)
- Fastest inference available (LPU hardware) — responses feel instant - Context: last 90 days transactions, budget status, goals, investments (anonymised)
- No local GPU/RAM needed - SSE streaming: `stream_chat()` yields `data: <chunk>\n\n`
- Models: `llama-3.3-70b-versatile` (best quality), `llama-3.1-8b-instant` (fastest) - Daily insight: non-streaming, stored in `ai_insights` table, max 400 tokens
### How It Works ### Receipt OCR (Vision)
``` - Model: `meta-llama/llama-4-scout-17b-16e-instruct`
User sends chat message - Input: base64-encoded image (JPEG/PNG/GIF/WEBP)
→ ai_service.py builds context (last 90 days transactions summary) - Prompt: structured JSON extraction (amount, date, merchant, category, notes)
→ POST to Groq API (`/chat/completions`) with Bearer token - Temperature: 0.1 (low, for consistent output)
→ Stream response back via SSE to browser - Post-processing: strips markdown fences, extracts JSON via regex fallback,
→ Response stored in ai_insights table normalises amount/date, maps category to system names
``` - Endpoints: `POST /transactions/ocr` (upload bytes), `POST /transactions/ocr-file` (stored file)
### Context Injection Strategy ### Free Tier Limits
```python | Metric | Limit |
# ai_service.py builds a prompt like: |--------|-------|
""" | Requests/day | 14,400 |
You are a personal finance assistant. Here is the user's financial data: | Tokens/minute | 500,000 |
| Cost | Free |
CURRENT MONTH SUMMARY:
- Total Income: $X
- Total Expenses: $Y
- Top spending categories: Food ($A), Transport ($B), ...
- Budget alerts: Entertainment 92% used
RECENT TRANSACTIONS (last 20):
[date] [category] [amount] [description]
...
NET WORTH: $Z
ACTIVE GOALS: Vacation Fund ($1,200 / $3,000)
User question: {user_message}
Answer concisely and specifically based on the data above.
"""
```
### Fallback
If Groq API key missing/invalid → show friendly message "AI assistant unavailable. Check GROQ_API_KEY in settings."
If Groq rate limit hit → show "AI rate limit reached. Try again shortly."
--- ---
## 7. USD → VND Exchange Rate ## 7. USD → VND Exchange Rate
> **Reference widget only** — independent of the app's transaction currency. All income/expenses/investments use the single configured app currency. This widget is informational display only. > **Reference widget only.** All transactions use the single configured app currency.
### Data Source — Free, No API Key Required ### Fetch Priority
1. DB cache — same day record in `fx_rates`
2. **yfinance** `USDVND=X` forex ticker (primary — most reliable)
3. `open.er-api.com` free REST API (fallback)
4. Last known DB record (stale fallback, shows ⚠ indicator)
Primary: **ExchangeRate-API open endpoint** ### Sanity Check
``` Rate must be `> 1000` — rejects garbage values (e.g. 1.0, 0.0) that some APIs return.
https://open.er-api.com/v6/latest/USD
```
Returns JSON with all rates including VND. Free tier, no key, 1,500 req/month.
Fallback: **Vietcombank (VCB) rate scrape** ### Dashboard Widget
``` - Shows rate, date, source
https://www.vietcombank.com.vn/en/KHCN/Chuyen-trang-KHCN/Pages/ty-gia.aspx - ↻ button → `POST /api/fx-refresh` → updates rate in-place without page reload
``` - Click widget → toggles 30-day Chart.js line chart
Scrape VCB's official buying/selling rate as backup. - `force_refresh()` used by daily cron — always bypasses cache
### DB Table: `fx_rates`
```sql
id INT AUTO_INCREMENT PRIMARY KEY
date DATE NOT NULL UNIQUE -- one record per day
usd_to_vnd DECIMAL(12,2) NOT NULL -- e.g. 25,450.00
source VARCHAR(50) -- 'exchangerate-api' | 'vcb' | 'manual'
fetched_at DATETIME
```
### `fx_service.py` Logic
```
get_today_rate():
1. Check fx_rates table for today's date
2. If found → return cached rate (no API call)
3. If not found → fetch from ExchangeRate-API
4. If API fails → try VCB scrape
5. If both fail → return last known rate from DB + show "rate may be outdated" flag
6. Save new rate to DB
```
### Dashboard Widget Display
- Card on dashboard header area (top bar or sidebar)
- Shows: `1 USD = 25,450 ₫` with date label
- Color: neutral/info (blue or grey — not green/red, it's informational)
- Click → opens 30-day rate history mini-chart (Chart.js, line chart)
- Stale indicator: if rate is >1 day old, show small warning icon
### 30-Day Rate History
- Stored in `fx_rates` table (one row/day, auto-accumulates)
- Chart available on dashboard click or Reports page
- Shows trend: flat/up/down with % change label
### Scheduled Job
- Daily 8AM fetch (after markets open in Vietnam)
- `scripts/fetch_fx_rate.py`
- systemd timer unit: `pfm-fxrate.timer`
--- ---
## 8. UI/UX Design Direction ## 8. UI/UX
- **Style**: Clean financial dashboard — dark sidebar, white/light content area - **Sidebar**: collapsible (desktop state saved in localStorage), mobile overlay
- **Charts**: Chart.js (CDN, no build step needed) - **Charts**: Chart.js 4.x (CDN)
- **Tables**: DataTables.js for sortable/searchable transaction tables - **Forms**: WTForms + Bootstrap 5.3
- **Forms**: WTForms + Bootstrap 5 - **Icons**: Bootstrap Icons 1.11
- **Icons**: Bootstrap Icons or Feather Icons - **Fonts**: DM Sans + DM Mono (Google Fonts CDN)
- **Color scheme**: Deep navy sidebar, white cards, green (income), red (expense), blue (investment) - **Color scheme**: `#0f172a` sidebar, `#f1f5f9` body, `#10b981` income, `#ef4444` expense, `#3b82f6` invest
- **Mobile**: Responsive (Bootstrap grid) - **CSS**: All inline in templates (no build step)
- **AI Chat**: Floating chat panel (slide-in from right), SSE streaming text - **SSE**: used for AI chat stream + FX refresh
--- ---
## 9. Authentication ## 9. Authentication
- Single-user app (self-hosted) - Single-user, Flask-Login, session-based
- Flask-Login with username/password - Hashed password (Werkzeug `generate_password_hash`)
- Session-based auth - `SESSION_COOKIE_SECURE=True` in production
- Password hashed with Werkzeug (bcrypt) - `SESSION_COOKIE_HTTPONLY=True`, `SESSION_COOKIE_SAMESITE='Lax'`
- Optional: IP whitelist via Nginx (allow only LAN access) - CSRF protection on all forms (Flask-WTF)
--- ---
## 10. Scheduled Jobs (systemd timers or APScheduler) ## 10. Scheduled Jobs
| Job | Schedule | Script | | Job | Schedule | Script | Notes |
|-----|----------|--------| |-----|----------|--------|-------|
| Process recurring transactions | Daily 6AM | `process_recurring.py` | | Process recurring transactions | Daily 6AM | `process_recurring.py` | Creates missed occurrences |
| Fetch USD/VND exchange rate | Daily 8AM | `fetch_fx_rate.py` | | Fetch USD/VND rate | Daily 8AM | `fetch_fx_rate.py` | force_refresh(), yfinance primary |
| Fetch investment prices | Daily 4PM | `fetch_prices.py` | | Fetch investment prices | Mon-Fri 4PM | `fetch_prices.py` | yfinance, all tickers |
| Save net worth snapshot | 1st of month | `daily_snapshot.py` | | Net worth snapshot | 1st of month 00:05 | `daily_snapshot.py` | Saves to net_worth_snapshots |
| Generate AI daily insight | Daily midnight | `daily_ai_insight.py` | | AI daily insight | Daily 00:01 | `daily_ai_insight.py` | Skips if already done today |
| DB backup | Daily 2AM | pfm-backup (systemd) | mysqldump → gzip, keep 30 days |
Recommend **APScheduler** inside Flask app (simpler) OR separate systemd timer units (more robust).
--- ---
## 11. Development Phases ## 11. Environment Variables (`.env`)
### Phase 1 — Foundation
- [ ] Flask app factory + config
- [ ] MySQL models + migrations
- [ ] Auth (login/logout)
- [ ] Base template + sidebar nav
### Phase 2 — Core Transactions
- [ ] Accounts CRUD
- [ ] Categories CRUD
- [ ] Transaction entry (income + expense)
- [ ] Transaction list with filter/search
- [ ] Dashboard basics (totals, recent feed)
- [ ] USD→VND rate widget + `fx_service.py` + daily fetch job
### Phase 3 — Budget & Goals
- [ ] Budget planner (set limits)
- [ ] Budget vs actual comparison
- [ ] Goals CRUD + contribution tracking
### Phase 4 — Investments
- [ ] Holdings CRUD
- [ ] Buy/sell transaction log
- [ ] yfinance price auto-fetch
- [ ] Portfolio charts
### Phase 5 — AI Assistant
- [ ] Groq API integration + context builder
- [ ] Chat UI with SSE streaming
- [ ] Daily auto-insight cron
### Phase 6 — Reports & Export
- [ ] Monthly summary page
- [ ] PDF export (WeasyPrint)
- [ ] CSV/Excel export
- [ ] Net worth history chart
### Phase 7 — Polish
- [ ] Recurring transaction engine
- [ ] Receipt upload
- [ ] CSV import
- [ ] Budget alerts + notifications
- [ ] Mobile responsiveness pass
---
## 12. Environment Variables (`.env`)
``` ```
SECRET_KEY=your-secret-key SECRET_KEY=your-secret-key
DATABASE_URL=mysql+pymysql://pfm_user:password@localhost/pfm_db DATABASE_URL=mysql+pymysql://pfm_user:password@localhost/pfm_db
GROQ_API_KEY=your-groq-api-key-here GROQ_API_KEY=your-groq-api-key-here
GROQ_MODEL=llama-3.3-70b-versatile GROQ_MODEL=llama-3.3-70b-versatile
UPLOAD_FOLDER=/home/pfm/app/uploads UPLOAD_FOLDER=/home/pfm/web/uploads
MAX_CONTENT_LENGTH=10485760 MAX_CONTENT_LENGTH=10485760
FLASK_ENV=production FLASK_ENV=production
APP_CURRENCY=USD # Single currency for all transactions (configurable) FLASK_APP=wsgi:app
APP_CURRENCY_SYMBOL=$ # Display symbol APP_CURRENCY=USD
APP_TIMEZONE=Asia/Ho_Chi_Minh # Server timezone for scheduled jobs APP_CURRENCY_SYMBOL=$
APP_TIMEZONE=Asia/Ho_Chi_Minh
``` ```
--- ---
## 12. Blueprints Registered (11 total)
| Blueprint | Prefix | Key routes |
|-----------|--------|------------|
| auth | /auth | login, logout |
| dashboard | / | index, api/fx-history, api/fx-refresh |
| accounts | /accounts | CRUD |
| categories | /categories | CRUD |
| transactions | /transactions | index, new, edit, delete, transfer, ocr, ocr-file |
| budgets | /budgets | index, new, edit, delete, copy |
| goals | /goals | index, new, edit, delete, contribute, contributions |
| investments | /investments | index, new, detail, edit, delete, add_transaction, refresh-prices, api/price |
| reports | /reports | monthly, quarterly, yearly, tax, export/csv|excel|pdf, snapshot |
| ai | /ai | index, stream (SSE), history, generate-insight |
| settings | /settings | index, profile, password, recurring, import, upload_receipt, delete_receipt, view_receipt |
---
## 13. Security Notes ## 13. Security Notes
- All routes protected by `@login_required` - All routes `@login_required`
- CSRF protection via Flask-WTF - CSRF on all POST forms
- SQL injection prevented by SQLAlchemy ORM - SQLAlchemy ORM (no raw SQL)
- File upload validation (type + size limit) - Receipt file path: `os.path.basename()` prevents path traversal
- Nginx: restrict access to local network if desired
- HTTPS via Let's Encrypt (Certbot) — `pfm.ngodanguyen.tech` - HTTPS via Let's Encrypt (Certbot) — `pfm.ngodanguyen.tech`
- Groq API receives only anonymized transaction summaries (no account names/personal details in prompt) - Groq receives anonymised transaction summaries (no account/personal names)
- GROQ_API_KEY stored in `.env`, never exposed to frontend - `GROQ_API_KEY` in `.env` (chmod 600), never in frontend
--- ---
## 14. Future Enhancements (Post-MVP) ## 14. Known Issues / Notes
- Mobile app companion (iOS — fits your skill set) - `wsgi.py` has `sys.path.insert(0, ...)` fix — required because app deploys at `/home/pfm/web/` which would otherwise be treated as a Python package
- Bank statement auto-import (parse PDF bank statements) - FX rate widget: `open.er-api.com` may return stale values; yfinance is the reliable primary
- Multi-currency with live FX rates (via free API) - WeasyPrint PDF: requires `libpango*` system libs (included in deploy.md apt install)
- Expense photo OCR (extract amount from receipt image via Groq vision model) - Import preview uses Flask session to pass rows to confirm step — requires `SECRET_KEY` to be set
- Email/SMS alerts (integrate Twilio — you already know it from FaxDesk)
- Shared household mode (2 users) ---
## 15. Post-MVP Roadmap
- [ ] iOS companion app
- [ ] Bank statement PDF auto-import (parse PDF → extract transactions)
- [ ] Budget alerts + Twilio SMS/email notifications
- [ ] Shared household mode (2 users, row-level isolation)
- [ ] Mobile responsiveness pass
+689 -2
View File
@@ -1,3 +1,690 @@
# Personal-Finance-Management # PFM — Personal Finance Manager
Personal-Finance-Management system A self-hosted personal finance web application. Track income, expenses, investments, budgets, and savings goals. AI-powered financial assistant and receipt OCR built in.
**Live at:** https://pfm.ngodanguyen.tech
---
## Table of Contents
1. [Features Overview](#features-overview)
2. [Getting Started](#getting-started)
3. [Dashboard](#dashboard)
4. [Transactions](#transactions)
5. [Accounts](#accounts)
6. [Categories](#categories)
7. [Budgets](#budgets)
8. [Goals & Savings](#goals--savings)
9. [Investments](#investments)
10. [AI Assistant](#ai-assistant)
11. [Reports & Export](#reports--export)
12. [Settings](#settings)
13. [Recurring Transactions](#recurring-transactions)
14. [CSV Import](#csv-import)
15. [Receipt OCR](#receipt-ocr)
16. [USD → VND Rate Widget](#usd--vnd-rate-widget)
17. [Keyboard Shortcuts & Tips](#keyboard-shortcuts--tips)
---
## Features Overview
| Module | What it does |
|--------|-------------|
| **Dashboard** | Net worth, cash flow chart, budget gauges, AI insight, USD/VND rate |
| **Transactions** | Income + expense entry, transfer, filter, search, receipt upload |
| **Accounts** | Multiple bank/cash/credit accounts, auto-calculated balances |
| **Categories** | Custom expense + income categories with color and icon |
| **Budgets** | Monthly spending limits per category with progress tracking |
| **Goals** | Savings goals with contribution tracking and projected completion |
| **Investments** | Stock/ETF/crypto/real estate portfolio with live price fetch |
| **AI Assistant** | Chat with your finances via Groq (free), daily auto-insight |
| **Receipt OCR** | Drag-drop a receipt image — AI extracts and fills the form |
| **Reports** | Monthly/quarterly/yearly summaries, net worth history, PDF/CSV/Excel export |
| **Settings** | Profile, currency, recurring rules, CSV import, password |
---
## Getting Started
### First Login
1. Navigate to `https://pfm.ngodanguyen.tech`
2. Enter the username and password created during setup (`python scripts/init_db.py`)
3. Check "Remember me" to stay logged in across browser sessions
### Recommended Setup Order
1. **Add accounts** — add your bank accounts, cash wallet, and credit cards first (Settings sidebar → Accounts)
2. **Review categories** — default categories are already seeded; add custom ones if needed
3. **Set your currency** — go to Settings → Profile and set your app currency (USD, VND, etc.)
4. **Add transactions** — start entering income and expenses
5. **Set budgets** — once you have categories, set monthly limits
6. **Create goals** — add savings goals and start contributing
7. **Add investments** — track your portfolio holdings
8. **Set recurring rules** — automate repeating income/expenses
---
## Dashboard
The dashboard is the home screen — accessible from the sidebar or by clicking the PFM logo.
### Period Selector
Three buttons in the top-right of the topbar:
- **This Month** — income/expenses for the current calendar month
- **Last Month** — previous calendar month
- **Custom** — opens a date picker modal; choose any start and end date
The period affects the summary cards and recent transactions feed. Charts always show fixed windows (6 months for cash flow, etc.).
### Summary Cards
Four cards across the top:
- **Income** — total income for the selected period (green)
- **Expenses** — total expenses for the selected period (red)
- **Net Cash Flow** — income minus expenses; green if positive, red if negative
- **Net Worth** — total assets minus liabilities across all accounts
### Cash Flow Chart
Bar chart showing the last 6 months of income (green bars) vs expenses (red bars). Hover over bars to see exact amounts.
### Top Spending
Right of the cash flow chart — shows your top 5 expense categories for the period with horizontal progress bars. Widest bar = highest spending.
### USD → VND Widget
Dark card below the top spending section. Shows today's USD to VND exchange rate for reference. Click the widget to reveal a 30-day trend chart. Click the ↻ button to force-refresh the rate without reloading the page. See [USD → VND Rate Widget](#usd--vnd-rate-widget) for details.
### AI Daily Insight
Dark card above the accounts section. Shows a 35 sentence AI-generated summary of your finances for the day. Generated automatically at midnight. Click "Open AI →" to go to the full chat interface.
### Accounts Panel
Left of the recent transactions — lists all active accounts with their current balance. Green = positive, red = negative (credit card debt). Click "+" to add a new account.
### Recent Transactions
Last 8 income and expense transactions. Click "View all" to go to the full transaction list.
### Quick Add Buttons
Two buttons in the page header:
- **+Income** (green) — opens new income form
- **+Expense** (red) — opens new expense form
---
## Transactions
### Viewing Transactions
Navigate via **Transactions** in the sidebar. Two tabs at the top:
- **Expenses** — shows expense transactions with a count badge
- **Income** — shows income transactions with a count badge
Transactions are ordered newest first, paginated at 30 per page.
### Filtering
A filter bar appears below the tabs:
- **Search** — matches against the description field (case-insensitive)
- **Category** — filter by a single category
- **Account** — filter by a single account
- **From / To** — date range filter
- Click the magnifying glass button to apply filters
- Click the ✕ button to clear all filters and reset to defaults
### Adding a Transaction
**From the sidebar:** click "Add Income" or "Add Expense"
**From the dashboard:** use the +Income / +Expense buttons
**From the transaction list:** use the topbar buttons
#### New Expense / Income Form Fields
| Field | Required | Notes |
|-------|----------|-------|
| Description | Yes | What the transaction was for |
| Amount | Yes | Positive number only |
| Date | Yes | Defaults to today |
| Account | Yes | Which account to debit/credit |
| Category | No | Helps with budgets and reports |
| Notes | No | Free text, up to 500 characters |
#### AI Receipt Scanner (on new expense form)
A purple panel sits above the form. Drop a receipt image onto it (or click to browse). The AI extracts the amount, date, merchant name, and category and fills the form fields automatically. Review the filled values before saving — highlighted fields (green flash) show what was auto-filled. See [Receipt OCR](#receipt-ocr) for details.
### Editing a Transaction
Click **Edit** on any transaction row. Same form as adding, with an additional receipt section:
- If a receipt is attached: shows filename with a **Re-extract** button (re-runs OCR on the stored file) and a **Remove** button
- If no receipt: shows an upload field; selecting an image file auto-triggers OCR
### Deleting a Transaction
Click **Del** on any transaction row. A confirmation prompt appears. Deletion is permanent and account balances are recalculated immediately.
### Transfers Between Accounts
Click **Transfer** in the topbar. Select source account, destination account, amount, and date. This creates a single transfer transaction that debits the source and credits the destination. Transfers do not appear on the Income or Expense tabs — they are excluded from income/expense totals.
---
## Accounts
### Adding an Account
Settings sidebar → **Accounts****New Account** (or from dashboard Accounts panel → "+").
| Field | Notes |
|-------|-------|
| Account Name | e.g. "Chase Checking", "Cash Wallet", "Visa Card" |
| Type | checking / savings / cash / credit_card / crypto / investment / other |
| Color | Click a color swatch — used for visual identification |
| Icon | Click an icon swatch — appears throughout the app |
| Notes | Optional description |
### Account Balances
Balances are **automatically calculated** from all transactions linked to that account — you do not enter a balance manually. When you add a transaction, the account balance updates instantly.
**Formula:**
```
Balance = sum(income) - sum(expenses) - sum(transfers_out) + sum(transfers_in)
```
For a brand-new account with no transactions, balance is 0. To set a starting balance, add an income transaction dated your desired start date with the description "Opening balance."
### Removing an Account
Click the three-dot menu (⋯) on an account card → **Remove**. This is a soft delete — the account is hidden but its transactions remain in the database and still affect totals. You cannot permanently delete an account with transactions.
---
## Categories
Navigate via the sidebar footer → **Categories**.
Two panels: **Expense Categories** (left) and **Income Categories** (right).
### Default Categories
**Expense (14):** Housing, Food & Dining, Transport, Utilities, Health, Entertainment, Shopping, Education, Insurance, Personal Care, Travel, Subscriptions, Gifts, Other
**Income (7):** Salary, Freelance, Business, Investment, Rental, Gift Received, Other Income
System categories (marked with a grey "system" badge) cannot be deleted but can be edited (color and icon only — name is protected).
### Adding a Category
Click **+Expense** or **+Income** in the topbar. Set a name, type, color (click a swatch), and icon (click an icon swatch).
**Category type options:**
- **Expense** — appears only in expense transaction dropdown
- **Income** — appears only in income transaction dropdown
- **Both** — appears in both dropdowns
### Editing a Category
Click **Edit** next to any category. Color and icon can always be changed. Name can only be changed for non-system categories.
### Deleting a Category
Click **Del** next to a non-system category. Categories with existing transactions cannot be hard-deleted — they are deactivated (hidden) instead. A warning message explains this.
---
## Budgets
Navigate via **Budgets** in the sidebar.
### Month Navigation
Use the **◀** and **▶** arrows to move between months. The current month is the default.
### Setting a Budget
Click **Add Budget** in the topbar (or "Set Budget" next to an unbudgeted category).
| Field | Notes |
|-------|-------|
| Category | Expense categories only |
| Monthly Limit | Maximum spend for this category this month |
| Roll over unused amount | If checked, unspent budget carries forward to next month |
### Budget Progress Bars
Each category row shows:
- Category name and icon
- Amount spent (colored by status)
- Budget limit
- Progress bar: green (< 80%) → amber (8099%) → red (100%+)
- Remaining amount (or over-budget amount in red)
### Unbudgeted Spending
Categories with spending but no budget appear at the bottom of the table with a yellow "no budget" badge and a "Set Budget" button.
### Copy from Previous Month
Click **Copy from YYYY-MM** button (top right or empty state) to duplicate all budget entries from the previous month. Existing budgets for the current month are not overwritten — only missing ones are created.
### Editing / Deleting a Budget
Click **Edit** or **Del** on any budget row. Deleting a budget does not delete the transactions — it only removes the limit.
---
## Goals & Savings
Navigate via **Goals** in the sidebar.
### Emergency Fund Tracker
A yellow card at the top (appears once you have 3+ months of expense data). Shows:
- Average monthly expense (last 3 months)
- Liquid assets (checking + savings + cash accounts combined)
- 3-month target and 6-month target with progress bars
- Months covered: how long your liquid assets would last at current spending
### Creating a Goal
Click **New Goal** in the topbar.
| Field | Notes |
|-------|-------|
| Goal Name | e.g. "Vacation Fund", "Emergency Fund", "New Laptop" |
| Target Amount | How much you want to save |
| Target Date | Optional deadline |
| Linked Account | Optional — associates the goal with a specific account |
| Description | Optional notes |
| Color | Visual identifier (circle swatches) |
| Icon | Visual identifier (icon swatches) |
### Goal Cards
Each active goal shows:
- Progress bar (fills with goal color)
- Current amount saved / target amount
- Completion percentage
- Projected completion date (calculated from average monthly contribution history — only shown after 2+ contributions)
- Description (if set)
- **Add Contribution** button (styled in goal color)
### Adding a Contribution
Click **Add Contribution** on a goal card or from the three-dot menu. Enter amount, date, and optional notes. When the total reaches the target, the goal auto-completes and moves to the "Completed Goals" section.
### Contribution History
Three-dot menu → **History** — shows all contributions with dates and amounts, plus the projected completion date. Individual contributions can be deleted (recalculates goal total and un-completes if needed).
### Completed Goals
Shown at the bottom of the page in a compact table. Last 5 completed goals displayed.
---
## Investments
Navigate via **Investments** in the sidebar.
### Portfolio Overview
Summary cards at the top:
- **Total Value** — current market value of all holdings
- **Total Cost** — total amount invested (cost basis)
- **Unrealized P&L** — gain or loss vs cost basis
- **Return** — percentage return
**Allocation chart** — doughnut chart showing portfolio split by asset type (stock, ETF, crypto, real estate, bond, cash, other). Each slice is color-coded. Click a row in the legend for exact values.
**Holdings table** — all active holdings. Click any row to go to the detail page.
### Adding a Holding
Click **Add Holding** in the topbar.
| Field | Notes |
|-------|-------|
| Asset Name | Full name, e.g. "Apple Inc.", "Bitcoin" |
| Asset Type | stock / ETF / crypto / real_estate / bond / cash / other |
| Ticker Symbol | Yahoo Finance format: AAPL, BTC-USD, ETH-USD, VNM. Click **Check** to verify |
After saving, you are taken to the detail page to record your first buy transaction.
### Ticker Format (Yahoo Finance)
- US Stocks: `AAPL`, `MSFT`, `GOOGL`
- ETFs: `VOO`, `QQQ`, `VTI`
- Crypto: `BTC-USD`, `ETH-USD`, `BNB-USD`
- Vietnamese stocks: `VNM` (VanEck Vietnam ETF on NYSE)
- Other markets: use Yahoo Finance suffix, e.g. `VIC.VN` for Vingroup on HOSE
### Holding Detail Page
Shows the holding summary (shares, avg cost, current price, market value, P&L) and transaction history. Current price timestamp is shown below the price.
### Recording a Transaction
On the detail page, click **Add Transaction** in the topbar.
| Transaction Type | What it does |
|-----------------|-------------|
| **Buy** | Adds shares, increases cost basis |
| **Sell** | Reduces shares, adjusts cost basis (FIFO) |
| **Dividend** | Records a dividend payment (does not change shares) |
| **Split** | Adds shares without changing cost (stock split) |
The **↓ button** next to the price field fetches the current live price from Yahoo Finance and fills it in. The "Estimated Total" preview updates as you type shares and price.
After saving, shares and average cost basis are automatically recalculated from the full transaction history.
### Refreshing Prices
Click **Refresh Prices** in the topbar on the portfolio page to fetch current prices for all holdings with ticker symbols. Prices are also automatically updated daily at 4PM (weekdays) via a background job.
### Removing a Holding
On the detail page, click **Remove** (top right of the holding card). This is a soft delete — the holding is hidden but transaction history is preserved.
---
## AI Assistant
Navigate via **AI Assistant** in the sidebar.
### Chat Interface
Type a question in the text box at the bottom and press **Enter** (or click the send button). The response streams word-by-word in real time.
**Shift+Enter** adds a new line without sending.
The AI has access to:
- Last 90 days of transactions (description, category, amount, date)
- Current month income/expense totals and budget status
- Active savings goals with progress
- Investment portfolio summary
- Net worth
No personal names, account names, or identifying details are sent to Groq — only aggregated financial figures.
### Suggested Questions
Eight suggestion buttons appear on the right panel. Click any to fill the chat input:
- "Where did I overspend this month?"
- "How is my budget looking?"
- "Am I on track for my goals?"
- "What's my biggest expense category?"
- "Summarize my finances"
- "How can I save more?"
- "What's my net worth trend?"
- "Review my investments"
### Daily Insight
A dark card on the right shows today's auto-generated insight (35 sentences covering spending, budget alerts, and a tip). Generated automatically at midnight. If not yet generated for today, click **Generate Now**. Click **Regenerate** to get a fresh one.
### Chat History
Click **History** in the topbar. Shows all past chat responses and daily summaries with timestamps and token counts. Paginated at 20 per page.
### AI Model
The model can be changed in Settings → Profile → AI Model:
- `llama-3.3-70b-versatile` — best quality, slightly slower
- `llama-3.1-8b-instant` — faster, slightly less detailed
### Error States
- **"AI assistant is not configured"** — GROQ_API_KEY is missing from .env
- **"Rate limit reached"** — too many requests; wait a moment and retry
- **"Invalid Groq API key"** — check the key at console.groq.com
---
## Reports & Export
Navigate via **Reports** in the sidebar.
### Report Types
Four tabs at the top:
**Monthly** — income, expenses, net, savings rate for a single month. Bar chart + expense doughnut.
**Quarterly** — same metrics for a quarter (Q1Q4), plus monthly breakdown bars within the quarter.
**Yearly** — full year summary with monthly breakdown, average monthly income/expense.
**Tax Year** — income by source and expenses by category for a full calendar year, plus a full list of all income transactions for the year. Useful for tax preparation.
### Selecting a Period
Use the year/month/quarter dropdowns next to the period tabs and click **Go**.
### Charts
**Period chart** — bar chart of income vs expenses for the selected period (or monthly breakdown for quarterly/yearly).
**Expense breakdown** — doughnut chart of expenses by category. Hover for exact amounts. Top 5 categories listed below with amounts.
**Net worth history** — line chart of net worth over time (requires monthly snapshots to be saved). Dashed line shows total assets. Click **Snapshot Now** to save today's values.
**Category spending trends** — line chart of top 6 expense categories over the last 6 months. Useful for spotting trends.
### Exporting
Three export buttons in the topbar:
| Button | Format | Contents |
|--------|--------|----------|
| **CSV** | `.csv` | All transactions for the selected period, plain text |
| **Excel** | `.xlsx` | Color-coded rows (green=income, red=expense), formatted amounts, totals row |
| **PDF** | `.pdf` | Clean printable report with summary cards and expense breakdown table |
Exports use the period selected in the current report view (year + month/quarter).
---
## Settings
Navigate via the gear icon at the bottom of the sidebar, or Settings → Profile.
### Profile
| Setting | Notes |
|---------|-------|
| Display Name | Shown in the topbar |
| Email | For reference only (no email features yet) |
| Timezone | Used for scheduled jobs |
| Currency | App-wide currency code (USD, VND, EUR, etc.) |
| Currency Symbol | Auto-set when currency is changed |
| AI Model | Groq model for chat and daily insights |
Click **Save Profile** to apply. Currency and symbol changes take effect immediately throughout the app.
### Changing Password
Settings → Password. Requires current password + new password (min 6 characters) + confirmation.
---
## Recurring Transactions
Navigate via Settings → **Recurring** (also accessible from Settings landing page).
### What It Does
Recurring rules automatically create transactions on a schedule. At 6AM daily, the system checks all active rules and creates any transactions that are due (including any missed occurrences if the server was down).
### Creating a Rule
Click **New Rule**.
| Field | Notes |
|-------|-------|
| Name | Label for the rule, e.g. "Monthly Rent" |
| Type | income or expense |
| Description | Used as the transaction description |
| Amount | Fixed amount each occurrence |
| Frequency | daily / weekly / biweekly / monthly / quarterly / yearly |
| Account | Which account to debit/credit |
| Category | Optional |
| Start Date | First occurrence date |
| End Date | Optional — leave blank for no end date |
### Managing Rules
The rules table shows all rules with their next run date, frequency, and status. Available actions:
- **Edit** — change any field (does not affect already-created transactions)
- **Pause / Enable** — temporarily disable without deleting
- **×** — delete the rule (already-created transactions are preserved)
### Run Now
Click **Run Now** in the topbar to immediately process all due rules. Useful after creating a new rule that has a past start date, or after the server was offline.
### Upcoming Preview
Right panel shows all upcoming recurring transactions for the next 30 days in date order, with amounts and type badges.
---
## CSV Import
Navigate via Settings → **Import**.
### CSV Format
```
date,type,description,category,account,amount,notes
2025-01-15,expense,Groceries,Food & Dining,Checking,85.50,Weekly shop
2025-01-16,income,Salary,Salary,Checking,3000.00,
```
**Required columns:** `date`, `type`, `description`, `amount`
**Optional columns:** `category`, `account`, `notes`
**Accepted date formats:** `YYYY-MM-DD`, `MM/DD/YYYY`, `DD/MM/YYYY`
**Type values:** must be exactly `income` or `expense` (lowercase)
### Import Process
1. Choose your CSV file
2. Select a **Default Account** — used when the account column is missing or the name doesn't match any of your accounts
3. Check **Skip duplicate transactions** (recommended) — skips rows where date + description + amount + type exactly match an existing transaction
4. Click **Preview Import**
5. Review the preview table:
- ⚠ yellow warning on category = category name not found (will be uncategorised)
- ⚠ yellow warning on account = account not found (will use default account or be unlinked)
6. Click **Confirm Import** to save all rows
Account and category matching is case-insensitive. If your CSV has `food & dining` it will match the `Food & Dining` system category.
---
## Receipt OCR
Receipt OCR uses the Groq `llama-4-scout` vision model to extract transaction data from a photo of a receipt.
### On New Expense Form
A purple panel with a dashed border appears above the transaction form:
1. **Drag and drop** a receipt image onto the panel, OR click the panel to browse for a file
2. Accepted formats: JPG, PNG, GIF, WEBP (max 10MB — PDF not supported for OCR)
3. The panel shows a spinning animation while scanning
4. On success: the panel turns green and form fields flash green to show what was filled:
- **Description** ← merchant name (e.g. "McDonald's")
- **Amount** ← total from receipt
- **Date** ← date on receipt (falls back to today if not found)
- **Category** ← AI's best guess matched to your system categories
- **Notes** ← brief description
5. Review all fields before clicking Save — OCR is not perfect, especially on low-quality photos
6. You can drop another receipt to re-scan and overwrite the filled values
### On Edit Transaction Form
**If a receipt is already attached:**
- A purple **Re-extract** button appears next to the receipt filename
- Click it to re-run OCR on the stored file and update the form fields
**If no receipt is attached:**
- An upload field appears
- Selecting a JPG/PNG/WEBP file automatically triggers OCR and opens the scanner panel
### Tips for Best OCR Results
- Use a well-lit, in-focus photo
- Make sure the total amount is clearly visible
- Flatten crumpled receipts before photographing
- Portrait orientation works better than landscape
- Higher resolution = better accuracy
- If OCR misreads the amount, correct it manually — amounts are the most important field
### What OCR Cannot Do
- Read multi-page receipts (only the uploaded image is processed)
- Handle PDF receipts (PDF format is excluded from OCR; it can still be uploaded as an attachment)
- Guarantee 100% accuracy — always review extracted values before saving
---
## USD → VND Rate Widget
The dark widget on the dashboard shows the current USD to VND exchange rate. This is **for reference only** — it does not affect any transactions or calculations in the app.
### How the Rate is Fetched
1. **Primary:** Yahoo Finance forex (`USDVND=X` via yfinance) — most reliable
2. **Fallback:** ExchangeRate API (`open.er-api.com`) — free, no key required
3. **Stale fallback:** Last known rate from database — shown with a ⚠ indicator
The rate is cached once per day. The daily cron job at 8AM always force-fetches a fresh rate.
### Refreshing Manually
Click the **↻** button (top-right of the widget) to force-fetch a fresh rate without reloading the page. The rate, date, and source label update in-place.
### 30-Day History Chart
Click anywhere on the widget (except the ↻ button) to toggle a compact line chart showing the rate trend over the last 30 days.
### Stale Indicator
If the cached rate is from a previous day and all live sources fail, a ⚠ symbol appears next to the date. This typically means the server has no internet access or the APIs are temporarily unavailable.
---
## Keyboard Shortcuts & Tips
### Navigation
- The sidebar collapses on desktop — click the ☰ button in the topbar to toggle. State is remembered across page loads.
- On mobile, the sidebar slides in as an overlay — tap anywhere outside to close it.
### Forms
- On the AI chat input: **Enter** sends the message, **Shift+Enter** adds a new line
- On transaction forms: the date field defaults to today — change it if entering a past transaction
- On the investment transaction form: the **↓** button next to price fetches the current live price
### Transaction Filters
- Filters persist within a tab session but reset when you switch tabs (Income ↔ Expense)
- Use the ✕ button to clear all filters at once
- Date range filter: both From and To are optional — leave one blank to filter from/to open-ended
### Budgets
- The budget list for an empty month shows a "Copy from previous month" button — use this at the start of each month instead of re-entering all budgets
- Rollover amounts appear as a blue "+rollover" badge — hover to see the exact amount
### Investments
- Click any row in the holdings table to go to the detail page
- Ticker symbols are case-insensitive on entry (auto-uppercased on save)
- The **Check** button on the ticker field verifies the ticker and shows the current price before you save
### Reports
- The "Snapshot Now" button on the Reports page saves today's net worth to the history chart — do this manually if you want more data points than the monthly automatic snapshots
- CSV and Excel exports use the period currently selected in the report view
### AI Assistant
- The AI does not have memory between sessions — each conversation starts fresh
- For best results, ask specific questions: "How much did I spend on food in March?" rather than "How am I doing?"
- The context includes the last 90 days of transactions — questions about older data may not be accurate
+235 -248
View File
@@ -1,5 +1,5 @@
# PFMS — Server Deployment Guide # PFM — Server Deployment Guide
> Ubuntu Server · Nginx · Gunicorn · MySQL · Groq API > Ubuntu Server · Nginx · Gunicorn · MySQL · Groq API · Certbot SSL
--- ---
@@ -9,11 +9,16 @@
|------|-------| |------|-------|
| OS | Ubuntu 22.04 LTS | | OS | Ubuntu 22.04 LTS |
| App user | `pfm` | | App user | `pfm` |
| App directory | `/home/pfm/app` | | Home dir | `/home/pfm` |
| Domain / IP | `pfm.ngodanguyen.tech` | | App directory | `/home/pfm/web` |
| Venv | `/home/pfm/venv` |
| Uploads | `/home/pfm/web/uploads` |
| Backups | `/home/pfm/web/backups` |
| Domain | `pfm.ngodanguyen.tech` |
| SSL | Let's Encrypt via Certbot | | SSL | Let's Encrypt via Certbot |
| MySQL version | 8.0 | | MySQL | 8.0 |
| Python | 3.11+ | | Python | 3.11+ |
| Gitea | `gitea.ngodanguyen.tech` |
--- ---
@@ -21,21 +26,22 @@
```bash ```bash
sudo apt update && sudo apt upgrade -y sudo apt update && sudo apt upgrade -y
sudo apt install -y python3 python3-pip python3-venv python3-dev \ sudo apt install -y \
python3 python3-pip python3-venv python3-dev \
build-essential libssl-dev libffi-dev \ build-essential libssl-dev libffi-dev \
nginx mysql-server git curl wget \ nginx mysql-server git curl wget unzip \
libpango-1.0-0 libpangoft2-1.0-0 libharfbuzz0b \ libpango-1.0-0 libpangoft2-1.0-0 libharfbuzz0b \
libffi-dev libjpeg-dev libopenjp2-7-dev libjpeg-dev libopenjp2-7-dev
``` ```
> `libpango*` and related libs are required by WeasyPrint for PDF generation. > `libpango*` libs required by WeasyPrint for PDF generation.
--- ---
## 2. Create App User ## 2. Create App User
```bash ```bash
# Skip if user pfm already exists # Skip if pfm already exists
id pfm &>/dev/null || sudo useradd -m -s /bin/bash pfm id pfm &>/dev/null || sudo useradd -m -s /bin/bash pfm
sudo usermod -aG www-data pfm sudo usermod -aG www-data pfm
``` ```
@@ -46,7 +52,7 @@ sudo usermod -aG www-data pfm
```bash ```bash
sudo mysql_secure_installation sudo mysql_secure_installation
# Follow prompts: set root password, remove anonymous users, disable remote root # Set root password, remove anonymous users, disable remote root
``` ```
```bash ```bash
@@ -66,18 +72,21 @@ EXIT;
## 4. App Directory & Virtual Environment ## 4. App Directory & Virtual Environment
```bash ```bash
sudo mkdir -p /home/pfm/app sudo mkdir -p /home/pfm/web
sudo chown pfm:www-data /home/pfm/app sudo chown pfm:www-data /home/pfm/web
sudo chmod 750 /home/pfm/app sudo chmod 750 /home/pfm/web
sudo -u pfm bash sudo -u pfm bash
cd /home/pfm/app cd /home/pfm
# Create venv at /home/pfm/venv
python3 -m venv venv python3 -m venv venv
source venv/bin/activate source venv/bin/activate
# Clone from your Gitea # Clone from Gitea
git clone https://gitea.yourdomain.com/youruser/pfm.git . cd /home/pfm/web
GIT_SSH_COMMAND='ssh -i /home/pfm/.ssh/gitea_deploy' \
git clone git@gitea.ngodanguyen.tech:youruser/pfm.git .
pip install --upgrade pip pip install --upgrade pip
pip install -r requirements.txt pip install -r requirements.txt
@@ -88,7 +97,7 @@ pip install -r requirements.txt
## 5. Environment File ## 5. Environment File
```bash ```bash
sudo -u pfm nano /home/pfm/app/.env sudo -u pfm nano /home/pfm/web/.env
``` ```
```env ```env
@@ -96,43 +105,53 @@ SECRET_KEY=generate-a-long-random-string-here
DATABASE_URL=mysql+pymysql://pfm_user:STRONG_PASSWORD_HERE@localhost/pfm_db DATABASE_URL=mysql+pymysql://pfm_user:STRONG_PASSWORD_HERE@localhost/pfm_db
GROQ_API_KEY=your-groq-api-key-here GROQ_API_KEY=your-groq-api-key-here
GROQ_MODEL=llama-3.3-70b-versatile GROQ_MODEL=llama-3.3-70b-versatile
UPLOAD_FOLDER=/home/pfm/app/uploads UPLOAD_FOLDER=/home/pfm/web/uploads
MAX_CONTENT_LENGTH=10485760 MAX_CONTENT_LENGTH=10485760
FLASK_ENV=production FLASK_ENV=production
FLASK_APP=wsgi:app
APP_CURRENCY=USD APP_CURRENCY=USD
APP_CURRENCY_SYMBOL=$ APP_CURRENCY_SYMBOL=$
APP_TIMEZONE=America/New_York APP_TIMEZONE=Asia/Ho_Chi_Minh
``` ```
```bash ```bash
chmod 600 /home/pfm/app/.env chmod 600 /home/pfm/web/.env
``` ```
Generate a secret key: Generate secret key:
```bash ```bash
python3 -c "import secrets; print(secrets.token_hex(32))" python3 -c "import secrets; print(secrets.token_hex(32))"
``` ```
--- ---
## 6. Initialize Database ## 6. Uploads & Backups Directories
```bash ```bash
cd /home/pfm/app sudo mkdir -p /home/pfm/web/uploads
source venv/bin/activate sudo mkdir -p /home/pfm/web/backups
sudo chown pfm:www-data /home/pfm/web/uploads
flask db upgrade # Run all migrations sudo chown pfm:pfm /home/pfm/web/backups
python scripts/init_db.py # Seed default categories + create admin user sudo chmod 750 /home/pfm/web/uploads
``` ```
--- ---
## 7. Upload Directory ## 7. Initialize Database
```bash ```bash
sudo mkdir -p /home/pfm/app/uploads cd /home/pfm/web
sudo chown pfm:www-data /home/pfm/app/uploads source /home/pfm/venv/bin/activate
sudo chmod 750 /home/pfm/app/uploads
# Remove stray __init__.py from project root if present
rm -f /home/pfm/web/__init__.py
flask db init
flask db migrate -m "initial schema"
flask db upgrade
# Seed categories + create admin user (interactive)
python scripts/init_db.py
``` ```
--- ---
@@ -145,16 +164,17 @@ sudo nano /etc/systemd/system/pfm.service
```ini ```ini
[Unit] [Unit]
Description=PFMS — Personal Finance Management System Description=PFM — Personal Finance Management System
After=network.target mysql.service After=network.target mysql.service
[Service] [Service]
User=pfm User=pfm
Group=www-data Group=www-data
WorkingDirectory=/home/pfm/app WorkingDirectory=/home/pfm/web
Environment="PATH=/home/pfm/app/venv/bin" Environment="PATH=/home/pfm/venv/bin"
EnvironmentFile=/home/pfm/app/.env EnvironmentFile=/home/pfm/web/.env
ExecStart=/home/pfm/app/venv/bin/gunicorn \ ExecStart=/home/pfm/venv/bin/gunicorn \
--chdir /home/pfm/web \
--workers 2 \ --workers 2 \
--bind unix:/run/pfm/pfm.sock \ --bind unix:/run/pfm/pfm.sock \
--timeout 120 \ --timeout 120 \
@@ -173,7 +193,6 @@ WantedBy=multi-user.target
``` ```
```bash ```bash
# Create log directory
sudo mkdir -p /var/log/pfm sudo mkdir -p /var/log/pfm
sudo chown pfm:www-data /var/log/pfm sudo chown pfm:www-data /var/log/pfm
@@ -195,7 +214,6 @@ sudo nano /etc/nginx/sites-available/pfm
server { server {
listen 80; listen 80;
server_name pfm.ngodanguyen.tech; server_name pfm.ngodanguyen.tech;
# Certbot will update this block automatically
return 301 https://$host$request_uri; return 301 https://$host$request_uri;
} }
@@ -203,7 +221,7 @@ server {
listen 443 ssl http2; listen 443 ssl http2;
server_name pfm.ngodanguyen.tech; server_name pfm.ngodanguyen.tech;
# Certbot will populate these after running certbot # Certbot populates these:
# ssl_certificate /etc/letsencrypt/live/pfm.ngodanguyen.tech/fullchain.pem; # ssl_certificate /etc/letsencrypt/live/pfm.ngodanguyen.tech/fullchain.pem;
# ssl_certificate_key /etc/letsencrypt/live/pfm.ngodanguyen.tech/privkey.pem; # ssl_certificate_key /etc/letsencrypt/live/pfm.ngodanguyen.tech/privkey.pem;
# include /etc/letsencrypt/options-ssl-nginx.conf; # include /etc/letsencrypt/options-ssl-nginx.conf;
@@ -221,7 +239,7 @@ server {
proxy_connect_timeout 10s; proxy_connect_timeout 10s;
} }
# AI chat uses SSE — disable buffering # AI chat SSE — disable buffering
location /ai/stream { location /ai/stream {
proxy_pass http://unix:/run/pfm/pfm.sock; proxy_pass http://unix:/run/pfm/pfm.sock;
proxy_set_header Host $host; proxy_set_header Host $host;
@@ -235,13 +253,14 @@ server {
} }
location /static/ { location /static/ {
alias /home/pfm/app/app/static/; alias /home/pfm/web/app/static/;
expires 30d; expires 30d;
add_header Cache-Control "public, immutable"; add_header Cache-Control "public, immutable";
} }
# Receipts — served via Flask send_from_directory (internal)
location /uploads/ { location /uploads/ {
alias /home/pfm/app/uploads/; alias /home/pfm/web/uploads/;
internal; internal;
} }
@@ -261,78 +280,46 @@ sudo systemctl reload nginx
## 9a. SSL — Certbot (Let's Encrypt) ## 9a. SSL — Certbot (Let's Encrypt)
```bash ```bash
# Install Certbot + Nginx plugin
sudo apt install -y certbot python3-certbot-nginx sudo apt install -y certbot python3-certbot-nginx
# Issue certificate — auto-edits the Nginx config above # Issue cert — auto-edits Nginx config
sudo certbot --nginx -d pfm.ngodanguyen.tech sudo certbot --nginx -d pfm.ngodanguyen.tech
# Follow prompts: # Verify
# - Enter email for renewal notices
# - Agree to TOS
# - Choose redirect HTTP → HTTPS (recommended)
# Verify cert
sudo certbot certificates sudo certbot certificates
```
Certbot automatically:
- Populates the `ssl_certificate` lines in the Nginx config
- Sets up HTTP → HTTPS redirect
- Installs a systemd timer for auto-renewal
```bash
# Test auto-renewal # Test auto-renewal
sudo certbot renew --dry-run sudo certbot renew --dry-run
# Check renewal timer
sudo systemctl status certbot.timer sudo systemctl status certbot.timer
``` ```
--- ---
## 10. Groq API Setup (Free AI) ## 10. Groq API Setup
No local installation needed. Groq runs in the cloud — just get a free API key.
### 10.1 Get API Key
1. Sign up at **https://console.groq.com** 1. Sign up at **https://console.groq.com**
2. Go to **API Keys****Create API key** 2. API Keys → Create API key
3. Copy the key into `/home/pfm/app/.env` as `GROQ_API_KEY` 3. Add to `/home/pfm/web/.env` as `GROQ_API_KEY`
### 10.2 Free Tier Limits ### Models in Use
| Metric | Limit | | Model | Used For |
|--------|-------| |-------|----------|
| Requests/day | 14,400 | | `llama-3.3-70b-versatile` | Chat + daily insights (default) |
| Tokens/minute | 500,000 | | `llama-3.1-8b-instant` | Fast queries (user-selectable) |
| Tokens/day | 500,000 (varies by model) | | `meta-llama/llama-4-scout-17b-16e-instruct` | Receipt OCR (vision) — hardcoded in ocr_service.py |
| Cost | Free |
More than enough for a personal finance assistant. ### Free Tier Limits
14,400 requests/day · 500,000 tokens/minute · Free
### 10.3 Recommended Models
| Model | Speed | Quality | Best For |
|-------|-------|---------|----------|
| `llama-3.3-70b-versatile` | Fast | High | Default — chat + insights |
| `llama-3.1-8b-instant` | Very fast | Good | Quick queries, daily summaries |
Set in `.env` via `GROQ_MODEL`. Switchable from Settings UI.
### 10.4 Verify Key Works
### Verify Key
```bash ```bash
curl https://api.groq.com/openai/v1/chat/completions -H "Authorization: Bearer YOUR_GROQ_API_KEY" -H "Content-Type: application/json" -d '{ curl https://api.groq.com/openai/v1/chat/completions \
"model": "llama-3.3-70b-versatile", -H "Authorization: Bearer YOUR_GROQ_API_KEY" \
"messages": [{"role": "user", "content": "Hello"}], -H "Content-Type: application/json" \
"max_tokens": 10 -d '{"model":"llama-3.3-70b-versatile","messages":[{"role":"user","content":"Hello"}],"max_tokens":10}'
}'
``` ```
Should return a JSON response with a greeting.
--- ---
## 11. Scheduled Jobs (systemd timers) ## 11. Scheduled Jobs (systemd timers)
@@ -342,28 +329,25 @@ Should return a JSON response with a greeting.
```bash ```bash
sudo nano /etc/systemd/system/pfm-recurring.service sudo nano /etc/systemd/system/pfm-recurring.service
``` ```
```ini ```ini
[Unit] [Unit]
Description=PFMS — Process Recurring Transactions Description=PFM — Process Recurring Transactions
After=mysql.service After=mysql.service
[Service] [Service]
Type=oneshot Type=oneshot
User=pfm User=pfm
WorkingDirectory=/home/pfm/app WorkingDirectory=/home/pfm/web
Environment="PATH=/home/pfm/app/venv/bin" Environment="PATH=/home/pfm/venv/bin"
EnvironmentFile=/home/pfm/app/.env EnvironmentFile=/home/pfm/web/.env
ExecStart=/home/pfm/app/venv/bin/python scripts/process_recurring.py ExecStart=/home/pfm/venv/bin/python scripts/process_recurring.py
``` ```
```bash ```bash
sudo nano /etc/systemd/system/pfm-recurring.timer sudo nano /etc/systemd/system/pfm-recurring.timer
``` ```
```ini ```ini
[Unit] [Unit]
Description=PFMS — Recurring Transactions Timer Description=PFM — Recurring Transactions Timer
[Timer] [Timer]
OnCalendar=*-*-* 06:00:00 OnCalendar=*-*-* 06:00:00
@@ -373,32 +357,61 @@ Persistent=true
WantedBy=timers.target WantedBy=timers.target
``` ```
### 11.2 Fetch Investment Prices (Daily 4PM weekdays) ### 11.2 Fetch USD/VND Rate (Daily 8AM)
```bash ```bash
sudo nano /etc/systemd/system/pfm-prices.service sudo nano /etc/systemd/system/pfm-fxrate.service
``` ```
```ini ```ini
[Unit] [Unit]
Description=PFMS — Fetch Investment Prices Description=PFM — Fetch USD/VND Exchange Rate
[Service] [Service]
Type=oneshot Type=oneshot
User=pfm User=pfm
WorkingDirectory=/home/pfm/app WorkingDirectory=/home/pfm/web
Environment="PATH=/home/pfm/app/venv/bin" Environment="PATH=/home/pfm/venv/bin"
EnvironmentFile=/home/pfm/app/.env EnvironmentFile=/home/pfm/web/.env
ExecStart=/home/pfm/app/venv/bin/python scripts/fetch_prices.py ExecStart=/home/pfm/venv/bin/python scripts/fetch_fx_rate.py
```
```bash
sudo nano /etc/systemd/system/pfm-fxrate.timer
```
```ini
[Unit]
Description=PFM — FX Rate Timer
[Timer]
OnCalendar=*-*-* 08:00:00
Persistent=true
[Install]
WantedBy=timers.target
``` ```
### 11.3 Fetch Investment Prices (Mon-Fri 4PM)
```bash
sudo nano /etc/systemd/system/pfm-prices.service
```
```ini
[Unit]
Description=PFM — Fetch Investment Prices
[Service]
Type=oneshot
User=pfm
WorkingDirectory=/home/pfm/web
Environment="PATH=/home/pfm/venv/bin"
EnvironmentFile=/home/pfm/web/.env
ExecStart=/home/pfm/venv/bin/python scripts/fetch_prices.py
```
```bash ```bash
sudo nano /etc/systemd/system/pfm-prices.timer sudo nano /etc/systemd/system/pfm-prices.timer
``` ```
```ini ```ini
[Unit] [Unit]
Description=PFMS — Price Fetch Timer Description=PFM — Price Fetch Timer
[Timer] [Timer]
OnCalendar=Mon-Fri *-*-* 16:00:00 OnCalendar=Mon-Fri *-*-* 16:00:00
@@ -408,32 +421,29 @@ Persistent=true
WantedBy=timers.target WantedBy=timers.target
``` ```
### 11.3 Net Worth Snapshot (1st of month) ### 11.4 Net Worth Snapshot (1st of month)
```bash ```bash
sudo nano /etc/systemd/system/pfm-snapshot.service sudo nano /etc/systemd/system/pfm-snapshot.service
``` ```
```ini ```ini
[Unit] [Unit]
Description=PFMS — Net Worth Snapshot Description=PFM — Net Worth Snapshot
[Service] [Service]
Type=oneshot Type=oneshot
User=pfm User=pfm
WorkingDirectory=/home/pfm/app WorkingDirectory=/home/pfm/web
Environment="PATH=/home/pfm/app/venv/bin" Environment="PATH=/home/pfm/venv/bin"
EnvironmentFile=/home/pfm/app/.env EnvironmentFile=/home/pfm/web/.env
ExecStart=/home/pfm/app/venv/bin/python scripts/daily_snapshot.py ExecStart=/home/pfm/venv/bin/python scripts/daily_snapshot.py
``` ```
```bash ```bash
sudo nano /etc/systemd/system/pfm-snapshot.timer sudo nano /etc/systemd/system/pfm-snapshot.timer
``` ```
```ini ```ini
[Unit] [Unit]
Description=PFMS Net Worth Snapshot Timer Description=PFM — Snapshot Timer
[Timer] [Timer]
OnCalendar=*-*-01 00:05:00 OnCalendar=*-*-01 00:05:00
@@ -443,33 +453,30 @@ Persistent=true
WantedBy=timers.target WantedBy=timers.target
``` ```
### 11.4 AI Daily Insight (Midnight) ### 11.5 AI Daily Insight (Midnight)
```bash ```bash
sudo nano /etc/systemd/system/pfm-aiinsight.service sudo nano /etc/systemd/system/pfm-aiinsight.service
``` ```
```ini ```ini
[Unit] [Unit]
Description=PFMS — AI Daily Insight Description=PFM — AI Daily Insight
After=network-online.target After=network-online.target
[Service] [Service]
Type=oneshot Type=oneshot
User=pfm User=pfm
WorkingDirectory=/home/pfm/app WorkingDirectory=/home/pfm/web
Environment="PATH=/home/pfm/app/venv/bin" Environment="PATH=/home/pfm/venv/bin"
EnvironmentFile=/home/pfm/app/.env EnvironmentFile=/home/pfm/web/.env
ExecStart=/home/pfm/app/venv/bin/python scripts/daily_ai_insight.py ExecStart=/home/pfm/venv/bin/python scripts/daily_ai_insight.py
``` ```
```bash ```bash
sudo nano /etc/systemd/system/pfm-aiinsight.timer sudo nano /etc/systemd/system/pfm-aiinsight.timer
``` ```
```ini ```ini
[Unit] [Unit]
Description=PFMS — AI Insight Timer Description=PFM — AI Insight Timer
[Timer] [Timer]
OnCalendar=*-*-* 00:01:00 OnCalendar=*-*-* 00:01:00
@@ -479,41 +486,6 @@ Persistent=true
WantedBy=timers.target WantedBy=timers.target
``` ```
### 11.5 Fetch USD/VND Rate (Daily 8AM)
```bash
sudo nano /etc/systemd/system/pfm-fxrate.service
```
```ini
[Unit]
Description=PFMS — Fetch USD/VND Exchange Rate
[Service]
Type=oneshot
User=pfm
WorkingDirectory=/home/pfm/app
Environment="PATH=/home/pfm/app/venv/bin"
EnvironmentFile=/home/pfm/app/.env
ExecStart=/home/pfm/app/venv/bin/python scripts/fetch_fx_rate.py
```
```bash
sudo nano /etc/systemd/system/pfm-fxrate.timer
```
```ini
[Unit]
Description=PFMS — FX Rate Fetch Timer
[Timer]
OnCalendar=*-*-* 08:00:00
Persistent=true
[Install]
WantedBy=timers.target
```
### Enable All Timers ### Enable All Timers
```bash ```bash
@@ -530,12 +502,56 @@ sudo systemctl list-timers --all | grep pfm
--- ---
## 12. Log Rotation ## 12. Database Backup (Daily 2AM)
```bash
sudo nano /etc/systemd/system/pfm-backup.service
```
```ini
[Unit]
Description=PFM — MySQL Backup
[Service]
Type=oneshot
User=pfm
ExecStart=/bin/bash -c 'mysqldump --defaults-file=/home/pfm/.my.cnf pfm_db | gzip > /home/pfm/web/backups/pfm_$(date +\%Y\%m\%d).sql.gz && find /home/pfm/web/backups -name "*.sql.gz" -mtime +30 -delete'
```
```bash
sudo nano /etc/systemd/system/pfm-backup.timer
```
```ini
[Unit]
Description=PFM — Backup Timer
[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true
[Install]
WantedBy=timers.target
```
```bash
sudo systemctl enable --now pfm-backup.timer
```
Store MySQL credentials for backup job:
```bash
sudo -u pfm bash -c 'cat > /home/pfm/.my.cnf << EOF
[client]
user=pfm_user
password=STRONG_PASSWORD_HERE
EOF
chmod 600 /home/pfm/.my.cnf'
```
---
## 13. Log Rotation
```bash ```bash
sudo nano /etc/logrotate.d/pfm sudo nano /etc/logrotate.d/pfm
``` ```
``` ```
/var/log/pfm/*.log { /var/log/pfm/*.log {
daily daily
@@ -553,59 +569,6 @@ sudo nano /etc/logrotate.d/pfm
--- ---
## 13. Database Backup (Daily)
```bash
sudo nano /etc/systemd/system/pfm-backup.service
```
```ini
[Unit]
Description=PFMS — MySQL Backup
[Service]
Type=oneshot
User=pfm
ExecStart=/bin/bash -c 'mysqldump --defaults-file=/home/pfm/.my.cnf pfm_db | gzip > /home/pfm/app/backups/pfm_$(date +\%Y\%m\%d).sql.gz'
```
```bash
sudo nano /etc/systemd/system/pfm-backup.timer
```
```ini
[Unit]
Description=PFMS — Daily Backup Timer
[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true
[Install]
WantedBy=timers.target
```
```bash
sudo mkdir -p /home/pfm/app/backups
sudo chown pfm:pfm /home/pfm/app/backups
sudo systemctl enable --now pfm-backup.timer
```
> Store MySQL credentials securely for the backup job:
> ```bash
> sudo -u pfm bash -c 'cat > /home/pfm/.my.cnf << EOF
> [client]
> user=pfm_user
> password=STRONG_PASSWORD_HERE
> EOF
> chmod 600 /home/pfm/.my.cnf'
> ```
> Keep last 30 days. Add a cleanup cron to delete older files:
> `find /home/pfm/app/backups -name "*.sql.gz" -mtime +30 -delete`
---
## 14. Firewall (UFW) ## 14. Firewall (UFW)
```bash ```bash
@@ -616,21 +579,30 @@ sudo ufw enable
sudo ufw status sudo ufw status
``` ```
> MySQL (3306) is NOT opened — internal only. Groq calls go outbound over HTTPS (port 443). > MySQL (3306) — internal only. Groq/yfinance — outbound HTTPS (443).
--- ---
## 15. Deploy Update Workflow ## 15. Gitea Deploy Key
```bash
sudo -u pfm ssh-keygen -t ed25519 -f /home/pfm/.ssh/gitea_deploy -N ""
cat /home/pfm/.ssh/gitea_deploy.pub
# Add to Gitea: repo → Settings → Deploy Keys
```
---
## 16. Deploy Update Workflow
```bash ```bash
# SSH into server
sudo -u pfm bash sudo -u pfm bash
cd /home/pfm/app cd /home/pfm/web
source venv/bin/activate source /home/pfm/venv/bin/activate
git pull origin main git pull origin main
pip install -r requirements.txt # if deps changed pip install -r requirements.txt # only if requirements changed
flask db upgrade # if migrations added flask db upgrade # only if new migrations
sudo systemctl restart pfm sudo systemctl restart pfm
sudo systemctl status pfm sudo systemctl status pfm
@@ -638,35 +610,50 @@ sudo systemctl status pfm
--- ---
## 16. Quick Status Check ## 17. First-Run Checklist
```bash ```bash
# All PFMS services # 1. Force fresh FX rate
sudo systemctl status pfm source /home/pfm/venv/bin/activate
sudo systemctl status nginx cd /home/pfm/web
sudo systemctl status mysql python scripts/fetch_fx_rate.py
# Should print: 1 USD = ₫26,xxx VND [yfinance]
# Logs (live) # 2. Verify all timers
sudo journalctl -u pfm -f sudo systemctl list-timers --all | grep pfm
sudo tail -f /var/log/pfm/error.log
sudo tail -f /var/log/nginx/pfm_error.log
# Groq API check # 3. Test AI
curl https://api.groq.com/openai/v1/models \ curl https://api.groq.com/openai/v1/models \
-H "Authorization: Bearer $(grep GROQ_API_KEY /home/pfm/app/.env | cut -d= -f2)" -H "Authorization: Bearer $(grep GROQ_API_KEY /home/pfm/web/.env | cut -d= -f2)"
# 4. Check app logs
sudo journalctl -u pfm -f
# 5. Full service status
sudo systemctl status pfm nginx mysql
``` ```
--- ---
## 17. Gitea Integration (your self-hosted) ## 18. Quick Status Reference
```bash ```bash
# On server — set up deploy key # Services
sudo -u pfm ssh-keygen -t ed25519 -f /home/pfm/.ssh/gitea_deploy -N "" sudo systemctl status pfm
cat /home/pfm/.ssh/gitea_deploy.pub sudo systemctl status nginx
# Add this public key to Gitea repo → Settings → Deploy Keys sudo systemctl status mysql
# Clone with deploy key # Live logs
sudo -u pfm GIT_SSH_COMMAND='ssh -i /home/pfm/.ssh/gitea_deploy' \ sudo journalctl -u pfm -f
git clone git@gitea.yourdomain.com:youruser/pfm.git /home/pfm/app sudo tail -f /var/log/pfm/error.log
sudo tail -f /var/log/nginx/pfm_error.log
# Timers
sudo systemctl list-timers --all | grep pfm
# Run a cron manually
source /home/pfm/venv/bin/activate && cd /home/pfm/web
python scripts/fetch_fx_rate.py
python scripts/process_recurring.py
python scripts/daily_ai_insight.py
``` ```