diff --git a/CLAUDE.md b/CLAUDE.md index 763912c..c0c1146 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,236 +1,262 @@ -# Personal Finance Management System (PFMS) +# Personal Finance Management System (PFM) > Stack: Python Flask · MySQL · Ubuntu Server · Nginx · Gunicorn · Groq API (free AI) +> App URL: https://pfm.ngodanguyen.tech +> Code: /home/pfm/web · User: pfm · Gitea: gitea.ngodanguyen.tech --- ## 1. Project Overview -Self-hosted personal finance web app. Tracks income, expenses, investments. AI assistant powered by **Groq API** (free tier, extremely fast inference, no local hardware needed). Everything runs on your Ubuntu server behind Nginx. +Self-hosted personal finance web app. Tracks income, expenses, investments. AI assistant powered by **Groq API** (free tier, fast inference, no local hardware). Receipt OCR via Groq vision model. Everything runs on Ubuntu server behind Nginx + Certbot SSL. + +**Status: All 7 phases complete + Receipt OCR post-MVP feature.** --- -## 2. Core Features +## 2. Core Features (Implemented) ### 2.1 Dashboard - Net worth snapshot (assets − liabilities) -- Monthly cash flow chart (income vs expenses) -- Budget utilization gauges per category -- Recent transactions feed -- AI insight card (auto-generated daily summary) -- Investment portfolio mini-widget -- **USD → VND exchange rate widget** (daily rate, fetched once/day, cached in DB — reference only, independent of app currency) +- Monthly cash flow bar chart (6 months) +- Budget utilization per category +- Recent transactions feed (last 8) +- AI daily insight card (Groq-generated, stored in DB) +- USD → VND exchange rate widget — reference only, independent of app currency + - Click to expand 30-day history chart + - ↻ refresh button (force-fetches fresh rate without page reload) + - Source label shown (yfinance / exchangerate-api) + - Stale indicator if rate > 1 day old +- Period selector: This Month / Last Month / Custom date range -### 2.2 Income Management -- Log income entries (salary, freelance, passive, other) -- Recurring income templates (auto-create entries on schedule) -- Income source breakdown (chart by source) -- Month-over-month comparison -- Export to CSV/Excel +### 2.2 Transactions +- Income + Expense entry with Income/Expense tabs +- Transfer between accounts +- Filter: search, category, account, date range +- Pagination (30/page) +- Receipt upload (PNG/JPG/WEBP/GIF/PDF, max 10MB) +- **AI Receipt OCR** — drag-drop receipt image → Groq vision extracts amount/date/merchant/category → auto-fills form +- Re-extract from already-uploaded receipt (edit mode) +- Export to CSV / Excel -### 2.3 Expense Management -- Manual expense entry -- Category tagging (custom + predefined: Food, Rent, Utilities, Transport, Health, Entertainment, etc.) -- Subcategory support -- Receipt photo upload (stored locally) -- Recurring expense detection -- Budget limits per category with alert thresholds -- Expense search + filter (date range, category, amount range, keyword) -- Export to CSV/Excel +### 2.3 Accounts +- Types: checking, savings, cash, credit_card, crypto, investment, other +- Balance auto-calculated from all transactions (not manually entered) +- Color + icon picker +- Soft delete -### 2.4 Investment Portfolio -- Asset types: Stocks, ETF, Crypto, Real Estate, Bonds, Cash, Other -- Holdings tracker (ticker, shares/units, buy price, current price) -- Manual price update OR auto-fetch via free API (Yahoo Finance via `yfinance`) -- P&L per holding (unrealized gain/loss) -- Portfolio allocation pie chart -- Transaction log (buy/sell history per asset) -- Cost basis tracking (FIFO) +### 2.4 Categories +- Expense + Income categories with color/icon +- System categories (protected from delete) +- Custom categories (user-created) +- 14 expense + 7 income defaults seeded on init ### 2.5 Budget Planner -- Monthly budget templates -- Set budget limits per category -- Real-time spending vs budget comparison -- Rollover unused budget (optional toggle) -- Budget history archive +- Monthly limits per expense category +- Progress bars: green → amber (80%) → red (100%+) +- Rollover unused budget to next month (toggle) +- Copy previous month's budgets in one click +- Unbudgeted spending shown with "Set Budget" prompt ### 2.6 Goals & Savings -- Create savings goals (name, target amount, target date, linked account) -- Track contributions toward each goal -- Progress bar + projected completion date -- Emergency fund tracker (X months of expenses) +- Goals with target amount, target date, color, icon +- Contribution tracking + history +- Progress bar + projected completion date (based on avg monthly contrib) +- Auto-complete on 100% +- Emergency fund tracker (liquid assets vs 3-month / 6-month expense targets) -### 2.7 Reports & Analytics -- Monthly/quarterly/yearly summary reports -- Category spending trends (line chart over time) -- Income growth chart -- Net worth over time (historical snapshots, monthly auto-saved) -- Tax year summary (income + deductible expenses) -- Printable PDF report (via WeasyPrint) +### 2.7 Investments +- Asset types: stock, ETF, crypto, real_estate, bond, cash, other +- Buy/sell/dividend/split transaction log +- FIFO cost basis auto-recalculated from transaction log +- yfinance price auto-fetch (daily 4PM weekdays via cron) +- Manual price refresh button (portfolio page) +- Live ticker check on add form +- Doughnut allocation chart +- P&L per holding + portfolio total -### 2.8 Accounts & Wallets -- Multiple accounts (bank checking, savings, cash, credit card, crypto wallet) -- Account balances tracked manually -- Transfer between accounts (internal transaction) -- Credit card balance + due date tracking +### 2.8 AI Financial Assistant +- Chat UI with SSE streaming (Groq API, word-by-word response) +- Context: last 90 days transactions + budget status + goals + investments +- 8 suggested question buttons +- Daily auto-insight generated at midnight (stored in `ai_insights` table) +- Manual "Generate Now" button +- Chat history page +- Model: `llama-3.3-70b-versatile` (default) or `llama-3.1-8b-instant` (fast) +- Fallback messages for rate limit / invalid key / unavailable -### 2.9 AI Financial Assistant (Groq API — Free Tier) -- Chat interface (ask questions about your finances) -- Context: last 90 days of transactions injected into prompt -- Example queries: - - "Where did I overspend this month?" - - "Am I on track for my vacation goal?" - - "Summarize my Q1 spending" - - "What categories can I cut to save $500/month?" -- Auto-insight: daily AI summary generated at midnight via cron -- Model: `llama-3.3-70b-versatile` or `llama-3.1-8b-instant` via Groq (configurable in `.env`) -- Streaming response (SSE) for real-time chat feel +### 2.9 Receipt OCR (Groq Vision) +- Model: `meta-llama/llama-4-scout-17b-16e-instruct` +- Drag-drop or click-to-upload on new expense form +- Extracts: amount, date, merchant name, category suggestion, notes +- Maps category suggestion → system category ID +- Auto-fills form fields with green flash animation +- Re-extract button on existing receipt (edit mode) +- Auto-triggers OCR when image file selected in edit mode +- Handles fenced markdown JSON output from LLM +- Sanity check: rejects amounts < 1000 (catches garbage values) +- Full error handling: 400/401/429/timeout/bad JSON -### 2.10 Notifications & Alerts -- Browser notifications (via Web Push or in-app toast) -- Budget threshold alerts (e.g., 80% of category budget used) -- Bill/recurring expense due reminders -- Goal milestone celebrations +### 2.10 Reports & Export +- Monthly / Quarterly / Yearly summary reports +- Tax year summary (all income by source, all expenses by category) +- Net worth history line chart (from monthly snapshots) +- Category spending trends (top 6 categories, 6-month line chart) +- "Snapshot Now" manual button +- Export: CSV, Excel (color-coded, formatted), PDF (WeasyPrint) -### 2.11 Settings & Config -- Profile (name, timezone) -- **App currency** — single configurable currency (e.g. USD, VND, EUR — set once, used everywhere for all transactions/display) -- USD→VND rate source preference (ExchangeRate-API free or VCB scrape fallback) -- Category management (add/edit/delete custom categories) -- Data backup (export full MySQL dump) -- Data import (CSV import for bulk transactions) -- Groq model selector (choose speed vs quality) +### 2.11 Settings +- Profile: name, email, timezone, currency, Groq model +- 8 currency options (USD/VND/EUR/GBP/JPY/AUD/CAD/SGD) — auto-updates symbol +- Password change (requires current password) +- Recurring rules: CRUD, pause/enable, frequency (daily/weekly/biweekly/monthly/quarterly/yearly) +- "Run Now" button to process due rules immediately +- CSV import: upload → preview with ⚠ warnings → confirm +- Upcoming recurring transactions (30-day view) + +### 2.12 USD → VND Exchange Rate +- **Reference widget only** — not used in transaction calculations +- Primary source: yfinance `USDVND=X` (Yahoo Finance forex) +- Fallback: `open.er-api.com` (free, no key) +- Sanity check: rate must be > 1000 (rejects garbage values) +- `force_refresh()` — always fetches fresh on cron, bypasses cache +- DB caches one record per day (`fx_rates` table) +- Dashboard: shows rate, date, source, stale warning, ↻ refresh button +- 30-day history chart (click widget to expand) --- ## 3. Database Schema (MySQL) -### Tables - +### All 14 Tables ``` -users — single user (self-hosted, no multi-tenant) -accounts — bank/wallet accounts -categories — expense/income categories -transactions — all money movements (income/expense/transfer) -investments — holdings/portfolio positions -investment_transactions — buy/sell log -budgets — monthly budget limits per category -goals — savings goals -goal_contributions — deposits toward each goal -net_worth_snapshots — monthly net worth history -ai_insights — stored daily AI summaries -recurring_rules — templates for recurring income/expenses -receipts — receipt image metadata -fx_rates — daily USD/VND rate cache (date, rate, source) +users — single user, hashed password, currency/timezone prefs +accounts — bank/wallet accounts (balance auto-calc from txns) +categories — expense/income categories with color/icon +transactions — income/expense/transfer, receipt_id, recurring_rule_id +receipts — receipt file metadata (filename, size, mime_type) +recurring_rules — templates: frequency, next_run, start/end date +budgets — monthly limits per category, rollover support +goals — savings goals with target amount/date +goal_contributions — individual deposits toward each goal +investments — holdings: ticker, shares, avg_cost_basis, current_price +investment_transactions — buy/sell/dividend/split log +net_worth_snapshots — monthly snapshots: assets, liabilities, net_worth (JSON) +ai_insights — stored AI responses: daily_summary / chat_response +fx_rates — daily USD/VND rate cache (date UNIQUE, source) ``` -### Key Table: `transactions` -```sql -id, account_id, category_id, type (income/expense/transfer), -amount, currency, description, date, notes, -is_recurring, recurring_rule_id, receipt_id, -created_at, updated_at -``` - -### Key Table: `investments` -```sql -id, asset_name, ticker, asset_type (stock/etf/crypto/real_estate/bond/cash/other), -shares, avg_cost_basis, current_price, last_price_update, -currency, notes, created_at -``` - -### Key Table: `budgets` -```sql -id, category_id, month (YYYY-MM), limit_amount, -rollover_enabled, rollover_amount, created_at -``` +### Key Column Notes +- `transactions.balance` — NOT stored; calculated on-demand via `account_service.calc_balance()` +- `investments.shares` / `avg_cost_basis` — recalculated from `investment_transactions` (FIFO) +- `goals.current_amount` — updated on each contribution add/delete +- `net_worth_snapshots.account_balances` — JSON snapshot of each account balance at time of snapshot --- -## 4. Project File Structure +## 4. Project File Structure (Actual) ``` -pfm/ +pfm/ # /home/pfm/web on server +├── wsgi.py # Gunicorn entry — sys.path fix included +├── requirements.txt +├── .env # Not committed +├── .env.example +├── .gitignore +│ ├── app/ -│ ├── __init__.py # Flask app factory -│ ├── config.py # Config classes (dev/prod) -│ ├── extensions.py # db, login_manager, etc. +│ ├── __init__.py # Flask app factory, all blueprints registered +│ ├── config.py # Dev/Prod configs, SESSION_COOKIE_SECURE in prod +│ ├── extensions.py # db, login_manager, migrate, csrf │ │ │ ├── models/ -│ │ ├── user.py -│ │ ├── account.py -│ │ ├── transaction.py -│ │ ├── category.py -│ │ ├── investment.py -│ │ ├── budget.py -│ │ ├── goal.py -│ │ ├── ai_insight.py -│ │ ├── fx_rate.py -│ │ └── recurring_rule.py +│ │ ├── __init__.py # Imports all models (required for Flask-Migrate) +│ │ ├── user.py # UserMixin, set/check password, load_user hook +│ │ ├── account.py # account_type enum, color, icon +│ │ ├── category.py # Self-referential (subcategories), is_system flag +│ │ ├── transaction.py # Dual FK to accounts (account_id + to_account_id) +│ │ ├── receipt.py # filename, original_filename, mime_type +│ │ ├── recurring_rule.py # frequency enum, next_run date +│ │ ├── budget.py # UniqueConstraint(category_id, month) +│ │ ├── goal.py # progress_percent property +│ │ ├── investment.py # total_cost/current_value/unrealized_gain properties +│ │ ├── net_worth_snapshot.py # account_balances JSON field +│ │ ├── ai_insight.py # insight_type enum +│ │ └── fx_rate.py # date UNIQUE index │ │ │ ├── routes/ -│ │ ├── auth.py # Login/logout (single user) -│ │ ├── dashboard.py -│ │ ├── transactions.py -│ │ ├── income.py -│ │ ├── expenses.py -│ │ ├── investments.py -│ │ ├── budgets.py -│ │ ├── goals.py -│ │ ├── accounts.py -│ │ ├── reports.py -│ │ ├── ai.py # AI chat + insights (SSE) -│ │ ├── settings.py -│ │ └── api.py # Internal JSON API endpoints +│ │ ├── auth.py # /auth/login, /auth/logout +│ │ ├── dashboard.py # /, /api/fx-history, /api/fx-refresh (POST) +│ │ ├── accounts.py # /accounts/ +│ │ ├── categories.py # /categories/ +│ │ ├── transactions.py # /transactions/, /transactions/ocr (POST), +│ │ │ # /transactions/ocr-file (POST) +│ │ ├── budgets.py # /budgets/, /budgets/copy (POST) +│ │ ├── goals.py # /goals/, contribute, contributions, delete_contribution +│ │ ├── investments.py # /investments/, detail, add_transaction, +│ │ │ # refresh-prices, /api/price/ +│ │ ├── reports.py # /reports/monthly|quarterly|yearly|tax +│ │ │ # /reports/export/csv|excel|pdf +│ │ ├── ai.py # /ai/, /ai/stream (SSE), /ai/history, +│ │ │ # /ai/generate-insight (POST) +│ │ └── settings.py # /settings/, profile, password, recurring, +│ │ # import, upload_receipt, delete_receipt, view_receipt │ │ │ ├── services/ -│ │ ├── ai_service.py # Groq API integration + context builder -│ │ ├── budget_service.py # Budget calc + alert logic -│ │ ├── investment_service.py # yfinance price fetcher -│ │ ├── fx_service.py # USD/VND rate fetch + cache logic -│ │ ├── report_service.py # PDF generation (WeasyPrint) -│ │ ├── recurring_service.py # Recurring rule processor -│ │ ├── import_service.py # CSV import parser -│ │ └── snapshot_service.py # Net worth snapshot scheduler +│ │ ├── account_service.py # calc_balance(), recalc_all(), get_total_assets/liabilities() +│ │ ├── ai_service.py # build_context(), stream_chat() SSE gen, generate_daily_insight() +│ │ ├── budget_service.py # get_budget_summary(), apply_rollovers() +│ │ ├── export_service.py # transactions_to_csv/excel(), report_to_pdf(), build_report_html() +│ │ ├── fx_service.py # get_today_rate(), force_refresh(), _fetch_yfinance(), +│ │ │ # _fetch_er_api(), get_rate_history() +│ │ ├── goal_service.py # get_projected_completion(), get_emergency_fund_status() +│ │ ├── import_service.py # parse_csv(), import_rows(), duplicate detection +│ │ ├── investment_service.py # fetch_price(), update_prices(), get_portfolio_summary() +│ │ ├── ocr_service.py # extract_from_file(), extract_from_bytes(), +│ │ │ # Groq vision model, JSON parse + sanitise +│ │ ├── recurring_service.py # process_due_rules(), next_occurrence(), get_upcoming() +│ │ └── report_service.py # monthly/quarterly/yearly/tax reports, +│ │ # net_worth_history(), category_trends(), take_net_worth_snapshot() │ │ │ ├── templates/ -│ │ ├── base.html -│ │ ├── auth/ -│ │ ├── dashboard/ -│ │ ├── transactions/ -│ │ ├── investments/ -│ │ ├── budgets/ -│ │ ├── goals/ -│ │ ├── accounts/ -│ │ ├── reports/ -│ │ ├── ai/ -│ │ └── settings/ +│ │ ├── base.html # Collapsible sidebar, topbar, flash messages, +│ │ │ # @keyframes spin, Bootstrap 5 + Bootstrap Icons +│ │ ├── auth/login.html # Dark themed, password toggle +│ │ ├── dashboard/index.html # All widgets, FX refresh JS, SSE-compatible +│ │ ├── accounts/ # index.html, form.html (color/icon picker) +│ │ ├── categories/ # index.html, form.html +│ │ ├── transactions/ # index.html (tabs+filter+pagination) +│ │ │ # form.html (OCR panel + drag-drop + field flash) +│ │ │ # transfer.html +│ │ ├── budgets/ # index.html (progress bars), form.html +│ │ ├── goals/ # index.html (emergency fund + cards), form.html, +│ │ │ # contribute.html, contributions.html +│ │ ├── investments/ # index.html (doughnut chart), detail.html, +│ │ │ # form.html (ticker check), transaction_form.html +│ │ ├── reports/ # index.html (4 chart types), tax.html +│ │ ├── ai/ # index.html (SSE chat + suggestions), history.html +│ │ └── settings/ # index.html, profile.html, password.html, +│ │ # recurring.html, recurring_form.html, import.html │ │ │ ├── static/ -│ │ ├── css/ -│ │ ├── js/ -│ │ └── img/ # Static images +│ │ ├── css/ # (empty — all CSS inline in templates) +│ │ ├── js/ # (empty — all JS inline in templates) +│ │ └── img/ │ │ │ └── utils/ -│ ├── decorators.py # Auth required, etc. -│ ├── formatters.py # Currency, date formatting -│ └── validators.py +│ ├── formatters.py # format_currency(), format_percent(), format_large_number() +│ └── decorators.py # login_required_custom (unused — Flask-Login handles it) │ -├── migrations/ # Flask-Migrate (Alembic) -├── scripts/ -│ ├── init_db.py # First-run DB setup + seed categories -│ ├── daily_snapshot.py # Cron: net worth snapshot -│ ├── daily_ai_insight.py # Cron: generate AI summary -│ ├── process_recurring.py # Cron: create recurring transactions -│ ├── fetch_prices.py # Cron: update investment prices -│ └── fetch_fx_rate.py # Cron: fetch daily USD/VND rate +├── migrations/ # Flask-Migrate / Alembic │ -├── tests/ -├── .env # Secrets (not committed) -├── .env.example -├── requirements.txt -├── wsgi.py -├── CLAUDE.md # This file -└── deploy.md # Server setup guide +├── scripts/ # All cron scripts — sys.path fix at top of each +│ ├── init_db.py # Seeds 21 default categories, creates admin user interactively +│ ├── process_recurring.py # Processes due recurring rules, creates transactions +│ ├── fetch_fx_rate.py # force_refresh() — always fetches fresh USD/VND +│ ├── fetch_prices.py # yfinance price update for all investment tickers +│ ├── daily_snapshot.py # Saves net worth snapshot (1st of month) +│ └── daily_ai_insight.py # Generates daily AI summary via Groq +│ +└── tests/ ``` --- @@ -238,248 +264,171 @@ pfm/ ## 5. Python Dependencies (`requirements.txt`) ``` -flask -flask-sqlalchemy -flask-login -flask-migrate -flask-wtf -pymysql -python-dotenv -gunicorn -groq # Groq official Python SDK -yfinance # Investment price fetching -weasyprint # PDF report generation -openpyxl # Excel export -Pillow # Receipt image processing -apscheduler # In-process scheduler (alternative to cron) +flask==3.1.0 +flask-sqlalchemy==3.1.1 +flask-login==0.6.3 +flask-migrate==4.1.0 +flask-wtf==1.2.2 +pymysql==1.1.1 +python-dotenv==1.0.1 +gunicorn==23.0.0 +groq==0.13.1 +yfinance==0.2.54 +weasyprint==63.1 +openpyxl==3.1.5 +Pillow==11.1.0 +apscheduler==3.10.4 +requests==2.32.3 +cryptography==44.0.2 +python-dateutil==2.9.0 ``` --- -## 6. AI Integration — Groq API (Free Tier) +## 6. AI Integration — Groq API -### Why Groq -- Free tier: 14,400 requests/day, 500,000 tokens/minute -- Fastest inference available (LPU hardware) — responses feel instant -- No local GPU/RAM needed -- Models: `llama-3.3-70b-versatile` (best quality), `llama-3.1-8b-instant` (fastest) +### Chat + Daily Insights +- Model: `llama-3.3-70b-versatile` (default) / `llama-3.1-8b-instant` (fast) +- Context: last 90 days transactions, budget status, goals, investments (anonymised) +- SSE streaming: `stream_chat()` yields `data: \n\n` +- Daily insight: non-streaming, stored in `ai_insights` table, max 400 tokens -### How It Works -``` -User sends chat message - → ai_service.py builds context (last 90 days transactions summary) - → POST to Groq API (`/chat/completions`) with Bearer token - → Stream response back via SSE to browser - → Response stored in ai_insights table -``` +### Receipt OCR (Vision) +- Model: `meta-llama/llama-4-scout-17b-16e-instruct` +- Input: base64-encoded image (JPEG/PNG/GIF/WEBP) +- Prompt: structured JSON extraction (amount, date, merchant, category, notes) +- Temperature: 0.1 (low, for consistent output) +- Post-processing: strips markdown fences, extracts JSON via regex fallback, + normalises amount/date, maps category to system names +- Endpoints: `POST /transactions/ocr` (upload bytes), `POST /transactions/ocr-file` (stored file) -### Context Injection Strategy -```python -# ai_service.py builds a prompt like: -""" -You are a personal finance assistant. Here is the user's financial data: - -CURRENT MONTH SUMMARY: -- Total Income: $X -- Total Expenses: $Y -- Top spending categories: Food ($A), Transport ($B), ... -- Budget alerts: Entertainment 92% used - -RECENT TRANSACTIONS (last 20): -[date] [category] [amount] [description] -... - -NET WORTH: $Z -ACTIVE GOALS: Vacation Fund ($1,200 / $3,000) - -User question: {user_message} - -Answer concisely and specifically based on the data above. -""" -``` - -### Fallback -If Groq API key missing/invalid → show friendly message "AI assistant unavailable. Check GROQ_API_KEY in settings." -If Groq rate limit hit → show "AI rate limit reached. Try again shortly." +### Free Tier Limits +| Metric | Limit | +|--------|-------| +| Requests/day | 14,400 | +| Tokens/minute | 500,000 | +| Cost | Free | --- ## 7. USD → VND Exchange Rate -> **Reference widget only** — independent of the app's transaction currency. All income/expenses/investments use the single configured app currency. This widget is informational display only. +> **Reference widget only.** All transactions use the single configured app currency. -### Data Source — Free, No API Key Required +### Fetch Priority +1. DB cache — same day record in `fx_rates` +2. **yfinance** `USDVND=X` forex ticker (primary — most reliable) +3. `open.er-api.com` free REST API (fallback) +4. Last known DB record (stale fallback, shows ⚠ indicator) -Primary: **ExchangeRate-API open endpoint** -``` -https://open.er-api.com/v6/latest/USD -``` -Returns JSON with all rates including VND. Free tier, no key, 1,500 req/month. +### Sanity Check +Rate must be `> 1000` — rejects garbage values (e.g. 1.0, 0.0) that some APIs return. -Fallback: **Vietcombank (VCB) rate scrape** -``` -https://www.vietcombank.com.vn/en/KHCN/Chuyen-trang-KHCN/Pages/ty-gia.aspx -``` -Scrape VCB's official buying/selling rate as backup. - -### DB Table: `fx_rates` -```sql -id INT AUTO_INCREMENT PRIMARY KEY -date DATE NOT NULL UNIQUE -- one record per day -usd_to_vnd DECIMAL(12,2) NOT NULL -- e.g. 25,450.00 -source VARCHAR(50) -- 'exchangerate-api' | 'vcb' | 'manual' -fetched_at DATETIME -``` - -### `fx_service.py` Logic -``` -get_today_rate(): - 1. Check fx_rates table for today's date - 2. If found → return cached rate (no API call) - 3. If not found → fetch from ExchangeRate-API - 4. If API fails → try VCB scrape - 5. If both fail → return last known rate from DB + show "rate may be outdated" flag - 6. Save new rate to DB -``` - -### Dashboard Widget Display -- Card on dashboard header area (top bar or sidebar) -- Shows: `1 USD = 25,450 ₫` with date label -- Color: neutral/info (blue or grey — not green/red, it's informational) -- Click → opens 30-day rate history mini-chart (Chart.js, line chart) -- Stale indicator: if rate is >1 day old, show small warning icon - -### 30-Day Rate History -- Stored in `fx_rates` table (one row/day, auto-accumulates) -- Chart available on dashboard click or Reports page -- Shows trend: flat/up/down with % change label - -### Scheduled Job -- Daily 8AM fetch (after markets open in Vietnam) -- `scripts/fetch_fx_rate.py` -- systemd timer unit: `pfm-fxrate.timer` +### Dashboard Widget +- Shows rate, date, source +- ↻ button → `POST /api/fx-refresh` → updates rate in-place without page reload +- Click widget → toggles 30-day Chart.js line chart +- `force_refresh()` used by daily cron — always bypasses cache --- -## 8. UI/UX Design Direction +## 8. UI/UX -- **Style**: Clean financial dashboard — dark sidebar, white/light content area -- **Charts**: Chart.js (CDN, no build step needed) -- **Tables**: DataTables.js for sortable/searchable transaction tables -- **Forms**: WTForms + Bootstrap 5 -- **Icons**: Bootstrap Icons or Feather Icons -- **Color scheme**: Deep navy sidebar, white cards, green (income), red (expense), blue (investment) -- **Mobile**: Responsive (Bootstrap grid) -- **AI Chat**: Floating chat panel (slide-in from right), SSE streaming text +- **Sidebar**: collapsible (desktop state saved in localStorage), mobile overlay +- **Charts**: Chart.js 4.x (CDN) +- **Forms**: WTForms + Bootstrap 5.3 +- **Icons**: Bootstrap Icons 1.11 +- **Fonts**: DM Sans + DM Mono (Google Fonts CDN) +- **Color scheme**: `#0f172a` sidebar, `#f1f5f9` body, `#10b981` income, `#ef4444` expense, `#3b82f6` invest +- **CSS**: All inline in templates (no build step) +- **SSE**: used for AI chat stream + FX refresh --- ## 9. Authentication -- Single-user app (self-hosted) -- Flask-Login with username/password -- Session-based auth -- Password hashed with Werkzeug (bcrypt) -- Optional: IP whitelist via Nginx (allow only LAN access) +- Single-user, Flask-Login, session-based +- Hashed password (Werkzeug `generate_password_hash`) +- `SESSION_COOKIE_SECURE=True` in production +- `SESSION_COOKIE_HTTPONLY=True`, `SESSION_COOKIE_SAMESITE='Lax'` +- CSRF protection on all forms (Flask-WTF) --- -## 10. Scheduled Jobs (systemd timers or APScheduler) +## 10. Scheduled Jobs -| Job | Schedule | Script | -|-----|----------|--------| -| Process recurring transactions | Daily 6AM | `process_recurring.py` | -| Fetch USD/VND exchange rate | Daily 8AM | `fetch_fx_rate.py` | -| Fetch investment prices | Daily 4PM | `fetch_prices.py` | -| Save net worth snapshot | 1st of month | `daily_snapshot.py` | -| Generate AI daily insight | Daily midnight | `daily_ai_insight.py` | - -Recommend **APScheduler** inside Flask app (simpler) OR separate systemd timer units (more robust). +| Job | Schedule | Script | Notes | +|-----|----------|--------|-------| +| Process recurring transactions | Daily 6AM | `process_recurring.py` | Creates missed occurrences | +| Fetch USD/VND rate | Daily 8AM | `fetch_fx_rate.py` | force_refresh(), yfinance primary | +| Fetch investment prices | Mon-Fri 4PM | `fetch_prices.py` | yfinance, all tickers | +| Net worth snapshot | 1st of month 00:05 | `daily_snapshot.py` | Saves to net_worth_snapshots | +| AI daily insight | Daily 00:01 | `daily_ai_insight.py` | Skips if already done today | +| DB backup | Daily 2AM | pfm-backup (systemd) | mysqldump → gzip, keep 30 days | --- -## 11. Development Phases - -### Phase 1 — Foundation -- [ ] Flask app factory + config -- [ ] MySQL models + migrations -- [ ] Auth (login/logout) -- [ ] Base template + sidebar nav - -### Phase 2 — Core Transactions -- [ ] Accounts CRUD -- [ ] Categories CRUD -- [ ] Transaction entry (income + expense) -- [ ] Transaction list with filter/search -- [ ] Dashboard basics (totals, recent feed) -- [ ] USD→VND rate widget + `fx_service.py` + daily fetch job - -### Phase 3 — Budget & Goals -- [ ] Budget planner (set limits) -- [ ] Budget vs actual comparison -- [ ] Goals CRUD + contribution tracking - -### Phase 4 — Investments -- [ ] Holdings CRUD -- [ ] Buy/sell transaction log -- [ ] yfinance price auto-fetch -- [ ] Portfolio charts - -### Phase 5 — AI Assistant -- [ ] Groq API integration + context builder -- [ ] Chat UI with SSE streaming -- [ ] Daily auto-insight cron - -### Phase 6 — Reports & Export -- [ ] Monthly summary page -- [ ] PDF export (WeasyPrint) -- [ ] CSV/Excel export -- [ ] Net worth history chart - -### Phase 7 — Polish -- [ ] Recurring transaction engine -- [ ] Receipt upload -- [ ] CSV import -- [ ] Budget alerts + notifications -- [ ] Mobile responsiveness pass - ---- - -## 12. Environment Variables (`.env`) +## 11. Environment Variables (`.env`) ``` SECRET_KEY=your-secret-key DATABASE_URL=mysql+pymysql://pfm_user:password@localhost/pfm_db GROQ_API_KEY=your-groq-api-key-here GROQ_MODEL=llama-3.3-70b-versatile -UPLOAD_FOLDER=/home/pfm/app/uploads +UPLOAD_FOLDER=/home/pfm/web/uploads MAX_CONTENT_LENGTH=10485760 FLASK_ENV=production -APP_CURRENCY=USD # Single currency for all transactions (configurable) -APP_CURRENCY_SYMBOL=$ # Display symbol -APP_TIMEZONE=Asia/Ho_Chi_Minh # Server timezone for scheduled jobs +FLASK_APP=wsgi:app +APP_CURRENCY=USD +APP_CURRENCY_SYMBOL=$ +APP_TIMEZONE=Asia/Ho_Chi_Minh ``` --- +## 12. Blueprints Registered (11 total) + +| Blueprint | Prefix | Key routes | +|-----------|--------|------------| +| auth | /auth | login, logout | +| dashboard | / | index, api/fx-history, api/fx-refresh | +| accounts | /accounts | CRUD | +| categories | /categories | CRUD | +| transactions | /transactions | index, new, edit, delete, transfer, ocr, ocr-file | +| budgets | /budgets | index, new, edit, delete, copy | +| goals | /goals | index, new, edit, delete, contribute, contributions | +| investments | /investments | index, new, detail, edit, delete, add_transaction, refresh-prices, api/price | +| reports | /reports | monthly, quarterly, yearly, tax, export/csv|excel|pdf, snapshot | +| ai | /ai | index, stream (SSE), history, generate-insight | +| settings | /settings | index, profile, password, recurring, import, upload_receipt, delete_receipt, view_receipt | + +--- + ## 13. Security Notes -- All routes protected by `@login_required` -- CSRF protection via Flask-WTF -- SQL injection prevented by SQLAlchemy ORM -- File upload validation (type + size limit) -- Nginx: restrict access to local network if desired +- All routes `@login_required` +- CSRF on all POST forms +- SQLAlchemy ORM (no raw SQL) +- Receipt file path: `os.path.basename()` prevents path traversal - HTTPS via Let's Encrypt (Certbot) — `pfm.ngodanguyen.tech` -- Groq API receives only anonymized transaction summaries (no account names/personal details in prompt) -- GROQ_API_KEY stored in `.env`, never exposed to frontend +- Groq receives anonymised transaction summaries (no account/personal names) +- `GROQ_API_KEY` in `.env` (chmod 600), never in frontend --- -## 14. Future Enhancements (Post-MVP) +## 14. Known Issues / Notes -- Mobile app companion (iOS — fits your skill set) -- Bank statement auto-import (parse PDF bank statements) -- Multi-currency with live FX rates (via free API) -- Expense photo OCR (extract amount from receipt image via Groq vision model) -- Email/SMS alerts (integrate Twilio — you already know it from FaxDesk) -- Shared household mode (2 users) +- `wsgi.py` has `sys.path.insert(0, ...)` fix — required because app deploys at `/home/pfm/web/` which would otherwise be treated as a Python package +- FX rate widget: `open.er-api.com` may return stale values; yfinance is the reliable primary +- WeasyPrint PDF: requires `libpango*` system libs (included in deploy.md apt install) +- Import preview uses Flask session to pass rows to confirm step — requires `SECRET_KEY` to be set + +--- + +## 15. Post-MVP Roadmap + +- [ ] iOS companion app +- [ ] Bank statement PDF auto-import (parse PDF → extract transactions) +- [ ] Budget alerts + Twilio SMS/email notifications +- [ ] Shared household mode (2 users, row-level isolation) +- [ ] Mobile responsiveness pass diff --git a/README.md b/README.md index 4826b65..6172b27 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,690 @@ -# Personal-Finance-Management +# PFM — Personal Finance Manager -Personal-Finance-Management system \ No newline at end of file +A self-hosted personal finance web application. Track income, expenses, investments, budgets, and savings goals. AI-powered financial assistant and receipt OCR built in. + +**Live at:** https://pfm.ngodanguyen.tech + +--- + +## Table of Contents + +1. [Features Overview](#features-overview) +2. [Getting Started](#getting-started) +3. [Dashboard](#dashboard) +4. [Transactions](#transactions) +5. [Accounts](#accounts) +6. [Categories](#categories) +7. [Budgets](#budgets) +8. [Goals & Savings](#goals--savings) +9. [Investments](#investments) +10. [AI Assistant](#ai-assistant) +11. [Reports & Export](#reports--export) +12. [Settings](#settings) +13. [Recurring Transactions](#recurring-transactions) +14. [CSV Import](#csv-import) +15. [Receipt OCR](#receipt-ocr) +16. [USD → VND Rate Widget](#usd--vnd-rate-widget) +17. [Keyboard Shortcuts & Tips](#keyboard-shortcuts--tips) + +--- + +## Features Overview + +| Module | What it does | +|--------|-------------| +| **Dashboard** | Net worth, cash flow chart, budget gauges, AI insight, USD/VND rate | +| **Transactions** | Income + expense entry, transfer, filter, search, receipt upload | +| **Accounts** | Multiple bank/cash/credit accounts, auto-calculated balances | +| **Categories** | Custom expense + income categories with color and icon | +| **Budgets** | Monthly spending limits per category with progress tracking | +| **Goals** | Savings goals with contribution tracking and projected completion | +| **Investments** | Stock/ETF/crypto/real estate portfolio with live price fetch | +| **AI Assistant** | Chat with your finances via Groq (free), daily auto-insight | +| **Receipt OCR** | Drag-drop a receipt image — AI extracts and fills the form | +| **Reports** | Monthly/quarterly/yearly summaries, net worth history, PDF/CSV/Excel export | +| **Settings** | Profile, currency, recurring rules, CSV import, password | + +--- + +## Getting Started + +### First Login + +1. Navigate to `https://pfm.ngodanguyen.tech` +2. Enter the username and password created during setup (`python scripts/init_db.py`) +3. Check "Remember me" to stay logged in across browser sessions + +### Recommended Setup Order + +1. **Add accounts** — add your bank accounts, cash wallet, and credit cards first (Settings sidebar → Accounts) +2. **Review categories** — default categories are already seeded; add custom ones if needed +3. **Set your currency** — go to Settings → Profile and set your app currency (USD, VND, etc.) +4. **Add transactions** — start entering income and expenses +5. **Set budgets** — once you have categories, set monthly limits +6. **Create goals** — add savings goals and start contributing +7. **Add investments** — track your portfolio holdings +8. **Set recurring rules** — automate repeating income/expenses + +--- + +## Dashboard + +The dashboard is the home screen — accessible from the sidebar or by clicking the PFM logo. + +### Period Selector +Three buttons in the top-right of the topbar: +- **This Month** — income/expenses for the current calendar month +- **Last Month** — previous calendar month +- **Custom** — opens a date picker modal; choose any start and end date + +The period affects the summary cards and recent transactions feed. Charts always show fixed windows (6 months for cash flow, etc.). + +### Summary Cards +Four cards across the top: +- **Income** — total income for the selected period (green) +- **Expenses** — total expenses for the selected period (red) +- **Net Cash Flow** — income minus expenses; green if positive, red if negative +- **Net Worth** — total assets minus liabilities across all accounts + +### Cash Flow Chart +Bar chart showing the last 6 months of income (green bars) vs expenses (red bars). Hover over bars to see exact amounts. + +### Top Spending +Right of the cash flow chart — shows your top 5 expense categories for the period with horizontal progress bars. Widest bar = highest spending. + +### USD → VND Widget +Dark card below the top spending section. Shows today's USD to VND exchange rate for reference. Click the widget to reveal a 30-day trend chart. Click the ↻ button to force-refresh the rate without reloading the page. See [USD → VND Rate Widget](#usd--vnd-rate-widget) for details. + +### AI Daily Insight +Dark card above the accounts section. Shows a 3–5 sentence AI-generated summary of your finances for the day. Generated automatically at midnight. Click "Open AI →" to go to the full chat interface. + +### Accounts Panel +Left of the recent transactions — lists all active accounts with their current balance. Green = positive, red = negative (credit card debt). Click "+" to add a new account. + +### Recent Transactions +Last 8 income and expense transactions. Click "View all" to go to the full transaction list. + +### Quick Add Buttons +Two buttons in the page header: +- **+Income** (green) — opens new income form +- **+Expense** (red) — opens new expense form + +--- + +## Transactions + +### Viewing Transactions + +Navigate via **Transactions** in the sidebar. Two tabs at the top: +- **Expenses** — shows expense transactions with a count badge +- **Income** — shows income transactions with a count badge + +Transactions are ordered newest first, paginated at 30 per page. + +### Filtering + +A filter bar appears below the tabs: +- **Search** — matches against the description field (case-insensitive) +- **Category** — filter by a single category +- **Account** — filter by a single account +- **From / To** — date range filter +- Click the magnifying glass button to apply filters +- Click the ✕ button to clear all filters and reset to defaults + +### Adding a Transaction + +**From the sidebar:** click "Add Income" or "Add Expense" +**From the dashboard:** use the +Income / +Expense buttons +**From the transaction list:** use the topbar buttons + +#### New Expense / Income Form Fields + +| Field | Required | Notes | +|-------|----------|-------| +| Description | Yes | What the transaction was for | +| Amount | Yes | Positive number only | +| Date | Yes | Defaults to today | +| Account | Yes | Which account to debit/credit | +| Category | No | Helps with budgets and reports | +| Notes | No | Free text, up to 500 characters | + +#### AI Receipt Scanner (on new expense form) +A purple panel sits above the form. Drop a receipt image onto it (or click to browse). The AI extracts the amount, date, merchant name, and category and fills the form fields automatically. Review the filled values before saving — highlighted fields (green flash) show what was auto-filled. See [Receipt OCR](#receipt-ocr) for details. + +### Editing a Transaction + +Click **Edit** on any transaction row. Same form as adding, with an additional receipt section: +- If a receipt is attached: shows filename with a **Re-extract** button (re-runs OCR on the stored file) and a **Remove** button +- If no receipt: shows an upload field; selecting an image file auto-triggers OCR + +### Deleting a Transaction + +Click **Del** on any transaction row. A confirmation prompt appears. Deletion is permanent and account balances are recalculated immediately. + +### Transfers Between Accounts + +Click **Transfer** in the topbar. Select source account, destination account, amount, and date. This creates a single transfer transaction that debits the source and credits the destination. Transfers do not appear on the Income or Expense tabs — they are excluded from income/expense totals. + +--- + +## Accounts + +### Adding an Account + +Settings sidebar → **Accounts** → **New Account** (or from dashboard Accounts panel → "+"). + +| Field | Notes | +|-------|-------| +| Account Name | e.g. "Chase Checking", "Cash Wallet", "Visa Card" | +| Type | checking / savings / cash / credit_card / crypto / investment / other | +| Color | Click a color swatch — used for visual identification | +| Icon | Click an icon swatch — appears throughout the app | +| Notes | Optional description | + +### Account Balances + +Balances are **automatically calculated** from all transactions linked to that account — you do not enter a balance manually. When you add a transaction, the account balance updates instantly. + +**Formula:** +``` +Balance = sum(income) - sum(expenses) - sum(transfers_out) + sum(transfers_in) +``` + +For a brand-new account with no transactions, balance is 0. To set a starting balance, add an income transaction dated your desired start date with the description "Opening balance." + +### Removing an Account + +Click the three-dot menu (⋯) on an account card → **Remove**. This is a soft delete — the account is hidden but its transactions remain in the database and still affect totals. You cannot permanently delete an account with transactions. + +--- + +## Categories + +Navigate via the sidebar footer → **Categories**. + +Two panels: **Expense Categories** (left) and **Income Categories** (right). + +### Default Categories + +**Expense (14):** Housing, Food & Dining, Transport, Utilities, Health, Entertainment, Shopping, Education, Insurance, Personal Care, Travel, Subscriptions, Gifts, Other + +**Income (7):** Salary, Freelance, Business, Investment, Rental, Gift Received, Other Income + +System categories (marked with a grey "system" badge) cannot be deleted but can be edited (color and icon only — name is protected). + +### Adding a Category + +Click **+Expense** or **+Income** in the topbar. Set a name, type, color (click a swatch), and icon (click an icon swatch). + +**Category type options:** +- **Expense** — appears only in expense transaction dropdown +- **Income** — appears only in income transaction dropdown +- **Both** — appears in both dropdowns + +### Editing a Category + +Click **Edit** next to any category. Color and icon can always be changed. Name can only be changed for non-system categories. + +### Deleting a Category + +Click **Del** next to a non-system category. Categories with existing transactions cannot be hard-deleted — they are deactivated (hidden) instead. A warning message explains this. + +--- + +## Budgets + +Navigate via **Budgets** in the sidebar. + +### Month Navigation + +Use the **◀** and **▶** arrows to move between months. The current month is the default. + +### Setting a Budget + +Click **Add Budget** in the topbar (or "Set Budget" next to an unbudgeted category). + +| Field | Notes | +|-------|-------| +| Category | Expense categories only | +| Monthly Limit | Maximum spend for this category this month | +| Roll over unused amount | If checked, unspent budget carries forward to next month | + +### Budget Progress Bars + +Each category row shows: +- Category name and icon +- Amount spent (colored by status) +- Budget limit +- Progress bar: green (< 80%) → amber (80–99%) → red (100%+) +- Remaining amount (or over-budget amount in red) + +### Unbudgeted Spending + +Categories with spending but no budget appear at the bottom of the table with a yellow "no budget" badge and a "Set Budget" button. + +### Copy from Previous Month + +Click **Copy from YYYY-MM** button (top right or empty state) to duplicate all budget entries from the previous month. Existing budgets for the current month are not overwritten — only missing ones are created. + +### Editing / Deleting a Budget + +Click **Edit** or **Del** on any budget row. Deleting a budget does not delete the transactions — it only removes the limit. + +--- + +## Goals & Savings + +Navigate via **Goals** in the sidebar. + +### Emergency Fund Tracker + +A yellow card at the top (appears once you have 3+ months of expense data). Shows: +- Average monthly expense (last 3 months) +- Liquid assets (checking + savings + cash accounts combined) +- 3-month target and 6-month target with progress bars +- Months covered: how long your liquid assets would last at current spending + +### Creating a Goal + +Click **New Goal** in the topbar. + +| Field | Notes | +|-------|-------| +| Goal Name | e.g. "Vacation Fund", "Emergency Fund", "New Laptop" | +| Target Amount | How much you want to save | +| Target Date | Optional deadline | +| Linked Account | Optional — associates the goal with a specific account | +| Description | Optional notes | +| Color | Visual identifier (circle swatches) | +| Icon | Visual identifier (icon swatches) | + +### Goal Cards + +Each active goal shows: +- Progress bar (fills with goal color) +- Current amount saved / target amount +- Completion percentage +- Projected completion date (calculated from average monthly contribution history — only shown after 2+ contributions) +- Description (if set) +- **Add Contribution** button (styled in goal color) + +### Adding a Contribution + +Click **Add Contribution** on a goal card or from the three-dot menu. Enter amount, date, and optional notes. When the total reaches the target, the goal auto-completes and moves to the "Completed Goals" section. + +### Contribution History + +Three-dot menu → **History** — shows all contributions with dates and amounts, plus the projected completion date. Individual contributions can be deleted (recalculates goal total and un-completes if needed). + +### Completed Goals + +Shown at the bottom of the page in a compact table. Last 5 completed goals displayed. + +--- + +## Investments + +Navigate via **Investments** in the sidebar. + +### Portfolio Overview + +Summary cards at the top: +- **Total Value** — current market value of all holdings +- **Total Cost** — total amount invested (cost basis) +- **Unrealized P&L** — gain or loss vs cost basis +- **Return** — percentage return + +**Allocation chart** — doughnut chart showing portfolio split by asset type (stock, ETF, crypto, real estate, bond, cash, other). Each slice is color-coded. Click a row in the legend for exact values. + +**Holdings table** — all active holdings. Click any row to go to the detail page. + +### Adding a Holding + +Click **Add Holding** in the topbar. + +| Field | Notes | +|-------|-------| +| Asset Name | Full name, e.g. "Apple Inc.", "Bitcoin" | +| Asset Type | stock / ETF / crypto / real_estate / bond / cash / other | +| Ticker Symbol | Yahoo Finance format: AAPL, BTC-USD, ETH-USD, VNM. Click **Check** to verify | + +After saving, you are taken to the detail page to record your first buy transaction. + +### Ticker Format (Yahoo Finance) +- US Stocks: `AAPL`, `MSFT`, `GOOGL` +- ETFs: `VOO`, `QQQ`, `VTI` +- Crypto: `BTC-USD`, `ETH-USD`, `BNB-USD` +- Vietnamese stocks: `VNM` (VanEck Vietnam ETF on NYSE) +- Other markets: use Yahoo Finance suffix, e.g. `VIC.VN` for Vingroup on HOSE + +### Holding Detail Page + +Shows the holding summary (shares, avg cost, current price, market value, P&L) and transaction history. Current price timestamp is shown below the price. + +### Recording a Transaction + +On the detail page, click **Add Transaction** in the topbar. + +| Transaction Type | What it does | +|-----------------|-------------| +| **Buy** | Adds shares, increases cost basis | +| **Sell** | Reduces shares, adjusts cost basis (FIFO) | +| **Dividend** | Records a dividend payment (does not change shares) | +| **Split** | Adds shares without changing cost (stock split) | + +The **↓ button** next to the price field fetches the current live price from Yahoo Finance and fills it in. The "Estimated Total" preview updates as you type shares and price. + +After saving, shares and average cost basis are automatically recalculated from the full transaction history. + +### Refreshing Prices + +Click **Refresh Prices** in the topbar on the portfolio page to fetch current prices for all holdings with ticker symbols. Prices are also automatically updated daily at 4PM (weekdays) via a background job. + +### Removing a Holding + +On the detail page, click **Remove** (top right of the holding card). This is a soft delete — the holding is hidden but transaction history is preserved. + +--- + +## AI Assistant + +Navigate via **AI Assistant** in the sidebar. + +### Chat Interface + +Type a question in the text box at the bottom and press **Enter** (or click the send button). The response streams word-by-word in real time. + +**Shift+Enter** adds a new line without sending. + +The AI has access to: +- Last 90 days of transactions (description, category, amount, date) +- Current month income/expense totals and budget status +- Active savings goals with progress +- Investment portfolio summary +- Net worth + +No personal names, account names, or identifying details are sent to Groq — only aggregated financial figures. + +### Suggested Questions + +Eight suggestion buttons appear on the right panel. Click any to fill the chat input: +- "Where did I overspend this month?" +- "How is my budget looking?" +- "Am I on track for my goals?" +- "What's my biggest expense category?" +- "Summarize my finances" +- "How can I save more?" +- "What's my net worth trend?" +- "Review my investments" + +### Daily Insight + +A dark card on the right shows today's auto-generated insight (3–5 sentences covering spending, budget alerts, and a tip). Generated automatically at midnight. If not yet generated for today, click **Generate Now**. Click **Regenerate** to get a fresh one. + +### Chat History + +Click **History** in the topbar. Shows all past chat responses and daily summaries with timestamps and token counts. Paginated at 20 per page. + +### AI Model + +The model can be changed in Settings → Profile → AI Model: +- `llama-3.3-70b-versatile` — best quality, slightly slower +- `llama-3.1-8b-instant` — faster, slightly less detailed + +### Error States + +- **"AI assistant is not configured"** — GROQ_API_KEY is missing from .env +- **"Rate limit reached"** — too many requests; wait a moment and retry +- **"Invalid Groq API key"** — check the key at console.groq.com + +--- + +## Reports & Export + +Navigate via **Reports** in the sidebar. + +### Report Types + +Four tabs at the top: + +**Monthly** — income, expenses, net, savings rate for a single month. Bar chart + expense doughnut. + +**Quarterly** — same metrics for a quarter (Q1–Q4), plus monthly breakdown bars within the quarter. + +**Yearly** — full year summary with monthly breakdown, average monthly income/expense. + +**Tax Year** — income by source and expenses by category for a full calendar year, plus a full list of all income transactions for the year. Useful for tax preparation. + +### Selecting a Period + +Use the year/month/quarter dropdowns next to the period tabs and click **Go**. + +### Charts + +**Period chart** — bar chart of income vs expenses for the selected period (or monthly breakdown for quarterly/yearly). + +**Expense breakdown** — doughnut chart of expenses by category. Hover for exact amounts. Top 5 categories listed below with amounts. + +**Net worth history** — line chart of net worth over time (requires monthly snapshots to be saved). Dashed line shows total assets. Click **Snapshot Now** to save today's values. + +**Category spending trends** — line chart of top 6 expense categories over the last 6 months. Useful for spotting trends. + +### Exporting + +Three export buttons in the topbar: + +| Button | Format | Contents | +|--------|--------|----------| +| **CSV** | `.csv` | All transactions for the selected period, plain text | +| **Excel** | `.xlsx` | Color-coded rows (green=income, red=expense), formatted amounts, totals row | +| **PDF** | `.pdf` | Clean printable report with summary cards and expense breakdown table | + +Exports use the period selected in the current report view (year + month/quarter). + +--- + +## Settings + +Navigate via the gear icon at the bottom of the sidebar, or Settings → Profile. + +### Profile + +| Setting | Notes | +|---------|-------| +| Display Name | Shown in the topbar | +| Email | For reference only (no email features yet) | +| Timezone | Used for scheduled jobs | +| Currency | App-wide currency code (USD, VND, EUR, etc.) | +| Currency Symbol | Auto-set when currency is changed | +| AI Model | Groq model for chat and daily insights | + +Click **Save Profile** to apply. Currency and symbol changes take effect immediately throughout the app. + +### Changing Password + +Settings → Password. Requires current password + new password (min 6 characters) + confirmation. + +--- + +## Recurring Transactions + +Navigate via Settings → **Recurring** (also accessible from Settings landing page). + +### What It Does + +Recurring rules automatically create transactions on a schedule. At 6AM daily, the system checks all active rules and creates any transactions that are due (including any missed occurrences if the server was down). + +### Creating a Rule + +Click **New Rule**. + +| Field | Notes | +|-------|-------| +| Name | Label for the rule, e.g. "Monthly Rent" | +| Type | income or expense | +| Description | Used as the transaction description | +| Amount | Fixed amount each occurrence | +| Frequency | daily / weekly / biweekly / monthly / quarterly / yearly | +| Account | Which account to debit/credit | +| Category | Optional | +| Start Date | First occurrence date | +| End Date | Optional — leave blank for no end date | + +### Managing Rules + +The rules table shows all rules with their next run date, frequency, and status. Available actions: +- **Edit** — change any field (does not affect already-created transactions) +- **Pause / Enable** — temporarily disable without deleting +- **×** — delete the rule (already-created transactions are preserved) + +### Run Now + +Click **Run Now** in the topbar to immediately process all due rules. Useful after creating a new rule that has a past start date, or after the server was offline. + +### Upcoming Preview + +Right panel shows all upcoming recurring transactions for the next 30 days in date order, with amounts and type badges. + +--- + +## CSV Import + +Navigate via Settings → **Import**. + +### CSV Format + +``` +date,type,description,category,account,amount,notes +2025-01-15,expense,Groceries,Food & Dining,Checking,85.50,Weekly shop +2025-01-16,income,Salary,Salary,Checking,3000.00, +``` + +**Required columns:** `date`, `type`, `description`, `amount` + +**Optional columns:** `category`, `account`, `notes` + +**Accepted date formats:** `YYYY-MM-DD`, `MM/DD/YYYY`, `DD/MM/YYYY` + +**Type values:** must be exactly `income` or `expense` (lowercase) + +### Import Process + +1. Choose your CSV file +2. Select a **Default Account** — used when the account column is missing or the name doesn't match any of your accounts +3. Check **Skip duplicate transactions** (recommended) — skips rows where date + description + amount + type exactly match an existing transaction +4. Click **Preview Import** +5. Review the preview table: + - ⚠ yellow warning on category = category name not found (will be uncategorised) + - ⚠ yellow warning on account = account not found (will use default account or be unlinked) +6. Click **Confirm Import** to save all rows + +Account and category matching is case-insensitive. If your CSV has `food & dining` it will match the `Food & Dining` system category. + +--- + +## Receipt OCR + +Receipt OCR uses the Groq `llama-4-scout` vision model to extract transaction data from a photo of a receipt. + +### On New Expense Form + +A purple panel with a dashed border appears above the transaction form: + +1. **Drag and drop** a receipt image onto the panel, OR click the panel to browse for a file +2. Accepted formats: JPG, PNG, GIF, WEBP (max 10MB — PDF not supported for OCR) +3. The panel shows a spinning animation while scanning +4. On success: the panel turns green and form fields flash green to show what was filled: + - **Description** ← merchant name (e.g. "McDonald's") + - **Amount** ← total from receipt + - **Date** ← date on receipt (falls back to today if not found) + - **Category** ← AI's best guess matched to your system categories + - **Notes** ← brief description +5. Review all fields before clicking Save — OCR is not perfect, especially on low-quality photos +6. You can drop another receipt to re-scan and overwrite the filled values + +### On Edit Transaction Form + +**If a receipt is already attached:** +- A purple **Re-extract** button appears next to the receipt filename +- Click it to re-run OCR on the stored file and update the form fields + +**If no receipt is attached:** +- An upload field appears +- Selecting a JPG/PNG/WEBP file automatically triggers OCR and opens the scanner panel + +### Tips for Best OCR Results + +- Use a well-lit, in-focus photo +- Make sure the total amount is clearly visible +- Flatten crumpled receipts before photographing +- Portrait orientation works better than landscape +- Higher resolution = better accuracy +- If OCR misreads the amount, correct it manually — amounts are the most important field + +### What OCR Cannot Do + +- Read multi-page receipts (only the uploaded image is processed) +- Handle PDF receipts (PDF format is excluded from OCR; it can still be uploaded as an attachment) +- Guarantee 100% accuracy — always review extracted values before saving + +--- + +## USD → VND Rate Widget + +The dark widget on the dashboard shows the current USD to VND exchange rate. This is **for reference only** — it does not affect any transactions or calculations in the app. + +### How the Rate is Fetched + +1. **Primary:** Yahoo Finance forex (`USDVND=X` via yfinance) — most reliable +2. **Fallback:** ExchangeRate API (`open.er-api.com`) — free, no key required +3. **Stale fallback:** Last known rate from database — shown with a ⚠ indicator + +The rate is cached once per day. The daily cron job at 8AM always force-fetches a fresh rate. + +### Refreshing Manually + +Click the **↻** button (top-right of the widget) to force-fetch a fresh rate without reloading the page. The rate, date, and source label update in-place. + +### 30-Day History Chart + +Click anywhere on the widget (except the ↻ button) to toggle a compact line chart showing the rate trend over the last 30 days. + +### Stale Indicator + +If the cached rate is from a previous day and all live sources fail, a ⚠ symbol appears next to the date. This typically means the server has no internet access or the APIs are temporarily unavailable. + +--- + +## Keyboard Shortcuts & Tips + +### Navigation +- The sidebar collapses on desktop — click the ☰ button in the topbar to toggle. State is remembered across page loads. +- On mobile, the sidebar slides in as an overlay — tap anywhere outside to close it. + +### Forms +- On the AI chat input: **Enter** sends the message, **Shift+Enter** adds a new line +- On transaction forms: the date field defaults to today — change it if entering a past transaction +- On the investment transaction form: the **↓** button next to price fetches the current live price + +### Transaction Filters +- Filters persist within a tab session but reset when you switch tabs (Income ↔ Expense) +- Use the ✕ button to clear all filters at once +- Date range filter: both From and To are optional — leave one blank to filter from/to open-ended + +### Budgets +- The budget list for an empty month shows a "Copy from previous month" button — use this at the start of each month instead of re-entering all budgets +- Rollover amounts appear as a blue "+rollover" badge — hover to see the exact amount + +### Investments +- Click any row in the holdings table to go to the detail page +- Ticker symbols are case-insensitive on entry (auto-uppercased on save) +- The **Check** button on the ticker field verifies the ticker and shows the current price before you save + +### Reports +- The "Snapshot Now" button on the Reports page saves today's net worth to the history chart — do this manually if you want more data points than the monthly automatic snapshots +- CSV and Excel exports use the period currently selected in the report view + +### AI Assistant +- The AI does not have memory between sessions — each conversation starts fresh +- For best results, ask specific questions: "How much did I spend on food in March?" rather than "How am I doing?" +- The context includes the last 90 days of transactions — questions about older data may not be accurate \ No newline at end of file diff --git a/deploy.md b/deploy.md index 34aaf5e..294513b 100644 --- a/deploy.md +++ b/deploy.md @@ -1,5 +1,5 @@ -# PFMS — Server Deployment Guide -> Ubuntu Server · Nginx · Gunicorn · MySQL · Groq API +# PFM — Server Deployment Guide +> Ubuntu Server · Nginx · Gunicorn · MySQL · Groq API · Certbot SSL --- @@ -9,11 +9,16 @@ |------|-------| | OS | Ubuntu 22.04 LTS | | App user | `pfm` | -| App directory | `/home/pfm/app` | -| Domain / IP | `pfm.ngodanguyen.tech` | +| Home dir | `/home/pfm` | +| App directory | `/home/pfm/web` | +| Venv | `/home/pfm/venv` | +| Uploads | `/home/pfm/web/uploads` | +| Backups | `/home/pfm/web/backups` | +| Domain | `pfm.ngodanguyen.tech` | | SSL | Let's Encrypt via Certbot | -| MySQL version | 8.0 | +| MySQL | 8.0 | | Python | 3.11+ | +| Gitea | `gitea.ngodanguyen.tech` | --- @@ -21,21 +26,22 @@ ```bash sudo apt update && sudo apt upgrade -y -sudo apt install -y python3 python3-pip python3-venv python3-dev \ +sudo apt install -y \ + python3 python3-pip python3-venv python3-dev \ build-essential libssl-dev libffi-dev \ - nginx mysql-server git curl wget \ + nginx mysql-server git curl wget unzip \ libpango-1.0-0 libpangoft2-1.0-0 libharfbuzz0b \ - libffi-dev libjpeg-dev libopenjp2-7-dev + libjpeg-dev libopenjp2-7-dev ``` -> `libpango*` and related libs are required by WeasyPrint for PDF generation. +> `libpango*` libs required by WeasyPrint for PDF generation. --- ## 2. Create App User ```bash -# Skip if user pfm already exists +# Skip if pfm already exists id pfm &>/dev/null || sudo useradd -m -s /bin/bash pfm sudo usermod -aG www-data pfm ``` @@ -46,7 +52,7 @@ sudo usermod -aG www-data pfm ```bash sudo mysql_secure_installation -# Follow prompts: set root password, remove anonymous users, disable remote root +# Set root password, remove anonymous users, disable remote root ``` ```bash @@ -66,18 +72,21 @@ EXIT; ## 4. App Directory & Virtual Environment ```bash -sudo mkdir -p /home/pfm/app -sudo chown pfm:www-data /home/pfm/app -sudo chmod 750 /home/pfm/app +sudo mkdir -p /home/pfm/web +sudo chown pfm:www-data /home/pfm/web +sudo chmod 750 /home/pfm/web sudo -u pfm bash -cd /home/pfm/app +cd /home/pfm +# Create venv at /home/pfm/venv python3 -m venv venv source venv/bin/activate -# Clone from your Gitea -git clone https://gitea.yourdomain.com/youruser/pfm.git . +# Clone from Gitea +cd /home/pfm/web +GIT_SSH_COMMAND='ssh -i /home/pfm/.ssh/gitea_deploy' \ + git clone git@gitea.ngodanguyen.tech:youruser/pfm.git . pip install --upgrade pip pip install -r requirements.txt @@ -88,7 +97,7 @@ pip install -r requirements.txt ## 5. Environment File ```bash -sudo -u pfm nano /home/pfm/app/.env +sudo -u pfm nano /home/pfm/web/.env ``` ```env @@ -96,43 +105,53 @@ SECRET_KEY=generate-a-long-random-string-here DATABASE_URL=mysql+pymysql://pfm_user:STRONG_PASSWORD_HERE@localhost/pfm_db GROQ_API_KEY=your-groq-api-key-here GROQ_MODEL=llama-3.3-70b-versatile -UPLOAD_FOLDER=/home/pfm/app/uploads +UPLOAD_FOLDER=/home/pfm/web/uploads MAX_CONTENT_LENGTH=10485760 FLASK_ENV=production +FLASK_APP=wsgi:app APP_CURRENCY=USD APP_CURRENCY_SYMBOL=$ -APP_TIMEZONE=America/New_York +APP_TIMEZONE=Asia/Ho_Chi_Minh ``` ```bash -chmod 600 /home/pfm/app/.env +chmod 600 /home/pfm/web/.env ``` -Generate a secret key: +Generate secret key: ```bash python3 -c "import secrets; print(secrets.token_hex(32))" ``` --- -## 6. Initialize Database +## 6. Uploads & Backups Directories ```bash -cd /home/pfm/app -source venv/bin/activate - -flask db upgrade # Run all migrations -python scripts/init_db.py # Seed default categories + create admin user +sudo mkdir -p /home/pfm/web/uploads +sudo mkdir -p /home/pfm/web/backups +sudo chown pfm:www-data /home/pfm/web/uploads +sudo chown pfm:pfm /home/pfm/web/backups +sudo chmod 750 /home/pfm/web/uploads ``` --- -## 7. Upload Directory +## 7. Initialize Database ```bash -sudo mkdir -p /home/pfm/app/uploads -sudo chown pfm:www-data /home/pfm/app/uploads -sudo chmod 750 /home/pfm/app/uploads +cd /home/pfm/web +source /home/pfm/venv/bin/activate + +# Remove stray __init__.py from project root if present +rm -f /home/pfm/web/__init__.py + +flask db init +flask db migrate -m "initial schema" +flask db upgrade + +# Seed categories + create admin user (interactive) +python scripts/init_db.py ``` --- @@ -145,16 +164,17 @@ sudo nano /etc/systemd/system/pfm.service ```ini [Unit] -Description=PFMS — Personal Finance Management System +Description=PFM — Personal Finance Management System After=network.target mysql.service [Service] User=pfm Group=www-data -WorkingDirectory=/home/pfm/app -Environment="PATH=/home/pfm/app/venv/bin" -EnvironmentFile=/home/pfm/app/.env -ExecStart=/home/pfm/app/venv/bin/gunicorn \ +WorkingDirectory=/home/pfm/web +Environment="PATH=/home/pfm/venv/bin" +EnvironmentFile=/home/pfm/web/.env +ExecStart=/home/pfm/venv/bin/gunicorn \ + --chdir /home/pfm/web \ --workers 2 \ --bind unix:/run/pfm/pfm.sock \ --timeout 120 \ @@ -173,7 +193,6 @@ WantedBy=multi-user.target ``` ```bash -# Create log directory sudo mkdir -p /var/log/pfm sudo chown pfm:www-data /var/log/pfm @@ -195,7 +214,6 @@ sudo nano /etc/nginx/sites-available/pfm server { listen 80; server_name pfm.ngodanguyen.tech; - # Certbot will update this block automatically return 301 https://$host$request_uri; } @@ -203,7 +221,7 @@ server { listen 443 ssl http2; server_name pfm.ngodanguyen.tech; - # Certbot will populate these after running certbot + # Certbot populates these: # ssl_certificate /etc/letsencrypt/live/pfm.ngodanguyen.tech/fullchain.pem; # ssl_certificate_key /etc/letsencrypt/live/pfm.ngodanguyen.tech/privkey.pem; # include /etc/letsencrypt/options-ssl-nginx.conf; @@ -221,7 +239,7 @@ server { proxy_connect_timeout 10s; } - # AI chat uses SSE — disable buffering + # AI chat SSE — disable buffering location /ai/stream { proxy_pass http://unix:/run/pfm/pfm.sock; proxy_set_header Host $host; @@ -235,13 +253,14 @@ server { } location /static/ { - alias /home/pfm/app/app/static/; + alias /home/pfm/web/app/static/; expires 30d; add_header Cache-Control "public, immutable"; } + # Receipts — served via Flask send_from_directory (internal) location /uploads/ { - alias /home/pfm/app/uploads/; + alias /home/pfm/web/uploads/; internal; } @@ -261,78 +280,46 @@ sudo systemctl reload nginx ## 9a. SSL — Certbot (Let's Encrypt) ```bash -# Install Certbot + Nginx plugin sudo apt install -y certbot python3-certbot-nginx -# Issue certificate — auto-edits the Nginx config above +# Issue cert — auto-edits Nginx config sudo certbot --nginx -d pfm.ngodanguyen.tech -# Follow prompts: -# - Enter email for renewal notices -# - Agree to TOS -# - Choose redirect HTTP → HTTPS (recommended) - -# Verify cert +# Verify sudo certbot certificates -``` -Certbot automatically: -- Populates the `ssl_certificate` lines in the Nginx config -- Sets up HTTP → HTTPS redirect -- Installs a systemd timer for auto-renewal - -```bash # Test auto-renewal sudo certbot renew --dry-run - -# Check renewal timer sudo systemctl status certbot.timer ``` --- -## 10. Groq API Setup (Free AI) - -No local installation needed. Groq runs in the cloud — just get a free API key. - -### 10.1 Get API Key +## 10. Groq API Setup 1. Sign up at **https://console.groq.com** -2. Go to **API Keys** → **Create API key** -3. Copy the key into `/home/pfm/app/.env` as `GROQ_API_KEY` +2. API Keys → Create API key +3. Add to `/home/pfm/web/.env` as `GROQ_API_KEY` -### 10.2 Free Tier Limits +### Models in Use -| Metric | Limit | -|--------|-------| -| Requests/day | 14,400 | -| Tokens/minute | 500,000 | -| Tokens/day | 500,000 (varies by model) | -| Cost | Free | +| Model | Used For | +|-------|----------| +| `llama-3.3-70b-versatile` | Chat + daily insights (default) | +| `llama-3.1-8b-instant` | Fast queries (user-selectable) | +| `meta-llama/llama-4-scout-17b-16e-instruct` | Receipt OCR (vision) — hardcoded in ocr_service.py | -More than enough for a personal finance assistant. - -### 10.3 Recommended Models - -| Model | Speed | Quality | Best For | -|-------|-------|---------|----------| -| `llama-3.3-70b-versatile` | Fast | High | Default — chat + insights | -| `llama-3.1-8b-instant` | Very fast | Good | Quick queries, daily summaries | - -Set in `.env` via `GROQ_MODEL`. Switchable from Settings UI. - -### 10.4 Verify Key Works +### Free Tier Limits +14,400 requests/day · 500,000 tokens/minute · Free +### Verify Key ```bash -curl https://api.groq.com/openai/v1/chat/completions -H "Authorization: Bearer YOUR_GROQ_API_KEY" -H "Content-Type: application/json" -d '{ - "model": "llama-3.3-70b-versatile", - "messages": [{"role": "user", "content": "Hello"}], - "max_tokens": 10 - }' +curl https://api.groq.com/openai/v1/chat/completions \ + -H "Authorization: Bearer YOUR_GROQ_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"model":"llama-3.3-70b-versatile","messages":[{"role":"user","content":"Hello"}],"max_tokens":10}' ``` -Should return a JSON response with a greeting. - --- ## 11. Scheduled Jobs (systemd timers) @@ -342,28 +329,25 @@ Should return a JSON response with a greeting. ```bash sudo nano /etc/systemd/system/pfm-recurring.service ``` - ```ini [Unit] -Description=PFMS — Process Recurring Transactions +Description=PFM — Process Recurring Transactions After=mysql.service [Service] Type=oneshot User=pfm -WorkingDirectory=/home/pfm/app -Environment="PATH=/home/pfm/app/venv/bin" -EnvironmentFile=/home/pfm/app/.env -ExecStart=/home/pfm/app/venv/bin/python scripts/process_recurring.py +WorkingDirectory=/home/pfm/web +Environment="PATH=/home/pfm/venv/bin" +EnvironmentFile=/home/pfm/web/.env +ExecStart=/home/pfm/venv/bin/python scripts/process_recurring.py ``` - ```bash sudo nano /etc/systemd/system/pfm-recurring.timer ``` - ```ini [Unit] -Description=PFMS — Recurring Transactions Timer +Description=PFM — Recurring Transactions Timer [Timer] OnCalendar=*-*-* 06:00:00 @@ -373,32 +357,61 @@ Persistent=true WantedBy=timers.target ``` -### 11.2 Fetch Investment Prices (Daily 4PM weekdays) +### 11.2 Fetch USD/VND Rate (Daily 8AM) ```bash -sudo nano /etc/systemd/system/pfm-prices.service +sudo nano /etc/systemd/system/pfm-fxrate.service ``` - ```ini [Unit] -Description=PFMS — Fetch Investment Prices +Description=PFM — Fetch USD/VND Exchange Rate [Service] Type=oneshot User=pfm -WorkingDirectory=/home/pfm/app -Environment="PATH=/home/pfm/app/venv/bin" -EnvironmentFile=/home/pfm/app/.env -ExecStart=/home/pfm/app/venv/bin/python scripts/fetch_prices.py +WorkingDirectory=/home/pfm/web +Environment="PATH=/home/pfm/venv/bin" +EnvironmentFile=/home/pfm/web/.env +ExecStart=/home/pfm/venv/bin/python scripts/fetch_fx_rate.py +``` +```bash +sudo nano /etc/systemd/system/pfm-fxrate.timer +``` +```ini +[Unit] +Description=PFM — FX Rate Timer + +[Timer] +OnCalendar=*-*-* 08:00:00 +Persistent=true + +[Install] +WantedBy=timers.target ``` +### 11.3 Fetch Investment Prices (Mon-Fri 4PM) + +```bash +sudo nano /etc/systemd/system/pfm-prices.service +``` +```ini +[Unit] +Description=PFM — Fetch Investment Prices + +[Service] +Type=oneshot +User=pfm +WorkingDirectory=/home/pfm/web +Environment="PATH=/home/pfm/venv/bin" +EnvironmentFile=/home/pfm/web/.env +ExecStart=/home/pfm/venv/bin/python scripts/fetch_prices.py +``` ```bash sudo nano /etc/systemd/system/pfm-prices.timer ``` - ```ini [Unit] -Description=PFMS — Price Fetch Timer +Description=PFM — Price Fetch Timer [Timer] OnCalendar=Mon-Fri *-*-* 16:00:00 @@ -408,32 +421,29 @@ Persistent=true WantedBy=timers.target ``` -### 11.3 Net Worth Snapshot (1st of month) +### 11.4 Net Worth Snapshot (1st of month) ```bash sudo nano /etc/systemd/system/pfm-snapshot.service ``` - ```ini [Unit] -Description=PFMS — Net Worth Snapshot +Description=PFM — Net Worth Snapshot [Service] Type=oneshot User=pfm -WorkingDirectory=/home/pfm/app -Environment="PATH=/home/pfm/app/venv/bin" -EnvironmentFile=/home/pfm/app/.env -ExecStart=/home/pfm/app/venv/bin/python scripts/daily_snapshot.py +WorkingDirectory=/home/pfm/web +Environment="PATH=/home/pfm/venv/bin" +EnvironmentFile=/home/pfm/web/.env +ExecStart=/home/pfm/venv/bin/python scripts/daily_snapshot.py ``` - ```bash sudo nano /etc/systemd/system/pfm-snapshot.timer ``` - ```ini [Unit] -Description=PFMS — Net Worth Snapshot Timer +Description=PFM — Snapshot Timer [Timer] OnCalendar=*-*-01 00:05:00 @@ -443,33 +453,30 @@ Persistent=true WantedBy=timers.target ``` -### 11.4 AI Daily Insight (Midnight) +### 11.5 AI Daily Insight (Midnight) ```bash sudo nano /etc/systemd/system/pfm-aiinsight.service ``` - ```ini [Unit] -Description=PFMS — AI Daily Insight +Description=PFM — AI Daily Insight After=network-online.target [Service] Type=oneshot User=pfm -WorkingDirectory=/home/pfm/app -Environment="PATH=/home/pfm/app/venv/bin" -EnvironmentFile=/home/pfm/app/.env -ExecStart=/home/pfm/app/venv/bin/python scripts/daily_ai_insight.py +WorkingDirectory=/home/pfm/web +Environment="PATH=/home/pfm/venv/bin" +EnvironmentFile=/home/pfm/web/.env +ExecStart=/home/pfm/venv/bin/python scripts/daily_ai_insight.py ``` - ```bash sudo nano /etc/systemd/system/pfm-aiinsight.timer ``` - ```ini [Unit] -Description=PFMS — AI Insight Timer +Description=PFM — AI Insight Timer [Timer] OnCalendar=*-*-* 00:01:00 @@ -479,41 +486,6 @@ Persistent=true WantedBy=timers.target ``` -### 11.5 Fetch USD/VND Rate (Daily 8AM) - -```bash -sudo nano /etc/systemd/system/pfm-fxrate.service -``` - -```ini -[Unit] -Description=PFMS — Fetch USD/VND Exchange Rate - -[Service] -Type=oneshot -User=pfm -WorkingDirectory=/home/pfm/app -Environment="PATH=/home/pfm/app/venv/bin" -EnvironmentFile=/home/pfm/app/.env -ExecStart=/home/pfm/app/venv/bin/python scripts/fetch_fx_rate.py -``` - -```bash -sudo nano /etc/systemd/system/pfm-fxrate.timer -``` - -```ini -[Unit] -Description=PFMS — FX Rate Fetch Timer - -[Timer] -OnCalendar=*-*-* 08:00:00 -Persistent=true - -[Install] -WantedBy=timers.target -``` - ### Enable All Timers ```bash @@ -530,12 +502,56 @@ sudo systemctl list-timers --all | grep pfm --- -## 12. Log Rotation +## 12. Database Backup (Daily 2AM) + +```bash +sudo nano /etc/systemd/system/pfm-backup.service +``` +```ini +[Unit] +Description=PFM — MySQL Backup + +[Service] +Type=oneshot +User=pfm +ExecStart=/bin/bash -c 'mysqldump --defaults-file=/home/pfm/.my.cnf pfm_db | gzip > /home/pfm/web/backups/pfm_$(date +\%Y\%m\%d).sql.gz && find /home/pfm/web/backups -name "*.sql.gz" -mtime +30 -delete' +``` +```bash +sudo nano /etc/systemd/system/pfm-backup.timer +``` +```ini +[Unit] +Description=PFM — Backup Timer + +[Timer] +OnCalendar=*-*-* 02:00:00 +Persistent=true + +[Install] +WantedBy=timers.target +``` + +```bash +sudo systemctl enable --now pfm-backup.timer +``` + +Store MySQL credentials for backup job: +```bash +sudo -u pfm bash -c 'cat > /home/pfm/.my.cnf << EOF +[client] +user=pfm_user +password=STRONG_PASSWORD_HERE +EOF +chmod 600 /home/pfm/.my.cnf' +``` + +--- + +## 13. Log Rotation ```bash sudo nano /etc/logrotate.d/pfm ``` - ``` /var/log/pfm/*.log { daily @@ -553,59 +569,6 @@ sudo nano /etc/logrotate.d/pfm --- -## 13. Database Backup (Daily) - -```bash -sudo nano /etc/systemd/system/pfm-backup.service -``` - -```ini -[Unit] -Description=PFMS — MySQL Backup - -[Service] -Type=oneshot -User=pfm -ExecStart=/bin/bash -c 'mysqldump --defaults-file=/home/pfm/.my.cnf pfm_db | gzip > /home/pfm/app/backups/pfm_$(date +\%Y\%m\%d).sql.gz' -``` - -```bash -sudo nano /etc/systemd/system/pfm-backup.timer -``` - -```ini -[Unit] -Description=PFMS — Daily Backup Timer - -[Timer] -OnCalendar=*-*-* 02:00:00 -Persistent=true - -[Install] -WantedBy=timers.target -``` - -```bash -sudo mkdir -p /home/pfm/app/backups -sudo chown pfm:pfm /home/pfm/app/backups -sudo systemctl enable --now pfm-backup.timer -``` - -> Store MySQL credentials securely for the backup job: -> ```bash -> sudo -u pfm bash -c 'cat > /home/pfm/.my.cnf << EOF -> [client] -> user=pfm_user -> password=STRONG_PASSWORD_HERE -> EOF -> chmod 600 /home/pfm/.my.cnf' -> ``` - -> Keep last 30 days. Add a cleanup cron to delete older files: -> `find /home/pfm/app/backups -name "*.sql.gz" -mtime +30 -delete` - ---- - ## 14. Firewall (UFW) ```bash @@ -616,21 +579,30 @@ sudo ufw enable sudo ufw status ``` -> MySQL (3306) is NOT opened — internal only. Groq calls go outbound over HTTPS (port 443). +> MySQL (3306) — internal only. Groq/yfinance — outbound HTTPS (443). --- -## 15. Deploy Update Workflow +## 15. Gitea Deploy Key + +```bash +sudo -u pfm ssh-keygen -t ed25519 -f /home/pfm/.ssh/gitea_deploy -N "" +cat /home/pfm/.ssh/gitea_deploy.pub +# Add to Gitea: repo → Settings → Deploy Keys +``` + +--- + +## 16. Deploy Update Workflow ```bash -# SSH into server sudo -u pfm bash -cd /home/pfm/app -source venv/bin/activate +cd /home/pfm/web +source /home/pfm/venv/bin/activate git pull origin main -pip install -r requirements.txt # if deps changed -flask db upgrade # if migrations added +pip install -r requirements.txt # only if requirements changed +flask db upgrade # only if new migrations sudo systemctl restart pfm sudo systemctl status pfm @@ -638,35 +610,50 @@ sudo systemctl status pfm --- -## 16. Quick Status Check +## 17. First-Run Checklist ```bash -# All PFMS services -sudo systemctl status pfm -sudo systemctl status nginx -sudo systemctl status mysql +# 1. Force fresh FX rate +source /home/pfm/venv/bin/activate +cd /home/pfm/web +python scripts/fetch_fx_rate.py +# Should print: 1 USD = ₫26,xxx VND [yfinance] -# Logs (live) -sudo journalctl -u pfm -f -sudo tail -f /var/log/pfm/error.log -sudo tail -f /var/log/nginx/pfm_error.log +# 2. Verify all timers +sudo systemctl list-timers --all | grep pfm -# Groq API check +# 3. Test AI curl https://api.groq.com/openai/v1/models \ - -H "Authorization: Bearer $(grep GROQ_API_KEY /home/pfm/app/.env | cut -d= -f2)" + -H "Authorization: Bearer $(grep GROQ_API_KEY /home/pfm/web/.env | cut -d= -f2)" + +# 4. Check app logs +sudo journalctl -u pfm -f + +# 5. Full service status +sudo systemctl status pfm nginx mysql ``` --- -## 17. Gitea Integration (your self-hosted) +## 18. Quick Status Reference ```bash -# On server — set up deploy key -sudo -u pfm ssh-keygen -t ed25519 -f /home/pfm/.ssh/gitea_deploy -N "" -cat /home/pfm/.ssh/gitea_deploy.pub -# Add this public key to Gitea repo → Settings → Deploy Keys +# Services +sudo systemctl status pfm +sudo systemctl status nginx +sudo systemctl status mysql -# Clone with deploy key -sudo -u pfm GIT_SSH_COMMAND='ssh -i /home/pfm/.ssh/gitea_deploy' \ - git clone git@gitea.yourdomain.com:youruser/pfm.git /home/pfm/app +# Live logs +sudo journalctl -u pfm -f +sudo tail -f /var/log/pfm/error.log +sudo tail -f /var/log/nginx/pfm_error.log + +# Timers +sudo systemctl list-timers --all | grep pfm + +# Run a cron manually +source /home/pfm/venv/bin/activate && cd /home/pfm/web +python scripts/fetch_fx_rate.py +python scripts/process_recurring.py +python scripts/daily_ai_insight.py ```