06/15 Update Claude.md

CLAUDE.md

@@ -662,22 +662,335 @@ Or create user with `IDENTIFIED WITH mysql_native_password BY '...'`.
 
 ---
 
-## 18. Build Roadmap
+## 18. Build Roadmap & To-Do List
 
-| Phase | Status | Deliverable |
-|---|---|---|
-| 1 — Foundation | ✅ Done | App factory, config, auth (register/verify/login/reset), RBAC, i18n scaffold, Turnstile, seed plans |
-| 2 — Listings core | ✅ Done | Categories + field_schema, listing CRUD, image pipeline, browse/search/radius, expiry sweep |
-| 3 — Messaging + favorites | ✅ Done | Conversations, messages, read tracking, contact masking, favorites, unread badge |
-| 4 — Monetization | 🔲 Next | Stripe Billing, plans/tier enforcement, Customer Portal, à la carte boosts, webhooks |
-| 5 — Ads & sponsors | 🔲 | Internal ad server, impression/click tracking, sponsor directory, promoted search |
-| 6 — Admin backend | 🔲 | Dashboard, moderation queue, user mgmt, category/plan editors, audit log, analytics |
-| 7 — Polish | 🔲 | Metro SEO pages, sitemaps, structured data, email notifications, reviews, perf tuning |
-| 8 — JSON API | 🔲 Optional | iOS app endpoints (mirrors JQC/CitizenReady pattern) |
+### ✅ Phase 1 — Foundation (Done)
+- [x] App factory, Dev/Prod config, env-driven settings
+- [x] Extensions: SQLAlchemy, Migrate, LoginManager, CSRF, Babel, Limiter
+- [x] ProxyFix for Nginx X-Forwarded headers
+- [x] `users`, `plans`, `trust_events` schema + Alembic migrations
+- [x] Argon2 password hashing, itsdangerous signed tokens
+- [x] Auth flows: register, email verify (+5 trust), login, logout, password reset
+- [x] Rate limiting on auth endpoints (Flask-Limiter + Redis)
+- [x] RBAC decorators: `role_required`, `admin_required`, `moderator_required`
+- [x] Trilingual i18n scaffold (EN/VI/ES), locale switcher `/lang/<code>`
+- [x] Accent-insensitive normalizer `utils/text.py` (phở→pho, ñ→n)
+- [x] Cloudflare Turnstile CAPTCHA hook (dev bypass when keys blank)
+- [x] `seed.py`: 4 plans with JSON limit configs + `--admin` flag
+- [x] systemd + Gunicorn + Nginx deploy files
+- [x] Smoke test: 14 checks
 
-**Phase 2 deferred items:**
-- Auto-translate button → `listing_translations` cache (DeepL/Google)
-- `listing_translations` table schema already defined above
+---
+
+### ✅ Phase 2 — Listings Core (Done)
+- [x] `categories` model (self-referential subcategories, `field_schema` JSON)
+- [x] `listings` + `listing_images` models (title_norm, lat/lng, hot columns, status, expiry)
+- [x] `zip_geo` + `metros` models
+- [x] 6 categories + subcategories seeded with field schemas
+- [x] 10 sample ZIP rows seeded (Westminster, Houston, San Jose, Miami, Falls Church…)
+- [x] Field schema validation service (text/number/select/bool, required, coerce)
+- [x] Hot-column denormalization (attr_condition, attr_job_type, attr_salary_*)
+- [x] Listing CRUD routes: create, edit, delete, mark-sold
+- [x] Tier enforcement: active-listing cap, image cap, listing life from plan config
+- [x] Image pipeline: MIME validate, Pillow re-encode to JPEG, strip EXIF, thumbnail, random filenames
+- [x] Browse + filter: category, state, price range, condition, keyword
+- [x] Accent-insensitive keyword search via `title_norm LIKE`
+- [x] Radius search: bounding-box SQL prefilter + exact haversine refine
+- [x] View counter on listing detail
+- [x] My-listings page with active/cap display
+- [x] `flask expire-listings` CLI + systemd timer (hourly sweep)
+- [x] Dev media serving route `/media/<path>`
+- [x] Smoke test: +6 checks (geocode, tier limit, attr validation, search, radius, image, expiry)
+
+---
+
+### ✅ Phase 3 — Messaging + Favorites (Done)
+- [x] `conversations` model (unique per listing+buyer, seller auto-set)
+- [x] `messages` model (append-only, read_at tracking)
+- [x] `favorites` model (unique user+listing)
+- [x] Messaging service: get-or-create conversation, send, mark-read, inbox, total-unread
+- [x] Self-message block (seller cannot contact own listing)
+- [x] Contact density heuristic (≥3 phone/email/URL signals → warning log)
+- [x] Contact masking service: mask phone/email/URL for low-trust users
+- [x] Trust-gated contact reveal (email_verified AND trust_tier ≥ trusted)
+- [x] Notification email to recipient on every new message
+- [x] Favorites service: toggle, is_favorited, paginated user favorites
+- [x] Inbox route with unread badge per conversation
+- [x] Conversation thread route (GET masked, POST send reply)
+- [x] Start-conversation from listing detail
+- [x] Toggle-favorite endpoint (AJAX-capable + form fallback)
+- [x] My-favorites page (saved listings grid)
+- [x] `/api/unread` JSON endpoint
+- [x] Unread count injected into every page via context processor (nav badge)
+- [x] Nav: Messages + Saved links added
+- [x] Listing detail: Contact seller + ♥ Save listing buttons for non-owners
+- [x] Smoke test: +15 checks (conversation, messages, unread, masking, favorites, routes)
+
+---
+
+### 🔲 Phase 4 — Monetization / Stripe (Next)
+
+**Models**
+- [ ] `subscriptions` table (stripe_customer_id, stripe_sub_id, status, period_end, cancel_at_period_end)
+- [ ] `transactions` table (type: subscription/boost/refund, amount_cents, stripe_object_id)
+- [ ] `boosts` table (listing_id, type: featured/bump/highlight/urgent, expires_at)
+- [ ] Alembic migration for all three tables
+
+**Stripe setup**
+- [ ] `stripe` pip package added to requirements.txt
+- [ ] Stripe keys in `.env` (`STRIPE_SECRET_KEY`, `STRIPE_PUBLISHABLE_KEY`, `STRIPE_WEBHOOK_SECRET`)
+- [ ] Create Products + Prices in Stripe dashboard for Basic/Pro/Business plans
+- [ ] Map `stripe_price_id` in `plans` table (via seed or admin)
+- [ ] Create boost Products in Stripe (featured, bump, highlight, urgent) + prices
+
+**Subscriptions**
+- [ ] `payments` blueprint: `/billing/subscribe/<plan_slug>` → Stripe Checkout Session
+- [ ] Redirect to Stripe-hosted checkout, success/cancel return URLs
+- [ ] Stripe Customer Portal route `/billing/portal` (self-serve upgrade/cancel/update card)
+- [ ] Webhook endpoint `/billing/webhook` — verify signature on every event
+- [ ] Handle `checkout.session.completed` → create/update `subscriptions` row, set `users.role=subscriber`, assign `tier_id`
+- [ ] Handle `customer.subscription.updated` → sync status, period_end, plan change
+- [ ] Handle `customer.subscription.deleted` → downgrade to free plan, clear tier
+- [ ] Handle `invoice.payment_failed` → flip subscription status to `past_due`, email user
+- [ ] Nightly reconcile job: compare local subscription status vs Stripe API (catch missed webhooks)
+- [ ] Stripe Tax enabled on checkout (auto US sales tax)
+
+**À la carte boosts (free + paid users)**
+- [ ] Boost purchase route `/listings/<id>/boost` → Stripe Payment Intent (one-off)
+- [ ] Boost type selector UI (featured / bump / highlight / urgent + price display)
+- [ ] On payment success: write `boosts` row with `expires_at`, flip `listing.is_featured` if featured boost
+- [ ] Boost expiry sweep added to `expire-listings` worker (clear expired boosts)
+- [ ] Browse query: order featured (boosted) listings first
+
+**Tier enforcement upgrades**
+- [ ] Enforce `featured_per_month` limit from plan config (count active featured boosts)
+- [ ] Enforce `scheduled_posting` — gate date-picker on plan check
+- [ ] Enforce `ad_free` — suppress ads for subscriber role
+- [ ] Upgrade prompt shown when free user hits any limit
+
+**UI**
+- [ ] Pricing page `/pricing` showing plan comparison table
+- [ ] Account billing page `/my/billing` (current plan, next renewal, manage button → portal)
+- [ ] Boost buttons on listing detail (owner-only) and my-listings page
+- [ ] Payment success / cancel flash pages
+
+**Tests**
+- [ ] Smoke test: Stripe webhook handler with test payload (mock signature)
+- [ ] Smoke test: boost creation writes correct `boosts` row
+- [ ] Smoke test: subscription downgrade clears tier
+
+---
+
+### 🔲 Phase 5 — Ads & Sponsors
+
+**Models**
+- [ ] `ads` table (slot, creative_path, target_url, lang, geo_state, schedule, impressions, clicks)
+- [ ] `sponsors` table (tier: directory/category, category_id, schedule)
+- [ ] Alembic migration
+
+**Ad server**
+- [ ] Ad service: `get_ad_for_slot(slot, lang, state)` — picks active ad matching targeting, returns one
+- [ ] Impression tracking: increment `ads.impressions` on serve (async-safe, batched to Redis then flush)
+- [ ] Click tracking: redirect endpoint `/ads/<id>/click` → increment `ads.clicks` → redirect to `target_url`
+- [ ] Ad slots in templates: header banner, sidebar (browse page), inline (every 6th listing card), footer
+- [ ] Ads suppressed for `role=subscriber` (check `plan.limit('ad_free')`)
+- [ ] Ad creative upload (admin): validate image, Pillow re-encode, store in `/media/ads/`
+
+**Sponsors**
+- [ ] Sponsor directory page `/sponsors`
+- [ ] Category sponsorship: "Jobs powered by X" banner on category browse
+- [ ] Sponsor logo in nav or footer (tier: directory)
+- [ ] Sponsor admin CRUD
+
+**Promoted search**
+- [ ] `promoted_keywords` table (keyword, listing_id, expires_at, priority)
+- [ ] Browse query: prepend promoted listings matching keyword before organic results
+- [ ] Admin UI to assign promoted keyword slots
+
+**Tests**
+- [ ] Smoke: ad served for correct slot/lang/state targeting
+- [ ] Smoke: click redirect increments counter
+- [ ] Smoke: subscriber sees no ads
+
+---
+
+### 🔲 Phase 6 — Admin Backend
+
+**Dashboard**
+- [ ] `/admin` dashboard: KPI cards (active listings, new users 7d/30d, MRR, ad revenue, flag-queue depth)
+- [ ] Mini charts: signups/day, listings/day (last 30d), revenue trend
+
+**User management**
+- [ ] `/admin/users` — searchable/filterable table (role, status, tier, trust)
+- [ ] User detail page: profile, listing history, subscription, trust events
+- [ ] Actions: ban/suspend/activate, tier override, trust score adjust, impersonate (logs to audit_log)
+- [ ] Bulk actions: ban selected, send email to selected
+
+**Listing moderation**
+- [ ] `/admin/listings` — flag queue sorted by `flag_count × recency`
+- [ ] Quick actions per listing: approve (clear flags), hide (flagged), remove, view
+- [ ] Bulk approve / bulk remove
+- [ ] Auto-flag threshold: N distinct-user flags → auto-flip to `flagged` (setting in `settings` table)
+- [ ] Keyword blocklist editor (stored in `settings`, checked on listing submit)
+- [ ] Duplicate body detection (hash `body` on submit, reject/flag if seen within 24h)
+
+**Reports queue**
+- [ ] `/admin/reports` — flagged content with reporter reasons
+- [ ] Mark resolved / escalate actions
+- [ ] `reports` table: unique per reporter+listing, reason enum
+
+**Category management**
+- [ ] `/admin/categories` — CRUD, reorder (drag or sort_order field)
+- [ ] Field schema editor per category (JSON form builder: add/remove fields, set type/required/options)
+- [ ] Preview of field schema as it would appear on listing form
+
+**Plan / pricing management**
+- [ ] `/admin/plans` — edit plan config JSON (limits), name, price, Stripe price ID, toggle active
+- [ ] No-redeploy: limits read at runtime from DB
+
+**Ads & sponsors management**
+- [ ] `/admin/ads` — upload creative, set slot/targeting/schedule, view impression/click stats
+- [ ] `/admin/sponsors` — CRUD sponsor entries, assign category
+- [ ] Ad performance report (impressions, clicks, CTR per ad)
+
+**Transactions & billing**
+- [ ] `/admin/transactions` — full log with filter by type/status/date
+- [ ] Refund action (calls Stripe Refund API, writes refund transaction row)
+- [ ] Failed payments list with retry action
+
+**Settings**
+- [ ] `/admin/settings` — toggle UI for all `settings` table keys:
+  - `registration_open` (bool)
+  - `ads_enabled` (bool)
+  - `maintenance_mode` (bool)
+  - `flag_threshold` (int, default 5)
+  - `new_user_trust_gate_days` (int)
+  - `contact_density_threshold` (int)
+
+**Audit log**
+- [ ] All admin write actions write to `audit_log` (actor, action, target, meta JSON)
+- [ ] `/admin/audit` — searchable audit trail
+
+**Analytics**
+- [ ] Traffic: page views/day, top pages, search terms used
+- [ ] Conversions: registrations, listings posted, messages sent, boosts purchased
+- [ ] Top categories by listing count and by view count
+- [ ] Top metros by listing count (Phase 7 SEO planning)
+
+**Tests**
+- [ ] Smoke: admin dashboard 200, non-admin gets 403
+- [ ] Smoke: flag threshold auto-flips listing status
+- [ ] Smoke: impersonate logs to audit_log
+
+---
+
+### 🔲 Phase 7 — Polish
+
+**SEO & discoverability**
+- [ ] Metro landing pages `/classifieds/<metro-slug>` (e.g. `/classifieds/orange-county`)
+- [ ] State landing pages `/classifieds/state/<state>`
+- [ ] Category landing pages `/classifieds/category/<slug>`
+- [ ] Dynamic `<title>` and `<meta description>` on all pages
+- [ ] JSON-LD structured data on listing detail (Product schema)
+- [ ] XML sitemap (`/sitemap.xml`) — listings + categories + metros, auto-updated
+- [ ] `robots.txt`
+- [ ] Open Graph tags (listing title, price, cover image) for social sharing
+
+**Email notifications**
+- [ ] Welcome email on registration (with verify link already sent — add branding)
+- [ ] Listing expiry warning email (3 days before `expires_at`)
+- [ ] Listing expired email (with renew CTA)
+- [ ] Message received email (already inline — move to RQ worker for async)
+- [ ] Payment failed email
+- [ ] Weekly digest email (new listings in saved categories/metro) — opt-in
+
+**Reviews**
+- [ ] `reviews` table (listing_id, author_id, rating TINYINT, body, created_at)
+- [ ] Leave review on completed transaction (mark-sold triggers prompt)
+- [ ] Seller aggregate rating on profile + listing detail
+- [ ] Moderation: flag/remove abusive reviews
+
+**Translation (deferred from Phase 2)**
+- [ ] Per-listing "Translate" button → DeepL/Google Translate API
+- [ ] Cache result in `listing_translations` (pay once per listing+lang)
+- [ ] Language filter on browse ("Show VI listings only")
+
+**Performance**
+- [ ] Switch sessions to Redis (`SESSION_TYPE=redis`)
+- [ ] Query-level caching for hot browse queries (Redis, 60s TTL)
+- [ ] Lazy-load listing images (native `loading="lazy"`)
+- [ ] Serve WebP thumbnails (Pillow WebP encode alongside JPEG)
+- [ ] Add MySQL `FULLTEXT(title, body)` index + switch keyword search to `MATCH ... AGAINST`
+- [ ] Pagination `LIMIT/OFFSET` → keyset pagination for large datasets
+
+**UX & mobile**
+- [ ] Responsive nav (hamburger menu on mobile)
+- [ ] Listing detail image gallery with lightbox
+- [ ] Infinite scroll OR "Load more" on browse (AJAX pagination)
+- [ ] Toast notifications (non-blocking flash messages)
+- [ ] "Back to results" link on listing detail (preserve filter state)
+- [ ] Listing preview before publish
+
+**i18n completion**
+- [ ] Extract all `_()` strings to `.pot` file
+- [ ] Translate all strings to Vietnamese (`vi`)
+- [ ] Translate all strings to Spanish (`es`)
+- [ ] Compile `.mo` files, test all three locales end-to-end
+
+**Tests**
+- [ ] Smoke: sitemap returns valid XML with listing URLs
+- [ ] Smoke: OG tags present on listing detail
+- [ ] Smoke: translation cached correctly in `listing_translations`
+
+---
+
+### 🔲 Phase 8 — JSON API (Optional, for iOS app)
+
+**Design**
+- [ ] RESTful JSON API under `/api/v1/`
+- [ ] JWT authentication (separate from session cookies)
+- [ ] API versioning strategy documented
+
+**Endpoints**
+- [ ] `POST /api/v1/auth/register`
+- [ ] `POST /api/v1/auth/login` → returns JWT
+- [ ] `GET  /api/v1/listings` (browse + all filters, returns paginated JSON)
+- [ ] `GET  /api/v1/listings/<id>`
+- [ ] `POST /api/v1/listings` (create, multipart for images)
+- [ ] `PUT  /api/v1/listings/<id>` (edit)
+- [ ] `DELETE /api/v1/listings/<id>`
+- [ ] `GET  /api/v1/messages` (inbox)
+- [ ] `GET  /api/v1/messages/<conv_id>`
+- [ ] `POST /api/v1/messages/<conv_id>` (send)
+- [ ] `POST /api/v1/listings/<id>/favorite`
+- [ ] `GET  /api/v1/categories`
+- [ ] `GET  /api/v1/zip/<zip>` (geocode lookup)
+- [ ] `GET  /api/v1/me` (profile + tier info)
+
+**Infrastructure**
+- [ ] JWT library added (`PyJWT` or `flask-jwt-extended`)
+- [ ] Rate limiting on all API endpoints (stricter than web)
+- [ ] API error responses: consistent `{"error": "...", "code": "..."}` JSON shape
+- [ ] CORS headers for iOS app origin
+- [ ] API smoke tests (separate from web smoke test)
+
+---
+
+### 🔲 Deferred Items (from earlier phases)
+
+- [ ] **Auto-translate** — per-listing translate button → DeepL/Google → `listing_translations` cache (moved to Phase 7)
+- [ ] **Full ZIP dataset** — replace 10-row sample with SimpleMaps US ZIP or Census ZCTA (~42k rows)
+- [ ] **MySQL spatial upgrade** — `POINT` generated column + `SPATIAL INDEX` + `ST_Distance_Sphere` (optional perf upgrade)
+- [ ] **MySQL FULLTEXT upgrade** — `FULLTEXT(title,body)` index + `MATCH ... AGAINST` (Phase 7)
+- [ ] **MySQL generated hot columns** — `GENERATED ALWAYS AS (JSON_EXTRACT(...))` replacing app-maintained `attr_*` columns (optional)
+- [ ] **RQ worker** — replace inline email sends + APScheduler with proper RQ queue + worker process
+- [ ] **Redis sessions** — switch from filesystem sessions to Redis-backed (`SESSION_TYPE=redis`)
+- [ ] **Bulk CSV listing upload** — Business tier feature (parse CSV, validate, batch-create listings)
+- [ ] **Scheduled posting** — Pro/Business: set `publish_at` datetime, worker flips to active
+- [ ] **Auto-bump/renew** — Pro (weekly) / Business (daily): worker resets `bump_at`, extends `expires_at`
+- [ ] **Storefront/profile page** — Pro/Business: public seller page `/seller/<username>` with all active listings
+- [ ] **Custom URL** — Business: vanity slug for storefront (e.g. `/shop/alices-cleaning`)
+- [ ] **Job-seeker posts** — reverse job listings ("I'm looking for work") under Jobs category
 
 ---
 
@@ -696,4 +1009,4 @@ Or create user with `IDENTIFIED WITH mysql_native_password BY '...'`.
 
 ---
 
-_End of spec. Phase 1–3 complete, 34 smoke-test checks green. Next: Phase 4 (Monetization / Stripe)._
+_End of spec. Phase 1–3 complete, 34 smoke-test checks green. Phases 4–8 fully detailed as to-do lists above. Next: Phase 4 (Monetization / Stripe)._