diff --git a/CLAUDE.md b/CLAUDE.md index fbc0641..9b2d89c 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -662,22 +662,335 @@ Or create user with `IDENTIFIED WITH mysql_native_password BY '...'`. --- -## 18. Build Roadmap +## 18. Build Roadmap & To-Do List -| Phase | Status | Deliverable | -|---|---|---| -| 1 — Foundation | ✅ Done | App factory, config, auth (register/verify/login/reset), RBAC, i18n scaffold, Turnstile, seed plans | -| 2 — Listings core | ✅ Done | Categories + field_schema, listing CRUD, image pipeline, browse/search/radius, expiry sweep | -| 3 — Messaging + favorites | ✅ Done | Conversations, messages, read tracking, contact masking, favorites, unread badge | -| 4 — Monetization | 🔲 Next | Stripe Billing, plans/tier enforcement, Customer Portal, à la carte boosts, webhooks | -| 5 — Ads & sponsors | 🔲 | Internal ad server, impression/click tracking, sponsor directory, promoted search | -| 6 — Admin backend | 🔲 | Dashboard, moderation queue, user mgmt, category/plan editors, audit log, analytics | -| 7 — Polish | 🔲 | Metro SEO pages, sitemaps, structured data, email notifications, reviews, perf tuning | -| 8 — JSON API | 🔲 Optional | iOS app endpoints (mirrors JQC/CitizenReady pattern) | +### ✅ Phase 1 — Foundation (Done) +- [x] App factory, Dev/Prod config, env-driven settings +- [x] Extensions: SQLAlchemy, Migrate, LoginManager, CSRF, Babel, Limiter +- [x] ProxyFix for Nginx X-Forwarded headers +- [x] `users`, `plans`, `trust_events` schema + Alembic migrations +- [x] Argon2 password hashing, itsdangerous signed tokens +- [x] Auth flows: register, email verify (+5 trust), login, logout, password reset +- [x] Rate limiting on auth endpoints (Flask-Limiter + Redis) +- [x] RBAC decorators: `role_required`, `admin_required`, `moderator_required` +- [x] Trilingual i18n scaffold (EN/VI/ES), locale switcher `/lang/` +- [x] Accent-insensitive normalizer `utils/text.py` (phở→pho, ñ→n) +- [x] Cloudflare Turnstile CAPTCHA hook (dev bypass when keys blank) +- [x] `seed.py`: 4 plans with JSON limit configs + `--admin` flag +- [x] systemd + Gunicorn + Nginx deploy files +- [x] Smoke test: 14 checks -**Phase 2 deferred items:** -- Auto-translate button → `listing_translations` cache (DeepL/Google) -- `listing_translations` table schema already defined above +--- + +### ✅ Phase 2 — Listings Core (Done) +- [x] `categories` model (self-referential subcategories, `field_schema` JSON) +- [x] `listings` + `listing_images` models (title_norm, lat/lng, hot columns, status, expiry) +- [x] `zip_geo` + `metros` models +- [x] 6 categories + subcategories seeded with field schemas +- [x] 10 sample ZIP rows seeded (Westminster, Houston, San Jose, Miami, Falls Church…) +- [x] Field schema validation service (text/number/select/bool, required, coerce) +- [x] Hot-column denormalization (attr_condition, attr_job_type, attr_salary_*) +- [x] Listing CRUD routes: create, edit, delete, mark-sold +- [x] Tier enforcement: active-listing cap, image cap, listing life from plan config +- [x] Image pipeline: MIME validate, Pillow re-encode to JPEG, strip EXIF, thumbnail, random filenames +- [x] Browse + filter: category, state, price range, condition, keyword +- [x] Accent-insensitive keyword search via `title_norm LIKE` +- [x] Radius search: bounding-box SQL prefilter + exact haversine refine +- [x] View counter on listing detail +- [x] My-listings page with active/cap display +- [x] `flask expire-listings` CLI + systemd timer (hourly sweep) +- [x] Dev media serving route `/media/` +- [x] Smoke test: +6 checks (geocode, tier limit, attr validation, search, radius, image, expiry) + +--- + +### ✅ Phase 3 — Messaging + Favorites (Done) +- [x] `conversations` model (unique per listing+buyer, seller auto-set) +- [x] `messages` model (append-only, read_at tracking) +- [x] `favorites` model (unique user+listing) +- [x] Messaging service: get-or-create conversation, send, mark-read, inbox, total-unread +- [x] Self-message block (seller cannot contact own listing) +- [x] Contact density heuristic (≥3 phone/email/URL signals → warning log) +- [x] Contact masking service: mask phone/email/URL for low-trust users +- [x] Trust-gated contact reveal (email_verified AND trust_tier ≥ trusted) +- [x] Notification email to recipient on every new message +- [x] Favorites service: toggle, is_favorited, paginated user favorites +- [x] Inbox route with unread badge per conversation +- [x] Conversation thread route (GET masked, POST send reply) +- [x] Start-conversation from listing detail +- [x] Toggle-favorite endpoint (AJAX-capable + form fallback) +- [x] My-favorites page (saved listings grid) +- [x] `/api/unread` JSON endpoint +- [x] Unread count injected into every page via context processor (nav badge) +- [x] Nav: Messages + Saved links added +- [x] Listing detail: Contact seller + ♥ Save listing buttons for non-owners +- [x] Smoke test: +15 checks (conversation, messages, unread, masking, favorites, routes) + +--- + +### 🔲 Phase 4 — Monetization / Stripe (Next) + +**Models** +- [ ] `subscriptions` table (stripe_customer_id, stripe_sub_id, status, period_end, cancel_at_period_end) +- [ ] `transactions` table (type: subscription/boost/refund, amount_cents, stripe_object_id) +- [ ] `boosts` table (listing_id, type: featured/bump/highlight/urgent, expires_at) +- [ ] Alembic migration for all three tables + +**Stripe setup** +- [ ] `stripe` pip package added to requirements.txt +- [ ] Stripe keys in `.env` (`STRIPE_SECRET_KEY`, `STRIPE_PUBLISHABLE_KEY`, `STRIPE_WEBHOOK_SECRET`) +- [ ] Create Products + Prices in Stripe dashboard for Basic/Pro/Business plans +- [ ] Map `stripe_price_id` in `plans` table (via seed or admin) +- [ ] Create boost Products in Stripe (featured, bump, highlight, urgent) + prices + +**Subscriptions** +- [ ] `payments` blueprint: `/billing/subscribe/` → Stripe Checkout Session +- [ ] Redirect to Stripe-hosted checkout, success/cancel return URLs +- [ ] Stripe Customer Portal route `/billing/portal` (self-serve upgrade/cancel/update card) +- [ ] Webhook endpoint `/billing/webhook` — verify signature on every event +- [ ] Handle `checkout.session.completed` → create/update `subscriptions` row, set `users.role=subscriber`, assign `tier_id` +- [ ] Handle `customer.subscription.updated` → sync status, period_end, plan change +- [ ] Handle `customer.subscription.deleted` → downgrade to free plan, clear tier +- [ ] Handle `invoice.payment_failed` → flip subscription status to `past_due`, email user +- [ ] Nightly reconcile job: compare local subscription status vs Stripe API (catch missed webhooks) +- [ ] Stripe Tax enabled on checkout (auto US sales tax) + +**À la carte boosts (free + paid users)** +- [ ] Boost purchase route `/listings//boost` → Stripe Payment Intent (one-off) +- [ ] Boost type selector UI (featured / bump / highlight / urgent + price display) +- [ ] On payment success: write `boosts` row with `expires_at`, flip `listing.is_featured` if featured boost +- [ ] Boost expiry sweep added to `expire-listings` worker (clear expired boosts) +- [ ] Browse query: order featured (boosted) listings first + +**Tier enforcement upgrades** +- [ ] Enforce `featured_per_month` limit from plan config (count active featured boosts) +- [ ] Enforce `scheduled_posting` — gate date-picker on plan check +- [ ] Enforce `ad_free` — suppress ads for subscriber role +- [ ] Upgrade prompt shown when free user hits any limit + +**UI** +- [ ] Pricing page `/pricing` showing plan comparison table +- [ ] Account billing page `/my/billing` (current plan, next renewal, manage button → portal) +- [ ] Boost buttons on listing detail (owner-only) and my-listings page +- [ ] Payment success / cancel flash pages + +**Tests** +- [ ] Smoke test: Stripe webhook handler with test payload (mock signature) +- [ ] Smoke test: boost creation writes correct `boosts` row +- [ ] Smoke test: subscription downgrade clears tier + +--- + +### 🔲 Phase 5 — Ads & Sponsors + +**Models** +- [ ] `ads` table (slot, creative_path, target_url, lang, geo_state, schedule, impressions, clicks) +- [ ] `sponsors` table (tier: directory/category, category_id, schedule) +- [ ] Alembic migration + +**Ad server** +- [ ] Ad service: `get_ad_for_slot(slot, lang, state)` — picks active ad matching targeting, returns one +- [ ] Impression tracking: increment `ads.impressions` on serve (async-safe, batched to Redis then flush) +- [ ] Click tracking: redirect endpoint `/ads//click` → increment `ads.clicks` → redirect to `target_url` +- [ ] Ad slots in templates: header banner, sidebar (browse page), inline (every 6th listing card), footer +- [ ] Ads suppressed for `role=subscriber` (check `plan.limit('ad_free')`) +- [ ] Ad creative upload (admin): validate image, Pillow re-encode, store in `/media/ads/` + +**Sponsors** +- [ ] Sponsor directory page `/sponsors` +- [ ] Category sponsorship: "Jobs powered by X" banner on category browse +- [ ] Sponsor logo in nav or footer (tier: directory) +- [ ] Sponsor admin CRUD + +**Promoted search** +- [ ] `promoted_keywords` table (keyword, listing_id, expires_at, priority) +- [ ] Browse query: prepend promoted listings matching keyword before organic results +- [ ] Admin UI to assign promoted keyword slots + +**Tests** +- [ ] Smoke: ad served for correct slot/lang/state targeting +- [ ] Smoke: click redirect increments counter +- [ ] Smoke: subscriber sees no ads + +--- + +### 🔲 Phase 6 — Admin Backend + +**Dashboard** +- [ ] `/admin` dashboard: KPI cards (active listings, new users 7d/30d, MRR, ad revenue, flag-queue depth) +- [ ] Mini charts: signups/day, listings/day (last 30d), revenue trend + +**User management** +- [ ] `/admin/users` — searchable/filterable table (role, status, tier, trust) +- [ ] User detail page: profile, listing history, subscription, trust events +- [ ] Actions: ban/suspend/activate, tier override, trust score adjust, impersonate (logs to audit_log) +- [ ] Bulk actions: ban selected, send email to selected + +**Listing moderation** +- [ ] `/admin/listings` — flag queue sorted by `flag_count × recency` +- [ ] Quick actions per listing: approve (clear flags), hide (flagged), remove, view +- [ ] Bulk approve / bulk remove +- [ ] Auto-flag threshold: N distinct-user flags → auto-flip to `flagged` (setting in `settings` table) +- [ ] Keyword blocklist editor (stored in `settings`, checked on listing submit) +- [ ] Duplicate body detection (hash `body` on submit, reject/flag if seen within 24h) + +**Reports queue** +- [ ] `/admin/reports` — flagged content with reporter reasons +- [ ] Mark resolved / escalate actions +- [ ] `reports` table: unique per reporter+listing, reason enum + +**Category management** +- [ ] `/admin/categories` — CRUD, reorder (drag or sort_order field) +- [ ] Field schema editor per category (JSON form builder: add/remove fields, set type/required/options) +- [ ] Preview of field schema as it would appear on listing form + +**Plan / pricing management** +- [ ] `/admin/plans` — edit plan config JSON (limits), name, price, Stripe price ID, toggle active +- [ ] No-redeploy: limits read at runtime from DB + +**Ads & sponsors management** +- [ ] `/admin/ads` — upload creative, set slot/targeting/schedule, view impression/click stats +- [ ] `/admin/sponsors` — CRUD sponsor entries, assign category +- [ ] Ad performance report (impressions, clicks, CTR per ad) + +**Transactions & billing** +- [ ] `/admin/transactions` — full log with filter by type/status/date +- [ ] Refund action (calls Stripe Refund API, writes refund transaction row) +- [ ] Failed payments list with retry action + +**Settings** +- [ ] `/admin/settings` — toggle UI for all `settings` table keys: + - `registration_open` (bool) + - `ads_enabled` (bool) + - `maintenance_mode` (bool) + - `flag_threshold` (int, default 5) + - `new_user_trust_gate_days` (int) + - `contact_density_threshold` (int) + +**Audit log** +- [ ] All admin write actions write to `audit_log` (actor, action, target, meta JSON) +- [ ] `/admin/audit` — searchable audit trail + +**Analytics** +- [ ] Traffic: page views/day, top pages, search terms used +- [ ] Conversions: registrations, listings posted, messages sent, boosts purchased +- [ ] Top categories by listing count and by view count +- [ ] Top metros by listing count (Phase 7 SEO planning) + +**Tests** +- [ ] Smoke: admin dashboard 200, non-admin gets 403 +- [ ] Smoke: flag threshold auto-flips listing status +- [ ] Smoke: impersonate logs to audit_log + +--- + +### 🔲 Phase 7 — Polish + +**SEO & discoverability** +- [ ] Metro landing pages `/classifieds/` (e.g. `/classifieds/orange-county`) +- [ ] State landing pages `/classifieds/state/` +- [ ] Category landing pages `/classifieds/category/` +- [ ] Dynamic `` and `<meta description>` on all pages +- [ ] JSON-LD structured data on listing detail (Product schema) +- [ ] XML sitemap (`/sitemap.xml`) — listings + categories + metros, auto-updated +- [ ] `robots.txt` +- [ ] Open Graph tags (listing title, price, cover image) for social sharing + +**Email notifications** +- [ ] Welcome email on registration (with verify link already sent — add branding) +- [ ] Listing expiry warning email (3 days before `expires_at`) +- [ ] Listing expired email (with renew CTA) +- [ ] Message received email (already inline — move to RQ worker for async) +- [ ] Payment failed email +- [ ] Weekly digest email (new listings in saved categories/metro) — opt-in + +**Reviews** +- [ ] `reviews` table (listing_id, author_id, rating TINYINT, body, created_at) +- [ ] Leave review on completed transaction (mark-sold triggers prompt) +- [ ] Seller aggregate rating on profile + listing detail +- [ ] Moderation: flag/remove abusive reviews + +**Translation (deferred from Phase 2)** +- [ ] Per-listing "Translate" button → DeepL/Google Translate API +- [ ] Cache result in `listing_translations` (pay once per listing+lang) +- [ ] Language filter on browse ("Show VI listings only") + +**Performance** +- [ ] Switch sessions to Redis (`SESSION_TYPE=redis`) +- [ ] Query-level caching for hot browse queries (Redis, 60s TTL) +- [ ] Lazy-load listing images (native `loading="lazy"`) +- [ ] Serve WebP thumbnails (Pillow WebP encode alongside JPEG) +- [ ] Add MySQL `FULLTEXT(title, body)` index + switch keyword search to `MATCH ... AGAINST` +- [ ] Pagination `LIMIT/OFFSET` → keyset pagination for large datasets + +**UX & mobile** +- [ ] Responsive nav (hamburger menu on mobile) +- [ ] Listing detail image gallery with lightbox +- [ ] Infinite scroll OR "Load more" on browse (AJAX pagination) +- [ ] Toast notifications (non-blocking flash messages) +- [ ] "Back to results" link on listing detail (preserve filter state) +- [ ] Listing preview before publish + +**i18n completion** +- [ ] Extract all `_()` strings to `.pot` file +- [ ] Translate all strings to Vietnamese (`vi`) +- [ ] Translate all strings to Spanish (`es`) +- [ ] Compile `.mo` files, test all three locales end-to-end + +**Tests** +- [ ] Smoke: sitemap returns valid XML with listing URLs +- [ ] Smoke: OG tags present on listing detail +- [ ] Smoke: translation cached correctly in `listing_translations` + +--- + +### 🔲 Phase 8 — JSON API (Optional, for iOS app) + +**Design** +- [ ] RESTful JSON API under `/api/v1/` +- [ ] JWT authentication (separate from session cookies) +- [ ] API versioning strategy documented + +**Endpoints** +- [ ] `POST /api/v1/auth/register` +- [ ] `POST /api/v1/auth/login` → returns JWT +- [ ] `GET /api/v1/listings` (browse + all filters, returns paginated JSON) +- [ ] `GET /api/v1/listings/<id>` +- [ ] `POST /api/v1/listings` (create, multipart for images) +- [ ] `PUT /api/v1/listings/<id>` (edit) +- [ ] `DELETE /api/v1/listings/<id>` +- [ ] `GET /api/v1/messages` (inbox) +- [ ] `GET /api/v1/messages/<conv_id>` +- [ ] `POST /api/v1/messages/<conv_id>` (send) +- [ ] `POST /api/v1/listings/<id>/favorite` +- [ ] `GET /api/v1/categories` +- [ ] `GET /api/v1/zip/<zip>` (geocode lookup) +- [ ] `GET /api/v1/me` (profile + tier info) + +**Infrastructure** +- [ ] JWT library added (`PyJWT` or `flask-jwt-extended`) +- [ ] Rate limiting on all API endpoints (stricter than web) +- [ ] API error responses: consistent `{"error": "...", "code": "..."}` JSON shape +- [ ] CORS headers for iOS app origin +- [ ] API smoke tests (separate from web smoke test) + +--- + +### 🔲 Deferred Items (from earlier phases) + +- [ ] **Auto-translate** — per-listing translate button → DeepL/Google → `listing_translations` cache (moved to Phase 7) +- [ ] **Full ZIP dataset** — replace 10-row sample with SimpleMaps US ZIP or Census ZCTA (~42k rows) +- [ ] **MySQL spatial upgrade** — `POINT` generated column + `SPATIAL INDEX` + `ST_Distance_Sphere` (optional perf upgrade) +- [ ] **MySQL FULLTEXT upgrade** — `FULLTEXT(title,body)` index + `MATCH ... AGAINST` (Phase 7) +- [ ] **MySQL generated hot columns** — `GENERATED ALWAYS AS (JSON_EXTRACT(...))` replacing app-maintained `attr_*` columns (optional) +- [ ] **RQ worker** — replace inline email sends + APScheduler with proper RQ queue + worker process +- [ ] **Redis sessions** — switch from filesystem sessions to Redis-backed (`SESSION_TYPE=redis`) +- [ ] **Bulk CSV listing upload** — Business tier feature (parse CSV, validate, batch-create listings) +- [ ] **Scheduled posting** — Pro/Business: set `publish_at` datetime, worker flips to active +- [ ] **Auto-bump/renew** — Pro (weekly) / Business (daily): worker resets `bump_at`, extends `expires_at` +- [ ] **Storefront/profile page** — Pro/Business: public seller page `/seller/<username>` with all active listings +- [ ] **Custom URL** — Business: vanity slug for storefront (e.g. `/shop/alices-cleaning`) +- [ ] **Job-seeker posts** — reverse job listings ("I'm looking for work") under Jobs category --- @@ -696,4 +1009,4 @@ Or create user with `IDENTIFIED WITH mysql_native_password BY '...'`. --- -_End of spec. Phase 1–3 complete, 34 smoke-test checks green. Next: Phase 4 (Monetization / Stripe)._ +_End of spec. Phase 1–3 complete, 34 smoke-test checks green. Phases 4–8 fully detailed as to-do lists above. Next: Phase 4 (Monetization / Stripe)._