LARAS
Which customer gets which offer, at what cost — and whether a randomized test really showed that it pays.
- Role
- Solo — requirements, data model, statistics, ML evaluation, UX; implemented with AI coding agents under my direction
- Stack
- FastAPI, PostgreSQL, Redis + Celery, scikit-learn, Next.js, Blender
- Timeline
- November 2026
- Status
- Tested locally · not deployed · no public repo
A generous offer is not a reason to break the rules.
Growth teams pick who sees which voucher. Three things fail quietly: an offer breaks consent or its terms; two requests spend one budget at once; a badly split test is read as a win.
All data is simulated for one fictional business, NusaPay Market: no real customers, merchants or money.
From the product specEligibility is a hard gate before learned ranking, and an auditable evidence bundle comes before any rollout decision.
Each step can refuse.
The model orders offers; rules, the database and experiment gates decide what happens.
- 01
Eligibility before ranking
Consent, opt-out, market, cooldown and frequency caps filter offers before ranking. If the model is late or down: an eligible baseline or no offer.
- 02
A budget that cannot go negative
Budget is a database constraint: with Rp10.000 left and two concurrent Rp7.000 requests, exactly one succeeds. Tests exposed a real deadlock, since fixed.
- 03
Offline evaluation, labelled offline
Temporal split, 7-day embargo, leakage audit; a planted leaky feature must block training. The gain comes from matching categories and is labelled not causal.
- 04
Experiments that can say “invalid”
Protocols are pre-registered and frozen. Gates run in order: logging, sample ratio, outcome window, estimate, guardrails. Any failure blocks rollout.
How each step refuses
- Eligibility: the ranker only orders offers that passed. The chosen offer is revalidated and reserved before display; a fallback answer carries a reason code.
- Budget: a reservation is a conditional update. Eleven ledger tests cover it, including a 24-thread stress run and an expiry-versus-redemption race; they exposed a real deadlock that was then fixed.
- Ranking: a gradient-boosted ranker is compared with popularity on held-out synthetic data. The gain comes from matching categories, and exposure bias is unresolved.
- Experiments: protocols are frozen in the database once submitted, then reviewed by a second role. Assignment is hashed and stable. The gates are logging, sample-ratio mismatch (SRM), outcome window, intention-to-treat (ITT) estimate and guardrails, run in order and are never skipped, and the server rejects a rollout that fails any of them.
A positive experiment that still may not roll out.
| Check | Result |
|---|---|
| Offline ranking (EVR-041), NDCG@5, popularity → candidate | 0.818 → 0.911 |
| EXP-102 ranker vs eligible baseline — contribution margin per eligible user (ITT) | Valid · +Rp4.906 [+3.510, +6.302] |
| EXP-102 incentive-cost guardrail | Breached → rollout blocked |
| EXP-108 with an injected client bug that loses treatment requests | Invalid · SRM p < 1e-4 · no estimate |
Seeded simulator, fictional business. Offline metrics measure agreement with biased logged exposure, not causal impact. EXP-108 is broken on purpose.
Confidence interval, calibration checks and run notes
| Check | Result |
|---|---|
| Difference in NDCG@5 [95% CI, user bootstrap; 1,170 queries, 1,076 users] | +0.093 [+0.079, +0.105] |
| A/A replay (identical policy in both arms, 2,000 users) | Inconclusive · CI covers 0 |
| Welch CI coverage over 600 simulated experiments | Within 92.5–97.5% |
Experiment results describe the simulator, not a market. EXP-108 is invalid by design: it exists to prove the gates catch a broken test. The screenshots below were taken from the same seeded run as these tables. An earlier seed, before a data fix, gave EXP-102 +Rp5.331 [+3.993, +6.668], with the same conclusion and the same guardrail breach.
A workbench that knows who is asking.
Six fictional personas see only what their role may do; a second tenant checks isolation.
All screens and roles
Screens: overview, audiences, campaigns and offers, the ranking lab, experiments (protocol, live monitor, analysis), decisions, monitoring and settings, plus a mobile brief. The overview lists what needs a person’s decision, derived from API data and her permissions. Roles: product manager, marketing analyst, data scientist, experiment reviewer, platform engineer and admin.
Two more screens: ranking lab and budget
Tested on one laptop. Not deployed.
Latest runs: 107 Python, 125 web unit and 88 browser tests. Also 16 decision records, a model card and a requirement-to-test table.
- Not yet: deployment, a public repository, single sign-on, drift detection, an external security or accessibility audit.
- Built with AI coding agents under my direction; results come from the recorded execution log.
Test detail and other gaps
- The Python tests are unit, statistical, ML and integration tests against a real PostgreSQL. The browser tests run in Chromium against the real API; 1 opt-in test was skipped. The slower end-to-end A/A replay passed in an earlier session.
- The 16 architecture decision records also log deviations from the spec; the traceability table maps each requirement to its test; the model card covers the candidate ranker.
- Also not yet: a manual screen-reader and 200% zoom pass, Safari and Firefox. A development identity provider stands in for single sign-on.
- Load figures (p95 84 ms at a 150 requests-per-second burst with four serving workers) are lab numbers from the same laptop, not a service-level objective.