THE CURIOUS CHARVAKAS

Product Requirements Document

Week 4 · Learning Tech & AI for non tech professionals

Building - and shipping - an MVP for Tech & AI Learning Platform for non tech professionals

Prepared by Manyata Manocha

Contents

• The Given Problem - Case Summary

Non-technical professionals are expected to work with AI every day, and the market's answer has been content — courses, videos, certifications — which leaves them intimidated, overwhelmed by choice, and unable to convert what they watched into capability they can use at work. The market is not short of content; it is drowning in it (4,500+ platforms in India alone). India is where the gap shows sharpest: 95% of Indian frontline white-collar workers use AI weekly against 74% globally (BCG), and India tops global GenAI course sign-ups — while free, credible, national-scale content completes at under 5% (§4). Adoption is finished. Capability did not follow. The battleground this PRD picks is therefore not "teach people AI." It is the gap between using AI and judging it.

The problem, as the case states it:

Non-technical professionals must now work with AI. Existing content-heavy platforms leave them overwhelmed and unable to build real capability.

The real question this PRD answers — the case hands us the problem, so the open question is not discovery, it is scope and proof:

What should the FIRST version solve for one specific professional, and how will we know it worked once it is live in front of real users?

Vantage's answer: build a consultant's judgment for mapping a client problem to an AI-approach category. The mechanic is guess-before-reveal. The measure is repeated practice (Weekly Engaged Learners), not content consumed. Vantage moves that one step to the front: bring a real client problem to Solve and get a structured, plain-language approach; build the judgment separately in Practice, where you must commit to an approach before the answer is revealed. We are not building another course. We are building the judgment a course never checks.

The battleground — why a client-facing consultant, not "non-technical professionals." "Non-technical professionals" is a market, not a user. Nine interviews across eight roles (§6) surfaced the same stall in different clothes; the consultant is the one where it is highest-stakes (her output reaches a paying client), highest-frequency (the trigger recurs weekly, not quarterly), and — decisively for a 9-day build — gradable: "which AI approach fits this problem" has a defensible answer key, where "was this a good prompt" does not. That is an evidence-led cut, and §7 defends it against the better-evidenced alternative.

The tension the case names, made concrete for her:

Information abundance vs directional paralysis — they can find a thousand AI resources and still not know which approach fits the brief on her desk.

Using AI vs judging AI — they produce client-ready work with ChatGPT but cannot tell whether the approach was right or merely fast.

Passive consumption vs practical capability — ten hours of AI content later, they still freeze on a genuinely novel problem.

Sporadic consumption vs habit-forming retention — they learn in bursts when a problem lands, and retains nothing in between.

For a client-facing consultant in India, early-career (0–8 years), already comfortable using AI — the hold-up is they cannot reliably tell which AI approach fits the client problem in front of her — because her confidence with AI comes from repeating familiar tasks, not from understanding how different approaches map to different problems.

2. Stakeholder Mapping

Vantage is self-serve. The learner is the user and the buyer.

Decided, not assumed: they is both user and buyer. Nothing in the research shows a manager requiring this or a firm budgeting for it — the pressure to get better is self-driven and career-protective. That rules out a paid-seat model for v1; it's free.

The competitor most products here overlook is the colleague they already asks. Courses give her content and chatbots give her answers, but when they is actually stuck her first move is to ask someone at her firm. So this product is not competing with other courses — it is competing with a free, fast, trusted person. §4 covers why that route fails her specifically for AI questions.

3. The Current Learning Journey - How Professionals Try to Learn Tech & AI Today

Symptom vs. root cause. The symptom: they keeps asking AI or a colleague for the same kind of judgment call. The root cause, from her own words (section 6): her confidence comes from repeating familiar tasks, not from understanding how approaches map to problems. A new problem gives her nothing to sort it into. A faster chatbot answer treats the symptom and repeats the root cause. It adds one more round of ask AI, take the answer.

4. Secondary Research

Market size and the capability gap

• 88% of enterprises use AI in at least one function. Only about 1% have reached real AI maturity, an estimated $5.5T in unrealised productivity (IDC). So what: the gap is applied judgment, not tool access.
• India: 95% of frontline white-collar workers use AI weekly vs. 74% globally (BCG); 80% weekly, 41% daily, the highest rate globally (ADP). So what: India is not an early market for this problem. It is the market where the post-adoption capability gap is already visible at scale.
• India also leads in AI pilot failure: 38% vs. 28% globally (Salesforce). So what: high adoption without judgment produces failure, not competence.

Why content-first approaches fail

• SWAYAM, India's free, government-backed, IIT-quality MOOC platform, has under 4% completion since 2017, flagged by a Parliamentary committee. So what: cost, trust, quality and access were all controlled for, and it still failed at 96%. The delivery model is wrong, not the learners.
• 62% of knowledge workers struggle to retain training even when they felt confident during it (Instructure). People forget about 70% of passive learning within 24 hours. So what: reading an answer creates false mastery. A chatbot's direct answer reproduces this exact failure.

Why active recall is the mechanism, not a design preference

• Reconstructing an answer beat reading a given answer for durable retention (multiple studies, group synthesis and Vantage's own Hypothesis.md). So what: this is the evidence base for guess-before-reveal. It is a retention mechanism, not a gamification add-on.
• Deslauriers' active-classroom study: students in active classes learned more but felt they learned less, and rated passive lectures higher. So what: a user preferring a faster chatbot answer is not evidence that the answer builds judgment. It may show the opposite.

Rework cost of the wrong tool-class choice

• 49% of general-purpose-AI users report significant rework needed, vs. 29% for specialised-tool users (BCG). So what: this supports Vantage's tool-class tier, general-purpose vs. specialised, sitting between category and named product. Category alone misses a choice that measurably drives rework.

5. Existing Tools - and the Seam They Miss

Everything they could use today, and where each one stops.

What none of them do: help her take a problem they have never seen before, decide which AI approach fits it, and find out whether they was right — privately, before a client sees the work.

Why this gap survives even though everyone has a chatbot. A chatbot will tell her which approach to use straight away. What it will not do is refuse to answer until they have committed to her own answer first. It has no reason to withhold, no correct answer to withhold, and no memory of whether her judgment is improving over time. That withholding is the entire structural difference here.

6. Primary Research - Right-Sized for an Open Problem

How it was done

Nine semi-structured interviews, conducted by me, across eight roles in eight different organisations. Each one was anchored on the last real occasion that person tried to learn or apply a tech or AI skill at work — what they tried, where they stopped, why — rather than what they would say they generally do.

The order was deliberate, and it is why the persona in §8 is narrow: interview across several candidate segments first, then choose one. The consultant was not the starting assumption. They are where the nine conversations pointed.

Participants are described by role, sector and experience only. Names and employers are withheld; the interviews were given in confidence.

Alongside this sits a small internal survey (n=3, two product owners and a consultant, 6–10 years' experience, all non-technical roles at tech companies). It is explicitly directional and not representative, and nothing in this document rests on it alone.

What came out of it

Findings are labelled by how far the evidence travels:

SAW — a participant said it, in their own words

THINK — my interpretation of what they said

ASSUMED — no evidence; carried as a hypothesis

What the wider set changed. Findings 6 and 7 were the weakest claims in the early thinking — 6 was our framing rather than a user's, and 7 had no user evidence behind it at all. Both now rest on something a participant said unprompted, from two different segments. They stay at Medium rather than High because neither person was the locked persona. Findings 3, 4 and 9 come entirely from participants who are not the persona, which is what makes them useful: they establish that the problem is structural rather than particular to one consultant.

What this research does not prove

The locked persona is one of the nine. The problem is evidenced across eight roles; the client-facing consultant specifically is a sample of one. Her weekly client cadence, her client stakes and her status costs all rest on a single conversation.

Nine is short of the 12–20 this case calls for. Treat the segment spread as directional. It was enough to converge on a persona; it is not enough to generalise to a population.

The sample skews early-career and urban. Six of the nine are under five years' experience, and none are in marketing, legal, or the public sector — a recruiting gap, not a finding.

"Category" is still our vocabulary. Participants described a matching problem. Calling it a taxonomy problem, and building a category map as the answer, is the largest interpretive step in this document.

Severity was never measured. The trigger recurs weekly, but what a wrong approach actually costs — an hour, a weaker deliverable, or nothing much — was never establitheyd.

The survey is n=3 and was not designed to test this problem statement. It is context, not proof, and no claim here depends on it.

Nobody was asked the decisive question. No participant was asked whether being made to guess would help them or annoy them. That can only be answered by behaviour, which is what §14 reports.

What it changed, and what is left to prove

Finding 1, combined with the status-cost data in §4, is what shaped the product: they has a real gap and nowhere safe to test herself. Findings 4 and 9 rule out the obvious alternative — an employer already bought tools, trained staff, and offered a budget, and none of it produced confident use. That is why this is a private commit-then-reveal loop rather than another content library.

The single assumption the pilot exists to settle is not whether the gap is real — nine people described versions of it. It is whether the fix works: does being made to guess actually build judgment, or does it just annoy people? No interview can answer that, because nobody was asked. Only behaviour can, which is why practice completion and first-guess accuracy are the first metrics watched in §12.

Where the evidence weight sits. The problem rests on nine interviews across eight roles. The persona rests on one of them. The fix rests on 17 external testers and 95 completed practice sessions (§14). Research located the problem; only behaviour tests the solution, and this document does not let one borrow the other's confidence.

7. Opportunity & Prioritisation

Everything §3 to §6 surfaced, written as for whom / through what / because it is still un-owned. The BECAUSE clause is the real test: if there is no explanation for why the gap survives despite everything that already exists, it is not a genuine opportunity.

How the scores were reached, so the table is not a black box.

Value = how much pain it removes × how hard it is to copy. O-1 and O-2 score L because they attack the one stage where they currently has no option at all (§3, step 5), and because withholding an answer is something a chatbot structurally will not do. O-6 scores M because the pain is real but a colleague's recommendation already solves it.

Effort = build cost in a nine-day sprint. O-1 is M because approach-selection can be marked against a pre-written question bank with no model training. O-5 and O-7 are both L because each needs a scoring rubric that would have to be invented and validated first.

Ties break toward the smaller build. O-2 has the same value as O-1 at lower effort — which is why it is delivered as a property of O-1 (the practice loop is private by default) rather than built as a separate feature.

The one bottleneck this version attacks

Seven candidates reduce to one. Naming the losers matters as much as naming the winner.

They cannot map an unfamiliar client problem to the right AI approach. Everything in v1 exists to move this, and nothing else.

Why this one:

It is the only opportunity that attacks the stall stage — the single point in §3 where her current process offers her nothing.

It is structurally defensible. The advantage is withholding the answer, which a chatbot will not do and a colleague cannot do without her paying a social cost.

It is markable in three minutes. A judgment can be graded; a piece of produced work cannot, not cheaply.

Why not prompt structuring (O-7)? Stated openly, because there is real evidence in that direction: BCG's 758-consultant experiment found AI made consultants 19% less likely to be correct on tasks outside its reliable range. It loses for two reasons. There is no stable answer key, since prompt craft changes with every model release and any content built on it decays. And it cannot be marked cheaply: "was this a good prompt" needs a rubric that does not exist. This is choosing the defensible problem over the loudest one, and it is a decision rather than an oversight.

Why not the most-felt pain (O-5, trust)? The trust gap is arguably the persona's strongest quote — "the human overlay is still important." It is deferred for one reason: measuring "can I trust this output" credibly requires a validated rubric and questions of comparable difficulty. Without both, the product would be inventing a confidence score and presenting it as fact. It is named in §10, not dropped.

Sizing the wedge, not the category

The §4 figures size the whole category — every Indian knowledge worker using AI. The wedge is narrower and is estimated directionally rather than borrowed from that number.

India has roughly 600,000–800,000 people in client-facing consulting and professional-services roles. The target is the early-career (0–8 years), non-technical slice already using AI on real client work — conservatively 25–35% of that, or about 180,000–250,000 people with this specific gap. (Directional estimate from industry headcount reporting, not a commissioned market study.) Round 1's cohort against that number is reported in §14.

The line from evidence to metric

Consultant interview (§6 #1) + status-cost data (§4)
→ Insight: they has a real gap and no safe place to test herself
→ Pain: the stall — "which approach fits?" goes unanswered (§3, step 5)
→ Bottleneck: cannot map an unfamiliar problem to an AI approach (§7)
→ Solution: withhold the answer until they commits (§9)
→ Feature: guess-then-reveal practice loop (§10)
→ Metric: practice completion + first-guess accuracy (§12)

8. Product Strategy - User, Buyer, Problem Statement

This is the lock point. Everything after this section is detailing, not a change of direction.

User and buyer — decided, not assumed

User = buyer. Pure self-serve. v1 is free.

This was decided rather than inherited from habit. The case does not hand over a B2B structure, so the question had to be answered from evidence: the pressure to get better is self-driven and career-protective, and §4 shows employers are not supplying the guidance — clients want AI used, under 20% say how, and nobody is teaching her which approach fits which problem. There is no credible L&D buyer to serve in a first version.

The interviews reinforce it from the other direction. The e-commerce category manager has a company learning budget and does not use it; the media account manager's employer bought AI tools and trained staff, and adoption was still uneven (§6 #4, #9). Employer money and employer training are both already present in this market and neither converts. Selling to the employer would be selling into a channel the research shows does not work yet.

Persona — the fluent stall

Non-technical, client-facing consultant in India. Early-career, 0–8 years. Uses AI daily as augmentation rather than delegation — comfortable with tools like Cursor and Databricks, learned by repeated hands-on practice rather than through any formal path. Meets clients weekly, so unfamiliar problems arrive at that rate. When they is unsure, they asks a colleague first, then reads or watches something.

They is not a beginner. That single fact rules out beginner framing, rules out teaching her tools they already uses, and is why the product opens with a question rather than a lesson.

What "capable" looks like to her: being able to look at a client problem they has not seen before and say which kind of AI approach fits it — without asking anyone, and without finding out they was wrong only after the work reaches a client.

How the problem statement narrowed

The starting hypothesis, written before the interviews, was broad:

"Professionals lack a personalized, practical way to understand what AI can reliably do for their specific work and apply it confidently."

Three things in the interviews narrowed it, and each narrowing is traceable:

The final statement is the descendant of the original, not a replacement for it. What survived is the observation that generic content does not connect to real work. What changed is who it is for, and which single decision it fixes.

The locked problem statement

For a client-facing consultant in India, early-career (0–8 years), who already uses AI daily, the biggest hold-up is they cannot reliably tell which AI approach fits the client problem in front of her, because her confidence with AI comes from repeating familiar tasks rather than from understanding how different approaches map to different problems, which matters because a new client problem gives her nothing to sort it into, so they guesses — and the cost of guessing lands on billable client work, every week.

Every clause is earned. The hold-up and the "because" come from the anchor interview (§6 #1, #5). The weekly frequency is stated directly by the same participant. The "nothing to sort it into" clause is my inference and is labelled as such in §1.

The core bet, stated so it can be proven wrong

If they commit to a guess about which approach fits before they are shown the answer, then they build a map they can apply to a problem they have never seen, unaided, because actively recalling an answer produces durable learning where recognising one does not.

The four sub-bets, and what would kill each:

§14 reports where each of these currently stands.

The success metric this is ultimately judged on

They correctly identify the right AI-approach category for a novel problem, unaided, within a defined timeframe.

"Novel" and "unaided" carry the weight. They are what separates judgment transferred to her from Vantage answered it once.

Locked scope

In: buckets 1 and 2 (UNDERSTAND, CHOOSE) — deciding what AI can do for a problem and which approach fits it, delivered through two surfaces (Solve for the problem in front of her now, Practice for the judgment that means they needs less help next time).

Out: buckets 3, 4 and 5 (STRUCTURE, VALIDATE, APPLY), named product recommendations, any employer or L&D layer, payments, and social or competitive features.

Locked on 23 August 2026 — problem, persona, scope. Everything after this section is detailing, not pivots.

The decision spine — every major call in this document

This is the lock point, so this is where every decision is recorded in one place. Each row names a decision, the evidence that forced it, and the finding that would reverse it. Rows 6–10 are settled in §12–§14; they are listed here because this is where the commitment was made, and nothing downstream contradicts this table.

9. Solution Ideation & Direction

Three genuinely different shapes were considered, not one idea written three ways.

Why C wins. It is the only direction with a structural advantage. A chatbot cannot withhold an answer — it has no reason to disagree with the user and no correct answer to withhold. C is also the only shape that implements §4 #6 literally: judgment is the markable unit, and committing before the reveal is the cheapest way to force recall.

Why the others lost. A would deepen the exact dependence the problem statement identifies — they would get faster answers and no better judgment. B fails on evidence already gathered: an employer bought AI tools and trained staff and adoption stayed uneven; a category manager has a learning budget he does not spend. Adding more content to that picture changes nothing.

The trade-off C forces, accepted openly. Being made to guess is friction, and someone under client deadline pressure does not want a quiz. So the product splits into two surfaces:

The consequence, stated because it governs how §12 is read: the judgment thesis lives in Practice only. H1, H2 and H4 are tested by Practice usage specifically. Solve usage on its own can never confirm the thesis.

One further rule. Vantage names categories, never products — "this is a classification problem," not "use Tool X." Named tools change constantly; categories are stable enough to build a mental model on. A tool-class distinction (general-purpose versus specialised) sits between them, justified by §4 #2.

End-to-end user flow

Vantage user flow

Today versus tomorrow:

The step that matters most is `practice_started` → `meaningful_activity_completed`. If people abandon at the guess, the core mechanic is being rejected — and that is the single most valuable thing this product could learn.

10. Product Detailing & MVP Scope

Features — each tied to a flow step and the evidence that earned it

Classified by product value: Must-have (the product is broken without it), Performance (more is better), Delighter (unexpected value).

The MVP in one line: a working web app where a client-facing consultant can bring a real client problem and get a structured plain-language approach — and, separately, practise AI-approach judgment through guess-then-reveal exercises that build the skill instead of just answering the question.

Information architecture

/ Landing → sign up / log in
├── /home The fork: Solve · Practice · Progress
├── /solve
│ ├── /new Ask, then pick which wording is solved
│ └── /[id]/solution Steps, tool classes, pro tips, rating
├── /practice Two top-level cards: Scenario · Quiz
│ └── /[surface] Difficulty → guess → reveal → continue
├── /progress Accuracy over time · strength by category
│ └── /history Every attempt, Correct / Missed
├── /settings Theme, practice preferences
└── /admin (admin accounts only)
├── /analytics Eight metric tiles + activation funnel
├── /questions Question bank
├── /quiz-review Human review gate
├── /daily-content Pipeline output by day
└── /users Signups, active / quiet / dormant

They always knows where they is because the two surfaces never mix: Solve is always about the problem in front of her, Practice is always about the judgment behind it, and Progress only ever reports on her own attempts.

Edge cases and failure states

Every view has a loading, empty, error and success state. Empty is designed, not an accident.

Deliberately not building

Buckets 3–5 (STRUCTURE, VALIDATE, APPLY) — reasoning in §1 and §7.

Named product recommendations — categories are stable, named tools churn.

Share and invite — designed in the mockup, never built. There is no referral backend, which is why referral rate is absent from §12, and it sits in genuine tension with the privacy principle.

Payments — a free v1 has no runtime cost to protect; payments would be pure friction.

Any employer or L&D layer — §8, and §6 #4 and #9 show employer-supplied tools and budgets are not converting.

11. UX & Product Design

Six components of great UX — self-check

Scored from the Round 1 pilot (17 external testers, unaided, no onboarding call), not from opinion. Where something was not measured, it says so.

The honest weak row is Efficiency. The honest structural weakness is discoverability: only 2 of 17 testers reached the scenario quiz (§14). Both are named here rather than left for a reviewer to find.

12. Analytics & Event Tracking

The full loop — acquisition to referral

The template asks for the whole AARRR funnel, not just the middle of it. Stated honestly, including the stage that does not exist:

Two of the five are honestly empty, and that is the reading. Referral is a missing feature, not a missing query — instrumenting it would measure a button that does not exist. Acquisition has a real number but not a real channel, which §14 states plainly rather than dressing a favour from a personal network as a growth loop.

North Star

Weekly Engaged Learners (WEL) — unique learners completing ≥3 meaningful practice sessions in a rolling 7 days. A meaningful session requires the learner to submit her own judgment and complete the feedback/reveal loop.

Why an absolute count, never a ratio. The tempting alternative — "% of active users practising 3+ times" — fails because the denominator moves:

The ratio doubled while the product reached the same 30 people and lost 50. A dashboard built on that ratio would show growth during a decline.

Why ≥3, not ≥1. One session delivers some value but does not achieve the promise — capability through repeated practice. Since the promise contains repetition, the metric must too. Three is an explicit hypothesis, to be validated by cohorting 1 / 2 / 3+ / 5+ against D30 retention.

Why the NSM is not the outcome metric. Measuring "judgment improved" credibly needs comparable difficulty, a validated rubric, and evidence the score tracks skill rather than quiz-gaming. Without those, an improvement percentage would look precise but mean nothing real. Behaviour is the honest proxy until then.

The sharpest objection to this NSM, and the answer. A count of completed sessions has no user-value term in it — so in principle WEL could rise while the product got worse, the way a usage metric always can. The clean fix used elsewhere is to multiply by a satisfaction rating ("sessions rated 4–5★"). Vantage cannot do that in Practice without breaking the product: a rating prompt after every reveal is exactly the kind of interruption that would depress the completion rate H1 is being tested on, and a learner who was just told they guessed wrong is the worst-placed person to rate the question fairly. So the value term is carried by three things instead, and WEL is never reported alone:

That is a deliberate trade, not an oversight: rating coverage is the honest weak point of this stack, because only one of the four surfaces asks. Adding a lightweight rating to the reveal — after the streak of questions, never between them — is the first Round 2 instrumentation change.

The stack

Every row states its arithmetic and what a reviewer should conclude from it. A metric whose "tells us" column cannot be written in one plain sentence does not belong on a dashboard.

Why week-over-week sits beside D7/D30 rather than replacing it. Week-over-week needs no cohort to age, so it produces a real number while every D-N figure is still honestly rendering "—" (§14). It is also the weaker claim: it compares counts, so it cannot tell you whether the same people came back or a fresh set replaced them. Both are kept, and neither is allowed to stand in for the other.

Segmentation is by practice, never attendance. Active (≥1 in 7d) / Inactive (0 in 7d) / Dormant (0 in 28d). Logging in never makes someone active. Inactive and dormant stay separate: someone quiet for a week may just be busy; someone quiet for a month has likely left.

Events

One table, analytics_events (event_name, user_id, metadata, created_at).

signup · login · ask_submitted · solve_started · practice_started · guess_locked · solution_generated · meaningful_activity_completed · solution_feedback · ask_refused

meaningful_activity_completed is the single canonical value event, fired from four surfaces tagged by metadata. source: solve, daily_challenge, quiz_general, quiz_scenario. `source` always means the surface — one canonical event is what keeps NSM, activation and segmentation consistent.

Implementation is real, not planned. Logging is fire-and-forget; computation is pure and unit-tested (60 tests covering threshold and window boundaries); empty denominators render "—". The owner dashboard at /admin/analytics computes every metric above live from the event table, and a login-free public view is submitted with this document so a reviewer can watch the numbers move.

13. Build & Deployment

Live demo: https://vantage-app-zeta.vercel.app

Public metrics, no login: https://vantage-app-zeta.vercel.app/demo/analytics

§11 shows the design; the two screenshots below are of the deployed app, taken from the live URL above rather than from a local build.

Landing Page

Home Page

Admin analytics

Frontend & backend: one Next.js 16 app (App Router, TypeScript, React 19) on Vercel — 24 routes.

Data & auth: Supabase (Postgres, email/password auth, row-level security so a user reads only her own rows). Admin accounts flagged via app_metadata.role.

Runtime engine: Groq, via a provider-agnostic adapter. Used for ask-refinement, grammar assistance, solution generation and the content pipeline.

Content: question banks are seeded and human-reviewed, not generated live per user — which keeps the experience fast and the quality controlled. Generation runs behind a review gate.

Testing: 268 unit tests across 27 files — 60 on the metrics layer alone, covering threshold boundaries and both eligibility traps, and 59 on the runtime engine.

The integrity model — the part that is not allowed to fail

A product whose whole promise is "we will tell you when you are wrong" dies the moment it is wrong itself. Three rules are enforced in code, not in review comments:

No question reaches a learner unreviewed. Generation is a pipeline that writes to a staging table; a human approves or rejects each item in /admin/quiz-review before it can ever be served. The gate is the integrity model — an unreviewed question is not a slightly-worse question, it is a product that marks a learner wrong for no reason.

A declined ask is never restated. Solve refuses harmful or abusive asks at the model boundary and shows one fixed, app-owned sentence. The model's own refusal wording is logged, never rendered — it is not written in this product's voice and is not safe to display unreviewed.

Empty denominators render "—", never 0%. A metric with nothing behind it reports absence, not failure. This is enforced in the metrics layer and covered by tests, because the tempting bug — showing 0% retention on day one — would report a failure that has not had the chance to occur.

Two decisions worth defending:

Seeded banks over live generation. Live generation would be more impressive and worse: unpredictable latency inside a timed loop, and no way to guarantee a question is fair before a user sees it. Generation still happens — it just happens ahead of time, behind rule 1 above.

Analytics is fire-and-forget. A tracking failure must never break the feature it measures. Metrics are computed as pure functions from the event table rather than incremented counters, so a definition change is a code change, not a data migration.

How it was verified — and what is still weak

Verified, not assumed. Before submission: the full suite runs green (268 passed, 5 skipped by design), TypeScript compiles clean across the repo, the linter passes, and every key route was checked serving live rather than merely building — /, /login, /solve/new, /practice, /settings, /admin, and the login-free /demo/analytics.

Known limitations, stated so nothing is discovered mid-demo:

Scope at ship — built / cut / deferred:

Built and live: signup/login, Solve (ask → pick the wording → solution) with voice dictation, Practice across daily challenge / Quiz Time / scenario quiz with difficulty routing and no-repeat, Progress and History, light/dark theming, the admin analytics dashboard, the content pipeline with its human review gate.

Stubbed or faked: nothing. Every screen in this document is backed by real data from the live database — no mock rows, no hardcoded numbers, no placeholder dashboards. The analytics tiles compute from the event table at request time, which is why a reviewer can open the public dashboard and watch them move. Where something has no data yet it renders "—" rather than a plausible-looking figure.

Cut: Share & invite — no referral backend, and against the privacy principle.

Deferred: Skill Improvement Rate scoring, pending a validated rubric. Automatic question-bank generation and approval — the pipeline can already generate candidates; everyone is manually triggered and manually approved for now. Auto-run and auto-approve come once there's enough reviewed volume to trust a confidence threshold instead of a human eye on every item.

14. Launch, Growth Loop & Iteration

This section reports the pilot as it actually stands on submission day — early, small-N, and honest. The live source of truth is the analytics dashboard submitted with this document, which a reviewer can open to watch every number below move.

Round 1 — 26 August 2026

17 signups. Admin account excluded throughout. Test window 25 Aug 23:51 → 27 Aug 04:40 UTC — 28.8 hours, with 16 of the 17 arriving in a single burst on the evening of 26 August.

Round 1 reached 17 of the 40–50 users the case asks for. Every figure below is measured on all 17, so the numbers are real rather than sampled — but the sample is thin, and the shortfall is time rather than interest. The whole pilot ran inside a 29-hour window. That is enough to read activation and completion honestly; it is not enough to read retention, which is why every D-N figure in this section is "—" rather than a number. Round 2's first job is the cohort size the case asks for, run across a full week.

Activation (first value within 24h of signup): 88% — 15 of 17. Every number above is read off analytics_events, not estimated.

Completion by surface

Counted per learner, not per event — event counting once made this read 148%, since quiz surfaces fire one completion per question but one start per session (risk #7 below).

Acquisition — how the first users were reached. By hand: direct messages to consultants and cohort peers, and WhatsApp professional groups, framed learner-to-learner rather than as a launch. Honestly, this is a manual, non-scalable push, not a repeatable channel — but it earned the product its first real users.

What auditing the numbers surfaced

A batch of signup/login rows carried no user_id. The cause: a test file exercising the login/signup server action mocked the auth client but not the tracking call, so every run of that test wrote real rows into the live table with no user attached — not a production defect, since the actual server code only falls back to a null id on a mocked response, never on a real Supabase one. Fixed and covered by a regression test asserting the tracking call receives the real user id (commit 8e17607).

No metric in this document was built from those rows. The metrics layer drops any event with a null user_id before it reaches a calculation — the same rule that makes an empty denominator render "—" instead of 0% (§13) — so nothing here needed correcting once the cause was found, only explaining. Worth naming anyway: a test suite writing into shared live infrastructure is a real process gap, distinct from and more honest than the "these two events can't be attributed" theory floated when the nulls were first spotted.

One more account sits outside every number above: the builder's own admin login, 9 events of which 3 are meaningful-activity completions — excluding it is why Sessions reads 95 rather than 98, and why the funnel's second step can't exceed its first.

What each hypothesis now reads

§8 named four bets the product rests on. One pilot's worth of behaviour has an answer for two of them:

Retention (D2, the 36–60h bracket §12 defines) has an eligible cohort of 0 — nobody has had 60 hours elapse since their first value yet, so it correctly renders "—" rather than a misleading 0%. H2 and H4, the two hypotheses that would actually confirm the product works, stay open until a second round of data exists.

Reach, not rejection

Only 2 of 17 testers opened the scenario quiz — the surface §7 names as the purest test of the judgment bottleneck, since it's built from real client-work scenarios rather than general concept questions. Both of the 2 who found it finished the loop. Rejection and reach demand opposite fixes: rejection means the guess mechanic itself needs to change, reach means the navigation does. Counting per learner instead of per event is what separated the two readings — an event-level count of the same data read 148% at one point, which would have pointed at the wrong fix (risk #7 above). Practice now shows Scenario and Quiz Time as two top-level cards instead of nested behind a menu; the same funnel is what will show whether that actually lifted reach next round.

Risks and open questions

Open questions Round 2 must answer:

• Do they describe her own problem in category terms, unprompted? (kills or confirms the product's central frame)

Is ≥3 the right NSM threshold? Cohort 1 / 2 / 3+ / 5+ against D30.

Is 17% accuracy appropriate difficulty or discouraging difficulty? Completion held for one sitting; a learner wrong four times running on day three may not return.

Does Solve genuinely feed Practice, or are they two products sharing a login?

Did the navigation fix actually lift scenario-quiz reach?

The growth loop — honest state

There is no referral loop firing yet.

Current growth is push, not pull. A real loop needs two things deliberately deferred: a "how did you hear about us?" field at signup, and a share moment — the latter in genuine tension with §11's privacy principle, which is why it was cut rather than merely postponed. The loop Vantage can own instead is internal: Practice is the high-frequency surface that keeps a low-frequency core (real client problems arrive weekly) in hand between real-work moments.

← Back to portfolio