Prove the important part before scaling the investment.
A proof should test a consequential question about a product or system. If it only demonstrates that a model can produce text, it is unlikely to justify a larger implementation.
What result would make you stop?
Define that result before building the proof. A useful validation can fail: it tests the assumption that matters and records the conditions under which the result holds.
The engagement at a glance
- When it fits
- Use when a consequential product or technical assumption needs a working test before a larger build.
- Bring to the work
- One written hypothesis and acceptance criteria
- Representative or labeled synthetic inputs
- Hard cases, constraints and the next investment decision
- Leave with
- One bounded proof, reproducible evidence, architecture and cost/quality observations, failures and remaining risks.
- Decision authority to agree
- Agree the hypothesis and criteria with the client; the evidence informs the client's next-stage investment decision.
- Stop or reframe
- Change the proof form, narrow the hypothesis or stop if the evidence does not support the next stage.
Design an experiment that can contradict the idea.
Illustrative hypothesis: a document-review workflow can preserve supporting evidence, enforce tenant roles and respond within an agreed latency boundary on representative cases.
Will people understand the workflow?
- Useful proof form
- Interactive UX prototype
- Limit of that proof
- Does not establish model accuracy or production readiness.
Will the model meet the task criteria?
- Useful proof form
- Evaluation harness
- Limit of that proof
- Does not establish the complete application or operating model.
Will the critical system seam work?
- Useful proof form
- Complete vertical slice
- Limit of that proof
- Does not make untested integrations or release paths production-ready.
Put hard cases into the design
Evidence is incomplete or conflicting
Inspect citations, qualifications and the behavior when support is insufficient.
A fluent answer cannot stand in for supported review.
A tenant role cannot access a source
Test the same workflow with the relevant identities and inspect disclosure.
A useful answer cannot excuse a broken access boundary.
Inputs or dependencies stress the workflow
Record observed latency, failure handling and the conditions of the test.
A successful easy case cannot establish the agreed operating boundary.
Illustrative experiment design, not reported results. Simulated APIs and synthetic inputs must be labeled in the actual proof.
Three legitimate outcomes
- Advance
- The agreed evidence supports a defined next stage; carry forward the remaining risks.
- Reframe
- The result suggests a narrower job, a different proof or a changed architecture.
- Stop
- The central assumption fails or the expected value no longer justifies the next investment.
The full argument for evidence before scale
For CTO, CIO, CPO, COO
The situation behind the request
A product team needs to show that a document review assistant can preserve evidence, enforce tenant roles and stay within acceptable latency. A full SaaS build would be premature, but slides will not answer these questions.
Where the obvious solution falls short
A prototype is often sold as nearly complete software. That obscures authentication, operational cost and failure handling still to be engineered.
How we would approach the work
Define the hypothesis, build one complete vertical slice around representative or synthetic inputs, test hard cases, record limitations and recommend the next funded stage.
What the engagement may include
- One written hypothesis and acceptance criteria.
- Interactive workflow or technical proof aligned to the hypothesis.
- Architecture and cost/quality observations.
- Evidence summary, failures, remaining risks and build options.
The choices worth making explicitly
A low-code prototype may suffice for a UX question. A model evaluation harness may be more appropriate for accuracy. Choose proof form by uncertainty, not visual impressiveness.
What we would test before calling it ready
Never present a validation build as secure production software. Label simulated APIs and synthetic data plainly. The evidence should be reproducible.
A sensible first step
Select the highest-risk assumption the team needs to resolve, then design a validation that can genuinely fail.