Last week, one of my project students told me they wanted to run a “pilot study”. When I asked for details, their intended participants were over 100 students but they had only just developed their digital education “prototype”. First round of testing. And when I asked if the prototype was tested for any bugs before conducting this pilot, they said only the co-supervisor (who was a non-technical person) tested it. What they were describing was not a pilot at all. It was, at best, a beta test – and one that had not even through an alpha test yet.
This mix-up is very common in digital health and digital education. Google search the terms “alpha vs beta testing” and you will probably be presented with a software engineering article. Search “pilot study” and you will realize that almost all the health tech studies in PubMed will call their evaluations by that term. Search “MVP vs prototype” and you will be flooded with startup articles. Yet, all are correct (in their own context). So what’s the difference?
Well, hopefully I’ll be able to de-conflict this concept in this post. I will go through each term, where it comes from, and more importantly – what question it is meant to answer.
Why are These Terms So Confusing?
The reason? These terms were from 3 different communities, decades apart, solving entirely different problems.
In Software Engineering, alpha and beta tests are words that describe build stability. Is the code buggy or ready to hand over to someone else yet?1
Then, in Product and Startup Culture, we have prototype, proof-of-concept (POC), minimum viable product (MVP), and a looser sense of “pilot” – words about validation. Is this idea worth building at all?2
And we also have Clinical and Education Research, which describes preliminary study, feasibility study, and the strict, ethics-board version of “pilot study” – which are words about evidence. Does this actually work for its purpose, is it safe, and can you prove it?3
To make things more complicated, there is another fourth term worth mentioning: Proof-of-value (POV), which is of another different context altogether – Enterprise IT Procurement and Vendor Evaluation. It poses another different question – not whether “does it work”, but “is it worth it to us (the users/stakeholders) specifically”.4,5
A digital health or digital education project usually has to pass through all 4, depending on the target audience/stakeholders. That’s 4 rulebooks being read all at once.
A Quick Glossary
For the digital health and digital education noob, here’s a quick glossary.1,3,4,5,6 But let’s face it… Sometimes even experts get these terms mixed up.
Prototype: It’s the rough build that proves whether your idea can function at all. It’s meant to be disposable, and not meant for real users.
Proof-of-Concept (POC): A small, scrappy test of whether the idea is technically or clinically feasible, run cheaply enough that if abandoned, it costs nothing.
Proof-of-Value (POV): Unlike POC, which is run by the developers who built the thing to test whether it works, a POV is run by people who are adopting an already-built thing, to test whether it is worth it for them. If your project is about whether your target audience will “adopt this existing tool”, rather than whether “should we build this” – then a POV is what you really need, not a POC.
Alpha Test: The first round of formal testing, run by your own team, on your own build. Bugs are expected. Usually, nobody outside the team is involved in an alpha test.
Beta Test: This is testing by people outside your team once the build is feature-complete, but not totally bug-free. This phase is about usability, not generating evidence.
Minimum Viable Product (MVP): This is the smallest version that actually solves the user’s core problem. It may not look like the finished product, but it’s useful. Usually, the MVP crosses over from “does it work” into real-world testing.
Feasibility / Preliminary Study: This is a quick check on whether your evaluation plan will work – can you recruit, does consent make sense, or do your instruments hold up. It is still not about whether the intervention works.
Pilot Study: This is about evaluating the entire evaluation protocol, run end-to-end on a small sample that represents, but is not part of, your real study population. A pilot tests whether your study can run smoothly. While most pilot studies in healthcare, education and digital health do report preliminary outcome data and often frame it as early evidence that the intervention works, it is usually underpowered to prove it.
In many cases, trials or summative evaluations with small samples sizes may be incorrectly labelled as a “pilot study” – when it is actually assessing effectiveness as a primary outcome. These “pilots” are methodologically just small underpowered trials wearing a “pilot” name.
Full Trial / Summative Evaluation: Unlike a pilot, which tests whether your study can run smoothly, or produces only preliminary hints of an effect; the full trial is properly powered to prove it. This is the study that definitively answers, with adequate statistical power and rigor, the question that everyone wants to know: “Does this actually work, for your population, at your scale?” If your pilot suggested an effect, this is where you find out whether that hint was right.
Compare at a Glance
A quick side-by-side comparison table for those who need a quick summary.
| Terminology | Where It’s From | What It Tests | Scope | Is Output Temporary? |
|---|---|---|---|---|
| Prototype | Product development | Does the core idea function? | Single build | Yes |
| Proof-of-Concept (POC) | Product development | Is it technically/clinically feasible? | Minimal | Yes |
| Proof-of-Value (POV)* | Enterprise IT adoption / Vendor evaluation (e.g. in healthcare IT procurement, institutional adoption) | Is it worth adopting, for us, specifically? | Limited, real-world context | Conditional – Yes, if value is not demonstrated |
| Alpha Test | Software engineering | Is the build stable? | Internal only | No, feeds beta |
| Beta Test | Software engineering | Is it usable in the real world? | External, limited | No, feeds release |
| Minimum Viable Product (MVP) | Product development | Does a stripped-down version solve the core problem? | Small, real users | No, the bridge |
| Feasibility / Preliminary Study | Clinical / Educational research | Will my evaluation procedures work? | Small, exploratory | No, informs pilot |
| Pilot Study | Clinical / Educational research | Does the full protocol run end-to-end? | Small, representative | No, informs trial |
| Full Trial / Summative Evaluation | Clinical / Educational research | Does it work, safely, at scale, with evidence? | Full powered sample | No, the deliverable |
Visualize the Flow
And below is a flowchart on how these play into a real project. Essentially, you have to consider the build readiness and evidence readiness. the MVP (or POV) is the link between them, but POVs are usually for someone adopting the tech, rather than developing.
Some Mistakes I See
- Calling a beta test a “pilot”. Both test different things, and the confusion shows up even in published industry case studies.
- Skipping straight from MVP to “pilot”. If the feasibility step is skipped, and you discover that the process and/or assumptions are flawed after the study has begun, it may be too expensive to fix by that time.
- Treating pilot data as proof. The sample size for a pilot is not meant to power the intervention study. The pilot results cannot, and should not be a definitive claim that your intervention works.
- Running a POC when you actually need a POV (or vice versa). If you are adopting an existing tool and need to know whether it is worth it in your context – this is a POV question, not a POC. If you are doing a POC, then you are probably re-proving the work that was already done by the vendor/developer already (answering the feasibility question).
Find Your Product Stage
You may still be a bit confused… That’s ok. Rather than make you guess, I have quickly built a small interactive tool through vibe coding to help you with your project stage identification (based on the flowchart above). Most projects sit at different points on the build and evidence readiness scales, so the tool reports them separately and tells you where your project stands right now and what completing that stage requires.
The Bottom Line
None of these terminology is more correct than the others. Each one answers a different question, and is asked by a different discipline. The key is knowing which question you are trying to answer right now, and naming your stage accordingly. Get this right, and your ethics application, your supervisor, your conference abstract, and your publication, will all be on the same page, even when your cited sources aren’t.
References:
- Wikipedia. Software release life cycle. June 2026. https://en.wikipedia.org/wiki/Software_release_life_cycle[↩][↩]
- Elam JB. PoC, MVP, pilot, prototype, alpha, beta — What’s the difference? Medium. 17 Mar 2025. https://john-elam.medium.com/poc-mvp-pilot-prototype-alpha-beta-whats-the-difference-70d53525c2ab[↩]
- Smith PG, Morrow RH, Ross DA. Preliminary studies and pilot testing. In: Field trials of health interventions: A toolbox. 3rd edition. OUP Oxford; 2015: Chap 13. https://www.ncbi.nlm.nih.gov/books/NBK305518/[↩][↩]
- Flory C. Proof of Concept vs Proof of Value. Medium. 14 May 2022. https://medium.com/@clive_59987/proof-of-concept-vs-proof-of-value-ccc10054ad99[↩][↩]
- Scotto J. Proof of Concept (PoC) vs. Proof of Value (PoV): What do they mean for your business? Tenable. 11 Mar 2019. https://www.tenable.com/blog/proof-of-concept-poc-vs-proof-of-value-pov-what-do-they-mean-for-your-business[↩][↩]
- Eldridge SM, Chan CL, Campbell MJ, et al. CONSORT 2010 statement: Extension to randomised pilot and feasibility trials. Pilot Feasibility Stud 2016;2:64. https://pmc.ncbi.nlm.nih.gov/articles/PMC5154046/[↩]