Write-up of “From Prompt to Product: A Human-Centered Benchmark of Agentic App Generation Systems”

I grew up with apps being critical — from True Skate and Brawl Stars to Instagram and Snapchat. I’d viewed app builders (probably) like the ancient Greeks and Romans viewed sculptors. That’s changed. Today, every person has access to marble, a chisel, and clear instructions — and that’s only for the people who still care to do the actual carving.

Quome Inc.’s recent paper, From Prompt to Product: A Human-Centered Benchmark of Agentic App Generation Systems, details the rapid evolution of LLM-accelerated, deployable web applications. Ortiz et al. evaluated three “prompt-to-app” generation systems — Replit, Bolt, and Firebase Studio — using a set of 96 prompts spanning domains like healthcare, legal, real estate, financial services, government, and education. Those prompts produced 288 unique application artifacts (96 from each platform), which were then assessed in a human-rater study of 205 participants and 1,071 quality-filtered pairwise comparisons.

The study used a two-stage design. In the isolated stage, raters evaluated each app on its own across two five-point measures: clarity (“How clear is the app’s purpose?”) and ease of use (“How easy or difficult was this task?”), the latter tied to completing a specific functional task inside the app. In the comparison stage, raters used both apps on the same task and then judged them side-by-side on visual appeal, visual appropriateness, comparative ease, and trust.

The isolated stage: near-identical performance

In isolation, the three systems performed nearly identically. All scores below are adjusted via Linear Mixed-Effects Models that control for participant grading bias (harsh vs. lenient raters), prompt difficulty, and presentation order.

The only statistically significant difference in the isolated stage was Firebase scoring lower than Bolt on ease of use (β = −0.130, p = 0.029). The Replit-vs-Bolt difference was not significant (p = 0.51). Across any single metric, the spread between platforms never exceeded 0.28 points on a 5-point scale.

Taken alone, these results would suggest a relatively commoditized market: little difference in user opinion on the quality of these products. Read in isolation, Firebase even looks like the weakest of the three on ease of use.

The comparison stage: a completely different story

Once raters used both apps on the same task and compared them directly, a clear hierarchy emerged. Firebase achieved the highest win rate across all four metrics.

The remaining percentages in each metric that don’t sum to 100% represent cases where users expressed no preference between the two platforms they were comparing.

Firebase’s advantages over Bolt were statistically significant across all four dimensions: Trust (β = 0.051), Visual Appropriateness (β = 0.049), Visual Appeal (β = 0.042), and Ease Overall (β = 0.034), all at p < 0.05. Replit, in turn, performed significantly worse than Bolt across most dimensions, particularly Trust and Ease Overall. Effect sizes (measured via Cliff’s Delta) were largest for Firebase over Bolt in Visual Appropriateness (Δ = −0.234) and Visual Appeal (Δ = −0.212), and for Firebase over Replit in Trust (Δ = 0.198) — all classified as “small.”

The paper’s final leaderboard combines normalized LMM scores with Bradley-Terry tournament win rates into a composite score.

Firebase ranked first at 1.933, Bolt second at 1.561, and Replit third at 1.302, forming a consistent Firebase > Bolt > Replit hierarchy across every statistical approach used.

Why the gap between stages matters

The authors draw two conclusions from this divergence. First, that independent assessments are statistically indistinguishable, suggesting raters struggle to provide reliable reference-free evaluations in this domain. This makes side-by-side comparison the only reliable methodology for establishing a meaningful benchmark. Second, that the systems “are not interchangeable” and the market “is not yet commoditized,” because a single system outperformed competitors across every measured axis.

That divergence implies something fundamental: human judgment about software quality is relative, not absolute. Without something to compare against, users can’t reliably detect differences that are nonetheless real. Their isolated ratings converge toward the middle of the scale regardless of actual platform quality.

The implication for capital

(The following is my interpretation. The paper itself does not discuss funding, cost, or capital allocation.)

From a funding standpoint, this distinction is critical. The isolated metrics’ implication that the market is commoditized is dispelled by the comparison metrics, which show significant differences in user preference. If the only signal capable of surfacing those differences is side-by-side human evaluation, then much of the evidence investors and procurement teams rely on, such as automated benchmarks, code-quality scores, and deployment success rates, are precisely the kind of isolated signals this study shows are insufficient to detect platform differences. The comparative signal that does surface them is not something that shows up in a pitch deck or a technical due-diligence checklist. That is to say: these results matter because they suggest capital may currently be allocated based on metrics that are structurally blind to user trust.