How we killed our own best idea with more data
The thesis was good. The evidence was real. It was still wrong, and the way it failed is worth more than the thesis would have been.
Our first pass at field service software sampled six apps and found 29 complaints about the app failing when connectivity dropped. Technicians in basements. Rural roads. Clock-ins that never registered. It produced a confident conclusion: these apps die where the work happens, and offline-first is the opening.
It is a satisfying thesis. It has a story, a villain and an obvious product. It was also an artefact of which six apps we happened to pick.
What happened as the sample grew
| Sample | Apps | Low-star reviews | Offline share |
|---|---|---|---|
| First pass | 6 | 1,589 | 29 mentions |
| Widened | 21 | 2,883 | 2.4% |
| Full sweep | 58 | 7,682 | 1.7% |
Widening the sample diluted the signal instead of scaling it. That is the whole tell. A theme that is genuinely market-wide holds its share as the sample grows, because it is present everywhere. One that shrinks was concentrated in the apps you started with — in this case Jobber and Housecall Pro, which do have unusually high offline complaint rates (3.1% and 3.3%) relative to everyone else.
What was actually wrong with those apps
Something much more boring, and much bigger. Reliability — plain crashes and freezes — is 19.5% of complaints, larger than the next two themes combined, and universal: every app measured with enough reviews falls between 11% and 41%.
| App | Rate |
|---|---|
| Salesforce Field Service | 41.0% |
| Workiz | 34.2% |
| ServiceTitan Mobile | 31.9% |
| Simpro Mobile | 26.1% |
| Buildertrend | 20.3% |
| Jobber | 19.5% |
| Housecall Pro | 13.9% |
| Joist | 11.1% |
And the specific thing we had under-weighted turned out to be twice the size of the thesis we were chasing: photo and attachment handling, 3.6%. In the trades, photos are the work product — proof of completion, warranty evidence, insurance. Every incumbent handles them badly, and the complaints are unusually precise:
“Needs an option to upload multiple photos from gallery at once, instead of one at a time. Needs ability to remove photos from tasks in case of accidental upload.”
UpKeep · 1★ · 54 votes
“Often times I'll attempt to upload pictures and it won't do it, OR it'll say it won't upload when it actually did and I end up uploading duplicates.”
Buildertrend · 1★ · 17 votes
That second quote is the offline thesis correctly scoped. The failure is not the absence of an offline mode — it is an upload queue that reports a state it is not in.
The test, in one sentence
Before building on a pattern, re-measure it on a deliberately wider sample and check whether its share holds. If it shrinks, it was never the market — it was your sample. Build the analysis so it can falsify the idea: rank the candidate pain against every competing pain, rather than counting hits for the one you like.
We now do this automatically, reporting per-app rates and flagging any theme concentrated in one or two vendors, so a single-vendor artefact cannot reach a conclusion unchallenged again.