How to turn app store reviews into product evidence (not a wishlist)
The short answer
Reviews become evidence when you stop summarising them and start extracting from them. Copy the user's exact words into a table with a source ID, a date and the app version. Tag each row with the job the person was trying to do and what stopped them. Then cluster by cause, not by vocabulary, and count. Two reviews that both say "login" can describe two unrelated bugs. Done this way, a pile of one-star reviews turns into a ranked, defensible list of problems with numbers attached.
The trap: reading reviews to form an impression
Most people read reviews the way they read a comment thread, scrolling until a feeling forms. The feeling is usually "people are angry about X", where X is whatever the three longest reviews happened to mention. Then X goes into a document as a priority and nobody can say how many people it affects, when it started, or whether it is one problem or three.
The fix is a little dull and completely reliable. Treat each review as a record and extract from it. Impressions cannot be checked by a colleague. A table can.
Keep the original words
The first discipline is to copy, not paraphrase. "Can't log in since the update" and "got logged out while paying a bill" both sound like login problems to a tired reader. In the user's own words they are plainly different: one person cannot get in at all after a version change, the other was thrown out mid-task. Paraphrasing merges them. Copying keeps them apart until you have a reason to join them.
Give every row a source ID so anyone can go back to the original, and record the date and app version. Half of all "the app is broken" reviews cluster around a single release once you can see the version column.
Extraction is not summary
Summary compresses. Extraction pulls out specific things in a fixed shape. For each review, extract three fields: the job the person was trying to do, what stopped them, and the words they used for the feeling. "I wanted to pay my electricity bill, the OTP never arrived, I was furious" becomes job: pay a bill; blocker: OTP not delivered; feeling: furious.
This is also the right way to use an AI model on reviews. Ask it to summarise and you get a smooth paragraph that sounds true and cannot be verified. Ask it to extract those three fields per review into a table and you get something you can audit row by row, correct, and count. Then spot-check a sample against the originals, because the model will occasionally invent a job the person never mentioned.
Cluster by cause, not by vocabulary
Now group the rows. The temptation is to group by the words people used, because that is what a word cloud does. Resist it. Group by what would have to change in the product for the complaint to disappear. "OTP never arrived" and "OTP arrived after it expired" share a word and may share a cause. "Logged out while paying" and "can't log in after update" share a word and almost certainly do not.
A cluster that is real has three properties: the rows describe the same user action, the same point of failure, and roughly the same period. If a cluster fails any of those tests, split it. Splitting is cheap. A wrong merge sends an engineer to fix the wrong thing.
Count, then size, then prioritise
With clean clusters you can finally count. Twenty-three rows about OTP delivery in the last thirty days, all on version 4.2, is a claim you can defend in a room. Three rows about dark mode across two years is also a claim, just a different one.
Counting reviews is not the same as sizing the problem, because people who leave reviews are not a sample of people who use the app. So the last step is to go from the review count to a product number: how many bill payments were attempted on 4.2, how many failed at OTP. Reviews tell you where to look. The product data tells you how big it is. Prioritise on the second number and cite the first as how you found it.
Do this week
- Export the last 100 reviews or support tickets for one product into a sheet with quote, source ID, date and version.
- Extract job, blocker and feeling for every row. Use a model if you like, then spot-check 15 rows by hand.
- Cluster by cause. Split anything that mixes two user actions or two failure points.
- For your top cluster, find the product number that sizes it, and write one paragraph that states the problem with both numbers.
Three terms, defined
- Extraction
- Pulling fixed fields out of each piece of feedback, as opposed to summarising the pile.
- Source ID
- A reference that lets anyone go back to the original review or ticket a row came from.
- Cluster
- A group of rows that describe the same action, the same failure point and the same period.
Questions people ask
How many reviews do I need before the method is worth it?
Around fifty gives you clusters you can trust. Below that, still keep the table, because the discipline of copying exact words pays off even with ten rows.
Can I use AI to read reviews for me?
Yes, for extraction into fixed fields, with a hand check on a sample. No, for summaries you intend to act on, because a summary cannot be audited and models will smooth over differences that matter.
What about feature requests in reviews?
Record them the same way, with the job the person was trying to do. A request for dark mode is usually a request about reading in bed, which may have other solutions. Prioritise the job, not the suggested feature.