Extraction beats summarisation
Lesson 16 of 24, level 01Index
Four thousand support tickets, eight hundred reviews and sixty sales calls, sitting in three different tools. None of it has ever been read systematically.
A summary is lossy and unfalsifiable
Ask a model to summarise a thousand tickets and you get three paragraphs you cannot check, sort, count or argue with. The evidence is gone.
Ask for a structured extraction and you get a thousand rows, each carrying its original quote. Now you can pivot, count, and trace any finding back to the person who said it. That is the difference between a finding and an opinion.
Decide your columns before you run anything: source, date, verbatim quote, problem label, severity, segment. A pipeline without a fixed schema produces rows you cannot group.
AI can turn messy material into a useful table, but a fluent summary can quietly merge people, erase disagreement, or invent a frequency. Extraction keeps the source visible.
Use your chosen course project throughout. The additional examples below are fictional practice cases; transfer the method to your own evidence.
Define a row before prompting
Choose the unit: one interview episode, support ticket, or distinct observation. Give each source a stable ID and define the fields you need. A theme is an interpretation; a direct quote is source material. Store them in separate columns so a reviewer can tell which is which.
Make absence representable
Include an unknown value when evidence does not support a field. Require exact source snippets and allow multiple interpretations. A forced category for every row makes missing information look like knowledge. Do not ask a model to estimate how many users share a problem from a handful of selected quotes.
Audit the transformation
Manually inspect a sample and every row used for a consequential claim. Count distinct source units, not mentions, if one person repeats the same issue. Record how the sample was collected. A reliable extraction from a biased sample still cannot establish population prevalence.
Keep the chain inspectable
- Source: Stable ID
- Extraction: Exact evidence
- Theme: Your interpretation
- Claim: Bounded conclusion
Follow a claim back to its source
Interactive teaching example · All numbers and tickets are fictional.
“I retried because nothing changed.” Fictional ticket A
- Observed action
- Retried the operation
- Interpretation, not a fact
- Possible missing feedback
Tickets A and B may share a feedback issue. Ticket C does not support that claim. Three selected tickets cannot establish how common the issue is across the product.
How Airbnb handled it
Airbnb
2009
Airbnb was flat in New York. The data showed listings were not converting and nobody knew why.
Gebbia and Chesky flew to New York, sat with hosts, and noticed the listing photos were terrible phone snaps. They rented a camera and shot the listings themselves. Revenue in New York doubled within a month.
The insight was invisible in the aggregate and obvious in the raw artefact. Which is exactly why your pipeline has to preserve the artefact rather than summarise it away.
Doing it with AI, and where it breaks
Write one extraction prompt with a strict JSON schema and run it in batches over the whole corpus. Load into a sheet and pivot.
Over a long batch, models drift. They start inventing new label values around row 300 despite your enumerated list. Add “labels must be exactly one of [list]; if nothing fits, use OTHER” and audit every OTHER by hand. The OTHERs are usually where the new finding is.
Your AI workbench
Start with your own notes or clearly labelled practice data. Remove private details before sharing. Replace the placeholders, run the prompt in your chosen AI tool, and keep the output beside its source.
Extract one row per source ID from the material below. Columns: source ID, exact quote, observed action, inferred theme, uncertainty, and follow-up question. Use unknown for absent evidence. Do not merge sources, rewrite quotes, or estimate population prevalence. After the table, identify claims needing manual verification. Sources: [paste anonymised material].
Before you use the output
- Quoted text matches the source exactly
- Counts use distinct source units
- Themes and facts occupy separate fields
Stuck? Try this next
Ask AI to return a machine-readable table, then compare a few rows against the source yourself. If it invents a field, revise the schema to explicitly permit unknown and rerun the audit.
Keep a brief AI log: input used, useful output, what you checked, and what you rejected. The decision remains yours.
Build it
A working extraction pipeline and a coded signal log of 200 or more rows.
Claude or ChatGPTGoogle Sheets
Checkpoint
If you did the build, these take two minutes. If you cannot answer one of them, that is the part to go back to.
Moving on marks this lesson complete. Finish the build first, it is the part that counts.