Automated Event Ingestion Pipeline
Built for a community discovery platform covering local events and groups
Overview
A community discovery platform was growing its coverage by hand. One person visited 52 different sources, read each listing, and typed events into the platform, which capped how fast new regions and categories could launch. This engagement replaced that with a five-stage ingestion pipeline: extract raw content from structured and unstructured sources, structure it into a fixed schema, deduplicate against what is already published, validate, then publish. The 52 sources collapse into four extractor archetypes, so onboarding a new site usually means choosing an archetype rather than writing another bespoke scraper. Code handles the high-volume work and the language model is reserved for the two places it earns its cost, structuring messy listings and resolving ambiguous fields, which keeps the per-event cost low enough to run continuously.
The Problem
Coverage was limited by how fast one person could type. Every new source needed its own extraction effort, every event needed manual validation, and expanding into a new region or category meant paying for all of it again in human hours. Growth was capped by data entry rather than by demand.
My Approach
Audited all 52 sources and found they reduce to four structural archetypes, which turned per-site scraping into archetype selection. Built the pipeline in five stages so each one can fail loudly and be re-run on its own: extraction, structuring, deduplication against published records, validation, and publication. Kept the language model out of the high-volume path and used it only for structuring unpredictable listing text and resolving fields the rules could not settle, so cost scales with difficulty rather than with volume.