Serverless Lakehouse
A CSV lands in a raw zone, a crawler infers its schema, and Athena rewrites it as columnar Parquet in a curated zone. The interesting part is not the pipeline, it is the measurement: the same query against curated data scans 156 bytes where the raw version scans 2.09 megabytes.
/01Problem
The argument for a curated zone is usually made in the abstract. Columnar storage is cheaper to scan, compression helps, everyone agrees, and nobody measures it on their own data, so the curated layer gets justified as a best practice rather than as a number.
The goal here was to build the smallest honest version of the pattern and then instrument the claim, so the economic case for the curated zone is a measurement the validator reproduces rather than an assertion in a README.
/02Approach
- One S3 bucket split by prefix into three zones: raw for CSV exactly as produced upstream, curated for Snappy-compressed Parquet, and a scratch prefix for Athena query output expired after seven days by a lifecycle rule so it cannot quietly accumulate cost.
- A Glue crawler infers the raw schema into the Data Catalog, so there is no hand-written DDL tracking a shape the crawler can discover.
- Athena reads the raw table and writes the curated table with a CTAS statement, which makes the curated schema an explicit contract rather than an inferred one.
- A deterministic 50,000-row synthetic orders generator seeded at 42, so the demo produces the same dataset and therefore the same measured numbers on every run.
/03Architecture
The design rule is crawl what you do not control, declare what you do. The raw zone changes shape whenever an upstream system changes, so letting a crawler own its schema means the catalog tracks reality instead of drifting from a hand-maintained definition. The curated zone is the opposite case: its schema is a contract the transform defines, so CTAS states it explicitly.
Splitting the zones by prefix rather than by bucket is a deliberate call for demo infrastructure: one lifecycle policy, one thing to empty at teardown, and a destroy that actually completes because the bucket empties itself.
/04Outcome
The validator runs five checks covering catalog registration, Parquet output, row reconciliation between zones, and the scan-cost claim itself. The same count and sum query scans 156 bytes against curated Parquet versus 2.09 megabytes against raw CSV, which is the entire economic argument for the curated zone expressed as a measurement rather than a principle.
Nothing runs between demos. There is no NAT gateway, no cluster, and no warehouse endpoint, so the idle cost is storage for a small dataset and the full deploy-demo-destroy session lands well under a quarter.