The medallion pattern separates raw data, validated detail, and business-facing
aggregates. In Verglas, those layers are resource and schema boundaries—not
clusters that stay running.
Bronze: preserve source facts
Wrap every incoming record with source metadata before sending it:
Create raw_events as a structured Stream. The binding only references the
resource; the immutable schema is part of Stream creation:
The first Pipeline statement copies that envelope to a raw Iceberg Sink:
Keep Bronze append-only. It is the input for audits and future reprocessing.
Silver: enforce the analytical contract
A second statement filters and projects accepted records from the same durable
source sequence:
Fan invalid rows to a quarantine Sink instead of silently losing them:
Gold: serve a business question
Attach a Query destination to the validated Pipeline output and define bounded
endpoints such as:
The Query Durable Object maintains declared aggregates as Pipeline batches
arrive. Use Gold for a specific dashboard or application question; keep Silver
as the reusable detailed dataset.
Why this shape works
- One Stream sequence feeds every layer, so raw and curated outputs share a
traceable input position.
- Sink batch IDs make Iceberg publication idempotent.
- Query replay receipts make aggregate updates idempotent.
- Each stateful component scales by its name instead of by a manually sized job
cluster.
Verglas Pipeline SQL is stateless and does not implement joins, windows, or
GROUP BY. Use a Query definition for maintained aggregates. For multi-source
joins, normalize each source first and join with an external Iceberg engine
until a declared stateful join component is available.