Skip to main content
The medallion pattern separates raw data, validated detail, and business-facing aggregates. In Verglas, those layers are resource and schema boundaries—not clusters that stay running.

Bronze: preserve source facts

Wrap every incoming record with source metadata before sending it:
Create raw_events as a structured Stream. The binding only references the resource; the immutable schema is part of Stream creation:
The first Pipeline statement copies that envelope to a raw Iceberg Sink:
Keep Bronze append-only. It is the input for audits and future reprocessing.

Silver: enforce the analytical contract

A second statement filters and projects accepted records from the same durable source sequence:
Fan invalid rows to a quarantine Sink instead of silently losing them:

Gold: serve a business question

Attach a Query destination to the validated Pipeline output and define bounded endpoints such as:
The Query Durable Object maintains declared aggregates as Pipeline batches arrive. Use Gold for a specific dashboard or application question; keep Silver as the reusable detailed dataset.

Why this shape works

  • One Stream sequence feeds every layer, so raw and curated outputs share a traceable input position.
  • Sink batch IDs make Iceberg publication idempotent.
  • Query replay receipts make aggregate updates idempotent.
  • Each stateful component scales by its name instead of by a manually sized job cluster.
Verglas Pipeline SQL is stateless and does not implement joins, windows, or GROUP BY. Use a Query definition for maintained aggregates. For multi-source joins, normalize each source first and join with an external Iceberg engine until a declared stateful join component is available.