Skip to main content
Use this pattern for payment, product, telemetry, or partner webhooks that must be acknowledged quickly but retained for downstream analysis.

1. Bind the Stream

Store the verification secret as an encrypted deployment secret, not in vars. Create partner_webhooks with an immutable structured schema. The binding references this resource; it does not define its schema:

2. Verify and ingest

Return 202 only after send resolves. At that point the Stream has durably acknowledged the record; lakehouse publication can happen independently.

3. Normalize in a Pipeline

Map webhook_events to an Iceberg Sink. The Sink and Catalog deduplicate Pipeline batch identities, so a crash and retry cannot publish the same batch as a second logical commit.

4. Keep the raw envelope

If audit or replay matters, fan out the same Pipeline input to a second Sink:
The raw Iceberg table preserves the source envelope. The normalized table is the stable analytical contract.

Production checks

  • Verify the signature before writing to the Stream.
  • Preserve the source event ID even if deduplication happens later.
  • Keep payloads below the 5 MiB Stream request limit.
  • Treat schema changes as versioned data contracts.
  • Alert on Stream validation errors and Pipeline cursor lag.