Bifrost

Bifrost is the analytical warehouse inside every Wyrd server. Every observation, verdict, trace, audit record, and custom row lands in it, and all of it is queryable with SQL. It runs inside wyrd-server; there is nothing extra to deploy.

How it is put together

Bifrost has four parts. Their names show up in logs, metrics, and deployment settings, so they are worth learning:

Part	Does
Gate	Authenticates every request, checks the permission, and routes it: writes to a Scribe, queries to an Oracle
Scribe	Takes writes, acknowledges them once they are in its write-ahead log, serves recent rows as the live tail, and publishes them as Parquet
Oracle	Runs SQL over the published tables plus each Scribe’s live tail, spreading large queries across peers
Forge	Compacts small files, applies retention, and commits new Iceberg snapshots in the background

Answers: What are the parts of Bifrost, and which way does data move?

Now

Every client enters through Gate. Choose Write path, Read path, or Background to follow one flow, or press Play.

Text version

Every client enters through Gate. Choose Write path, Read path, or Background to follow one flow, or press Play.
Write path. Writes arrive over gRPC at Gate, which authenticates the credential, checks the permission, and routes the batch to a Scribe. Scribe acknowledges once the batch is in its write-ahead log, which is when flush() returns, then publishes the rows to object storage as Parquet.
Read path. A query arrives over HTTP at Gate, which routes it to an Oracle. Oracle pins the published Iceberg snapshot and asks each Scribe for its live tail, the rows not published yet. Large queries fan out across peer Oracles. Results stream back and end with a terminal status: success, degraded, or failed.
Background. Forge runs on its own. One Forge leader decides which tables are due; Forge workers promote Scribe output into Iceberg snapshots, compact small files, and apply retention. Postgres holds the catalog and elects the leader. You never call Forge directly.

Where each part runs

Every pod runs the same wyrd-server binary. WYRD_TARGET picks which parts it runs, and the default, all, runs everything. A single-pod install never needs to set it. A team that outgrows one pod splits the parts so each scales on its own signal: Scribes with write volume, Oracles with query load, Forge workers with the compaction backlog.

Answers: Which WYRD_TARGET runs Gate, Scribe, Oracle, and Forge, and which do I scale?

Now

Every pod is the same wyrd-server binary. WYRD_TARGET picks which parts it runs. The default, all, runs everything, which is what a single-pod install uses. Pick a part to see where it lives.

SelectPick Gate, Scribe, Oracle, Forge leader, or Forge worker above.

Text version

Every pod is the same wyrd-server binary. WYRD_TARGET picks which parts it runs. The default, all, runs everything, which is what a single-pod install uses. Pick a part to see where it lives.
One way a team splits Bifrost: Scribes on persistent disks for writes, autoscaled Oracles for queries, one server pod holding the Forge leader, and workers sized to the compaction backlog. Pick a part to follow it.
Gate. The front door. Authenticates every request, checks the permission and records the decision in audit, then routes it: writes to a Scribe, queries to an Oracle, on the same pod or a peer.
Scribe. Takes writes. Acknowledges a batch once it is in the write-ahead log, holds recent rows, serves them to queries as the live tail, and publishes them to object storage as Parquet.
Oracle. Runs SQL. Pins a published Iceberg snapshot, asks every ready Scribe for its live tail, and fans large queries out across peer Oracles.
Forge leader. Decides maintenance. Exactly one is active per deployment, elected through Postgres; it tracks which tables are due for compaction and retention and hands them to workers.
Forge worker. Does the rewriting. Pulls due tables from the leader, compacts small files into larger ones, applies retention, and commits new Iceberg snapshots.

scribe and oracle pods each run half of the read path, so they only work with peers: pods that reach each other over mutual TLS. Exactly one Forge leader is active at a time, elected through Postgres; a second all or server pod waits as a standby.

How it stores data

Tables are Apache Iceberg tables of Parquet files in the object store you configured with WYRD_STORAGE_URL. Postgres holds the catalog, scoped per tenant. Queries run on Apache DataFusion.

Each table has a namespace and a name, such as vala.drift.observations. Wyrd owns every namespace except vala.datasets, which is yours. See Write and query custom data with Bifrost for the full list.

Every row carries managed columns Wyrd fills in: card_uid, run_id, sub_principal_id (who the work was for), act_principal_id (who performed it), wyrd_request_id, and wyrd_ingested_at. Each table also has one time column that partitions it, so a query that filters on time reads only the files it needs.

Two ways in

Path	Use it from	Rows carry	Writes to
The observations API, `run.observe.*`	Your running service	The run and Card that produced them	Drift samples, verification inputs and results, and your `vala.datasets` tables
The Bifrost client, `Bifrost`	Batch jobs, backfills, notebooks, analysis	No run, unless you pass a correlation	Your `vala.datasets` tables

They share one writer underneath, and both read with the same SQL. The difference is attribution. The observations API stamps every row with the exact Card version and run from the bundle your service loaded, which is what lets a verdict point back to what ran. The Bifrost client is for data that did not come from a run, and for reading anything you have access to.

Record a run and observations and Write and query custom data with Bifrost show each path.

Writes go to the server over gRPC; queries go over HTTP.

Written versus queryable

These are two different moments, and the gap between them explains most “where is my row” questions.

Answers: I flushed a row. Why does my query not return it yet?

Time

Where the row is Queued in your process Not sent yet

A query returns ✕ no rows Nothing reached the server

Survives a restart ✕ No Lost if the process dies before flush

Now

The row is queued in the SDK. insert() and observe.* return immediately so your request is not slowed down. Nothing is stored yet.

Text version

insert(). The row is queued in the SDK. insert() and observe.* return immediately so your request is not slowed down. Nothing is stored yet.
flush() returns. flush() has returned, so the row is durable. Oracle reads it from the holding Scribe as live tail, on a best-effort basis. An empty result now is not proof the row is missing. The dashed box marks where it will appear in the table.
published. The row is part of the published table. Every query returns it from here on.
compacted. Forge has merged small files into larger ones. Nothing changes for your query except speed.

A write is stored when a Scribe acknowledges it. That is what flush, shutdown, and write_batch wait for. From then on the row survives a server restart.

A row is queryable once it is published into the table, which happens shortly after. An Oracle query reads the published data plus the live tail: recent rows still held by the Scribe that received them. The live tail is best effort, so a row written seconds ago may not appear yet. In rare cases a row being published during the query can be missed or counted twice. Re-run the query; the next one sees a settled table.

This applies to everything that reads Bifrost, including Verifiers. A scheduled drift window that closes the instant a row is written may score without it.

Query results have a terminal

A query streams rows and then ends with a terminal status: success, degraded, or failed.

success means every part of the plan completed.
degraded means a Scribe holding recent rows was unreachable, so its live tail may be missing. Published data is complete.
failed means the query did not finish. Partial rows are never returned as a success.

The SDKs check the terminal for you and raise on failed. The QueryResult keeps it as terminal if you need to tell degraded from success.

What Bifrost does not do

No SQL writes. INSERT, UPDATE, DELETE, and CREATE TABLE AS are refused. Write through the observations API or the Bifrost client.
No schema changes. A table’s columns and layout are fixed at registration. A new schema needs a new table.
No dropping tables. Data is removed by retention, not by you.
No writes outside vala.datasets. Drift, verification, telemetry, gateway, and audit tables are written only by Wyrd.

Permissions

To	You need
Query a table	`bifrost_query:read`
Read Gateway request and response payloads	`gateway_payload:read` as well
List or describe tables	`bifrost_table:read`
Register a table	`bifrost_table:write`
Write rows	`bifrost_record:write`

viewer can query and describe. workload can also register and write. See Identity and authorization.

Last updated October 16, 2026