The Three Layers of the Medallion Architecture

01Why the Medallion Architecture exists

Raw data arriving from the outside world is almost always a mess. Files land in inconsistent formats, columns get renamed at the source, duplicate records slip through, timestamps arrive in three different timezones, and every so often an upstream system ships you a batch of malformed rows. If you point your BI dashboards and ML models directly at that raw feed, everything downstream inherits the chaos — and every consumer ends up re-solving the same cleaning problems in their own way.

The Medallion Architecture is a data design pattern that brings order to this by organising data into a series of layers, each one progressively more refined than the last. Data is incrementally improved as it flows through Bronze → Silver → Gold, so that data quality and business value rise at every hop.

Each layer does exactly one job and hands cleaner data to the next. Nobody downstream has to re-solve a problem an upstream layer already solved.

The names come from Olympic medals — but the more useful way to read them is as a quality gradient. Bronze is raw and untouched, Silver is validated and conformed, Gold is curated and ready to serve. It is not a rigid rule handed down by the platform; it is a convention that gives a team a shared vocabulary and a predictable place for every kind of transformation to live.

02The big picture — one pipeline, three layers

Before we go layer by layer, hold the whole shape in your head. A single logical pipeline reads from a source, lands raw data in Bronze, refines it into Silver, then aggregates it into Gold. Every layer is stored as Delta Lake tables — the format never changes between layers, only the degree of refinement does.

03Bronze — the raw landing zone

The Bronze layer is where data lands exactly as it arrived from the source system, with as little transformation as possible. Its job is not to be clean — its job is to be a complete, faithful, replayable record of everything the source ever sent you.

What Bronze is for

• Fidelity to the source: data is stored as-is, so nothing is lost. If your Silver logic has a bug, you can reprocess from Bronze without re-pulling from the source.
• Append-only history: Bronze is typically an ever-growing log of raw records. It captures what the source said, when it said it.
• Ingestion metadata: this is the one thing you do add — columns like the source file name and an ingest timestamp, so every record is traceable back to its origin.
• A cheap, durable buffer: decoupling ingestion from processing means a spike in incoming data never breaks your transformation logic.

Bronze is usually populated with Auto Loader (cloudFiles) for incremental file ingestion, or COPY INTO for simpler batch loads. Schema is handled leniently — Auto Loader can infer and evolve the schema and route unexpected fields into a rescued-data column rather than failing the load.

# Ingest raw JSON into a Bronze Delta table, as-is
from pyspark.sql.functions import current_timestamp, input_file_name

bronze_df = (spark.readStream
  .format("cloudFiles")
  .option("cloudFiles.format", "json")
  .option("cloudFiles.schemaLocation", "/chk/orders_schema")
  .option("cloudFiles.inferColumnTypes", "true")
  .load("/landing/orders/")
  # the only transformation: attach ingestion metadata
  .withColumn("_ingest_ts", current_timestamp())
  .withColumn("_source_file", input_file_name()))

(bronze_df.writeStream
  .format("delta")
  .option("checkpointLocation", "/chk/orders_bronze")
  .trigger(availableNow=True)
  .toTable("main.bronze.orders"))

04Silver — the cleansed, conformed layer

The Silver layer is where raw Bronze data becomes trustworthy. This is the layer most of your engineering effort lives in, because turning raw records into a clean, consistent, queryable model is genuinely hard work. Silver gives the organisation an "enterprise view" of its key entities — customers, orders, events — that any downstream team can build on.

What happens in Silver

• Cleaning & validation: null handling, dropping or quarantining malformed rows, enforcing expected value ranges.
• Type enforcement: casting strings to proper dates, decimals, and integers so the schema is reliable.
• Deduplication: removing repeated records that the source or ingestion may have produced.
• Conforming: standardising column names, units, and formats so everything speaks the same language.
• Enrichment & joins: combining related sources into cleaner, more useful entities — while still keeping records at an atomic grain.

Silver is the natural home for MERGE, because you are usually upserting the latest state of a record (including CDC feeds from the source) rather than blindly appending. MERGE also makes loads idempotent, so a re-run does not create duplicates or violate keys.

-- Clean, cast, dedupe, then MERGE into Silver
MERGE INTO main.silver.orders AS t
USING (
  SELECT
    CAST(order_id AS BIGINT)        AS order_id,
    CAST(order_ts AS TIMESTAMP)     AS order_ts,
    CAST(amount   AS DECIMAL(12,2)) AS amount,
    upper(country)                    AS country
  FROM main.bronze.orders
  WHERE order_id IS NOT NULL          -- validate
    AND amount   >= 0                  -- quality rule
  QUALIFY row_number() OVER
    (PARTITION BY order_id ORDER BY order_ts DESC) = 1  -- dedupe
) AS s
ON t.order_id = s.order_id
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *;

05Gold — the business-ready layer

The Gold layer is where clean Silver data is shaped into curated, consumption-ready tables built for a specific purpose. Where Silver answers "what is the clean, complete truth?", Gold answers "what does this dashboard, this report, this model need?"

What lives in Gold

• Business-level aggregates: daily revenue by region, active users per cohort, KPIs and metrics.
• Dimensional models: star-schema fact and dimension tables optimised for BI tools.
• Denormalised, project-specific marts: a wide, ready-to-query table for one team's use case.
• ML feature tables: engineered features shaped for training and inference.

Gold tables are typically smaller in row count but far higher in value. They are read by many consumers and are the layer BI tools and analysts point at — so they are tuned for fast reads.

-- Business-level aggregate: daily revenue by country
CREATE OR REPLACE TABLE main.gold.daily_revenue AS
SELECT
  DATE(order_ts)      AS order_date,
  country,
  count(*)             AS order_count,
  sum(amount)          AS total_revenue,
  avg(amount)          AS avg_order_value
FROM main.silver.orders
GROUP BY DATE(order_ts), country;

06How data flows through the pipeline

The mechanics are simpler than the concept. Each layer is a Delta table; each transformation reads from the previous layer and writes to the next. Nothing reads from the source twice, and nothing skips a layer without a reason.

1. Source → Bronze: Auto Loader or COPY INTO incrementally ingests raw files and appends them, with ingestion metadata attached.
2. Bronze → Silver: a transformation cleans, casts, deduplicates and validates, then MERGEs the result into the Silver table.
3. Silver → Gold: aggregation and dimensional modelling produce business-ready tables consumers can query directly.

Because every layer is Delta, the whole chain can run incrementally — Structured Streaming and Auto Loader use checkpoints to process only new data at each stage, so the pipeline handles continuous and batch workloads with the same code.

07Building it with Declarative Pipelines

Medallion pairs naturally with Lakeflow Spark Declarative Pipelines (formerly Delta Live Tables). You declare each layer as a table or view, describe the dependency between them, and the runtime figures out the execution order, incremental processing, and data-quality enforcement for you.

• Streaming tables for incremental Bronze and Silver ingestion.
• Materialized views for Gold aggregates that recompute efficiently.
• Expectations (@dlt.expect) to enforce quality rules right in the pipeline — drop, quarantine, or fail on bad rows.

import dlt
from pyspark.sql.functions import col

# --- Bronze: raw ingest ---
@dlt.table(name="bronze_orders")
def bronze_orders():
    return (spark.readStream.format("cloudFiles")
        .option("cloudFiles.format", "json")
        .load("/landing/orders/"))

# --- Silver: clean + validate with an expectation ---
@dlt.table(name="silver_orders")
@dlt.expect_or_drop("valid_amount", "amount >= 0")
def silver_orders():
    return (dlt.read_stream("bronze_orders")
        .withColumn("amount", col("amount").cast("decimal(12,2)"))
        .dropDuplicates(["order_id"]))

# --- Gold: business aggregate ---
@dlt.table(name="gold_daily_revenue")
def gold_daily_revenue():
    return (dlt.read("silver_orders")
        .groupBy("country").sum("amount"))

08Bronze vs. Silver vs. Gold

The exam loves to give you a scenario and ask which layer it describes. This table is the fastest way to lock in the distinctions:

09Design decisions & common mistakes

Knowing the definitions passes the exam. Knowing where teams go wrong is what makes you an architect. These are the mistakes that show up again and again in real lakehouse builds:

• Transforming in Bronze. Cleaning here destroys your ability to replay from source. Bronze must stay raw.
• Aggregating in Silver. Rolling up to totals in Silver couples your clean model to one consumer's needs. Keep Silver atomic; aggregate in Gold.
• Treating three layers as mandatory. Medallion is a flexible pattern, not a law. You can have many Silver tables, multiple Gold marts, or skip a layer where it genuinely adds nothing.
• Forgetting ingestion metadata. Without a source-file and timestamp column in Bronze, you lose traceability the moment something goes wrong.
• Letting consumers read Silver directly for reporting. That is what Gold is for — a stable, curated contract that shields consumers from model changes.

10Exam tips & practice questions

Key facts to memorise

• Three layers in order: Bronze → Silver → Gold; data quality rises left to right.
• Bronze = raw, as-ingested, append-only, full history, ingestion metadata.
• Silver = cleaned, deduplicated, validated, conformed, enriched — still atomic grain.
• Gold = aggregated, curated, business-ready; powers BI, reporting, and ML.
• All three are Delta Lake tables — the pattern is a convention, not a table type.
• Common tooling: Auto Loader / COPY INTO → Bronze; MERGE → Silver; aggregation → Gold.

11Quick-reference cheat sheet

That is everything you need for Subdomain 3.1. The core message is simple: each layer refines the last. Bronze preserves the raw truth, Silver turns it into a clean and trustworthy model, and Gold shapes that model into exactly what each consumer needs. Know the direction, know the job of each layer, and know that all three are just Delta tables playing different roles — and the exam questions on this topic become easy points.

Good luck on the exam!