Advantages of Lakeflow Spark Declarative Pipelines for ETL

01Two ways to build a pipeline

There are two fundamentally different ways to build an ETL pipeline on Databricks. The imperative way — hand-written Spark and Structured Streaming jobs — means you write every step yourself: the reads, the transformations, the checkpoints, the write logic, the orchestration order, the retry handling, the data-quality checks, and the monitoring. You describe how everything happens.

The declarative way — Lakeflow Spark Declarative Pipelines — flips this. You declare the datasets you want to exist and the queries that define them, and the engine works out the rest: dependency order, parallelism, incremental computation, retries, and quality enforcement. You describe what you want, not how to get there.

Imperative code tells the machine every step. Declarative code tells the machine the destination and lets it find the route.

02What Lakeflow SDP actually is

Lakeflow Spark Declarative Pipelines is a declarative framework for building batch and streaming pipelines in SQL or Python. You define a small number of object types, and the pipeline runtime assembles and runs them:

• Streaming tables — Unity Catalog-managed tables that are streaming targets, ideal for incremental ingestion (e.g. via Auto Loader).
• Materialized views — UC-managed tables that hold the results of a batch query and recompute efficiently when inputs change.
• Flows — the processing steps that move data into a table (append flows, AUTO CDC flows).
• Sinks — targets for writing streaming results out to external systems such as Kafka or Event Hubs.
• Expectations — declarative data-quality constraints attached to a dataset.

A pipeline is the container for all of these. When it runs, it analyses the dependencies between your datasets, builds an execution graph, and orchestrates the whole thing automatically.

03Declarative vs. imperative — the core shift

The single biggest advantage is captured in one number: SDP can reduce hundreds or thousands of lines of manual Spark and Structured Streaming code down to a handful. Compare the same Bronze→Silver step written both ways.

# You wire up reads, checkpoints, MERGE logic, and orchestration yourself
def upsert_to_silver(batch_df, batch_id):
    batch_df.createOrReplaceTempView("updates")
    batch_df.sparkSession.sql("""
      MERGE INTO silver.orders t USING updates s
      ON t.order_id = s.order_id
      WHEN MATCHED THEN UPDATE SET *
      WHEN NOT MATCHED THEN INSERT *""")

(spark.readStream.format("cloudFiles")
   .option("cloudFiles.format", "json")
   .load("/landing/orders/")
   .filter("amount >= 0")                     # hand-rolled quality check
   .writeStream
   .foreachBatch(upsert_to_silver)
   .option("checkpointLocation", "/chk/silver")  # you manage state
   .trigger(availableNow=True)
   .start())
# ...then separately schedule + order every job, add retries, add monitoring

from pyspark import pipelines as dp

@dp.table(name="silver_orders")                    # declares a streaming table
def silver_orders():
    return (spark.readStream.table("bronze_orders")
              .filter("amount >= 0"))
# orchestration, checkpoints, retries, lineage — all handled for you

04Advantage 1 — automatic orchestration

With hand-written jobs, you decide the order things run in — Bronze before Silver, Silver before Gold — usually by wiring tasks together in a scheduler. Get the order wrong, or forget a dependency, and the pipeline breaks or produces stale data.

SDP reads the queries you declared, works out the dependency graph automatically, and runs each step in the correct order and with the maximum safe parallelism. Because a Silver table reads from a Bronze table, SDP simply knows Bronze must run first — you never declare the order explicitly.

05Advantage 2 — built-in data quality

In hand-written pipelines, data-quality checks are something you bolt on manually — a filter here, a count there, a custom alert somewhere else. SDP makes quality a first-class, declarative feature through expectations: constraints you attach to a dataset that are enforced and tracked automatically.

You choose what happens when a row violates a rule: keep it and record the metric, drop the bad row, or fail the update entirely. The pass/fail rates are captured in the pipeline's event log, so quality is observable, not guesswork.

CREATE OR REFRESH STREAMING TABLE silver_orders (
  -- drop rows that fail; violations tracked as metrics
  CONSTRAINT valid_amount   EXPECT (amount >= 0) ON VIOLATION DROP ROW,
  CONSTRAINT valid_order_id EXPECT (order_id IS NOT NULL) ON VIOLATION FAIL UPDATE
) AS SELECT * FROM STREAM(bronze_orders);

06Advantage 3 — incremental & unified batch/stream

Efficient pipelines process only new data, not the entire dataset every run. Doing that by hand means managing checkpoints, offsets, and state yourself — exactly the kind of thing that's easy to get subtly wrong.

• Streaming tables incrementally ingest and process new rows as they arrive, tracking progress automatically.
• Materialized views recompute their results efficiently when upstream data changes, rather than from scratch.
• One framework for both: the same SDP code handles batch and streaming — the difference is often just spark.read vs. spark.readStream. You don't maintain two separate codebases.

07Advantage 4 — automatic retries & recovery

Transient failures — a flaky network call, a briefly unavailable source — are a fact of production life. In hand-written jobs you write your own retry logic and hope you covered every case. SDP retries transient failures automatically, and it does so from the most granular, cheapest unit first: it retries the failed Spark task, then the flow, and only if needed the whole pipeline.

Combined with checkpoint-based state, this means a pipeline can recover from an interruption and pick up where it left off — without you writing a single line of recovery code.

08Advantage 5 — lineage, observability & governance

Because SDP knows every dataset and every dependency, it produces a live pipeline graph and end-to-end lineage for free. You can see exactly how Gold traces back to Silver traces back to Bronze — no separate lineage tooling required.

• Automatic lineage: the dependency graph is visible in the UI and captured in Unity Catalog.
• Observability: a structured event log records runs, data-quality metrics, and failures.
• Governance built in: streaming tables and materialized views are Unity Catalog-managed tables, so permissions, auditing, and lineage come through UC natively.

09Advantage 6 — CDC & SCD without hand-rolled MERGE

Change data capture and slowly changing dimensions are notoriously fiddly to implement by hand — you juggle MERGE statements, ordering columns, and out-of-order events. SDP provides AUTO CDC (the successor to APPLY CHANGES) to handle this declaratively, including both SCD Type 1 (overwrite with latest) and SCD Type 2 (retain full history).

You point AUTO CDC at a source of changes, tell it the key and the sequencing column, and it maintains the target dimension correctly — deduplicating, ordering, and applying inserts, updates, and deletes for you.

10When hand-written Spark still wins

Declarative isn't a silver bullet, and a good architect knows the edges. Hand-written Spark still makes sense when:

• You need fine-grained control the framework doesn't expose — custom state handling, unusual sinks, or bespoke streaming logic.
• The workload isn't really a table-producing pipeline — one-off scripts, complex ML training loops, or arbitrary Python that doesn't fit the dataset model.
• You can't use the Premium plan or the managed runtime SDP requires.
• You're integrating with external orchestration where you deliberately want to own the control flow.

For the large majority of production ETL that reads sources, transforms through Bronze/Silver/Gold, and enforces quality, SDP removes far more work than it costs — which is exactly why it's the recommended default.

11SDP vs. hand-written Spark

12Exam tips & practice questions

Key facts to memorise

• SDP = declarative: you declare datasets; the engine handles orchestration, retries, incremental compute, and quality.
• Naming: Delta Live Tables → Lakeflow Spark Declarative Pipelines; open-sourced in Apache Spark 4.1.
• Objects: streaming tables, materialized views, flows, sinks, expectations — inside a pipeline.
• Outputs are UC-managed tables → automatic governance and lineage.
• Expectations = declarative quality: default / DROP ROW / FAIL UPDATE.
• AUTO CDC (formerly APPLY CHANGES) handles SCD Type 1 & 2 declaratively.
• Requires the Premium plan.

13Quick-reference cheat sheet

That's Subdomain 3.3. The one-line takeaway: declarative pipelines let you describe the destination, not the route. For standard production ETL — ingest, clean through Bronze/Silver/Gold, enforce quality, serve — Lakeflow SDP removes the orchestration, retry, incremental, quality, and lineage code you'd otherwise write and maintain by hand. Know the six advantages, know the objects, and remember the naming has changed but the concepts haven't.

Good luck on the exam!