Classify Cluster Type & Configuration for Optimal Performance

01Why compute choice is the biggest lever

Every notebook you run, every pipeline you schedule, and every dashboard query executes on compute — a cluster of virtual machines running Apache Spark. Pick the wrong type of cluster and you either burn money on idle machines or throttle a production job. Size it wrong and you spend twice as long — and twice as much — as you needed to.

Compute cost is measured in DBUs (Databricks Units) — a unit of processing consumed per hour, on top of the raw cloud VM cost. Different cluster types are billed at different DBU rates, so the type you choose has a direct, permanent effect on your bill.

Right compute is a design decision, not an afterthought. It sets the ceiling on how fast, cheap, and reliable everything downstream can be.

02The two cluster types at a glance

Databricks gives you two fundamental kinds of general-purpose Spark compute, and the difference is entirely about who creates them and how long they live:

• All-purpose clusters are created by a person, can be shared by many users, and stay running until stopped. Built for interactive work.
• Job clusters are created automatically by the scheduler when a job starts, and are torn down the instant the job finishes. Built for automated work.

A third compute type, the SQL Warehouse, is dedicated to SQL and BI workloads and comes in Classic, Pro, and Serverless flavours — worth knowing it exists, but the two clusters above are the heart of this subdomain.

03All-purpose clusters — interactive work

All-purpose (interactive) clusters are for exploration and collaboration: writing and testing code in notebooks, ad-hoc analysis, debugging, and shared development. Their defining trait is that they persist — you start one, several engineers attach to it, and it keeps running across many notebook runs until someone stops it or it auto-terminates.

Characteristics

• Created manually via the UI, CLI, or API, and can be restarted.
• Shared by multiple users at once, which is great for collaboration.
• Higher DBU rate — you pay a premium for that always-on interactivity.
• Auto-termination is the key cost control: set an idle timeout so the cluster shuts itself down after N minutes of inactivity instead of running all night.

04Job clusters — automated workloads

Job clusters (also called automated clusters) are created on demand by the Databricks job scheduler. When a scheduled or triggered job runs, a fresh cluster spins up, executes the job, and is automatically terminated when the job completes. You cannot restart a job cluster or attach a notebook to it interactively.

Why they're the right default for production

• Cheaper: job compute is billed at a lower DBU rate than all-purpose compute.
• Isolated: a clean environment every run means no leftover state, libraries, or variables from a previous run to cause surprises.
• Zero idle cost: the cluster exists only for the duration of the job — nothing is left running afterwards.
• Reliable & reproducible: the same cluster spec is provisioned identically each time the job runs.

05Autoscaling — sizing that adapts

Rather than pinning a fixed number of workers, autoscaling lets Databricks add and remove worker nodes automatically within a range you define (a minimum and a maximum). When a stage needs more parallelism, workers are added; when the cluster goes quiet, idle workers are released to save money.

• You set the min and max worker count; Databricks manages everything in between based on load.
• It scales the number of workers only — the driver node stays a fixed size.
• It's ideal for variable or unpredictable workloads where a fixed size would either bottleneck or waste money.

06Spot instances — cheap, but interruptible

Spot instances use a cloud provider's spare capacity at a steep discount — often a fraction of on-demand price. The catch: the provider can reclaim (evict) them at short notice when it needs the capacity back. That makes spot fantastic for cost savings and risky for anything that can't tolerate a node vanishing mid-task.

The golden rule of spot

Put workers on spot, keep the driver on-demand.

If a spot worker is reclaimed, Spark simply re-runs its lost tasks on the remaining nodes — a slowdown, not a failure. But if the driver is lost, the entire cluster dies and the job fails. So the standard, cost-optimised pattern is spot workers (ideally with fallback to on-demand if spot capacity is unavailable) and an on-demand driver.

07Driver & worker sizing

A Spark cluster is one driver node coordinating many worker nodes. Knowing what each does tells you which one to make bigger when performance suffers.

Size the driver up when…

• You pull large results back to the driver — collect(), toPandas(), or displaying huge outputs.
• You broadcast a large table in a join (the driver stages the broadcast).
• You run many concurrent notebooks / streams on one cluster, or heavy single-node (non-Spark) Python.

Scale the workers out when…

• Your data is large and the bottleneck is distributed compute — more workers means more parallelism.
• Jobs are shuffle- or CPU-heavy and would finish faster with more executors.

Choosing an instance family

• Memory-optimised — for shuffle-heavy joins, aggregations, and caching-heavy workloads.
• Compute-optimised — for CPU-bound work with modest memory needs.
• Storage-optimised — for workloads that benefit from fast local disk caching (Delta / disk cache).

08Configuring for optimal performance

Once you've picked a type, a handful of settings do most of the work of balancing cost against speed:

1. Auto-termination on every all-purpose cluster — the simplest way to kill idle spend.
2. Autoscaling for variable batch workloads, with a sensible min/max range.
3. Spot workers + on-demand driver for cost-tolerant jobs.
4. Instance pools — keep a set of ready, idle VMs warm so clusters start in seconds instead of minutes, cutting job latency.
5. Photon — Databricks' vectorised query engine that accelerates SQL and DataFrame workloads; it costs extra DBUs but often improves overall price/performance.
6. Right-size, don't over-provision — pick node families and counts that match the actual bottleneck (memory vs CPU vs disk).

// A job cluster: autoscaling workers + spot with on-demand driver
{
  "new_cluster": {
    "spark_version": "15.4.x-scala2.12",
    "node_type_id": "Standard_DS4_v2",
    "autoscale": { "min_workers": 2, "max_workers": 8 },
    "azure_attributes": {
      "availability": "SPOT_WITH_FALLBACK_AZURE",  // workers on spot
      "first_on_demand": 1                        // driver stays on-demand
    }
  }
}

09All-purpose vs. job — the table

The exam frequently hands you a scenario and asks which cluster fits. Lock in the distinctions:

10Design decisions & common mistakes

• Production jobs on all-purpose clusters. Higher DBU rate plus idle-cost risk. Use job clusters for anything scheduled.
• Driver on spot. One eviction kills the whole cluster. Keep the driver on-demand; only workers go on spot.
• Scaling out when the driver is the bottleneck. Adding workers won't help a collect()-heavy or broadcast-heavy job — that needs a bigger driver.
• Assuming autoscaling always helps. Steady streaming and cache-dependent jobs may see little benefit or even disruption.
• No auto-termination. The most common source of surprise bills is an interactive cluster left running overnight.
• Over-provisioning "to be safe." Bigger isn't faster if the bottleneck is elsewhere; match node family to the actual constraint.

11Exam tips & practice questions

Key facts to memorise

• All-purpose = interactive, shared, manual, persists, restartable, higher DBU cost.
• Job cluster = automated, single job, terminates on completion, not restartable, lower DBU cost.
• Autoscaling adjusts workers between min/max; the driver is fixed.
• Spot = cheap but reclaimable → workers on spot, driver on-demand.
• Driver coordinates + collects results; workers execute tasks in parallel.
• Single-node cluster = driver only, no workers; for small / single-machine work.
• Auto-termination controls idle cost; pools cut start-up time; Photon speeds SQL/DataFrame work.

12Quick-reference cheat sheet

That's Subdomain 3.2 in full. The mental model is simple: match the compute to the workload. Interactive and collaborative work goes on all-purpose clusters with auto-termination; automated production work goes on cheaper, self-terminating job clusters. Then tune with autoscaling, spot workers behind an on-demand driver, pools, and Photon. Get the type right first, size second, and the exam's cluster questions become predictable.

Good luck on the exam!