Platforms and distributed systems

Shared-nothing architecture distributes data and computation across nodes that do not share memory or storage. Each node processes its local partition independently, communicating only to shuffle data between partitions (for joins and aggregations). This architecture scales linearly: doubling the number of nodes approximately doubles throughput, because each additional node handles an additional partition.

Apache Spark replaced Hadoop MapReduce by keeping intermediate results in memory (RAM) rather than writing them to disk between each computation step. For iterative algorithms like machine learning training, where the same data is processed many times, this in-memory execution reduces job time by 10-100x compared to disk-bound MapReduce. Spark introduced resilient distributed datasets (RDDs) and later DataFrames and Datasets, which provide a columnar, SQL-friendly API while retaining the underlying parallel execution model.

Spark's lazy evaluation model builds a directed acyclic graph (DAG) of transformations before executing any computation. When an action (such as count or write) is triggered, the Catalyst optimiser rewrites the DAG to eliminate redundant steps, push filters as early as possible, and select efficient physical execution plans. This query optimisation is why writing idiomatic Spark code (using the DataFrame API, not RDDs) consistently outperforms manually optimised lower-level code.

Snowflake uses a hybrid of shared-disk and shared-nothing ideas. Persisted data sits in a central repository accessible by compute clusters, while query processing runs on massively parallel compute clusters. Virtual warehouses can be sized and paused independently, which makes workload isolation and cost control board-level design decisions rather than database administrator tuning details.

Google BigQuery is a serverless analytical warehouse: users submit SQL queries without managing clusters. Its operating discipline is different from Snowflake because query cost and performance are strongly shaped by bytes scanned, partitioning, clustering, slot reservations, and governance of who can launch expensive queries.

Databricks builds around Apache Spark, Delta Lake, notebooks, jobs, SQL, and ML tooling. Its strength is mixed workload development: the same platform can ingest events, transform lake data, train models, serve features, and expose SQL analytics. The leadership trade-off is integration versus dependency: one platform can reduce hand-offs, but it also concentrates governance, skills, and commercial risk.

A simple decision test keeps platform selection honest. If the workload is governed SQL analytics with many independent business users, start with a warehouse pattern. If the workload mixes engineering, streaming, ML, and feature pipelines, start with a lakehouse pattern. If sub-second operational decisions are required, keep a serving store or stream processor in the design rather than forcing every decision through the analytical warehouse.

Data lakes (raw files in object storage, schema-on-read) offered cheap scalable storage but no ACID guarantees, no schema enforcement, and poor query performance on small files. Data warehouses offered ACID and performance but were expensive and required schema-on-write. The lakehouse pattern combines cheap open-format storage with warehouse-quality reliability by adding a transactional metadata layer over the data lake.

Delta Lake (Databricks, open-sourced 2019) adds a transaction log to Parquet files stored in object storage. Every write creates a new log entry describing the files added or removed. This provides ACID transactions (readers never see partial writes), time travel (querying the state of the table at any past timestamp), schema evolution (adding columns without rewriting history), and efficient upserts and deletes. Delta Lake is the foundation of the Databricks Lakehouse Platform.

Apache Iceberg (created at Netflix and now an Apache project) solves the same class of problems with a snapshot and manifest architecture rather than a single sequential log. Iceberg's hidden partitioning allows queries to prune data without requiring users to know the partition layout. Its main strategic value is engine neutrality: Spark, Flink, Trino, Presto, Hive, and Impala can safely work with the same tables.

The architectural distinction is important for executives as well as engineers. A lakehouse is not just a new name for a data lake. It is a commitment to table-level contracts: schema change rules, ACID writes, retention and vacuum rules, lineage, access control, compaction, snapshot retention, and recovery. Without those controls, the organisation still has cheap files, not trusted analytical tables.