Platforms and distributed systems

Shared-nothing architecture distributes data and computation across nodes that do not share memory or storage. Each node processes its local partition independently, communicating only to shuffle data between partitions (for joins and aggregations). This architecture scales linearly: doubling the number of nodes approximately doubles throughput, because each additional node handles an additional partition.

Apache Spark replaced Hadoop MapReduce by keeping intermediate results in memory (RAM) rather than writing them to disk between each computation step. For iterative algorithms like machine learning training, where the same data is processed many times, this in-memory execution reduces job time by 10-100x compared to disk-bound MapReduce. Spark introduced resilient distributed datasets (RDDs) and later DataFrames and Datasets, which provide a columnar, SQL-friendly API while retaining the underlying parallel execution model.

Spark's lazy evaluation model builds a directed acyclic graph (DAG) of transformations before executing any computation. When an action (such as count or write) is triggered, the Catalyst optimiser rewrites the DAG to eliminate redundant steps, push filters as early as possible, and select efficient physical execution plans. This query optimisation is why writing idiomatic Spark code (using the DataFrame API, not RDDs) consistently outperforms manually optimised lower-level code.

Snowflake (launched 2014) introduced the multi-cluster, shared data architecture. Storage and compute are separated: data is stored in cloud object storage (S3, Azure Blob, GCS); compute clusters (virtual warehouses) are spun up on demand, independently sized, and can be paused when not in use. Multiple warehouses can query the same data simultaneously without contention. This separation means storage cost is always on (data must persist) but compute cost is only incurred when queries run.

Google BigQuery (launched 2010, publicly available 2012) is a serverless query engine: there are no clusters to manage. Users submit SQL queries and are billed per terabyte of data scanned. BigQuery's Dremel execution engine distributes queries across thousands of servers automatically. The trade-off is that query cost is proportional to data scanned, which incentivises partitioning and clustering tables to reduce scan scope.

Databricks (founded 2013, based on Apache Spark) unifies batch processing, streaming, and machine learning in a single platform. Databricks Runtime optimises Spark performance with proprietary improvements to the Catalyst optimiser and Delta Engine. It is the platform of choice for organisations whose workloads blend data engineering (ETL), data science (ML training), and SQL analytics in a single environment.

Data lakes (raw files in object storage, schema-on-read) offered cheap scalable storage but no ACID guarantees, no schema enforcement, and poor query performance on small files. Data warehouses offered ACID and performance but were expensive and required schema-on-write. The lakehouse pattern combines cheap open-format storage with warehouse-quality reliability by adding a transactional metadata layer over the data lake.

Delta Lake (Databricks, open-sourced 2019) adds a transaction log to Parquet files stored in object storage. Every write creates a new log entry describing the files added or removed. This provides ACID transactions (readers never see partial writes), time travel (querying the state of the table at any past timestamp), schema evolution (adding columns without rewriting history), and efficient upserts and deletes. Delta Lake is the foundation of the Databricks Lakehouse Platform.

Apache Iceberg (Netflix, Apache project since 2020) solves the same problems with a different architecture: a tree of metadata files (manifest lists, manifests, and data files) rather than a sequential log. Iceberg's hidden partitioning allows queries to scan only relevant partitions without requiring the user to specify the partition scheme in the query. Iceberg is engine-agnostic: it works with Spark, Flink, Trino, and BigQuery. It has become the open standard alternative to Delta Lake in vendor-neutral architectures.