In this post, Andreas Paech, Head of Data Engineering, argues for a pragmatic stance: SQL-first is not nostalgia. It is the lowest-friction way to share and version the executable definitions behind metrics and pipelines, so analysts, engineers, and power users can review intent, detect semantic drift, and keep velocity and trust intact.
SQL as the Lowest-Friction Interface for Shared Meaning
In most data organisations, “speed” problems are often interpretation problems: teams hesitate because they do not trust what a metric means, whether a change altered semantics, or whether today is comparable to yesterday. SQL’s advantage is that it is declarative: semantic meaning is readable, diffable, and testable across functions.
In practice, the query is the contract: it encodes join rules, filter boundaries, and null-handling. When treated as a versioned, code-reviewed artefact, SQL reduces context decay by keeping logic close to the number. SQL-first is therefore not only an engineering preference; it is a data governance primitive that makes trust reproducible.
Why the Cloud Makes Query Discipline More, Not Less, Important
Cloud elasticity changes failure modes. In on-premise systems, an inefficient query often fails loudly because capacity is fixed. In the cloud, inefficient queries often “succeed,” but at the cost of slow feedback loops and expensive runs. That is a more dangerous outcome: the system appears functional while quietly taxing the organisation with longer cycles, higher DBU/compute consumption, and slower time-to-insight.
Managed features also create a psychological trap. It is tempting to assume that auto-scaling and serverless compute compensate for suboptimal SQL. They do not. They can mask symptoms and inflate cost. Query planners cannot infer business semantics, and distributed engines cannot parallelise workloads effectively if the underlying logic produces unnecessary shuffles, Cartesian blow-ups, or oversized partitions. The engine can only optimise what you actually express. When the expression is sloppy, the bill and the latency will eventually make that visible.
A Real Incident: When a “Stable” ETL Job Turns Into a Six-Hour Outage
Treat query regressions as reliability incidents. In one production pipeline, an ETL job that normally ran 40-50 minutes spiked to 6.91 hours and 6.11 hours on consecutive days while processing a comparable volume (~20.2 million records). This was not a demand surge; it was a failure in disguise.
Instead of blaming transient infrastructure, we inspected the query execution plan. The transformation’s nested subqueries and join logic amplified skew under a specific file distribution, creating redundant scans and expensive shuffles.
The fix was boring and decisive: rewrite into explicit Common Table Expressions (CTEs), tighten join predicates, and harden null-handling; then align the table layout with the access pattern (partitioning). Runtime dropped to 2 minutes and 21 seconds, including a quick clustering step including statistics update. The point is not the rewrite; it is that a SQL-first posture turns “cloud slowness” into deterministic levers.
Data Layout Is Part of the Query
Query optimisation and data layout optimisation are two halves of the same performance problem. Clean SQL can still run poorly if files, partitions, or clustering prevent parallelism or amplify shuffles; a clean layout cannot rescue fundamentally inefficient logic.
A simple example is file distribution: a handful of oversized files caps parallelism; thousands of tiny files shift the bottleneck to metadata and scheduling. Practical layout literacy therefore includes file size distribution, locality (partitioning/clustering aligned to dominant filters and join keys), and statistics refresh when the optimiser depends on them.
A Pragmatic Process Model: Optimise Logic, Then Layout, Finally Automate
