Your Databricks query got slower. The console won't say so.

The page shows you what ran, and for how long. Not what's on fire, or what's getting worse.

The Databricks Query History page is a good log. It lists recent statements, their durations, and their status, and you can sort by the slowest. What it cannot do is answer the two questions you actually ask in an incident: which query or cluster is causing pain right now, and is this specific query slower than it usually is. The first needs a fast read across two different kinds of compute; the second needs each statement compared to its own history, not to the other statements on the screen. Both are one plain SELECT over the same Unity Catalog system.* tables the rest of the audit reads.

Starting point: the audit's heaviest statement shapes, rolled up by fingerprint — it finds the shapes that cost the most cumulatively. This post adds the two that come next: is one hurting now, and is one getting worse.

A log tells you what happened. It never tells you which of those things is still happening — or which one is slower than it was yesterday.

"Which statements just went heavy?" The leading edge of query.history, not the whole log.

Sorting the whole history by duration finds yesterday's worst query, which is no help while a warehouse is slow now. Narrow the window to the last few minutes and keep the statements that already crossed a heaviness line, newest first. One fact shapes how you read the result: system.query.history records only terminated statements — its execution_status is FINISHED, FAILED or CANCELED, with no RUNNING — so these are the leading edge of the recent past: the ones to diagnose, cancel the follow-ups for, or decide on, not queries you catch mid-flight. To find and stop something running this second, the Query History UI (it has a Cancel button) and the Query History REST API carry a live RUNNING status; this table does not.

SELECT
  statement_id,
  CASE WHEN executed_by LIKE '%@%'
       THEN concat(substr(executed_by, 1, 2), '****@****')
       ELSE executed_by END                            AS executed_by,
  compute.type                                         AS compute_type,
  compute.warehouse_id                                 AS warehouse_id,
  statement_type,
  execution_status,
  start_time,
  end_time,
  execution_duration_ms,
  waiting_for_compute_duration_ms,
  spilled_local_bytes,
  substr(statement_text, 1, 160)                       AS statement_head
FROM system.query.history
WHERE start_time >= dateadd(MINUTE, -:lookback_minutes, current_timestamp())
  AND from_result_cache = false
  AND execution_status IN ('FINISHED', 'FAILED')  -- terminal states only: this table has no RUNNING
  AND execution_duration_ms >= :heavy_ms
ORDER BY end_time DESC, execution_duration_ms DESC
LIMIT :top_n;

Defaults: :lookback_minutes 15 · :heavy_ms 60000 (one minute) · :top_n 100.

Read it like this: newest heavy statements first. A non-zero spilled_local_bytes is a statement that spilled to disk — memory pressure worth chasing — and a large waiting_for_compute_duration_ms means it spent its time queued for a warehouse, not computing. That distinction changes the fix: tune the query, or add a warehouse. A run of FAILED rows in the same window is its own signal — something started breaking a few minutes ago.

Four things this lens can't see:

• system.query.history publishes no ingestion-latency figure — and the one delay the docs do quantify, up to 24 hours, is the unrelated wait for customer-managed-key encrypted fields, not row ingestion. In practice, completed rows trail live execution by an unspecified, minutes-order lag, and the table records only terminated statements. So read it as the leading edge of the recent past, not a live console — to catch a statement mid-run, use the Query History UI or the Query History REST API (filter_by.statuses = RUNNING or QUEUED), which this table has no equivalent of.
• It only sees SQL-warehouse and serverless statements. Work on classic all-purpose or job clusters is never recorded in query.history at all — that is exactly the blind spot the node lens in the next section covers.
• executed_by is masked here (email to first-two-plus-stars), matching the audit library's identity-masking default; drop the CASE if you own the workspace and want the full name.
• from_result_cache = false keeps cache hits out — a query answered from cache is fast and harmless, and would only pad the list.

"Which cluster is saturated this minute?" The same node table the audit uses to find idle boxes — read the other way.

The audit's node utilization profile ranks clusters by average CPU ascending, to find the oversized boxes running at 8%. Flip the sort and the threshold, shorten the window to now, and the same system.compute.node_timeline table answers the opposite question: which classic cluster is saturated this minute. The tell is not only CPU — a high cpu_wait_percent means the cores are stalled waiting on IO or locks, which is contention you feel as slowness even when CPU looks unremarkable.

SELECT
  cluster_id,
  node_type,
  driver,
  COUNT(*)                                             AS slices,
  ROUND(AVG(cpu_user_percent + cpu_system_percent), 1) AS avg_cpu_pct,
  ROUND(MAX(cpu_user_percent + cpu_system_percent), 1) AS peak_cpu_pct,
  ROUND(AVG(mem_used_percent), 1)                      AS avg_mem_pct,
  ROUND(AVG(cpu_wait_percent), 1)                      AS avg_cpu_wait_pct
FROM system.compute.node_timeline
WHERE start_time >= dateadd(MINUTE, -:lookback_minutes, current_timestamp())
GROUP BY cluster_id, node_type, driver
HAVING AVG(cpu_user_percent + cpu_system_percent) >= :hot_cpu_pct
    OR AVG(mem_used_percent)                       >= :hot_mem_pct
    OR AVG(cpu_wait_percent)                        >= :hot_wait_pct
ORDER BY avg_cpu_pct DESC, avg_mem_pct DESC
LIMIT :top_n;

Defaults: :lookback_minutes 15 · :hot_cpu_pct 85 · :hot_mem_pct 85 · :hot_wait_pct 20 · :top_n 100.

Read it like this: one row per cluster, node type and driver/worker role, worst CPU first. A high avg_cpu_pct is a busy box; a high avg_cpu_wait_pct with unremarkable CPU is a box waiting on something — IO, shuffle, a lock — which is the more interesting finding. Cross-reference the driver row: a saturated driver with idle workers is a plan problem, not a sizing one.

Three caveats on the node lens:

• A node that has run less than roughly ten minutes may not have emitted a slice yet, so a very fresh cluster can read as absent rather than quiet.
• mem_used_percent includes background processes, so it is not pure workload memory — treat it as a pressure signal, not an accounting figure.
• A high peak_cpu_pct over a low average is a burst, not sustained load; the short window already leans toward "now," but glance at peak against average before you act.

"Is this statement running slower than it used to?" Thirty days of one shape, each day against its own trailing-7-day baseline.

This is the question the console structurally can't answer, because it compares each statement to the others on the screen, never to itself. Pick one statement shape — its statement_fingerprint, the sha2 of its literal-stripped text, straight from the grouped costly-statements query — pull every run of it over the last month, roll up per day, and compare each day to the average of the seven days before it. Anything more than :regress_pct above that baseline is a regression; anything below is an improvement; the rest is steady.

WITH runs AS (
  SELECT
    date(start_time)       AS day,
    execution_duration_ms
  FROM system.query.history
  WHERE start_time >= dateadd(DAY, -:period_days, current_date())
    AND execution_status = 'FINISHED'
    AND from_result_cache = false
    AND execution_duration_ms > 0
    -- same de-value recipe as the grouped query, so the fingerprint matches:
    AND sha2(
          regexp_replace(
            regexp_replace(statement_text,
              '[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+[.][A-Za-z]{2,}', '<email>'),
            concat(chr(39), '[^', chr(39), ']*', chr(39)), '?'
          ), 256
        ) = :fingerprint
),
daily AS (
  SELECT day,
         COUNT(*)                                             AS runs,
         CAST(AVG(execution_duration_ms) AS BIGINT)           AS avg_ms,
         CAST(percentile_approx(execution_duration_ms, 0.95) AS BIGINT) AS p95_ms
  FROM runs
  GROUP BY day
),
based AS (
  SELECT day, runs, avg_ms, p95_ms,
         CAST(AVG(avg_ms) OVER (
           ORDER BY day
           RANGE BETWEEN INTERVAL 7 DAYS PRECEDING AND INTERVAL 1 DAY PRECEDING
         ) AS BIGINT)                                         AS baseline_7d_ms,
         COUNT(*) OVER (
           ORDER BY day
           RANGE BETWEEN INTERVAL 7 DAYS PRECEDING AND INTERVAL 1 DAY PRECEDING
         )                                                    AS baseline_days
  FROM daily
)
SELECT day, runs, avg_ms, p95_ms, baseline_7d_ms, baseline_days,
       ROUND(100.0 * (avg_ms - baseline_7d_ms) / baseline_7d_ms, 1) AS pct_vs_7d,
       CASE
         WHEN baseline_7d_ms IS NULL OR baseline_days < :min_baseline_days THEN 'NO_BASELINE'
         WHEN avg_ms > baseline_7d_ms * (1 + :regress_pct / 100.0)        THEN 'REGRESSED'
         WHEN avg_ms < baseline_7d_ms * (1 - :regress_pct / 100.0)        THEN 'IMPROVED'
         ELSE 'STEADY'
       END                                                              AS verdict
FROM based
ORDER BY day;

Defaults: :period_days 30 · :regress_pct 10 · :min_baseline_days 3 · :fingerprint from the grouped query.

Read it like this: one row per day. pct_vs_7d is how far that day sits above or below the trailing week — a run of REGRESSED rows with a rising pct_vs_7d is a genuine slowdown, and if p95_ms moves before avg_ms does, the tail is degrading first, which usually means data growth or a plan flip rather than a change everyone feels. The baseline is the seven calendar days before each day, excluding the day itself, and baseline_days tells you how many of those actually had runs — anything under :min_baseline_days is honestly reported as NO_BASELINE rather than a confident zero.

The receipt. Two tables, three SELECTs, and "it feels slow" becomes a row you can point at.

None of this needed a monitoring product. It needed system.query.history for the statement lens and the regression series, system.compute.node_timeline for the node lens, and one window function to give each day its own trailing baseline — the same two tables the audit's Performance and Compute domains already read for cumulative cost and right-sizing. Run the two live-load queries when something feels wrong now; run the regression query when someone says a report "used to be faster," and let the ±:regress_pct band decide whether they're right.

These two sit next to the hundred that came before them. The full library — a hundred plain SELECTs across seven domains — is open on GitHub, and the worked reference is the Databricks audit chapter on Crosshire Learn. Two companions from the same pass: jobs that fail on the expensive clock and what an anomaly alert's system row actually says.

Every query reads Databricks Unity Catalog system.* tables via plain SELECT; figures are durations and utilization, directional, and read as answer shapes rather than billed numbers. Empty is treated as not assessed. Thresholds are field heuristics — tune them to your account. — Crosshire