Part 2 of this series tested a directed question — “why is COMPUTE_WH expensive?” — where there is a known right answer. This round is the harder test: an open-ended sweep with no single correct scope. Coverage is the game.

My prediction: skilled-Opus — frontier model plus five playbooks plus an anomaly-discovery checklist — would produce a superset of everything worth finding, and naive-Opus would be a noisier subset. I would spend the round grading skilled’s report against my manual notes and call it done.

The skilled sweep (trace 234345, 31 tool calls, 9.6 minutes, $0.7595) surfaced the idle-burn attribution, a single query pattern responsible for 84.9% of attributed compute on COMPUTE_WH (the v_cpu_heavy family), 145.8 GB of active storage held in duplicate table names across 11 names, the security posture (zero MFA on both users, ACCOUNTADMIN as the running role for 79.3% of all queries), and several non-default account parameters with governance implications — cross-region AI processing enabled, session-scoped temp objects disabled.

Naive-Opus finished its sweep with substantial overlap with skilled on those findings. The majority of naive’s findings, however, were things skilled never looked at. They included the two items that falsified my own notes, which I will come to in §4. They also included a queuing crisis and a compile-time anomaly that skilled skipped entirely, because neither appears in my playbooks as a first-pass check.

And naive-Opus missed the headline idle-burn number — the finding every skilled run leads with, because the playbook’s first query computes it. Without the playbook, naive did not look at QUERY_ATTRIBUTION_HISTORY. The most important number in the account was invisible to it.

Neither report dominated. Skilled and naive did not produce better and worse versions of the same analysis. They explored different regions of the same territory, because one was guided by a checklist of known places and the other by whatever the data suggested.

The naive sweep flagged a queuing problem on the stress-test warehouses. Adjudication against QUERY_HISTORY confirmed it — and the real numbers are more varied than the draft summary suggested. The adaptive warehouses (QTM2, QTM4) had 96.1% of their SELECT queries queued; QTM10 was at 94.1%. The classic stress warehouses (MC1/MC10_AUTO/MC4_AUTO) sat at 84.3% queued; MC4_PRE at 41.2%; MC10_PRE at 0%. Average queue waits ranged from 4.6 seconds to 115.6 seconds depending on warehouse.

The compile-time ratio on COMPUTE_WH is a separate data point from the same sweep: 49.9% of total elapsed time on that warehouse was compilation rather than execution, against a 5.8% account-wide baseline. Half the warehouse’s time was planning, not running. Neither of these showed up in the skilled sweep.

The two sweeps disagreed outright on cache. Skilled: average cache hit roughly 0.6%, near zero. Naive: several warehouses at 99%-plus cache. Both cannot be correct.

Adjudication took one query. The PERCENTAGE_SCANNED_FROM_CACHE column in this export is a fraction (0.0–1.0), not a percentage. A sample row returns 0.6787 — meaning 67.9% of data scanned from cache, not 0.6787%. Skilled-Opus read the column as a percentage. Naive-Opus, with no schema documentation to anchor on, sampled the values and read them correctly.

The verified numbers: average cache fraction 0.6194 (61.9%) across 2,458 queries that scanned any data; 1,366 queries had a cache fraction above 0.9 (above 90%). That is a well-cached account, not a near-zero one.

The configuration with more documentation made the units error, because partial documentation licenses assumptions. My schema notes recorded time units in milliseconds and said nothing about the cache column. Naive, which assumes nothing because it was told nothing, sampled the values and got it right.

This is now a prompt rule in Part 5: document the unit of every numeric column explicitly, or leave the column undocumented rather than risk a silent assumption. Partial documentation is not always safer than no documentation.

While adjudicating the agent disagreements, I compared the naive findings against my own manual analysis — done carefully, before any agent ran. My report had a “checks that came back clean” section. Two entries: query queuing and disk spill.

Both were wrong. The queuing crisis is real — verified at 84–96% queued on the stress warehouses, with the exact per-warehouse spread above. The disk spill is real to the decimal: 63.55 GB total local spill across the window, the single worst query responsible for 26.75 GB on its own.

I had written “clean” for those entries because I had, in honesty, never properly run the checks. I knew the shape of what to look for but had not executed the queries. The naive agent, with one sentence of context and no playbook, queried those columns and found what I had asserted was not there.

Every analyst had blind spots. Every analyst made at least one outright error. And the union of all three, after a four-query adjudication pass, beat every individual — including the human.

The instinct when two agent runs disagree is to treat the discrepancy as a reliability problem — the system is inconsistent, pick the better report, hide the gap. The data says the opposite. Every contradiction in this experiment, when adjudicated, exposed something real: a units bug, a documentation gap, a human’s unexamined assumption.

Each contradiction also came with a built-in test: run the SQL and see who is right. The adjudication is cheap. In my case, four queries settled all the disagreements. Shipping either report alone means shipping its blind spots unexamined. Running both and diffing them surfaces the errors in both — including the human’s.

The operating consequence, which Part 5 builds into a process: on open-ended work, run both configurations, diff the findings, adjudicate every conflict deterministically, and publish the union. The skilled agent finds what its playbook points at. The naive agent follows the data wherever it goes. Neither covers the full territory. Both together, with a human reading the diff, do.

SELECT
  SUM(BYTES_SPILLED_TO_LOCAL_STORAGE)  AS total_spill_bytes,
  MAX(BYTES_SPILLED_TO_LOCAL_STORAGE)  AS max_single_query_bytes,
  SUM(BYTES_SPILLED_TO_REMOTE_STORAGE) AS remote_spill_bytes
FROM SNOWFLAKE.ACCOUNT_USAGE.QUERY_HISTORY
WHERE START_TIME >= DATEADD('day', -30, CURRENT_TIMESTAMP());

On this account the query returned 63.55 GB local spill (max single query 26.75 GB) and 0.11 GB remote. Any result above a few GB on a warehouse smaller than Large is worth investigating: spill means the warehouse ran out of memory and had to write to disk, which compounds into slow queries and inflated metering on a warehouse already under pressure.

-- PERCENTAGE_SCANNED_FROM_CACHE is a fraction (0.0–1.0), not a percentage.
-- Multiply by 100 to display as percent.
SELECT
  ROUND(AVG(PERCENTAGE_SCANNED_FROM_CACHE) * 100, 1) AS avg_cache_pct,
  COUNT_IF(PERCENTAGE_SCANNED_FROM_CACHE > 0.9)       AS queries_above_90pct,
  COUNT(*)                                               AS scanning_queries
FROM SNOWFLAKE.ACCOUNT_USAGE.QUERY_HISTORY
WHERE BYTES_SCANNED > 0
  AND   START_TIME >= DATEADD('day', -30, CURRENT_TIMESTAMP());

The comment in that query is the fix for the units bug: state the scale explicitly. When this column appears in the skilled agent’s schema documentation without a unit note, it reads as a percentage because the column name says “percentage.” The naive agent sampled the values (0.67, 0.91, 0.43) and inferred the correct scale. The lesson is in Part 5’s prompt rules.