How Profiling Works¶

One Pass Over the Data¶

Whatever an inspection ends up reporting, it reads your data once:

1. Build a snapshot     what rows are in scope for this date
2. Materialise          optionally, into a work table
3. Measure              row counts, then the enabled statistics
4. Store                one value per dataset, column, and statistic

The modules then work from what step 3 produced. Data Anomalies and Data Analytics read the statistics; Data Validation, when it is enabled, evaluates its own rules against the same snapshot in the same pass, so it too needs no separate trip to the source.

Step 1 – The Snapshot¶

The snapshot is the set of rows the inspection considers. How it is built depends on what the data source is:

For tables and views, an optional snapshot filter narrows it further. This is the place to exclude rows that should never count toward any measurement — test records, soft-deleted rows, a partition that is not yet complete.

Snapshot query tags¶

A snapshot query is a template, not a fixed statement. Tags let one query serve every inspection date and stay portable between environments:

select * from #db_name#.#schema_name#.orders
where load_date = '#date#'

The date offsets are what let a source be described whose data lands a day late, or whose window spans a boundary:

select * from #db_name#.#schema_name#.events
where ts >= date('#date#') and ts < date('#date+1#')

An offset that cannot be resolved is reported as an error rather than quietly falling back to the unshifted date — a wrong window would silently measure the wrong rows.

This is what makes inspections per-date and re-runnable. Each date is measured on its own, a range can be backfilled, and re-inspecting a date replaces that date's results rather than duplicating them.

Step 2 – Work Tables¶

A snapshot may be read many times in one inspection — once per statistic, or once per dataset. Whether it is materialised first is a property of the database connection:

Session is the mode to prefer where it is available: the snapshot is evaluated once instead of being re-evaluated by every statistic, and nothing is left behind afterwards. Permanent achieves the same thing on technologies that do not offer a usable temporary table, at the cost of creating and dropping a real object.

The work table is dropped even when profiling fails, so a failed inspection does not leave an object behind that blocks the next one.

Step 3 – Measuring¶

Each dataset's filter and grouping expressions are applied, and every enabled statistic becomes an aggregate over the result:

SELECT
    <grouping expression>,
    <statistic>
FROM
    (<snapshot>)
WHERE
    <filter expression>
GROUP BY
    <grouping expression>

Query mode¶

How those aggregates are packed into SQL is set per data source:

Combined reduces the number of queries and the round trips they cost, and is the better fit when the source can comfortably hold the combined query in memory. Single issues more queries, each with a smaller memory footprint, which is what large data sources need when a combined query risks running into out-of-memory or spool limits.

Query mode changes only how the work is packaged. The statistics it produces are identical either way, so switching modes on an existing data source does not create a discontinuity in its history.

Row Counts¶

Row counting is the one measurement that is not conditional on any module.

Every inspection counts the rows in each dataset and in the data source as a whole, and stores those counts as statuses at dataset and data source level. This happens with no module enabled, because a data source that delivered no rows is worth knowing about regardless of what else you have configured — and because it is what Timeliness keys off to decide whether a delivery happened at all.

These counts have their own switch in a subscription, so they can be routed or silenced separately from anomaly findings.

Report Empty Datasets¶

A data source carries a Report Empty Datasets option. With it enabled, digna raises an alert when the snapshot query returns no rows at all for the inspected date — the signal of a missing extraction or a failed upstream job, which is otherwise easy to mistake for a quiet day.

Step 4 – Storage and Re-Running¶

Each measured value is stored against the dataset, column, statistic, and date it belongs to. Nothing is aggregated away, so any statistic can be read back later as a full time series.

Before writing, an inspection removes its own previous results for the dates in scope. That is what makes a data source safe to re-inspect: running the same date twice leaves one set of results, not two. Delivery history for Timeliness is deliberately exempt — re-inspecting an old date must not erase the record of what was delivered that day.

Getting Profiling Right¶

1. Categorise columns deliberately. The attribute category decides which statistics a column gets. The default set is a reasonable start, not an answer for every column.
2. Add the statistic that would move. Work backwards from the failure you want to catch — a swapped column moves string lengths, a truncated load moves row counts, a broken lookup moves the distribution.
3. Turn off what nobody acts on. A statistic producing findings nobody investigates makes the whole data source look unhealthy and trains people to ignore the colour.
4. Use the snapshot filter for rows that should never count, and dataset filters for subsets you want measured separately. They are different tools: the first removes rows from all measurement, the second creates a measurement of its own.

Related Pages¶

• Profiling – The Foundation
• Statistics
• Datasets
• Data Anomalies – How It Works
• Statuses and Alerts