Why do dashboards show wrong numbers?

The most common causes are upstream: events that stopped firing after a release, duplicate rows from replayed pipelines, timezone mismatches between systems, silent schema changes, and two teams using different definitions of the same metric. The dashboard is usually the messenger, not the culprit.

How often should data quality checks run?

Volume, freshness, and schema checks should run on every pipeline execution — typically daily or hourly — with alerts to a channel someone actually reads. Deeper reconciliation against source systems (billing, CRM) is usually weekly or monthly. Checks that never fire alerts are as bad as no checks: tune thresholds until alerts are rare and real.

The 12-point data quality checklist to run before you trust a dashboard

Every team has lived this: a dashboard says revenue dipped, three people investigate, and the answer turns out to be a tracking bug from last Tuesday's release. Trust, once lost, takes months to rebuild. The fix isn't a prettier dashboard — it's a checklist the pipeline has to pass before numbers reach a decision-maker.

Collection checks

1. Event loss: compare client-side event counts against server-side or backend truth (orders in the warehouse vs. orders in the database). Ad blockers and consent commonly cost 10–30% — know your number.
2. Duplicates: count events per unique ID; retries and pipeline replays create duplicates that quietly inflate everything.
3. Schema drift: validate events against the tracking plan; a renamed property is invisible until a chart flatlines.
4. Release regression: after every deploy, check the volume of each core event; most tracking is broken by releases, not by tools.

Pipeline checks

5. Freshness: every table has an expected update time; alert when it's missed, before someone screenshots stale data.
6. Volume anomalies: row counts within an expected band per run — a 0-row day and a 3x day are both bugs.
7. Referential integrity: no orphaned keys (events for users that don't exist, orders without customers).
8. Timezone and currency normalization: one convention, enforced at the boundary, documented once.

Definition checks

9. One metric, one definition: "active user," "session," and "conversion" defined in the transformation layer, not per-dashboard.
10. Reconciliation to money: revenue metrics reconciled against billing exports on a schedule — finance's number wins.
11. Segment sanity: the parts must sum to the whole; if channel revenue doesn't add up to total revenue, attribution logic is leaking.
12. Survivorship of history: yesterday's numbers shouldn't change when pipelines rerun; if they must, the dashboard should say so.

Make the checks visible

Tools like dbt tests, Great Expectations, or plain SQL assertions can implement all twelve. The part teams skip is the social layer: publish check status next to the dashboard itself. A small "data verified as of 06:00 UTC" badge does more for trust than any redesign — and when a check fails, the dashboard should say that too, loudly, before an executive finds out the hard way.

Run the checklist once and you'll find at least two surprises. Automate it and dashboards stop being arguments — which is the entire point of building them well.