Technology / Data governance / Financial controls

Databricks: A simpler approach to data lineage

By John Biersmith · October 25, 2026 · 5 min read

How data lineage can connect finance and technology—and why a lineage graph alone cannot establish financial-reporting reliability.

When a number reaches a financial report, its journey is often more complicated than the report suggests. It may have passed through source applications, ingestion jobs, transformation logic, shared tables, and a spreadsheet before a reviewer sees it.

My interest in data lineage starts with a practical leadership question: can finance and technology explain that journey together? A diagram has value when it helps people understand dependencies, assess changes, and investigate unexpected results.

Databricks can make that conversation easier. The more interesting opportunity is how an organization uses that visibility to make decisions about financial-reporting risk.

What lineage helps us see

Databricks Unity Catalog captures runtime lineage for supported queries, including table relationships and, where supported, column mappings. Catalog Explorer presents upstream and downstream relationships. Capture and visibility depend on the workload, registration, configuration, and permissions.^[1]

For a finance leader, this can provide a useful starting point for questions such as: which reports depend on this table? Which transformation feeds the amount under review? Who should be involved before the source changes?

Those questions turn a technical map into a shared working view. Finance can explain the significance of a balance. Engineering can explain the processing path. Control owners can identify where an error could enter and what would detect it.

Trace a financial-reporting use case

Consider a hypothetical monthly revenue reconciliation. Billing activity enters a data platform, transformations prepare it for analysis, and a report compares the output with the general ledger. The review is intended to identify differences requiring investigation.

Start with the reconciliation’s purpose and trace backward from the report. Which columns drive the comparison? Which tables supply those columns? Which transformations alter transaction status, product classification, currency, or reporting period?

Then trace forward from a critical source. If a field changes, which financial outputs could be affected? That second direction can reveal dependencies that a report-by-report inventory overlooks.

Neither direction replaces a walkthrough with the people operating the process. The map helps focus the conversation; the team still needs to understand accounting rules, interfaces, manual steps, and the timing of processing.

Separate visibility from reliability

A lineage relationship shows a dependency. It does not establish that the source population is complete, the accounting treatment is appropriate, or the transformation is correct.

For the hypothetical reconciliation, I would want to understand how the process handles missing files, duplicate transactions, rejected records, late arrivals, and reruns. I would also ask whether joins can multiply records, whether filters remove valid transactions, and whether manual adjustments are governed.

These are different questions from “does the graph show a connection?” They require evidence suited to the risk: for example, reconciliations, rule validation, exception handling, and controlled changes. The right mix depends on the process and how the information is used.

This distinction matters for SOX. A tool can help identify relevant dependencies, but assigning control reliance requires evaluating the control’s design, operation, and supporting information. I would use lineage to inform that assessment rather than treat the feature as a SOX certification.

Understand the map’s boundaries

Databricks documents capture limitations, including unsupported workloads and cases where column mappings are unavailable. Permissions also affect what a reviewer can explore. A missing connection should trigger investigation; it does not prove that a dependency is absent.^[1]

Teams should identify known gaps explicitly. A journey may begin in an upstream application outside the captured environment or end with a manual export. If the most important financial adjustment happens after the data leaves the platform, the review needs to cover that step too.

For each critical report, pair the technical lineage with a short business explanation: what the report is for, which financial-reporting risk it supports, who owns it, and where the mapped coverage ends. That creates a more useful record than a screenshot alone.

Connect lineage to change management

One leadership use case I find compelling is evaluating the downstream effect of a proposed change. A transformation can be technically valid and still alter the meaning of a financial report.

Before changing a shared table or calculation, I would want the team to identify affected financial outputs, involve the appropriate owners, define the expected result, and validate the change against that expectation. The scope of review should reflect the risk rather than every change receiving an identical checklist.

Lineage can also help structure recurring monitoring. Databricks provides table and column lineage system tables for programmatic analysis; its documentation notes that these contain only events for which lineage can be inferred and retain a rolling one-year window.^[2]

I would distinguish those platform capabilities from any monitoring process built around them. If a team uses lineage changes to generate alerts, it should establish the expected coverage, baseline, review responsibility, and escalation path. An alert becomes useful when someone can decide what it means and what action to take.

Design evidence around the question

For a financial control, I would decide in advance what evidence needs to be reproducible: the reporting period, relevant assets, transformation version, observed dependencies, and the reviewer’s conclusion. Retention should follow the organization’s obligations and policy, with the platform’s available history understood before reliance is placed on it.

I would also keep a distinction between documenting a data path and proving that the related control operated. Both may matter, but they answer different questions.

The executive value is shared understanding

Start with one important financial report and establish a repeatable approach before expanding across the enterprise. Bring its finance owner, data engineer, application owner, and controls partner together. Trace the path, challenge the gaps, and decide which improvements would materially strengthen confidence.

The result should be a clearer view of the process and its ownership. Over time, that can support more focused reviews, better change decisions, and less dependence on individual memory.

For me, that is the opportunity in lineage: helping leaders connect technology architecture with financial accountability. The graph opens the conversation. The quality of the decisions that follow gives it business value.