Recovering data lineage from code
A method for turning executable programs into an explainable map of data transformations.
The problem extends beyond tables
A view limited to reads and writes misses the transformations between them. Expressions, conditional branches, function calls and intermediate structures all participate in the path taken by data.
Build an analysable representation
The first step is to parse the language and turn its syntax into elements with explicit roles. This representation must retain details useful to lineage without becoming a complete copy of the source language.
Connect definitions and uses
Analysis follows variables, fields, parameters and results across scopes and calls. A graph can then connect sources, operations and destinations while retaining the provenance of each relationship.
Make the result verifiable
Useful lineage should explain why a relationship exists. Source locations, traversed operations and known analysis limits make the result reviewable by an engineer.