Flat File

The Flat File integration is used to upload a single flat file (like csv, parquet, and feather) into the data lineage graph.

Web App

Fields

Field	Value	Example
source	The name of the source, see sources	my-source
Namespace	Namespace for the connection, see namespaces	default

Python Library

The dbt integration can be run as a standalone python library to extract data lineage from flat files like csv and parquet.

The library is available via pip

pip install grai_source_flat_file

More information about the API is available here.

Example

The library is split into a few distinct functions but if you only wish to extract nodes/edges you can do so as follows:

  from grai_source_flat_file import FlatFileIntegration
  from grai_schemas.v1.source import SourceV1
 
  source = SourceV1(name="my-source", type="my-type")
 
  integration = FlatFileIntegration(source=source, namespace="dbt", file_name="/path/to/file.csv")
 
  nodes, edges = integration.get_nodes_and_edges()

Miscellaneous OpenLineage

Did this doc help you?