Getting Started

Install DataEngineX and run your first pipeline.

Installation

pipinstalldataenginex

Create a config file

Create dex.yaml:

project:name:my-first-pipelinedata:sources:raw_users:type:csvpath:data/users.csvpipelines:clean_users:source:raw_usersdestination:silver.users

Validate and run

dexvalidatedex.yaml

fromdataenginex.engineimportDexEngineengine=DexEngine("dex.yaml")result=engine.run_pipeline("clean_users")print(result.rows_written)

Optional extras

pipinstall"dataenginex[cloud]"# S3, GCS, BigQuery pipinstall"dataenginex[postgres]"# PostgreSQL lineage pipinstall"dataenginex[qdrant]"# Qdrant vector store pipinstall"dataenginex[ml]"# scikit-learn + XGBoost pipinstall"dataenginex[kafka]"# Kafka connector pipinstall"dataenginex[rabbitmq]"# RabbitMQ backend