Skip to content

Quickstart

1. Create a config file

Create dex.yaml:

project:
  name: my-first-pipeline
  version: "0.1.0"

data:
  sources:
    raw_users:
      type: csv
      path: data/users.csv
  pipelines:
    clean_users:
      source: raw_users
      transforms:
        - type: filter
          condition: "age > 0"
      destination: silver.users

2. Validate the config

dex validate dex.yaml

3. Use DexEngine in your application

from dataenginex.engine import DexEngine

engine = DexEngine("dex.yaml")

# Run a pipeline
result = engine.run_pipeline("clean_users")

# Query sources
schema = engine.source_schema("raw_users")
sample = engine.source_sample("raw_users", limit=5)

# Inspect warehouse
layers = engine.warehouse_layers()
tables = engine.warehouse_tables("silver")

# Check run history
runs = engine.store.list_pipeline_runs(limit=10)

4. Build an HTTP layer (optional)

dataenginex does not bundle a server. If you need HTTP endpoints, create a FastAPI app and call DexEngine directly:

pip install fastapi uvicorn

from fastapi import FastAPI
from dataenginex.engine import DexEngine
import uvicorn

engine = DexEngine("dex.yaml")
app = FastAPI()

@app.get("/health")
def health():
    return engine.health()

@app.post("/pipelines/{name}/run")
def run_pipeline(name: str):
    return engine.run_pipeline(name)

if __name__ == "__main__":
    uvicorn.run(app, host="0.0.0.0", port=17000)

See examples/02_api_quickstart.py for a complete working example.

Next Steps