DataEngineX¶

The complete platform for Data + ML + AI engineering. Config-driven, self-hosted, production-ready.

Define your entire data pipeline, ML lifecycle, and AI agents in a single dex.yaml config file.

Quick Links¶

Getting Started — Install and run in 5 minutes
Framework — Core architecture and patterns
API Reference — Python API autodoc
Studio — DEX Studio web UI
Deploy — Kubernetes deployment guide
Contributing — How to contribute

Ecosystem¶

Component	Description
dataenginex	Core framework — config, CLI, ML registry, LLM routing, AI agents, DuckDB lakehouse
dex-studio	B2B web UI — pipelines, ML experiments, AI playground, SQL console (FastAPI/Jinja2)
careerdex	B2C career AI — job matching, resume analysis, interview prep
infradex	K3s / Helm / Terraform — Authentik, Langfuse, Qdrant, Prometheus, Grafana

Install¶

pip install dataenginex

With extras:

pip install "dataenginex[cloud]"          # S3 · GCS · BigQuery connectors
pip install "dataenginex[observability]"  # Langfuse LLM tracing
pip install 'litellm>=1.83.3' --no-deps   # 100+ LLM providers (separate install)