Airbyte - Dagster - LangChain Proof of Concept 2025

Requirements

• Python 3.10 or higher (successfully tested with Python 3.12.3)
• Python venv module for creating virtual environments
• Python pip for installing Python dependencies
• Docker (might be installed with abctl, see below)
• kubectl for managing Kubernetes
• kind
• Airbyte CLI (abctl)
• OpenAI API Key

Quickstart

This will set up everything locally.

Airbyte Installation

Install Airbyte and disable the login page. Refer to the Airbyte documentation.

abctl local install --values ./airbyte-values.yaml

Configure Airbyte Connection

Open Airbyte Web UI: http://localhost:8000.

Create a new connection:

• Source: Sample Data (Faker)
• Destination: Posgresl with:

  Host:               airbyte-db-svc
  Port:               5432
  DB Name:            postgres
  Default Schema:     public
  User:               airbyte
  SSL modes:          disable
  SSH Tunnel Method:  No Tunnel
  Password:           airbyte
  

Build & Start Dagster

Forward database port:

kubectl --kubeconfig $HOME/.airbyte/abctl/abctl.kubeconfig port-forward --namespace airbyte-abctl airbyte-db-0 5432:5432

Set up the environment:

python3 -m venv .venv
source .venv/bin/activate
pip3 install -r requirements.txt
export OPENAI_API_KEY=YOUR_OPENAI_API_KEY
dagster dev -f ingest.py

Run and Materialize

Visit http://127.0.0.1:3000/asset-groups and materialize all assets.

Query

python3 query.py

Ask a question, e.g., "What do we sell?".