Airbyte Integration Test Suite

.env.airbyte_integration_test

@@ -0,0 +1,49 @@
+# Airbyte Integration Test Configuration
+
+# Set to true to enable Airbyte integration tests
+# Uncomment and set to true to explicitly enable tests
+# AIRBYTE_TEST_ENABLED=true
+
+# Comma-separated list of destination types to test (use lowercase values)
+# Options: postgres, local-csv, s3
+AIRBYTE_TEST_DESTINATIONS_EXAMPLE=postgres
+
+# Comma-separated list of source types to test (use lowercase values)
+# Options: file, postgres, google-sheets, kobo-toolbox
+AIRBYTE_TEST_SOURCES_EXAMPLE=file
+
+# Timeout settings (in seconds)
+AIRBYTE_TEST_TIMEOUT_EXAMPLE=300
+AIRBYTE_SYNC_WAIT_INTERVAL_EXAMPLE=10
+AIRBYTE_MAX_WAIT_CYCLES_EXAMPLE=30
+
+# PostgreSQL source configuration
+AIRBYTE_TEST_PG_HOST_EXAMPLE=localhost
+AIRBYTE_TEST_PG_PORT_EXAMPLE=5432
+AIRBYTE_TEST_PG_DB_EXAMPLE=test
+AIRBYTE_TEST_PG_USER_EXAMPLE=postgres
+AIRBYTE_TEST_PG_PASSWORD_EXAMPLE=changeme
+
+# PostgreSQL destination configuration
+AIRBYTE_TEST_PG_DEST_HOST_EXAMPLE=localhost
+AIRBYTE_TEST_PG_DEST_PORT_EXAMPLE=5432
+AIRBYTE_TEST_PG_DEST_DB_EXAMPLE=test
+AIRBYTE_TEST_PG_DEST_USER_EXAMPLE=postgres
+AIRBYTE_TEST_PG_DEST_PASSWORD_EXAMPLE=changeme
+
+# Google Sheets configuration
+# JSON credentials with access to the spreadsheet
+AIRBYTE_TEST_GSHEETS_CREDS_EXAMPLE={"type":"service_account","project_id":"your-project-id"}
+# Spreadsheet ID from the URL
+AIRBYTE_TEST_GSHEETS_ID_EXAMPLE=your-spreadsheet-id
+
+# KoboToolbox configuration
+AIRBYTE_TEST_KOBO_HOST_EXAMPLE=https://kobo.humanitarianresponse.info
+AIRBYTE_TEST_KOBO_TOKEN_EXAMPLE=your-token-here
+
+# S3 destination configuration
+AIRBYTE_TEST_S3_BUCKET_EXAMPLE=test-bucket
+AIRBYTE_TEST_S3_PATH_EXAMPLE=test-path
+AIRBYTE_TEST_S3_REGION_EXAMPLE=us-east-1
+AIRBYTE_TEST_S3_ACCESS_KEY_EXAMPLE=your-access-key
+AIRBYTE_TEST_S3_SECRET_KEY_EXAMPLE=your-secret-key 

README.md

@@ -231,3 +231,43 @@ If using M1-based MacBook  run this before building image
 
 -  `docker compose -p dalgo_backend -f Docker/docker-compose.yml --env-file Docker/.env.docker up -d`
 -  `docker compose -p dalgo_backend -f Docker/docker-compose.yml --env-file Docker/.env.docker down`
+
+## Running Airbyte Integration Tests
+
+The project includes a comprehensive integration test suite for Airbyte that tests end-to-end operations including creating workspaces, sources, destinations, connections, and running syncs.
+
+### Prerequisites
+
+1. A running Airbyte server (as described in Setup Step 4)
+2. Configure the test settings by creating a `.env.airbyte_integration_test` file or adding the settings to your existing `.env` file.
+
+### Configuration
+
+The Airbyte integration tests are configured through environment variables. A template file `.env.airbyte_integration_test` is provided with all the available options.
+
+Key settings:
+- `AIRBYTE_TEST_ENABLED=true` - Set to true to enable tests (they are skipped by default)
+- `AIRBYTE_TEST_DESTINATIONS` - Comma-separated list of destination types to test (must be lowercase values)
+- `AIRBYTE_TEST_SOURCES` - Comma-separated list of source types to test (must be lowercase values)
+
+### Running the Tests
+
+To run only the Airbyte integration tests:
+
+```sh
+python -m pytest ddpui/tests/integration_tests/test_airbyte_full_integration.py -v
+```
+
+To run a specific test:
+
+```sh
+python -m pytest ddpui/tests/integration_tests/test_airbyte_full_integration.py::TestAirbyteFullIntegration::test_01_create_destinations -v
+```
+
+### Notes
+
+- These tests create real Airbyte resources (workspaces, sources, destinations, connections) and clean them up afterward.
+- Test execution might take several minutes, especially when running syncs.
+- The tests require proper Airbyte server access as configured in your `.env` file.
+- For production environments, consider keeping these tests disabled to avoid unintended resource creation.
+- **WARNING: DO NOT run these tests against production workspaces!** The tests perform destructive operations such as setting "overwrite" sync modes and deleting workspaces and connections.