Using ingestr with CrateDB: `ValueError: Unsupported destination scheme: cratedb`

[amotl]

Problem

@hlcianfagna reported:

Regarding using CrateDB with ingestr, when I try ingestr ingest with a --dest-uri using crate:// or cratedb://, I receive the error Unsupported destination scheme.

Question

Is this supposed to work or it can only be invoked using ctk load table?

[amotl]

We are able to confirm the observation.

docker run --rm -it --name=cratedb \
  --publish=4200:4200 --publish=5432:5432 \
  --env=CRATE_HEAP_SIZE=2g crate/crate:nightly \
  -Cdiscovery.type=single-node

uvx ingestr ingest --yes \
  --source-uri 'frankfurter://?base=EUR' \
  --interval-start '2025-03-20' \
  --interval-end '2025-03-28' \
  --source-table 'exchange_rates' \
  --dest-uri 'cratedb://crate:na@localhost:5432' \
  --dest-table 'my_schema.exchange_rates'

ValueError: Unsupported destination scheme: cratedb

uvx crash -c "SELECT * from my_schema.exchange_rates"

SchemaUnknownException[Schema 'my_schema' unknown]

[amotl]

The invocation works well when using uvx --prerelease=allow ingestr@latest ingest ... when it is using the minimum required version of ingestr v0.13.61 that supports CrateDB as a destination database. Installation instructions for uv/uvx are here.

We've observed you will need to allow installing pre-release packages to permit installation of sqlalchemy-cratedb==0.42.0.dev2 until we've wrapped up the ingredients of crate/sqlalchemy-cratedb#215.

Those samples clearly demonstrate that it is easy to pick a version that is too old, even when using ingestr@latest.

uvx "ingestr@latest" version
v0.13.52

uvx --prerelease=allow "ingestr@latest" version
v0.13.63

ingestr versions v0.13.60 and earlier will present the error message Unsupported destination scheme: cratedb.

[amotl]

We've successfully exercised a vanilla pip install ingestr installation in both Python 3.11 and 3.12 virtualenv environments now, which takes ages to complete, but works out of the box:

pip install 'ingestr>=0.13.61'

ingestr version
v0.13.63

[amotl]

This invocation sample also clearly demonstrates we need to get our SQLAlchemy dialect GA-released so that recent versions of ingestr can be installed with uv/uvx without needing to add the --prerelease=allow option.

$ uvx --python 3.13 'ingestr>=0.13.61' version
  × No solution found when resolving tool dependencies:
  ╰─▶ Because there is no version of sqlalchemy-cratedb==0.42.0.dev2 and ingestr>=0.13.61 depends on sqlalchemy-cratedb==0.42.0.dev2, we can conclude that
      ingestr>=0.13.61 cannot be used.
      And because only the following versions of ingestr are available:
          ingestr<=0.13.61
          ingestr==0.13.62
          ingestr==0.13.63
      and you require ingestr>=0.13.61, we can conclude that your requirements are unsatisfiable.

      hint: `sqlalchemy-cratedb` was requested with a pre-release marker (e.g., sqlalchemy-cratedb==0.42.0.dev2), but pre-releases weren't enabled (try:
      `--prerelease=allow`)

uv selects an older version of ingestr by default even when requesting latest. It is the last version of the package available which does not include the dependency to sqlalchemy-cratedb==0.42.0.dev2.

$ uvx --python 3.13 'ingestr@latest' version
v0.13.52

[amotl]

A CI run also displays the problem demonstrated above.

[amotl]

We've added the --prerelease=allow option to permit installation,

• Dependencies and documentation crate/cratedb-toolkit#478

and additionally submitted another patch to improve dependencies of ingestr.

• Dependencies: Downgrade to sqlalchemy-cratedb==0.41.0 GA bruin-data/ingestr#277