Changelog

Unreleased

  • Settings: Stop flagging gateway.recover_after_time as a difference when both gateway.expected_nodes and gateway.expected_data_nodes are unset (-1).

  • Admin: Added XMover - CrateDB shard analyzer and movement tool. Thanks, @WalBeh.

2025/08/19 v0.0.41

  • I/O: Updated to influxio-0.6.0. Thanks, @ZillKhan.

    • The target table is now the measurement name when importing ILP files.

    • The InfluxDB source URL accepts a timeout query parameter (seconds) to configure the network timeout when talking to the InfluxDB API.

    • For ILP imports, the CrateDB URL no longer needs a table component; you can point it at the schema only (the measurement determines the table).

2025/08/19 v0.0.40

  • I/O: Fixed MongoDB CDC invocation. Thanks, Mỹ Duyên.

2025/08/19 v0.0.39

  • OCI: Started producing image ghcr.io/crate/cratedb-toolkit-ingest

  • I/O: Added drivers for ODBC and Oracle to cratedb-toolkit-ingest

  • I/O: Updated BSON library to support ARM64

2025/08/14 v0.0.38

  • I/O: Updated to ingestr>=0.13.61

  • CFR: Improved log output

  • CFR: Fixed double quoting of table name. Thanks, @karynzv.

  • CFR: When importing, started using replace strategy instead of append

  • CFR: Improved importing data re. type mapping without NumPy

  • CFR: Truncated target table before importing, using append strategy again, because replace doesn’t do the right DDL.

  • I/O: Tuned down ingestr, it masked native I/O adapters

2025/07/01 v0.0.37

  • Settings: Fixed comparison of 0s vs 0ms. Thanks, @hlcianfagna.

  • DMS: Provided a recipe file to relay primary key and column type map information

  • DMS: Provided a recipe option to ignore processing DMS control DDL events

  • DMS: Started using the “direct” column mapping by default, retaining the “universal” column mapping optionally.

  • Dependencies: Updated to commons-codec>=0.0.23

  • I/O: Adapter for PostgreSQL full-load using ingestr

  • I/O: Added documentation about ingestr adapter

2025/06/23 v0.0.36

  • Dependencies: Migrated from zyp to tikray. It’s effectively the same, but provided using a dedicated package now

  • Dependencies: Updated to croud-1.14

  • Dependencies: Updated to async-kinesis-2.0.0. Thanks, @hampsterx.

  • CDC: Added canonical SQL example for PostgreSQL from Ibis

  • CDC: Enabled loading DMS events from Kinesis streams and stream-dump files

  • CDC: Added subcommand ctk dms table-mappings

2025/05/13 v0.0.35

  • Added lost pytest dependencies to cratedb-toolkit[testing]

2025/05/13 v0.0.34

  • Downgraded to sqlalchemy-cratedb 0.41, version 0.42 is not GA yet

2025/05/12 v0.0.33

  • CFR: Enhanced job statistics with optional reporting database support. Thanks, @WalBeh.

  • Settings: Added settings comparison utility. Thanks, @WalBeh.

  • Meta: Added parser for https://cratedb.com/releases.json file. Thanks, @WalBeh.

  • CFR: Added the ability to anonymize queries recorded by collect

  • Cloud API: SDK and CLI for CrateDB Cloud Cluster and Import APIs. Supports headless/unattended operations on CrateDB Cloud clusters, covering deploy/start/resume and data import procedures using fluent API and CLI.

  • Cloud API: Added JWT authentication to client API and ctk shell.

  • Cloud API: Added health and ping subcommands to ctk cluster

  • CLI: Downgraded to Click 8.1, as the code is not compatible with 8.2 yet

Breaking changes

Naming things for CLI options and environment variables:

  • Converged --cratedb-sqlalchemy-url vs. --cratedb-http-url options into single --cluster-url

  • Converged CRATEDB_SQLALCHEMY_URL vs. CRATEDB_HTTP_URL env vars into single CRATEDB_CLUSTER_URL

2025/04/23 v0.0.32

  • MCP: Add subsystem providing a few server and client utilities through the ctk query mcp {list,inquire,launch} subcommands.

  • Docs API: Added extractors for CrateDB functions and settings

  • Connect: Respect sslmode URI parameter when converting SQLAlchemy connection URLs to http(s)://

2025/01/31 v0.0.31

  • Fixed connectivity for jobstats collect

  • Refactored code and improved CLI interface of ctk info vs. ctk cfr

  • Dependencies: Updated to crate-2.0.0, which uses orjson for JSON marshalling

  • CFR: Job statistics and slow-query exploration per Marimo notebook. Thanks, @WalBeh.

2025/01/13 v0.0.30

  • Dependencies: Minimize dependencies of core installation, defer polars to cratedb-toolkit[io].

  • Fixed ctk cfr info record about too large values of ulimit_hard

  • Improved ctk shell to also talk to CrateDB standalone databases

  • Added basic utility command ctk tail, for tailing a database table, and optionally following the tail

  • Table Loader: Added capability to load InfluxDB Line Protocol (ILP) files

  • Query Collector: Now respects CRATEDB_CLUSTER_URL environment variable

2024/10/13 v0.0.29

  • MongoDB: Added Zyp transformations to the CDC subsystem, making it more symmetric to the full-load procedure.

  • Query Converter: Added very basic expression converter utility with CLI interface

  • DynamoDB: Added query expression converter for relocating object references, to support query migrations after the breaking change with the SQL DDL schema, by v0.0.27.

2024/10/09 v0.0.28

  • IO: Improved BulkProcessor when running per-record operations by also checking rowcount for handling INSERT OK, 0 rows responses

  • MongoDB: Fixed BSON decoding of {"$date": 1180690093000} timestamps by updating to commons-codec 0.0.21.

  • Testcontainers: Don’t always pull the OCI image before starting. It is unfortunate in disconnected situations.

2024/10/01 v0.0.27

  • MongoDB: Updated to pymongo 4.9

  • DynamoDB: Change CrateDB data model to use (pk, data, aux) columns Attention: This is a breaking change.

2024/09/26 v0.0.26

  • MongoDB: Configure MongoDBCrateDBConverter after updating to commons-codec 0.0.18

  • DynamoDB CDC: Fix MODIFY operation to also propagate deleted attributes

2024/09/22 v0.0.25

  • Table Loader: Improved conditional handling of “transformation” parameter

  • Table Loader: Improved status reporting and error logging in BulkProcessor

  • MongoDB: Improve error reporting

  • MongoDB Full: Polars’ read_ndjson doesn’t load MongoDB JSON data well, use fsspec and orjson instead

  • MongoDB Full: Improved initialization of transformation subsystem

  • MongoDB Adapter: Improved performance of when computing collection cardinality by using collection.estimated_document_count()

  • MongoDB Full: Optionally use limit parameter as number of total records

  • MongoDB Adapter: Evaluate _id filter field by upcasting to bson.ObjectId, to convey a filter that makes ctk load table process a single document, identified by its OID

  • MongoDB Dependencies: Update to commons-codec 0.0.17

2024/09/19 v0.0.24

  • MongoDB Full: Refactor transformation subsystem to commons-codec

  • MongoDB: Update to commons-codec v0.0.16

2024/09/16 v0.0.23

  • MongoDB: Unlock processing multiple collections, either from server database, or from filesystem directory

  • MongoDB: Unlock processing JSON files from HTTP resource, using https+bson://

  • MongoDB: Optionally filter server collection using MongoDB query expression

  • MongoDB: Improve error handling wrt. bulk operations vs. usability

  • DynamoDB CDC: Add ctk load table interface for processing CDC events

  • DynamoDB CDC: Accept a few more options for the Kinesis Stream: batch-size, create, create-shards, start, seqno, idle-sleep, buffer-time

  • DynamoDB Full: Improve error handling wrt. bulk operations vs. usability

2024/09/10 v0.0.22

  • MongoDB: Rename columns with leading underscores to use double leading underscores

  • MongoDB: Add support for UUID types

  • MongoDB: Improve reading timestamps in previous BSON formats

  • MongoDB: Fix processing empty arrays/lists. By default, assume TEXT as inner type.

  • MongoDB: For ctk load table, use “partial” scan for inferring the collection schema, based on the first 10,000 documents.

  • MongoDB: Skip leaking UNKNOWN fields into SQL DDL. This means relevant column definitions will not be included into the SQL DDL.

  • MongoDB: Make ctk load table use the data OBJECT(DYNAMIC) mapping strategy.

  • MongoDB: Sanitize lists of varying objects

  • MongoDB: Add treatment option for applying special treatments to certain items on real-world data

  • MongoDB: Use pagination on source collection, for creating batches towards CrateDB

  • MongoDB: Unlock importing MongoDB Extended JSON files using file+bson://...

2024/09/02 v0.0.21

  • DynamoDB: Add special decoding for varied lists. Store them into a separate OBJECT(IGNORED) column in CrateDB.

  • DynamoDB: Add pagination support for full-load table loader

2024/08/27 v0.0.20

  • DMS/DynamoDB: Fix table name quoting within CDC processor handler

2024/08/26 v0.0.19

  • MongoDB: Fix and verify Zyp transformations

  • DMS/DynamoDB/MongoDB I/O: Use SQL with parameters instead of inlining values

2024/08/21 v0.0.18

  • Dependencies: Unpin commons-codec, to always use the latest version

  • Dependencies: Unpin lorrystream, to always use the latest version

  • MongoDB: Improve type mapper by discriminating between INTEGER and BIGINT

  • MongoDB: Improve type mapper by supporting BSON DatetimeMS, Decimal128, and Int64 types

2024/08/19 v0.0.17

  • Processor: Updated Kinesis Lambda processor to understand AWS DMS

  • MongoDB: Fix missing output on STDOUT for migr8 export

  • MongoDB: Improve timestamp parsing by using python-dateutil

  • MongoDB: Converge _id input field to id column instead of dropping it

  • MongoDB: Make user interface use stderr, so stdout is for data only

  • MongoDB: Make migr8 extract write to stdout by default

  • MongoDB: Make migr8 translate read from stdin by default

  • MongoDB: Improve user interface messages

  • MongoDB: Strip single leading underscore character from all top-level fields

  • MongoDB: Map OID types to CrateDB TEXT columns

  • MongoDB: Make migr8 extract and migr8 export accept the --limit option

  • MongoDB: Fix indentation in prettified SQL output of migr8 translate

  • MongoDB: Add capability to give type hints and add transformations

  • Dependencies: Adjust code for lorrystream version 0.0.3

  • Dependencies: Update to lorrystream 0.0.4 and commons-codec 0.0.7

  • DynamoDB: Add table loader for full-load operations

2024/07/25 v0.0.16

  • ctk load table: Added support for MongoDB Change Streams

  • Fix dependency with the kaggle package, downgrade to kaggle==1.6.14

  • DynamoDB CDC: Add demo to support reading DynamoDB change data capture

2024/07/08 v0.0.15

  • IO: Added the if-exists query parameter by updating to influxio 0.4.0.

  • Rockset: Added CrateDB Rockset Adapter, a HTTP API emulation layer

  • MongoDB: Added adapter amalgamating PyMongo to use CrateDB as backend

  • SQLAlchemy: Clean up and refactor SQLAlchemy polyfills to cratedb_toolkit.util.sqlalchemy

  • CFR: Build as a self-contained program using PyInstaller

  • CFR: Publish self-contained application bundle to GitHub Workflow Artifacts

2024/06/18 v0.0.14

  • Add ctk info and ctk cfr diagnostics programs

  • Remove support for Python 3.7

  • SQLAlchemy dialect: Use sqlalchemy-cratedb>=0.37.0 This includes the fix to the get_table_names() reflection method.

2024/06/11 v0.0.13

  • Dependencies: Migrate from crate[sqlalchemy] to sqlalchemy-cratedb

2024/05/30 v0.0.12

  • Fix InfluxDB Cloud <-> CrateDB Cloud connectivity by using ssl=true query argument also for influxdb2:// source URLs.

2024/05/30 v0.0.11

  • Fix InfluxDB Cloud <-> CrateDB Cloud connectivity by propagating ssl=true query argument. Update dependencies to influxio>=0.2.1,<1.

2024/04/10 v0.0.10

2024/03/22 v0.0.9

  • Dependencies: Use dask[dataframe]

2024/03/11 v0.0.8

  • datasets: Fix compatibility with Python 3.7

2024/03/07 v0.0.7

  • datasets: Fix dataset loader

2024/03/07 v0.0.6

  • Added cratedb_toolkit.datasets subsystem, for acquiring datasets from cratedb-datasets and Kaggle.

2024/02/12 v0.0.5

  • Do not always activate pytest11 entrypoint to pytest fixture cratedb_service, as it depends on the testcontainers package, which is not always installed.

2024/02/10 v0.0.4

  • Packaging: Use cloud extra to install relevant packages

  • Dependencies: Add testing extra, which installs testcontainers only

  • Testing: Export cratedb_service fixture as pytest11 entrypoint

  • Sandbox: Reduce number of extras by just using all

2024/01/18 v0.0.3

  • Add SQL runner utility primitives to io.sql namespace

  • Add import_csv_pandas and import_csv_dask utility primitives

  • data: Add subsystem for “loading” data.

  • Add SDK and CLI for CrateDB Cloud Data Import APIs ctk load table ...

  • Add migr8 program from previous repository

  • InfluxDB: Add adapter for influxio

  • MongoDB: Add migr8 program from previous repository

  • MongoDB: Improve UX by using ctk load table mongodb://...

  • load table: Refactor to use more OO

  • Add examples/cloud_import.py

  • Adapt testcontainers to be agnostic of the testing framework. Thanks, @pilosus.

2023/11/06 v0.0.2

  • CLI: Upgrade to click-aliases>=1.0.2, fixing erroring out when no group aliases are specified.

  • Add support for Python 3.12

  • SQLAlchemy: Improve UNIQUE constraints polyfill to accept multiple column names, for emulating unique composite keys.

2023/10/10 v0.0.1

  • SQLAlchemy: Add a few patches and polyfills, which do not fit well into the vanilla Python driver / SQLAlchemy dialect.

  • Retention: Refactor strategies delete, reallocate, and snapshot, to standalone variants.

  • Retention: Bundle configuration and runtime settings into Settings entity, and use more OO instead of weak dictionaries: Add RetentionStrategy, TableAddress, and Settings entities, to improve information passing throughout the application and the SQL templates.

  • Retention: Add --schema option, and CRATEDB_EXT_SCHEMA environment variable, to configure the database schema used to store the retention policy table. The default value is ext.

  • Retention: Use full-qualified table names everywhere.

  • Retention: Fix: Compensate for DROP REPOSITORY now returning RepositoryMissingException when the repository does not exist. With previous versions of CrateDB, it was RepositoryUnknownException.

2023/06/27 v0.0.0