Skip to content

Releases: questdb/questdb

7.3.4

07 Nov 21:20
@goodroot goodroot
7.3.4
a66f0dc
Compare
Choose a tag to compare
7.3.4

What's Changed

Better. Faster. Stronger. And even more good looking. 😎

Highlighting our latest release:

  • Window functions. These flexible functions provide moving averages, running totals, or Top-N results within a group, and more via a "window", defined by an OFFSET against a RANGE or ROWS. Unlike aggregate functions, window functions do not cause rows to become grouped into a single output row — the rows retain their separate identities. Additional perks? Performance and an easing of query complexity.

  • More functions. We've added:first(boolean), last(boolean), first(string), last(string), mathematical exponent exp(D) and first_not_null(), last_not_null(), which operate on most nullable data types, except IPv4 and GeoHash.

  • Performance. Our goal is top performance. LIKE/ILIKE queries now perform over 80% faster.

  • Web Console Refresh. The best tools provide the help that you need without getting in your way. We've tuned the Web Console UI to put what you need closer to where you need it. Less used UI elements have had their significance reduced and overall navigation has been streamlined. On top of that, a discrete news feed will send you all the latest, so that you don't miss a beat.

Before:

Screenshot 2023-11-07 at 1 00 40 PM

After:

web-console-after

New features

  • feat(sql): add first and last boolean aggregation functions by @rexf in #3873
  • feat(sql): add math exp function by @rexf in #3903
  • feat(sql): add first and last string aggregation functions by @rexf in #3879
  • perf(sql): speed up simple LIKE/ILIKE filters by @puzpuzpuz in #3897
  • feat(core): add metrics around worker by @marregui in #3872
  • feat(web-console): general UI refresh, new News UI, new CSV Import UI #199
  • feat(sql): sql functions to support traffic light widget in the Web Console by @bluestreak01 (#3918)
  • feat(sql): add first_not_null and last_not_null aggregation functions for major data types by @rexf in #3883
  • feat(sql): add vwap aggregation function by @rexf in #3885
  • feat(sql): moving avg window function by @bluestreak01 @bziobrowski in #3763
  • feat(sql): vectorized aggregation of short columns by @bluestreak01 (#3924)

UI

  • feat(web-console): general UI refresh, new News UI, new CSV Import UI questdb/ui#199

Improvements and bug fixes

  • fix(http): REST interface sends HTTP error replies on runtime errors by @jerrinot (#3908)
  • fix(sql): fix duplicate column error for implicit timestamp by @bziobrowski #3921
  • fix(sql): prevent predicate pushdown into subquery with window clause by @bziobrowski (#3923)
  • fix(sql): fix segmentation fault in non-keyed max(int) vectorised implementation by @bziobrowski (#3914)
  • fix(sql): fix server hang on evaluating large integer addition expression by @bziobrowski (#3922)
  • fix(sql): fix 'Invalid column' error for columns used in non-equality join conditions by @bziobrowski (#3912)
  • fix(core): fix random file descriptor leak on WAL table drop by @ideoma (#3919)
  • fix(core): fix connection error to HTTP TCP port on MacOS by @jerrinot in #3830
  • fix(http): prometheus exporter occasionally fails to send all metrics by @jerrinot in #3835
  • fix(sql): ensure EXPLAIN (FORMAT JSON) returns valid JSON by @biosfood in #3756
  • fix(sql): fix ArrayIndexOutOfBoundsException in select distinct with duplicate columns by @puzpuzpuz in #3834
  • fix(core): fix intermittent writing failures by @jerrinot in #3844
  • fix(core): fix write error appearing after symbol column drop by @ideoma in #3876
  • fix(sql): rename column from name to table_name in cursor function tables by @marregui in #3857
  • fix(sql): fix sql error [col in (1,2)] when col is type short by @ideoma in #3887
  • fix(http): large prometheus metrics responses split across multiple chunks by @amunra in #3843
  • fix(sql): fix error on UPDATE statement with WAL tables after ALTER by @ideoma in #3894
  • fix(sql): fix comparing Date columns with date literals by @jerrinot in #3862
  • fix(core): add FreeBSD UFS (0x35) to the list of supported file systems by @nitram509 in #3899
  • fix(sql): fix symbol value comparison with <, <=, >, >= operators by @bziobrowski in #3909
  • fix(sql): fix modulo int operator result with nulls by @bziobrowski in #3910

Documentation

  • docs(core): add section about the cloud to Spanish README by @javier in #3831

New Contributors

Full Changelog: 7.3.3...7.3.4

Contributors

javier, jerrinot, and 10 other contributors
Loading

7.3.3

06 Oct 18:47
@bluestreak01 bluestreak01
7.3.3
3c54a41
Compare
Choose a tag to compare
7.3.3

Overview

Thanks for all who contributed to our most recent release.

This release provides a set of stability fixes and usability improvements.

We are excited to highlight the following areas:

  • Improved operations: A trio of features including improved snapshots to replace existing backup functionality, connection count gauge metrics and a new mkdir.mode for detached partitions make QuestDB easier to operate.

  • InfluxDB Line Protocol improvements: The InfluxDB Line Protocol is commonly used for rapid ingest. We’ve made it better, by introducing more flexible and less error prone timestamps. In addition, it also authenticates faster and disconnects on error when using WAL tables.

  • New aggregate functions: Thanks to community member @charlespnh QuestDB now offers variance(), var_samp() and var_pop(), stddev(), stddev_pop() functions.

What is new

CORE

  • allow using SNAPSHOT statements as an alternative to BACKUP by @puzpuzpuz in #3761
  • add connection count gauge metrics for all network interfaces by @puzpuzpuz in #3815
  • add mkdir.mode permission config for detached partitions by @debanjanc01 in #3818

SQL

ILP

  • introduce less error-prone timestamp methods in Java ILP client by @puzpuzpuz in #3755

What is fixed

CORE

  • prevent connection reset during inactivity by @jerrinot in #3734
  • fix incorrect metadata pool sizing, which could cause ingestion performance to degrade by @bluestreak01 in #3762
  • fix race leading to occasionally frozen WAL transaction processing on some tables by @puzpuzpuz in #3785
  • workaround for broken file locking on Docker Desktop on MacOS by @jerrinot in #3797
  • fix wal apply suspending table on edge condition by @ideoma in #3808
  • fix transient select error after symbol column rename sequence by @ideoma in #3787
  • use the same GC algorithm in Docker image as with other QuestDB distros by @debanjanc01 in #3819

SQL

ILP

HTTP

  • http min works even when the main http server is disabled by @jerrinot in #3732

PGWire

  • fix pg-wire protocol errors caused by network IO fragmentation by @ideoma in #3751
  • fix pgwire transaction flag not cleaned on context reuse by @ideoma in #3777

New Contributors

Full Changelog: 7.3.2...7.3.3

Contributors

jerrinot, ideoma, and 9 other contributors
Loading

7.3.2

11 Sep 13:34
@bluestreak01 bluestreak01
7.3.2
19059de
Compare
Choose a tag to compare
7.3.2

Overview

Maintenance release improving stability of the deduplication logic, which we introduced in 7.3. For anyone relying on
deduplication or considering using it, we recommend to upgrade at the earliest convenience.

What is new

SQL

What is fixed

CORE

SQL

PGWire

HTTP

UI

New Contributors

Full Changelog: 7.3.1...7.3.2

Contributors

jerrinot, marregui, and 9 other contributors
Loading

7.3.1

15 Aug 18:24
@bluestreak01 bluestreak01
7.3.1
bc5d9c0
Compare
Choose a tag to compare
7.3.1

Overview

Follow-up release for the data deduplication logic and IPv4 type, first released in 7.3.
This version brings the dedup logic for CSV Import alongside minor fixes and additional functions compatible with IPv4.

We also update the Web Console with a new "create table" UI to create new tables interactively. We would love your feedback on this new UI item; let us know your thoughts.

What is new

SQL

UI

What is fixed

CORE

  • avoid unbounded growth of transactions in O3 lag of WAL tables by @puzpuzpuz in #3620
  • reduce footprint of QuestDB data structures in Java Heap by @jerrinot in #3596

SQL

HTTP

  • avoid running out of memory when importing partitioned table over REST API by @puzpuzpuz in #3638
  • make HTTP text import for WAL tables to write via WAL by @ideoma in #3648

PGWire

UI

New Contributors

Full Changelog: 7.3...7.3.1

Contributors

jerrinot, insmac, and 5 other contributors
Loading

7.3

03 Aug 11:18
@bluestreak01 bluestreak01
7.3
e63d675
Compare
Choose a tag to compare
7.3

Overview

Data deduplication

This release includes the highly anticipated data deduplication logic, which is now supported across all ingestion protocols. Its primary objective is to simplify the process of restarting data streams. If a data stream is interrupted, clients can easily resolve the issue by re-publishing an arbitrarily long window of data. The server will then handle the task of retaining only the missing data.

Data deduplication functions as an upsert operation by preserving the newer version of a record when duplicates are encountered.

To control data deduplication the following SQL syntax is available:

-- deduplicate on 2 columns
CREATE TABLE no_ts_dups_table (timestamp ts, l long, symbol sym, uuid id, string s) timestamps(ts) 
PARTITION BY DAY WAL 
DEDUPLICATE UPSERT KEYS(ts, id)

-- deduplication on 3 columns
ALTER TABLE no_ts_dups_table DEDUP UPSERT KEYS(ts, id, l);

-- deduplication on 4 columns
ALTER TABLE no_ts_dups_table DEDUP UPSERT KEYS(ts, sym, id, l);

We are thrilled to assure you that data deduplication has a negligible impact on ingestion performance, allowing it to be enabled at all times.

New IPV4 data type

This release introduces a new data type called ipv4. The primary objective behind this addition is to assist users working with IPv4 addresses in alleviating storage constraints and enhancing the overall SQL execution performance. Please see this the original pull request for more details.

What is new

CORE

  • data deduplication by @ideoma in #3524, #3566
  • add ipv4 as supported column type by @allegraharris in #3523
  • enable prepending jvm flags from docker run by @yifanstefanwang in #3567
  • make wal apply more fair in case of skewed transaction frequences by @ideoma in #3592

SQL

  • implement function to convert two longs to UUID by @jerrinot in #3559

What is fixed

CORE

  • fix wal to non-wal table conversion leaving table non-writable by @ideoma in #3572
  • fix physical rows written metric on partition split by @ideoma in #3577
  • fix vacuum command deleting live table columns by @ideoma in #3589

SQL

UI

  • fix incorrect rendering of last few columns in Web Console grid

New Contributors

Full Changelog: 7.2.1...7.3

Contributors

jerrinot, ideoma, and 6 other contributors
Loading

7.2.1

11 Jul 16:33
@bluestreak01 bluestreak01
7.2.1
cd22e5c
Compare
Choose a tag to compare
7.2.1

Overview

This is a maintenance release ahead of the next major release. Consider upgrading if you are affected by the issues fixed below.

What is new

SQL

What is fixed

CORE

SQL

  • fix thread and reader leak on invalid filter in WHERE clause by @puzpuzpuz in #3490
  • fix latest by factories not reacting to bind variable value change by @bziobrowski in #3521
  • fix constant StringIndexOutOfBoundsException in alert socket writer by @bziobrowski in #3529
  • better handling invalid SUM function parameters by @xplosunn in #3528
  • fix like/ilike filter on bind variable expression by @puzpuzpuz in #3542

PGWire

  • gracefully handle a client disconnecting during authentication by @jerrinot in #3485

ILP

New Contributors

Full Changelog: 7.2...7.2.1

Contributors

jerrinot, marregui, and 6 other contributors
Loading

7.2

09 Jun 16:51
@bluestreak01 bluestreak01
7.2
da47b26
Compare
Choose a tag to compare
7.2

Overview

In QuestDB 7.2, we address a key issue faced by our users regarding write amplification and performance under heavy loads.

Previously, QuestDB utilized fixed-size time partitions to store data in ascending chronological order. However, this approach resulted in write amplification, particularly towards the end of each time interval. To tackle this problem, we have introduced implicit variable-size time partitions in this release.

Before

image

After

image

The new feature intelligently splits partitions at strategic points, ensuring smaller partition sizes and reducing the surface area for copy-on-write operations. This minimizes write amplification and maintains high write performance, even during heavy load scenarios.

To further optimize the system, we have implemented a sophisticated statistical model to balance the split and squash logic. As a result, partitions statistically unlikely to be modified again are asynchronously squashed, reducing the number of files on disk and alleviating stress on the OS metadata catalog.

Write amplification is reduced by orders of magnitude. Users can now experience sustained write performance, even in demanding environments.

Gotchas

This is a breaking change. If you decide to downgrade from this release, partitions, created by 7.2 may not be understood by earlier versions. To proceed with the downgrade you may need to "squash" partitions on each table. To do that, run the following SQL before the downgrade:

ALTER TABLE <tab> SQUASH PARTITIONS

What is new

SQL

  • add auxiliary meta functions to support integration with third party library superset by @marregui in #3390
  • add string comparison operator by @puzpuzpuz in #3458
  • support squashing partitions using SQL by @ideoma in #3328

ILP

  • adjust ILP msg buffer size to max measurement size if needed by @glasstiger in #3408

UI

What is fixed

CORE

  • fix rare / by 0 error in WAL table writing by @ideoma in #3375
  • fix WAL segment housekeeping edge case leading to a failure to write to WAL table by @amunra in #3381
  • increase max open files limit on MacOS by @bowen-xu in #3416
  • fix stuck writing on rare concurrency condition by @ideoma in #3446
  • fix incorrect truncation of WAL table partition leading to read errors by @ideoma in #3451
  • fix wal apply error by @ideoma in #3460

SQL

ILP

  • fix ILP errors when column deleted in middle of line processing by @ideoma in #3453

HTTP

  • fix case sensivity in duplicate column validation of CSV import by @puzpuzpuz in #3444

UI

New Contributors

Full Changelog: 7.1.3...7.2

Contributors

marregui, amunra, and 8 other contributors
Loading

7.1.3

19 May 18:42
@bluestreak01 bluestreak01
7.1.3
aab510e
Compare
Choose a tag to compare
7.1.3

Overview

This release includes several minor hotfixes before the next major release, 7.2.

What we fixed

CORE

  • fix stuck ingestion by @ideoma in #3360
  • allow server to perform graceful shutdown under heavy ILP load by @bluestreak01 in #3361
  • fix wal table error applying transactions after some table alters by @ideoma in #3366
  • fix column not found error after column rename by @ideoma in #3368

SQL

  • fix unhandled error when compiling group-by SQLs that contain table aliases by @bluestreak01 in #3352

ILP

UI

  • small visual updates for the schema editor UI in CSV Import, along with code refactoring for simplicity by @insmac in #3365
  • allow editing column types on existing tables by @insmac in #3365

New Contributors

Full Changelog: 7.1.2...7.1.3

Contributors

insmac, ideoma, and 3 other contributors
Loading

7.1.2

16 May 13:08
@bluestreak01 bluestreak01
7.1.2
5b42946
Compare
Choose a tag to compare
7.1.2

Overview

We ship a new user interface for CSV imports in the Web Console. This stability release collates bug fixes for issues raised by our community. We decided to cut this release ahead of the feature-packed 7.2.

What is new

UI

CORE

SQL

HTTP

What we fixed

CORE

SQL

  • fix display of interval scans in EXPLAIN's output by @bziobrowski in #3203
  • fix bad rewrite of limit -N by @bziobrowski in #3261
  • fix column type handling in table_partitions() function by @bziobrowski in #3251
  • fix sample by over subquery with explicit timestamp by @bziobrowski in #3186
  • fix crash in hash outer join with filter factory by @bziobrowski in #3303
  • unhandled ArrayIndexOutOfBoundsException in ASOF and LT JOIN on empty table by @puzpuzpuz in #3302
  • fix COPY performance for pre-created tables with non-default symbol capacity by @puzpuzpuz in #3305
  • fix UPDATE sql timeout issue for WAL tables by @bluestreak01 in #3311
  • NullPointerException on join with function filter by @puzpuzpuz in #3333
  • UnsupportedOperationException on join with constant non-boolean filter expression by @puzpuzpuz in #3334
  • timeout of CREATE TABLE AS SELECT leaves leaked entry in table registry by @puzpuzpuz in #3338

HTTP

  • fix(http): format query parameter ignored by /imp endpoint by @puzpuzpuz in #3325

PGRWire

New Contributors

Full Changelog: 7.1.1...7.1.2

Contributors

javier, dragonlee8, and 12 other contributors
Loading

7.1.1

12 Apr 19:17
@bluestreak01 bluestreak01
7.1.1
a523400
Compare
Choose a tag to compare
7.1.1

Overview

This release was prompted by the library dependency that QuestDB 7.1 introduced in the Web Console. This dependency has affected a lot of users, and we have decided to fix it as soon as possible.

What is new

SQL

What we fixed

CORE

  • fix issues with UTF8 table and column names by @bluestreak01 in #3155
  • fix table truncate bug resulting in incorrect data inserted by @ideoma in #3165

SQL

UI

HTTP

  • fix(http): fix unhandled error when importing CSV to table with dropped and recreated columns by @puzpuzpuz in #3175

New Contributors

Full Changelog: 7.1...7.1.1

Contributors

javier, ideoma, and 3 other contributors
Loading