Skip to content

Support clickhouse v2 #1261

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 16 commits into
base: master
Choose a base branch
from
Open

Support clickhouse v2 #1261

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
6b84fed
merge conflict
kokizzu Mar 11, 2025
1596f36
add rizal patch
kokizzu Apr 17, 2025
bf4106c
fix gomod merge
kokizzu Apr 17, 2025
3095dc1
linter fix
kokizzu Apr 17, 2025
a4d96db
another linter fix
kokizzu Apr 17, 2025
11bbd04
remove database query string until clickhouse-go for patch merged
kokizzu Apr 17, 2025
4dfeff2
Merge branch 'master' into support-clickhouse-v2
kokizzu Apr 17, 2025
b90713b
use filtered custom query string
kokizzu Apr 18, 2025
2a55cd9
Merge branch 'support-clickhouse-v2' of github.com:kokizzu/migrate in…
kokizzu Apr 18, 2025
4711dff
fix other query string that no longer supported
kokizzu Apr 18, 2025
259fddb
downgrade docker to 27
kokizzu Apr 18, 2025
aa6d06b
downgrade clickhouse/v2 to downgrade docker
kokizzu Apr 19, 2025
2b7d958
go mod tidy
kokizzu Apr 19, 2025
5191ae4
Revert "go mod tidy"
kokizzu Apr 19, 2025
039e35b
Revert "downgrade clickhouse/v2 to downgrade docker"
kokizzu Apr 19, 2025
09a326d
Revert "downgrade docker to 27"
kokizzu Apr 19, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 16 additions & 14 deletions database/clickhouse/clickhouse.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
"strings"
"time"

"github.com/ClickHouse/clickhouse-go/v2"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add a v2 db driver similar to how we use pgxv4 and pgxv5? That way users can decide when to migrate themselves.

Copy link
Author

@kokizzu kokizzu Apr 17, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i don't think it's possible, clickhouse v2 autoregister as "clickhouse" same as v1 (that already deprecated)
while pgx it using different name "pgx5"

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've confirmed this behavior in the code:

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

finally green btw @dhui

"go.uber.org/atomic"

"github.com/golang-migrate/migrate/v4"
Expand Down Expand Up @@ -73,11 +74,12 @@
return nil, err
}
q := migrate.FilterCustomQuery(purl)
q.Scheme = "tcp"
conn, err := sql.Open("clickhouse", q.String())
q.Scheme = "clickhouse"
opt, err := clickhouse.ParseDSN(purl.String())
if err != nil {
return nil, err
}
conn := clickhouse.OpenDB(opt)

multiStatementMaxSize := DefaultMultiStatementMaxSize
if s := purl.Query().Get("x-multi-statement-max-size"); len(s) > 0 {
Expand Down Expand Up @@ -167,7 +169,7 @@
var (
version int
dirty uint8
query = "SELECT version, dirty FROM `" + ch.config.MigrationsTable + "` ORDER BY sequence DESC LIMIT 1"
query = "SELECT version, dirty FROM " + quoteIdentifier(ch.config.DatabaseName) + "." + quoteIdentifier(ch.config.MigrationsTable) + " ORDER BY sequence DESC LIMIT 1"
)
if err := ch.conn.QueryRow(query).Scan(&version, &dirty); err != nil {
if err == sql.ErrNoRows {
Expand All @@ -192,7 +194,7 @@
return err
}

query := "INSERT INTO " + ch.config.MigrationsTable + " (version, dirty, sequence) VALUES (?, ?, ?)"
query := "INSERT INTO " + quoteIdentifier(ch.config.DatabaseName) + "." + quoteIdentifier(ch.config.MigrationsTable) + " (version, dirty, sequence) VALUES (?, ?, ?)"
if _, err := tx.Exec(query, version, bool(dirty), time.Now().UnixNano()); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
Expand Down Expand Up @@ -234,18 +236,18 @@
// if not, create the empty migration table
if len(ch.config.ClusterName) > 0 {
query = fmt.Sprintf(`
CREATE TABLE %s ON CLUSTER %s (
version Int64,
dirty UInt8,
sequence UInt64
) Engine=%s`, ch.config.MigrationsTable, ch.config.ClusterName, ch.config.MigrationsTableEngine)
CREATE TABLE %s.%s ON CLUSTER %s (
version Int64,
dirty UInt8,
sequence UInt64
) Engine=%s`, quoteIdentifier(ch.config.DatabaseName), quoteIdentifier(ch.config.MigrationsTable), ch.config.ClusterName, ch.config.MigrationsTableEngine)
} else {
query = fmt.Sprintf(`
CREATE TABLE %s (
version Int64,
dirty UInt8,
sequence UInt64
) Engine=%s`, ch.config.MigrationsTable, ch.config.MigrationsTableEngine)
CREATE TABLE %s.%s (
version Int64,
dirty UInt8,
sequence UInt64
) Engine=%s`, quoteIdentifier(ch.config.DatabaseName), quoteIdentifier(ch.config.MigrationsTable), ch.config.MigrationsTableEngine)
}

if strings.HasSuffix(ch.config.MigrationsTableEngine, "Tree") {
Expand Down Expand Up @@ -291,14 +293,14 @@
}

func (ch *ClickHouse) Lock() error {
if !ch.isLocked.CAS(false, true) {

Check failure on line 296 in database/clickhouse/clickhouse.go

View workflow job for this annotation

GitHub Actions / lint 

SA1019: ch.isLocked.CAS is deprecated: Use CompareAndSwap. (staticcheck)
return database.ErrLocked
}

return nil
}
func (ch *ClickHouse) Unlock() error {
if !ch.isLocked.CAS(true, false) {

Check failure on line 303 in database/clickhouse/clickhouse.go

View workflow job for this annotation

GitHub Actions / lint 

SA1019: ch.isLocked.CAS is deprecated: Use CompareAndSwap. (staticcheck)
return database.ErrNotLocked
}

Expand Down
2 changes: 1 addition & 1 deletion database/clickhouse/clickhouse_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ import (
"log"
"testing"

_ "github.com/ClickHouse/clickhouse-go"
_ "github.com/ClickHouse/clickhouse-go/v2"
"github.com/dhui/dktest"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database/clickhouse"
Expand Down
44 changes: 23 additions & 21 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,12 @@ require (
cloud.google.com/go/spanner v1.56.0
cloud.google.com/go/storage v1.38.0
github.com/Azure/go-autorest/autorest/adal v0.9.16
github.com/ClickHouse/clickhouse-go v1.4.3
github.com/ClickHouse/clickhouse-go/v2 v2.33.0
github.com/aws/aws-sdk-go v1.49.6
github.com/cenkalti/backoff/v4 v4.1.2
github.com/cenkalti/backoff/v4 v4.3.0
github.com/cockroachdb/cockroach-go/v2 v2.1.1
github.com/dhui/dktest v0.4.4
github.com/docker/docker v27.2.0+incompatible
github.com/docker/docker v28.0.1+incompatible
github.com/fsouza/fake-gcs-server v1.17.0
github.com/go-sql-driver/mysql v1.5.0
github.com/gobuffalo/here v0.6.0
Expand All @@ -31,38 +31,42 @@ require (
github.com/nakagami/firebirdsql v0.0.0-20190310045651-3c02a58cfed8
github.com/neo4j/neo4j-go-driver v1.8.1-0.20200803113522-b626aa943eba
github.com/snowflakedb/gosnowflake v1.6.19
github.com/stretchr/testify v1.9.0
github.com/stretchr/testify v1.10.0
github.com/xanzy/go-gitlab v0.15.0
go.mongodb.org/mongo-driver v1.7.5
go.uber.org/atomic v1.7.0
go.mongodb.org/mongo-driver v1.11.4
go.uber.org/atomic v1.11.0
golang.org/x/oauth2 v0.18.0
golang.org/x/tools v0.24.0
golang.org/x/tools v0.30.0
google.golang.org/api v0.169.0
modernc.org/ql v1.0.0
modernc.org/sqlite v1.18.1
)

require (
github.com/ClickHouse/ch-go v0.65.1 // indirect
github.com/distribution/reference v0.6.0 // indirect
github.com/docker/go-connections v0.5.0 // indirect
github.com/felixge/httpsnoop v1.0.4 // indirect
github.com/go-faster/city v1.0.1 // indirect
github.com/go-faster/errors v0.7.1 // indirect
github.com/go-logr/logr v1.4.2 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/jackc/puddle/v2 v2.2.1 // indirect
github.com/jackc/puddle/v2 v2.2.2 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/kr/text v0.2.0 // indirect
github.com/moby/docker-image-spec v1.3.1 // indirect
github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421 // indirect
github.com/modern-go/reflect2 v1.0.2 // indirect
github.com/rogpeppe/go-internal v1.12.0 // indirect
github.com/montanaflynn/stats v0.6.6 // indirect
github.com/paulmach/orb v0.11.1 // indirect
github.com/segmentio/asm v1.2.0 // indirect
github.com/stretchr/objx v0.5.2 // indirect
go.opentelemetry.io/auto/sdk v1.1.0 // indirect
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.49.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.54.0 // indirect
go.opentelemetry.io/otel v1.29.0 // indirect
go.opentelemetry.io/otel v1.35.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.29.0 // indirect
go.opentelemetry.io/otel/metric v1.29.0 // indirect
go.opentelemetry.io/otel/sdk v1.29.0 // indirect
go.opentelemetry.io/otel/trace v1.29.0 // indirect
go.opentelemetry.io/otel/metric v1.35.0 // indirect
go.opentelemetry.io/otel/trace v1.35.0 // indirect
go.opentelemetry.io/proto/otlp v1.3.1 // indirect
)

Expand All @@ -83,7 +87,7 @@ require (
github.com/Azure/go-autorest/logger v0.2.1 // indirect
github.com/Azure/go-autorest/tracing v0.6.0 // indirect
github.com/Microsoft/go-winio v0.6.2 // indirect
github.com/andybalholm/brotli v1.0.4 // indirect
github.com/andybalholm/brotli v1.1.1 // indirect
github.com/apache/arrow/go/v10 v10.0.1 // indirect
github.com/apache/thrift v0.16.0 // indirect
github.com/aws/aws-sdk-go-v2 v1.16.16 // indirect
Expand All @@ -101,7 +105,6 @@ require (
github.com/aws/smithy-go v1.13.3 // indirect
github.com/census-instrumentation/opencensus-proto v0.4.1 // indirect
github.com/cespare/xxhash/v2 v2.2.0 // indirect
github.com/cloudflare/golz4 v0.0.0-20150217214814-ef862a3cdc58 // indirect
github.com/cncf/xds/go v0.0.0-20240318125728-8a4994d93e50 // indirect
github.com/cznic/mathutil v0.0.0-20180504122225-ca4c9f2c1369 // indirect
github.com/danieljoos/wincred v1.1.2 // indirect
Expand All @@ -113,7 +116,6 @@ require (
github.com/envoyproxy/protoc-gen-validate v1.0.4 // indirect
github.com/form3tech-oss/jwt-go v3.2.5+incompatible // indirect
github.com/gabriel-vasile/mimetype v1.4.1 // indirect
github.com/go-stack/stack v1.8.0 // indirect
github.com/goccy/go-json v0.9.11 // indirect
github.com/godbus/dbus v0.0.0-20190726142602-4481cbc300e2 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
Expand Down Expand Up @@ -145,7 +147,7 @@ require (
github.com/kardianos/osext v0.0.0-20190222173326-2bc1f35cddc0 // indirect
github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51 // indirect
github.com/klauspost/asmfmt v1.3.2 // indirect
github.com/klauspost/compress v1.15.11 // indirect
github.com/klauspost/compress v1.18.0 // indirect
github.com/klauspost/cpuid/v2 v2.0.9 // indirect
github.com/mattn/go-colorable v0.1.6 // indirect
github.com/mattn/go-isatty v0.0.16 // indirect
Expand All @@ -159,13 +161,13 @@ require (
github.com/onsi/gomega v1.15.0 // indirect
github.com/opencontainers/go-digest v1.0.0 // indirect
github.com/opencontainers/image-spec v1.1.0 // indirect
github.com/pierrec/lz4/v4 v4.1.16 // indirect
github.com/pierrec/lz4/v4 v4.1.22 // indirect
github.com/pkg/browser v0.0.0-20210911075715-681adbf594b8 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/remyoudompheng/bigfft v0.0.0-20200410134404-eec4a21b6bb0 // indirect
github.com/rqlite/gorqlite v0.0.0-20230708021416-2acd02b70b79
github.com/shopspring/decimal v1.2.0 // indirect
github.com/shopspring/decimal v1.4.0 // indirect
github.com/sirupsen/logrus v1.9.3 // indirect
github.com/xdg-go/pbkdf2 v1.0.0 // indirect
github.com/xdg-go/scram v1.1.1 // indirect
Expand All @@ -176,7 +178,7 @@ require (
go.opencensus.io v0.24.0 // indirect
golang.org/x/crypto v0.36.0 // indirect
golang.org/x/exp v0.0.0-20230315142452-642cacee5cc0 // indirect
golang.org/x/mod v0.21.0 // indirect
golang.org/x/mod v0.23.0 // indirect
golang.org/x/net v0.38.0 // indirect
golang.org/x/sync v0.12.0 // indirect
golang.org/x/sys v0.31.0 // indirect
Expand Down
Loading
Loading