Skip to content

[Bug] [Connector-V2] sink trying to update primary key in distribute sink database #9293

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
3 tasks done
papadave66 opened this issue May 8, 2025 · 0 comments · Fixed by #9298
Closed
3 tasks done
Labels

Comments

@papadave66
Copy link
Contributor

papadave66 commented May 8, 2025

Search before asking

  • I had searched in the issues and found no similar issues.

What happened

I am trying to running seatunnel with standalone mysql database and cluster postgresql sink database.
And it gives following ERROR:

It seems that it is trying to update id's value with id condition. how to prevent updating primary columns?

SeaTunnel Version

2.3.10

SeaTunnel Config

env {
  job.name = "mysql-cdc-test"
  parallelism = 1
  job.mode = "STREAMING"
  checkpoint.interval = 5000
}

source {
  MySQL-CDC {
    server-id = 5652
    username = "system"
    password = "xxxxxxxxx"
    table-names = ["seatunnel.role"]
    base-url = "jdbc:mysql://10.xx.xx.xx:3306"
  }
}

transform {
}

sink {
  jdbc {
    url = "jdbc:postgresql://xxxxxxxxx:11345/tmodel"
    driver = "org.postgresql.Driver"
    user = "xxxxx"
    password = "xxxxxxxx"
    generate_sink_sql = true
    data_save_mode = DROP_DATA
    database = "tmodel"
    table = "data.${table_name}"
    primary_keys = ["${primary_key}"]
    enable_upsert = false
    support_upsert_by_query_primary_key_exist = false
  }
}

Running Command

`seatunnel.sh --config mysql-cdc-test.conf `

Error Exception

Caused by: java.sql.BatchUpdateException: Batch entry 0 UPDATE "tmodel"."data"."role" SET "id" = ('15'::int4), "type" = ('1'::int4), "role_name" = ('NORMAL_ROLE2'), "description" = ('Normal User'), "create_time" = ('2025-03-24 08:45:55.804+08'), "update_time" = ('2025-05-08 13:31:13.768+08') WHERE "id" = ('15'::int4) was aborted: ERROR: Distributed column "id" can't be updated in current version  Call getNextException to see other errors in the batch.
        at org.postgresql.jdbc.BatchResultHandler.handleCompletion(BatchResultHandler.java:186)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:591)
        at org.postgresql.jdbc.PgStatement.internalExecuteBatch(PgStatement.java:889)
        at org.postgresql.jdbc.PgStatement.executeBatch(PgStatement.java:913)
        at org.postgresql.jdbc.PgPreparedStatement.executeBatch(PgPreparedStatement.java:1739)
        at org.apache.seatunnel.shade.com.zaxxer.hikari.pool.ProxyStatement.executeBatch(ProxyStatement.java:127)
        at org.apache.seatunnel.shade.com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeBatch(HikariProxyPreparedStatement.java)
        at org.apache.seatunnel.connectors.seatunnel.jdbc.internal.executor.FieldNamedPreparedStatement.executeBatch(FieldNamedPreparedStatement.java:540)
        at org.apache.seatunnel.connectors.seatunnel.jdbc.internal.executor.InsertOrUpdateBatchStatementExecutor.executeBatch(InsertOrUpdateBatchStatementExecutor.java:108)
        at org.apache.seatunnel.connectors.seatunnel.jdbc.internal.executor.BufferReducedBatchStatementExecutor.executeBatch(BufferReducedBatchStatementExecutor.java:89)
        at org.apache.seatunnel.connectors.seatunnel.jdbc.internal.JdbcOutputFormat.attemptFlush(JdbcOutputFormat.java:172)
        at org.apache.seatunnel.connectors.seatunnel.jdbc.internal.JdbcOutputFormat.flush(JdbcOutputFormat.java:136)
        ... 9 more
Caused by: org.postgresql.util.PSQLException: ERROR: Distributed column "id" can't be updated in current version
        at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2733)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2420)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:580)
        ... 19 more

        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:220)
        ... 2 more
2025-05-08 13:31:41,309 INFO  [s.c.s.s.c.ClientExecuteCommand] [SeaTunnel-CompletableFuture-Thread-0] - run shutdown hook because get close signal

Zeta or Flink or Spark Version

zeta with seatunnel v2.3.10

Java or Scala Version

java version "1.8.0_391"
Java(TM) SE Runtime Environment (build 1.8.0_391-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.391-b13, mixed mode)

Screenshots

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

@papadave66 papadave66 added the bug label May 8, 2025
papadave66 added a commit to papadave66/seatunnel that referenced this issue May 9, 2025
papadave66 added a commit to papadave66/seatunnel that referenced this issue May 9, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant