Skip to content

Refactor CS memory configuration (#19708) #22608

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 9 commits into from
Aug 13, 2025
Merged

Refactor CS memory configuration (#19708) #22608

merged 9 commits into from
Aug 13, 2025

Conversation

Vladilen
Copy link
Collaborator

@Vladilen Vladilen commented Aug 8, 2025 •
edited
Loading

Changelog entry

Added compaction memory limit configuration in memory controller instead of column shard separate configuration

Changelog category

  • Not for changelog (changelog entry is not required)

Description for reviewers

Added unit tests and fixed some problems

@Vladilen Vladilen requested a review from a team as a code owner August 8, 2025 14:29
@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 8, 2025 •
edited
Loading

🟢 2025-08-08 15:00:55 UTC The validation of the Pull Request description is successful.

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 8, 2025 •
edited
Loading

⚪ 2025-08-08 14:33:02 UTC Pre-commit check linux-x86_64-relwithdebinfo for c57ba56 has started.
⚪ 2025-08-08 14:33:16 UTC Artifacts will be uploaded here
⚪ 2025-08-08 14:37:04 UTC ya make is running...
⚫ 2025-08-08 14:57:21 UTC Check cancelled

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 8, 2025 •
edited
Loading

⚪ 2025-08-08 14:33:24 UTC Pre-commit check linux-x86_64-release-asan for c57ba56 has started.
⚪ 2025-08-08 14:33:38 UTC Artifacts will be uploaded here
⚪ 2025-08-08 14:37:19 UTC ya make is running...
⚫ 2025-08-08 14:57:21 UTC Check cancelled

@Vladilen Vladilen requested a review from kunga August 8, 2025 14:57
@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 8, 2025 •
edited
Loading

⚪ 2025-08-08 15:02:49 UTC Pre-commit check linux-x86_64-relwithdebinfo for b66cc78 has started.
⚪ 2025-08-08 15:03:40 UTC Artifacts will be uploaded here
⚪ 2025-08-08 15:08:03 UTC ya make is running...
🔴 2025-08-08 15:10:16 UTC Build failed, see the logs. Also see fail summary

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 8, 2025 •
edited
Loading

⚪ 2025-08-08 15:09:46 UTC Pre-commit check linux-x86_64-release-asan for b66cc78 has started.
⚪ 2025-08-08 15:10:00 UTC Artifacts will be uploaded here
⚪ 2025-08-08 15:13:43 UTC ya make is running...
🔴 2025-08-08 15:16:15 UTC Build failed, see the logs. Also see fail summary

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 8, 2025 •
edited
Loading

⚪ 2025-08-08 15:53:43 UTC Pre-commit check linux-x86_64-release-asan for a173c9c has started.
⚪ 2025-08-08 15:53:58 UTC Artifacts will be uploaded here
⚪ 2025-08-08 15:57:55 UTC ya make is running...
🟡 2025-08-08 18:41:13 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
16781 16212 0 209 332 28

🟢 2025-08-08 18:42:45 UTC Build successful.
🟢 2025-08-08 18:43:14 UTC ydbd size 4.0 GiB changed* by -18.3 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash main: 07b141b merge: a173c9c diff diff %
ydbd size 4 249 042 952 Bytes 4 249 024 232 Bytes -18.3 KiB -0.000%
ydbd stripped size 1 470 291 864 Bytes 1 470 280 600 Bytes -11.0 KiB -0.001%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 8, 2025 •
edited
Loading

⚪ 2025-08-08 15:57:55 UTC Pre-commit check linux-x86_64-relwithdebinfo for a173c9c has started.
⚪ 2025-08-08 15:58:08 UTC Artifacts will be uploaded here
⚪ 2025-08-08 16:01:49 UTC ya make is running...
🟡 2025-08-08 17:55:21 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
39375 36548 0 31 2757 39

⚪ 2025-08-08 17:58:58 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-08-08 18:12:14 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
1297 (only retried tests) 1174 0 26 75 22

⚪ 2025-08-08 18:12:28 UTC ya make is running... (failed tests rerun, try 3)
🔴 2025-08-08 18:26:12 UTC Some tests failed, follow the links below.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
702 (only retried tests) 626 0 4 50 22

🟢 2025-08-08 18:26:22 UTC Build successful.
🟢 2025-08-08 18:26:42 UTC ydbd size 2.2 GiB changed* by -9.7 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash main: 07b141b merge: a173c9c diff diff %
ydbd size 2 415 130 056 Bytes 2 415 120 120 Bytes -9.7 KiB -0.000%
ydbd stripped size 505 007 080 Bytes 505 004 904 Bytes -2.1 KiB -0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

kunga
kunga reviewed Aug 11, 2025
View reviewed changes
kunga
kunga reviewed Aug 11, 2025
View reviewed changes
@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 11, 2025 •
edited
Loading

⚪ 2025-08-11 09:43:22 UTC Pre-commit check linux-x86_64-relwithdebinfo for 8a8054d has started.
⚪ 2025-08-11 09:43:26 UTC Artifacts will be uploaded here
⚪ 2025-08-11 09:47:07 UTC ya make is running...
🟡 2025-08-11 11:44:05 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
39381 36584 0 2 2757 38

⚪ 2025-08-11 11:47:38 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-08-11 12:04:16 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
848 (only retried tests) 762 0 0 59 27

🟢 2025-08-11 12:04:27 UTC Build successful.
🟢 2025-08-11 12:04:49 UTC ydbd size 2.2 GiB changed* by -9.7 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash main: fb7f1f5 merge: 8a8054d diff diff %
ydbd size 2 415 153 208 Bytes 2 415 143 272 Bytes -9.7 KiB -0.000%
ydbd stripped size 505 012 296 Bytes 505 010 120 Bytes -2.1 KiB -0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 11, 2025 •
edited
Loading

⚪ 2025-08-11 09:44:34 UTC Pre-commit check linux-x86_64-release-asan for 8a8054d has started.
⚪ 2025-08-11 09:44:49 UTC Artifacts will be uploaded here
⚪ 2025-08-11 09:48:43 UTC ya make is running...
🟡 2025-08-11 12:24:57 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
16787 16326 0 179 258 24

🟢 2025-08-11 12:26:31 UTC Build successful.
🟢 2025-08-11 12:27:03 UTC ydbd size 4.0 GiB changed* by -14.3 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash main: fb7f1f5 merge: 8a8054d diff diff %
ydbd size 4 249 059 800 Bytes 4 249 045 176 Bytes -14.3 KiB -0.000%
ydbd stripped size 1 470 307 192 Bytes 1 470 300 024 Bytes -7.0 KiB -0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@lll-phill-lll
Copy link
Member

A reminder to change this param also:

ydb/ydb/tests/functional/tpc/large/test_tpch_spilling.py

Line 29 in 1c268ab

'column_tables_read_execution_limit_bytes': 10 * 1073741824,

@Vladilen Vladilen requested a review from kunga August 13, 2025 14:57
@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 13, 2025 •
edited
Loading

⚪ 2025-08-13 15:02:45 UTC Pre-commit check linux-x86_64-release-asan for 2c880b9 has started.
⚪ 2025-08-13 15:03:15 UTC Artifacts will be uploaded here
⚪ 2025-08-13 15:07:22 UTC ya make is running...
⚫ 2025-08-13 17:19:43 UTC Check cancelled

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 13, 2025 •
edited
Loading

⚪ 2025-08-13 15:04:30 UTC Pre-commit check linux-x86_64-relwithdebinfo for 2c880b9 has started.
⚪ 2025-08-13 15:04:44 UTC Artifacts will be uploaded here
⚪ 2025-08-13 15:08:29 UTC ya make is running...
🟡 2025-08-13 17:08:09 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
39428 36675 0 2 2710 41

⚪ 2025-08-13 17:11:33 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-08-13 17:19:53 UTC ydbd size 2.3 GiB changed* by -14.0 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash main: 52aa81f merge: 2c880b9 diff diff %
ydbd size 2 425 616 128 Bytes 2 425 601 792 Bytes -14.0 KiB -0.001%
ydbd stripped size 506 672 264 Bytes 506 668 808 Bytes -3.4 KiB -0.001%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation
⚫ 2025-08-13 17:19:55 UTC Check cancelled

dorooleg
dorooleg previously approved these changes Aug 13, 2025
View reviewed changes
@Vladilen Vladilen enabled auto-merge (squash) August 13, 2025 17:21
@Vladilen Vladilen disabled auto-merge August 13, 2025 17:21
@Vladilen Vladilen enabled auto-merge (squash) August 13, 2025 17:21
@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 13, 2025 •
edited
Loading

⚪ 2025-08-13 17:23:33 UTC Pre-commit check linux-x86_64-relwithdebinfo for d2a694d has started.
⚪ 2025-08-13 17:23:39 UTC Artifacts will be uploaded here
⚪ 2025-08-13 17:27:30 UTC ya make is running...
🟡 2025-08-13 18:51:57 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
39428 36679 0 5 2706 38

⚪ 2025-08-13 18:55:27 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-08-13 19:08:42 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
325 (only retried tests) 297 0 1 0 27

⚪ 2025-08-13 19:08:52 UTC ya make is running... (failed tests rerun, try 3)
🟢 2025-08-13 19:22:41 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
284 (only retried tests) 259 0 0 0 25

🟢 2025-08-13 19:22:54 UTC Build successful.
🟢 2025-08-13 19:23:16 UTC ydbd size 2.3 GiB changed* by -13.7 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash main: 51cf589 merge: d2a694d diff diff %
ydbd size 2 425 616 128 Bytes 2 425 602 064 Bytes -13.7 KiB -0.001%
ydbd stripped size 506 672 264 Bytes 506 668 936 Bytes -3.2 KiB -0.001%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 13, 2025 •
edited
Loading

⚪ 2025-08-13 17:24:08 UTC Pre-commit check linux-x86_64-release-asan for d2a694d has started.
⚪ 2025-08-13 17:24:22 UTC Artifacts will be uploaded here
⚪ 2025-08-13 17:28:06 UTC ya make is running...
🟡 2025-08-13 19:21:54 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
16822 16442 0 96 257 27

🟢 2025-08-13 19:23:29 UTC Build successful.
🟢 2025-08-13 19:23:55 UTC ydbd size 4.0 GiB changed* by -21.6 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash main: 51cf589 merge: d2a694d diff diff %
ydbd size 4 268 147 352 Bytes 4 268 125 224 Bytes -21.6 KiB -0.001%
ydbd stripped size 1 478 297 496 Bytes 1 478 287 960 Bytes -9.3 KiB -0.001%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@Vladilen Vladilen merged commit 639441b into ydb-platform:main Aug 13, 2025
12 checks passed
dorooleg pushed a commit to dorooleg/ydb that referenced this pull request Aug 14, 2025
@Vladilen @dorooleg
Refactor CS memory configuration (ydb-platform#22608)
ed16048
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dorooleg dorooleg dorooleg approved these changes

@kunga kunga Awaiting requested review from kunga

Assignees
No one assigned
Labels
not-for-changelog
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@Vladilen @lll-phill-lll @kunga @dorooleg