cmake: Split test cases to improve parallelism #1760

hebasto · 2025-10-16T14:24:57Z

This PR implements the approach suggested in #1734 (review).

Here is an example of the ctest output:

$ ctest --test-dir build -j $(nproc)
Test project /home/hebasto/dev/secp256k1/secp256k1/build
      Start  1: secp256k1_noverify::ellswift
      Start  2: secp256k1_noverify::ecmult_constants
      Start  3: secp256k1_noverify::ecmult
      Start  4: secp256k1_noverify::integer
      Start  5: secp256k1_noverify::scalar
      Start  6: secp256k1_noverify::field
      Start  7: secp256k1_noverify::group
      Start  8: secp256k1_noverify::ec
      Start  9: secp256k1_noverify::ecdh
      Start 10: secp256k1_noverify::ecdsa
      Start 11: secp256k1_noverify::recovery
      Start 12: secp256k1_noverify::extrakeys
      Start 13: secp256k1_noverify::schnorrsig
      Start 14: secp256k1_noverify::musig
      Start 15: secp256k1_noverify::general,hash,utils
      Start 16: secp256k1_verify::ellswift
 1/36 Test #11: secp256k1_noverify::recovery .............***Skipped   0.00 sec
      Start 17: secp256k1_verify::ecmult_constants
 2/36 Test #12: secp256k1_noverify::extrakeys ............   Passed    0.01 sec
      Start 18: secp256k1_verify::ecmult
 3/36 Test  #8: secp256k1_noverify::ec ...................   Passed    0.01 sec
      Start 19: secp256k1_verify::integer
 4/36 Test  #5: secp256k1_noverify::scalar ...............   Passed    0.02 sec
      Start 20: secp256k1_verify::scalar
 5/36 Test #15: secp256k1_noverify::general,hash,utils ...   Passed    0.02 sec
      Start 21: secp256k1_verify::field
 6/36 Test #13: secp256k1_noverify::schnorrsig ...........   Passed    0.03 sec
      Start 22: secp256k1_verify::group
 7/36 Test  #9: secp256k1_noverify::ecdh .................   Passed    0.04 sec
      Start 23: secp256k1_verify::ec
 8/36 Test #23: secp256k1_verify::ec .....................   Passed    0.03 sec
      Start 24: secp256k1_verify::ecdh
 9/36 Test #20: secp256k1_verify::scalar .................   Passed    0.05 sec
      Start 25: secp256k1_verify::ecdsa
10/36 Test #14: secp256k1_noverify::musig ................   Passed    0.10 sec
      Start 26: secp256k1_verify::recovery
11/36 Test #26: secp256k1_verify::recovery ...............***Skipped   0.00 sec
      Start 27: secp256k1_verify::extrakeys
12/36 Test #27: secp256k1_verify::extrakeys ..............   Passed    0.02 sec
      Start 28: secp256k1_verify::schnorrsig
13/36 Test #24: secp256k1_verify::ecdh ...................   Passed    0.12 sec
      Start 29: secp256k1_verify::musig
14/36 Test #28: secp256k1_verify::schnorrsig .............   Passed    0.07 sec
      Start 30: secp256k1_verify::general,hash,utils
15/36 Test #30: secp256k1_verify::general,hash,utils .....   Passed    0.02 sec
      Start 31: secp256k1_exhaustive
16/36 Test #29: secp256k1_verify::musig ..................   Passed    0.24 sec
      Start 32: secp256k1_example::ecdsa
17/36 Test #32: secp256k1_example::ecdsa .................   Passed    0.00 sec
      Start 33: secp256k1_example::ecdh
18/36 Test #33: secp256k1_example::ecdh ..................   Passed    0.00 sec
      Start 34: secp256k1_example::schnorr
19/36 Test #34: secp256k1_example::schnorr ...............   Passed    0.00 sec
      Start 35: secp256k1_example::ellswift
20/36 Test #35: secp256k1_example::ellswift ..............   Passed    0.00 sec
      Start 36: secp256k1_example::musig
21/36 Test #36: secp256k1_example::musig .................   Passed    0.00 sec
22/36 Test #10: secp256k1_noverify::ecdsa ................   Passed    0.89 sec
23/36 Test  #6: secp256k1_noverify::field ................   Passed    1.38 sec
24/36 Test #21: secp256k1_verify::field ..................   Passed    1.47 sec
25/36 Test  #7: secp256k1_noverify::group ................   Passed    1.51 sec
26/36 Test #25: secp256k1_verify::ecdsa ..................   Passed    1.68 sec
27/36 Test #19: secp256k1_verify::integer ................   Passed    2.76 sec
28/36 Test  #4: secp256k1_noverify::integer ..............   Passed    3.16 sec
29/36 Test  #3: secp256k1_noverify::ecmult ...............   Passed    3.39 sec
30/36 Test  #1: secp256k1_noverify::ellswift .............   Passed    3.74 sec
31/36 Test #22: secp256k1_verify::group ..................   Passed    3.97 sec
32/36 Test  #2: secp256k1_noverify::ecmult_constants .....   Passed    5.15 sec
33/36 Test #18: secp256k1_verify::ecmult .................   Passed    6.18 sec
34/36 Test #31: secp256k1_exhaustive .....................   Passed    7.53 sec
35/36 Test #17: secp256k1_verify::ecmult_constants .......   Passed    9.50 sec
36/36 Test #16: secp256k1_verify::ellswift ...............   Passed   12.04 sec

100% tests passed, 0 tests failed out of 36

Label Time Summary:
secp256k1_example       =   0.01 sec*proc (5 tests)
secp256k1_exhaustive    =   7.53 sec*proc (1 test)
secp256k1_noverify      =  19.46 sec*proc (15 tests)
secp256k1_verify        =  38.16 sec*proc (15 tests)

Total Test time (real) =  12.05 sec

The following tests did not run:
	 11 - secp256k1_noverify::recovery (Skipped)
	 26 - secp256k1_verify::recovery (Skipped)

real-or-random · 2025-10-17T07:02:25Z

The output is nice for sure, but wouldn't it be much simpler to just invoke the test binaries with -j?

My concern is about the list of tests here. It will require maintenance work, and it's easy to forget a test.

sipa · 2025-10-17T13:06:50Z

The output is nice for sure, but wouldn't it be much simpler to just invoke the test binaries with -j?

I don't think that's desirable, because it lacks the ability to share the granularity with other test targets.

Specifically, I'm thinking in the context of the Bitcoin Core build, which has tons of test cases of its own. The ideal scenario is that you have a single process pool (controlled by the -j argument to cmake), and it is used to run both the Bitcoin Core and libsecp256k1 (and other) tests. This will generally result in all processes making progress throughout the entire run, until cmake runs out of test tasks to schedule.

In contrast, if you push the task scheduling down to libsecp256k1's own test binary, you at best end up with a situation where you first run all Bitcoin Core tests, wait for those to complete (temporarily running out of tasks in the process), and then run libsecp256k1's tests with -j. I don't know if that's actually possible to do in cmake's supported generators, and more likely you'll end up in a situation where the entire libsecp256k1 test run (-j and all) is seen as a single process from the perspective of the build system - meaning it'd simultaneously run N-1 Bitcoin Core tests, plus 1 libsecp256k1 test (which itself runs in N processes), temporarily raising the parallellism to 2N-1.

furszy · 2025-10-17T15:28:38Z

Regarding the maintenance topic, wouldn't be better to call --list_tests to fetch the test names instead of manually adding them one by one? We could make --list_tests=2 output the tests names separated by commas to make your life easier at the parsing side.

I experimented a bit with this idea and was able to dynamically generate a file containing all the tests names. See https://github.com/furszy/secp256k1/tree/2025_ci_concurrent_tests (just need to run the tests_discover target).
The next step would be to connect this to run separate ctest targets in some way, which doesn't seem to be so simple because the binary does not exist during configure time but ctest targets are added there.. (chicken-egg problem). Maybe we could execute a custom script that reads the file and launches per target ctest runs dynamically too.

furszy · 2025-10-17T14:04:49Z

src/CMakeLists.txt

+  function(add_executable_and_tests verify)
+    if(verify)
+      set(verify_definition VERIFY)
+      set(exe_name tests)
+    else()
+      set(verify_definition "")
+      set(exe_name noverify_tests)
+    endif()
+    add_executable(${exe_name} tests.c)
+    target_link_libraries(${exe_name} secp256k1_precomputed secp256k1_asm)
+    target_compile_definitions(${exe_name} PRIVATE ${verify_definition} ${TEST_DEFINITIONS})
+    add_test(NAME secp256k1_${exe_name} COMMAND ${exe_name})
+  endfunction()
+
+  add_executable_and_tests(NO)
  if(NOT CMAKE_BUILD_TYPE STREQUAL "Coverage")
-    add_executable(tests tests.c)
-    target_compile_definitions(tests PRIVATE VERIFY ${TEST_DEFINITIONS})
-    target_link_libraries(tests secp256k1_precomputed secp256k1_asm)
-    add_test(NAME secp256k1_tests COMMAND tests)
+    add_executable_and_tests(YES)


Could write this shorter as:

function(add_executable_and_tests bin_name verify_definition) add_executable(${bin_name} tests.c) target_link_libraries(${bin_name} secp256k1_precomputed secp256k1_asm) target_compile_definitions(${bin_name} PRIVATE ${verify_definition} ${TEST_DEFINITIONS}) add_test(NAME secp256k1_${bin_name} COMMAND ${bin_name}) endfunction() add_executable_and_tests(noverify_tests "") if(NOT CMAKE_BUILD_TYPE STREQUAL "Coverage") add_executable_and_tests(tests VERIFY) endif()

Also, probably the target_link_libraries should be PRIVATE too.

See 00f20f6 (I applied it on top of your first commit - could directly squash it there)

Also, probably the target_link_libraries should be PRIVATE too.

All target_link_libraries commands should use the same signature. I think this change should be made in a separate PR.

Could write this shorter as: ...

Thanks! Reworked.

hebasto · 2025-10-21T07:36:26Z

The output is nice for sure, but wouldn't it be much simpler to just invoke the test binaries with -j?

I don't think that's desirable, because it lacks the ability to share the granularity with other test targets.

Specifically, I'm thinking in the context of the Bitcoin Core build, which has tons of test cases of its own. The ideal scenario is that you have a single process pool (controlled by the -j argument to cmake), and it is used to run both the Bitcoin Core and libsecp256k1 (and other) tests. This will generally result in all processes making progress throughout the entire run, until cmake runs out of test tasks to schedule.

I share the same perspective.

hebasto · 2025-10-21T07:39:29Z

Regarding the maintenance topic, wouldn't be better to call --list_tests to fetch the test names instead of manually adding them one by one? We could make --list_tests=2 output the tests names separated by commas to make your life easier at the parsing side.

I experimented a bit with this idea and was able to dynamically generate a file containing all the tests names. See https://github.com/furszy/secp256k1/tree/2025_ci_concurrent_tests (just need to run the tests_discover target). The next step would be to connect this to run separate ctest targets in some way, which doesn't seem to be so simple because the binary does not exist during configure time but ctest targets are added there.. (chicken-egg problem). Maybe we could execute a custom script that reads the file and launches per target ctest runs dynamically too.

@purpleKarrot's approach could be applied as a follow-up.

Co-authored-by: furszy <matiasfurszyfer@protonmail.com>

hebasto · 2025-10-21T07:58:01Z

Feedback from @furszy has been addressed.

real-or-random · 2025-10-22T09:59:37Z

src/CMakeLists.txt

+  list(APPEND test_targets "ecmult_pre_g wnaf point_times_order ecmult_near_split_bound ecmult_chain ecmult_gen_blind ecmult_const_tests ecmult_multi_tests ec_combine")
+  list(APPEND test_names   ecmult)
+  # Continue with the remaining tests.
+  list(APPEND test_targets integer scalar field group ec ecdh ecdsa recovery extrakeys schnorrsig musig)


Is there a reason why you have quotation marks in the first line here but not in the last?

hebasto mentioned this pull request Oct 16, 2025

Slow unit tests delay functional tests and leave CPU unused bitcoin/bitcoin#32770

Open

real-or-random added ci tweak/refactor labels Oct 17, 2025

furszy reviewed Oct 17, 2025

View reviewed changes

hebasto and others added 3 commits October 21, 2025 09:49

cmake, refactor: De-duplicate test-related code

7a9dfed

Co-authored-by: furszy <matiasfurszyfer@protonmail.com>

cmake: Split test cases to improve parallelism

7600f5f

cmake: Set LABELS property for tests

2d8da55

hebasto force-pushed the 251016-ctest-opt branch from c08bb65 to 2d8da55 Compare October 21, 2025 07:57

real-or-random reviewed Oct 22, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

cmake: Split test cases to improve parallelism #1760

cmake: Split test cases to improve parallelism #1760

Uh oh!

hebasto commented Oct 16, 2025

Uh oh!

real-or-random commented Oct 17, 2025

Uh oh!

sipa commented Oct 17, 2025

Uh oh!

furszy commented Oct 17, 2025

Uh oh!

furszy Oct 17, 2025 •

edited

Loading

Uh oh!

hebasto Oct 21, 2025

Uh oh!

hebasto Oct 21, 2025

Uh oh!

hebasto commented Oct 21, 2025

Uh oh!

hebasto commented Oct 21, 2025

Uh oh!

hebasto commented Oct 21, 2025

Uh oh!

real-or-random Oct 22, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

cmake: Split test cases to improve parallelism #1760

Are you sure you want to change the base?

cmake: Split test cases to improve parallelism #1760

Uh oh!

Conversation

hebasto commented Oct 16, 2025

Uh oh!

real-or-random commented Oct 17, 2025

Uh oh!

sipa commented Oct 17, 2025

Uh oh!

furszy commented Oct 17, 2025

Uh oh!

furszy Oct 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hebasto Oct 21, 2025

Choose a reason for hiding this comment

Uh oh!

hebasto Oct 21, 2025

Choose a reason for hiding this comment

Uh oh!

hebasto commented Oct 21, 2025

Uh oh!

hebasto commented Oct 21, 2025

Uh oh!

hebasto commented Oct 21, 2025

Uh oh!

real-or-random Oct 22, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

furszy Oct 17, 2025 •

edited

Loading