test_dgemmt (actually DGEMM) failure on ppc64le with GCC 15

[sharkcz]

Looks like the test_extensions/test_dgemmt test has some failure when OpenBLAS is built with GCC 15 on ppc64le (Fedora 42). Building with GCC 14 (Fedora 41) there is no such issue.

...
TEST 1197/1522 dgemmt:c_api_rowmajor_lower_beta_one [OK]
TEST 1198/1522 dgemmt:c_api_rowmajor_lower_alpha_zero [OK]
TEST 1199/1522 dgemmt:c_api_rowmajor_lower_M_50_K_50_a_trans_b_trans [OK]
TEST 1200/1522 dgemmt:c_api_rowmajor_lower_M_50_K_100_a_notrans_b_trans [OK]
TEST 1201/1522 dgemmt:c_api_rowmajor_lower_M_100_K_50_a_trans_b_notrans [OK]
TEST 1202/1522 dgemmt:c_api_rowmajor_lower_M_50_K_50_a_notrans_b_notrans [OK]
TEST 1203/1522 dgemmt:c_api_rowmajor_upper_beta_one [OK]
TEST 1204/1522 dgemmt:c_api_rowmajor_upper_alpha_zero [OK]
TEST 1205/1522 dgemmt:c_api_rowmajor_upper_M_50_K_50_a_trans_b_trans [OK]
TEST 1206/1522 dgemmt:c_api_rowmajor_upper_M_50_K_100_a_notrans_b_trans [OK]
TEST 1207/1522 dgemmt:c_api_rowmajor_upper_M_100_K_50_a_trans_b_notrans [FAIL]
  ERR: test_extensions/test_dgemmt.c:820  expected 0.000e+00, got 6.352e-02 (diff -6.352e-02, tol 1.000e-13)
TEST 1208/1522 dgemmt:c_api_rowmajor_upper_M_50_K_50_a_notrans_b_notrans [OK]
TEST 1209/1522 dgemmt:c_api_colmajor_lower_beta_one [OK]
TEST 1210/1522 dgemmt:c_api_colmajor_lower_alpha_zero [OK]
TEST 1211/1522 dgemmt:c_api_colmajor_lower_M_50_K_50_a_trans_b_trans [OK]
TEST 1212/1522 dgemmt:c_api_colmajor_lower_M_50_K_100_a_notrans_b_trans [OK]
TEST 1213/1522 dgemmt:c_api_colmajor_lower_M_100_K_50_a_trans_b_notrans [OK]
TEST 1214/1522 dgemmt:c_api_colmajor_lower_M_50_K_50_a_notrans_b_notrans [OK]
TEST 1215/1522 dgemmt:c_api_colmajor_upper_beta_one [OK]
TEST 1216/1522 dgemmt:c_api_colmajor_upper_alpha_zero [OK]
TEST 1217/1522 dgemmt:c_api_colmajor_upper_M_50_K_50_a_trans_b_trans [OK]
TEST 1218/1522 dgemmt:c_api_colmajor_upper_M_50_K_100_a_notrans_b_trans [OK]
TEST 1219/1522 dgemmt:c_api_colmajor_upper_M_100_K_50_a_trans_b_notrans [OK]
TEST 1220/1522 dgemmt:c_api_colmajor_upper_M_50_K_50_a_notrans_b_notrans [OK]
TEST 1221/1522 dgemmt:lower_beta_one [OK]
TEST 1222/1522 dgemmt:lower_alpha_zero [OK]
TEST 1223/1522 dgemmt:lower_M_50_K_50_a_trans_b_trans [OK]
TEST 1224/1522 dgemmt:lower_M_50_K_100_a_notrans_b_trans [FAIL]
  ERR: test_extensions/test_dgemmt.c:431  expected 0.000e+00, got 2.585e-01 (diff -2.585e-01, tol 1.000e-13)
TEST 1225/1522 dgemmt:lower_M_100_K_50_a_trans_b_notrans [OK]
TEST 1226/1522 dgemmt:lower_M_50_K_50_a_notrans_b_notrans [OK]
TEST 1227/1522 dgemmt:upper_beta_one [OK]
TEST 1228/1522 dgemmt:upper_alpha_zero [OK]
...
RESULTS: 1522 tests (1520 ok, 2 failed, 0 skipped) ran in 973 ms
make[1]: *** [Makefile:87: run_test] Chyba 2
make[1]: Opouští se adresář „/mnt/dan/OpenBLAS/utest“
make: *** [Makefile:177: tests] Chyba 2

In both cases I am doing make TARGET=POWER9 , but it fails with POWER8 kernel as well.

[sharkcz]

I can reproduce that locally with 0.3.29, but the same version passes the tests in the Fedora official build (https://koji.fedoraproject.org/koji/packageinfo?packageID=15277) even with GCC 15. Perhaps the distro compiler flags make the difference ...

[martin-frbg]

Reproduced with gcc15.1 built from source. Actually that appears to be the DGEMM kernel getting miscompiled and throwing off the comparison ("LESS THAN HALF ACCURATE" warning in test/DBLAT3.SUMM)

[sharkcz]

Hmm, would you be able to open an upstream (or Fedora) gcc bug? Would any item from https://gcc.gnu.org/wiki/FAQ#misoptimization help?

[martin-frbg]

That would probably be the bug report from hell, as I probably can't reduce it to a small self-contained reproducer... The relevant DGEMM kernel files are in assembly and I think they have been reviewed often enough to exclude any undeclared clobbers of crucial registers that a more aggressive optimizer might try to exploit.

Everything works when I switch the compilation option to -O2 instead of -O3 in Makefile.power (leaving the -mvsx -fno-fast-math unchanged), but I cannot get it to work by adding #pragma GCC optimize ("O2") to all sources on the DGEMM codepath (and/or build the dblas3 test with -O2).

Curiously, everything also works when I use -O2 -mvsx -fno-fast-math plus all the individual options that -O3 adds over -O2 according to the gcc online docs at https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html - I wonder if something was added in 15.1 that is not yet reflected in the docs ? (The only other rational explanation would be that gcc 15.1 passes different options to the assembler at -O3 than it did in 14.x - I have not yet explored that. Mind you I have not updated binutils between the two GCC versions.)

[sharkcz]

Thanks for all the details, I will use my contacts and reach to the IBM toolchain team about it.

[martin-frbg]

So the problem most likely is somewhere in the compilation of the DGEMMINCOPY and DGEMMOTCOPY kernels (from generic C sources in kernel/generic/gemm_ncopy_16.c and kernel/generic/gemm_tcopy_4.c respectively).

These yield different assembly files when compiled at -O3 with 14.2 and 15.1, and the problem is not resolved by adding #pragma GCC optimize("O2") at the top of the sources - although compiling the entire codebase at -O2 makes it work.

There appears to be significant difference between the assembly files generated by 15.1 at O2 and O3 even with the pragma requesting O2 in place (files in the O2-s and O3-s folders of the tarball) though annoyingly there is also a difference in register numbers used.

issue5372.tar.gz

[sharkcz]

The recommendation I got from the internal toolchain team is more or less generic:

• -O3 is certainly not equivalent to -O2 with tons of enabled options (especially if from manual), it affects stuff like vectorization cost model, -O2 vectorizes significantly less stuff
• I'd say just the normal test reduction suggestions apply, try to build it without -flto, if it still reproduces and have working vs. non-working set of options, bisect which .o file is affected, once ideally a single problematic .o file is found, preprocess it (if C++ ideally with -D_GLIBCXX_DO_NOT_USE_BUILTIN_TRAITS), try some optimize pragmas or noinline attributes etc. to narrow it down further, next step can be bisect compiler and/or if narrowed down to a single routine try to construct self-contained runtime testcase (noipa on the function if possible, caller sets minimal environment to call the function and noipa also on functions which weren't inlined but are still used in the function and stub their behavior so that it still reproduces)

FWIW quite some time ago (wow, it has been 10 years ...) I wrote https://sharkcz.livejournal.com/13754.html

[martin-frbg]

Guess I can try a build with -mno-lra if gcc still supports that, but if it doesn't then I'll just force the build to use O2 if compiling with 15.1 (and revisit at 15.2 or 16 if necessary). None of the generic suggestions seem applicable, and I certainly don't have the time to bisect the gcc codebase

[sharkcz]

Yeah, the GCC guys have pre-built compilers handy, so it's not that difficult for them to do "compiler bisecting". But I will try to find some time to do the test reduction.

[martin-frbg]

One more datapoint - the problem already occurs with a GCC15 built from snapshot 2025-02-02

[martin-frbg]

actually fails with 2025-01-05 snapshot (corresponding to gcc git f5351b38 ) too - found me a mirror site now that keeps snapshots from 2024 as well, so will try 2024-07-07 next. (This may seem silly, but I find it less of a hassle than picking a suitable starting point for a proper bisect from the full git log)
Update: fails with gcc15-2024-07-07 (4594d55) and 2024-06-02 (4782729) too
gcc-2024-05-05 aka 21e7aa5f works though - gcc-2024-05-19 (8713d5e) doesn't

[martin-frbg]

Seems to have been something committed in the week between gcc-2024-05-12 (13b6ac4e) and gcc-2024-05-19 (8713d5e) , bisecting now

[martin-frbg]

1e0ae1f52741f7e0133661659ed2d210f939a398 is the first bad commit
commit 1e0ae1f52741f7e0133661659ed2d210f939a398
Author: Richard Biener rguenther@suse.de
Date: Wed May 15 18:32:37 2024 +0200

tree-optimization/79958 - make DSE track multiple paths

DSE currently gives up when the path we analyze forks.  This leads
to multiple missed dead store elimination PRs.  The following fixes
this by recursing for each path and maintaining the visited bitmap
to avoid visiting CFG re-merges multiple times.  The overall cost
is still limited by the same bound, it's just more likely we'll hit
the limit now.  The patch doesn't try to deal with byte tracking
once a path forks but drops info on the floor and only handling
fully dead stores in that case.

    PR tree-optimization/79958
    PR tree-optimization/109087
    PR tree-optimization/100314
    PR tree-optimization/114774
    * tree-ssa-dse.cc (dse_classify_store): New forwarder.
    (dse_classify_store): Add arguments cnt and visited, recurse
    to track multiple paths when we end up with multiple defs.

    * gcc.dg/tree-ssa/ssa-dse-48.c: New testcase.
    * gcc.dg/tree-ssa/ssa-dse-49.c: Likewise.
    * gcc.dg/tree-ssa/ssa-dse-50.c: Likewise.
    * gcc.dg/tree-ssa/ssa-dse-51.c: Likewise.
    * gcc.dg/graphite/pr80906.c: Avoid DSE of last data reference
    in loop.
    * g++.dg/ipa/devirt-24.C: Adjust for extra DSE.
    * g++.dg/warn/Wuninitialized-pr107919-1.C: Use more important
    -O2 optimization level, -O1 regresses.

:040000 040000 c1f085377eb94065e875949c6d3c07fae0d47a0e 89eb61d53a73a8ab5c6b8401ff04e4810acf430d M gcc

[JeevithaPalanisamy]

GCC's dead store elimination (DSE) causes an issue in the dgemv_n function. The following loop populates a temporary xbuffer[0..3] before calling dgemv_kernel_4x4:

for (int i = 0; i < n1; i++) {
xbuffer[0] = x_ptr[0];
x_ptr += inc_x;
xbuffer[1] = x_ptr[0];
x_ptr += inc_x;
xbuffer[2] = x_ptr[0];
x_ptr += inc_x;
xbuffer[3] = x_ptr[0];
x_ptr += inc_x;
dgemv_kernel_4x4(NB, a_ptr, lda, xbuffer, ybuffer, alpha);
a_ptr += lda4;
}

However, GCC's ipa-modref analysis incorrectly identifies stores to xbuffer[1], xbuffer[2], and xbuffer[3] as dead, removing them:

ipa-modref: call to dgemv_kernel_4x4/36 does not use ref: xbuffer[2] alias sets: 1->1
ipa-modref: call to dgemv_kernel_4x2/37 does not use ref: xbuffer[2] alias sets: 1->1
ipa-modref: call to dgemv_kernel_4x1/38 does not use ref: xbuffer[2] alias sets: 1->1
Deleted dead store: xbuffer[3] = _24;
Deleted dead store: xbuffer[2] = _21;
Deleted dead store: xbuffer[1] = _18;

To confirm: declaring xbuffer as volatile prevents DSE from removing these stores, and the test case no longer fails.

Checking the behavior of ipa-modref analysis.