chore(ci): check for performance regression and create report

[soonum]

After running performances regression benchmarks, a performance changes checking is executed. It will fetch results data with an external tool then it will look for anomaly in changes. Finally it will produce a report as an issue comment with any anomaly display in a Markdown array. A folded section of the report message contains all the results from the benchmark.

Note that a fully custom benchmark triggered from an issue comment would not generate a report. In addition HPU performance regression benchmark is not supported yet.

closes: please link all relevant issues

PR content/description

Check-list:

• Tests for the changes have been added (for bug fixes / features)
• Docs have been added / updated (for bug fixes / features)
• Relevant issues are marked as resolved/closed, related issues are linked in the description
• Check for breaking changes (including serialization changes) and add them to commit message following the conventional commit specification

---

This change is 

[github-actions]

No performance regression detected. 🎉
Configuration

• backend: gpu
• regression-profile: default

View All Benchmarks

Operation	Current (ms)	Baseline (ms)	Baseline Stddev (ms)	Change (%)	Status
mul	152 ms	156 ms	3.42 ms	-2.33%	➖ no changes
dex::swap_claim::no_cmux	902 ms	934 ms	16.1 ms	-3.46%	✅ minor improvement

[IceTDrinker]

@IceTDrinker reviewed 4 of 7 files at r1, 13 of 13 files at r2, all commit messages.
Reviewable status: all files reviewed, 8 unresolved discussions (waiting on @tmontaigu)

---

ci/perf_regression.py line 524 at r2 (raw file):

            # Between +/- MINOR_CHANGE_SCALE_FACTOR * Std_dev we consider there is no change
            self.change_type = PerfChange.NoChange
        if self.head_branch_value < self.baseline_mean - minor_threshold:

elif ?

---

ci/perf_regression.py line 548 at r2 (raw file):

def convert_value_to_readable_text(value: float, max_digits=3):

value_nanos

---

.github/workflows/benchmark_perf_regression.yml line 51 at r2 (raw file):

      issues: write
      # Needed to react to a comment in a pull-request
      pull-requests: write

do you need both ? when you say react it's to comment/answer ?

---

.github/workflows/benchmark_perf_regression.yml line 73 at r2 (raw file):

      - name: Install Python requirements
        run: |
          python3 -m pip install -r ci/perf_regression_requirements.txt

maybe even creat a dedicated subdir ?

---

.github/workflows/benchmark_perf_regression.yml line 307 at r2 (raw file):

        uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
        with:
          fetch-depth: 0

to be checked elsewhere : we may not need the full depth

---

.github/workflows/benchmark_perf_regression.yml line 326 at r2 (raw file):

          --hardware "${HARDWARE_NAME}" \
          --branch "${REF_NAME}" \
          --time-span 60

--time-span-days ? to make it clear those are days

---

.github/workflows/benchmark_perf_regression.yml line 351 at r2 (raw file):

    name: benchmark_perf_regression/comment-on-failure
    runs-on: ubuntu-latest
    if: ${{ failure() && github.event_name == 'issue_comment' }}

you probably need a "needs" directive to be able to check the previous status

---

ci/data_extractor/src/connector.py line 62 at r2 (raw file):

    def _override_with_env(self):
        self.host = os.environ.get("DATABASE_HOST", self.host)

maybe make the env var a bit more specific to your code

[github-actions]

No performance regression detected. 🎉
Configuration

• backend: cpu
• regression-profile: default

View All Benchmarks

Operation	Current (ms)	Baseline (ms)	Baseline Stddev (ms)	Change (%)	Status
add_parallelized	106 ms	107 ms	1.29 ms	-0.68%	➖ no changes
dex::swap_claim::no_cmux	2.26 s	2.33 s	52.2 ms	-2.99%	➖ no changes
erc20::transfer::whitepaper	283 ms	281 ms	12.7 ms	+0.75%	➖ no changes

[github-actions]

No performance regression detected. 🎉
Configuration

• backend: cpu
• regression-profile: default

View All Benchmarks

Operation	Current (ms)	Baseline (ms)	Baseline Stddev (ms)	Change (%)	Status
add_parallelized	105 ms	107 ms	1.29 ms	-2.13%	➖ no changes
dex::swap_claim::no_cmux	2.42 s	2.33 s	52.2 ms	+3.94%	➖ no changes
erc20::transfer::whitepaper	295 ms	281 ms	12.7 ms	+4.8%	➖ no changes

[IceTDrinker]

One tiny thing

@IceTDrinker reviewed 3 of 8 files at r3, 3 of 4 files at r4, 2 of 2 files at r6, 2 of 2 files at r7, all commit messages.
Reviewable status: all files reviewed, 2 unresolved discussions (waiting on @tmontaigu)

---

ci/data_extractor/src/benchmark_specs.py line 100 at r7 (raw file):

            case "ks_pbs":
                return CoreCryptoOperation.KeyswitchPBS
            case "multi_bit_ks_pbs" | "multi_bit_deterministic_ks_pbs":

to be checked with a newer PR that gets rid of it

---

.github/workflows/benchmark_perf_regression.yml line 49 at r7 (raw file):

    permissions:
#      # Needed to react to benchmark command in issue comment
#      issues: write

comment to remove then ?

[soonum]

Reviewable status: all files reviewed, 1 unresolved discussion (waiting on @IceTDrinker and @tmontaigu)

---

.github/workflows/benchmark_perf_regression.yml line 51 at r2 (raw file):

Previously, IceTDrinker (Arthur Meyre) wrote…

do you need both ? when you say react it's to comment/answer ?

I removed the pull-requests permission since it's just to react to an existing issue comment.
To test that, this code needs to be in main. We'll see how it unfolds.

---

ci/data_extractor/src/benchmark_specs.py line 100 at r7 (raw file):

Previously, IceTDrinker (Arthur Meyre) wrote…

to be checked with a newer PR that gets rid of it

The PR you're referring to is not concerned by this change.
This is intended to cover all cases from data_extractor point of view.

[IceTDrinker]

Huge achievement, thanks a lot !!!

@IceTDrinker reviewed 1 of 1 files at r8, all commit messages.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on @IceTDrinker and @tmontaigu)

[IceTDrinker]

go for mege!