[Feature] Investigate dependabot package upgrade failure

[dentiny]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

A detailed failure case:

• A failed package upgrade: Bump google.golang.org/protobuf from 1.34.2 to 1.36.6 in /proto #3396
• A successful package upgrade: bump protobuf version #3443

The only difference lies in golang and golang toolchain upgrade.
It would be nice (and somehow necessary) to investigate how to configure dependabot properly, otherwise we have to manually fix all the upgrade breakage ourselves.

Use case

No response

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[kenmcheng]

In #3396, the protobuf version for the apiserver was v1.36.5 whereas v1.36.6 in #3443.

While PR #3396 was for bumping protobuf version for ./proto, go mod download in the workflow bumped up the version in the ./apiserver as well (v1.36.5 -> v1.36.6). However, go mod download didn't handle the protobuf version in the go.mod correctly which led to a build failure.

The version bump wasn't required in #3443, the breakage was avoided.
I can successfully build apiserver by running go mod tidy instead.

Repro steps:

1. Checkout on commit d40d1f6
2. cd apiserver
3. go mod download
4. go build ./..., and the output is

% go mod download
% go build ./...
$HOME/go/pkg/mod/google.golang.org/grpc@v1.65.0/credentials/credentials.go:33:2: missing go.sum entry for module providing package google.golang.org/protobuf/proto (imported by github.com/ray-project/kuberay/apiserver/test/e2e); to add:
        go get github.com/ray-project/kuberay/apiserver/test/e2e
$HOME/go/pkg/mod/google.golang.org/grpc@v1.65.0/internal/pretty/pretty.go:27:2: missing go.sum entry for module providing package google.golang.org/protobuf/encoding/protojson (imported by github.com/ray-project/kuberay/apiserver/cmd); to add:
        go get github.com/ray-project/kuberay/apiserver/cmd
$HOME/go/pkg/mod/google.golang.org/grpc@v1.65.0/internal/pretty/pretty.go:28:2: missing go.sum entry for module providing package google.golang.org/protobuf/protoadapt (imported by google.golang.org/grpc/encoding/proto); to add:
        go get google.golang.org/grpc/encoding/proto@v1.65.0
$HOME/go/pkg/mod/google.golang.org/grpc@v1.65.0/binarylog/grpc_binarylog_v1/binarylog.pb.go:28:2: missing go.sum entry for module providing package google.golang.org/protobuf/reflect/protoreflect (imported by github.com/grpc-ecosystem/grpc-gateway/v2/runtime); to add:
        go get github.com/grpc-ecosystem/grpc-gateway/v2/runtime@v2.20.0
...

run go mod tidy instead:
3. go mod tidy
4. go build ./...

% go mod tidy
% go build ./...
%  # no error message

The difference was that protobuf version wasn't updated correctly in the go.mod

go mod download

go mod tidy

I think using go mod tidy is a more robust way if it doesn't break other dependencies.

[dentiny]

I think using go mod tidy is a more robust way if it doesn't break other dependencies.

Thanks for the thorough investigation and experiment! Do you think it's possible to add go mod tidy to dependabot?

[kenmcheng]

Thanks for the thorough investigation and experiment! Do you think it's possible to add go mod tidy to dependabot?

The behavior is configured in the .github/workflows/test-job.yaml. We may rather change that file.

kuberay/.github/workflows/test-job.yaml

Lines 56 to 62 in 89e980f

	- name: Get dependencies
	run: go mod download
	working-directory: ${{env.working-directory}}
	- name: Build
	run: go build ./...
	working-directory: ${{env.working-directory}}

[kenmcheng]

Update: By default, dependabot performs go mod tidy for each update.
reference: https://github.blog/changelog/2020-10-19-dependabot-go-mod-tidy-and-vendor-support/

However, this isssue occurred when the sub-project was being checked was different from the sub-project that the PR targeted at, leading to a mismatch in the library version in the go.sum after running go mod download.

--

Another solution is adding grouping configuration such as:

      google-golang:
        patterns:
          - "google.golang.org/*"

The related updates will be grouped into one PR which avoids inconsistency across sub-projects.

[dentiny]

Re-open this issue, because I've seen a few more upgrade failure cases.

[dentiny]

@kenmcheng Are you willing to put out a PR? Currently I think manually resolve dependency issue is too labor intensive.

[kenmcheng]

Hi @dentiny, I have to take a deeper look into this issue since PRs came with different fail reasons. For instance, there was a codegen issue in #3472 which may not have a simple fix of modifying dependabot config file. (maybe adjusting github action workflow would help)
We may try our best effort to mitigate but hardly eradicate the issue.

[dentiny]

We may try our best effort to mitigate but hardly eradicate the issue.

Thank you @kenmcheng for the help! I really appreciate and admire your technical skill to reduce upgrade PR, and the methodology to approach the problem!

My personal opinion is as long as we're going towards the right direction we're good, so no need to propose a perfect solution.
If you want a discussion I'm here anytime, I want to learn more from you genius!

[kenmcheng]

#3509 another codegen fail.

The following reference from the k8s-sigs may solve the codegen issue from the dependabot PRs.
https://github.com/kubernetes-sigs/cluster-api-provider-azure/blob/9e05057d543263fc0fbfa4dedc568def72db7897/.github/workflows/dependabot-code-gen.yml