Skip to content

Add Kubernetes API coverage test consuming traces #20181

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

AwesomePatrol
Copy link
Contributor

It pulls a file from testdata with traces collected by Jaeger and then iterates over them in order to group them into those which use the contract interface, are known (and usually valid) uses outside of it. Later it can further analyze the parameters used over the run of Kubernetes conformance tests.

It requires patched Kubernetes
with:
AwesomePatrol/kubernetes@5a18a66 or at:
https://github.com/AwesomePatrol/kubernetes/tree/add-kubernetes-etcd-contract-tracker

Example result:

=== RUN   TestInterfaceUse
=== RUN   TestInterfaceUse/all_conformance_traces_2025-06-16a.json
=== RUN   TestInterfaceUse/all_conformance_traces_2025-06-16a.json/interface_bypass
    coverage_test.go:61: Traces which did not go through the interface: 4489 / 44064
    coverage_test.go:93: contract bypass count by operation: map[etcdserverpb.KV/Compact:11 etcdserverpb.Lease/LeaseGrant:318 etcdserverpb.Maintenance/Status:84 etcdserverpb.Watch/Watch:23]
=== RUN   TestInterfaceUse/all_conformance_traces_2025-06-16a.json/contract_methods_set_rev_when_needed
=== RUN   TestInterfaceUse/all_conformance_traces_2025-06-16a.json/contract_methods_set_rev_when_needed/Get_kubernetesEtcdContract
    coverage_test.go:117: Found 23231 traces matching Get kubernetesEtcdContract operation
    coverage_test.go:127: Distribution of revision values: map[missing:0 set:0 unset:23231]
=== RUN   TestInterfaceUse/all_conformance_traces_2025-06-16a.json/contract_methods_set_rev_when_needed/OptimisticPut_kubernetesEtcdContract
    coverage_test.go:117: Found 13576 traces matching OptimisticPut kubernetesEtcdContract operation
    coverage_test.go:127: Distribution of revision values: map[missing:0 set:9808 unset:3768]
=== RUN   TestInterfaceUse/all_conformance_traces_2025-06-16a.json/contract_methods_set_rev_when_needed/OptimisticDelete_kubernetesEtcdContract
    coverage_test.go:117: Found 1430 traces matching OptimisticDelete kubernetesEtcdContract operation
    coverage_test.go:127: Distribution of revision values: map[missing:0 set:1430 unset:0]
=== RUN   TestInterfaceUse/all_conformance_traces_2025-06-16a.json/contract_methods_set_rev_when_needed/List_kubernetesEtcdContract
    coverage_test.go:117: Found 6726 traces matching List kubernetesEtcdContract operation
    coverage_test.go:127: Distribution of revision values: map[missing:0 set:5641 unset:1085]
=== RUN   TestInterfaceUse/all_e2e_traces_2025-06-17a.json
    coverage_test.go:51: WARNING: some traces are present in Etcd only: 1004 > 0
=== RUN   TestInterfaceUse/all_e2e_traces_2025-06-17a.json/interface_bypass
    coverage_test.go:61: Traces which did not go through the interface: 6417 / 40802
    coverage_test.go:93: contract bypass count by operation: map[etcdserverpb.KV/Compact:14 etcdserverpb.Lease/LeaseGrant:454 etcdserverpb.Maintenance/Status:122 etcdserverpb.Watch/Watch:6]
=== RUN   TestInterfaceUse/all_e2e_traces_2025-06-17a.json/contract_methods_set_rev_when_needed
=== RUN   TestInterfaceUse/all_e2e_traces_2025-06-17a.json/contract_methods_set_rev_when_needed/OptimisticPut_kubernetesEtcdContract
    coverage_test.go:117: Found 12557 traces matching OptimisticPut kubernetesEtcdContract operation
    coverage_test.go:127: Distribution of revision values: map[missing:0 set:10765 unset:1792]
=== RUN   TestInterfaceUse/all_e2e_traces_2025-06-17a.json/contract_methods_set_rev_when_needed/OptimisticDelete_kubernetesEtcdContract
    coverage_test.go:117: Found 906 traces matching OptimisticDelete kubernetesEtcdContract operation
    coverage_test.go:127: Distribution of revision values: map[missing:0 set:906 unset:0]
=== RUN   TestInterfaceUse/all_e2e_traces_2025-06-17a.json/contract_methods_set_rev_when_needed/List_kubernetesEtcdContract
    coverage_test.go:117: Found 3903 traces matching List kubernetesEtcdContract operation
    coverage_test.go:127: Distribution of revision values: map[missing:0 set:2864 unset:1039]
=== RUN   TestInterfaceUse/all_e2e_traces_2025-06-17a.json/contract_methods_set_rev_when_needed/Get_kubernetesEtcdContract
    coverage_test.go:117: Found 17610 traces matching Get kubernetesEtcdContract operation
    coverage_test.go:127: Distribution of revision values: map[missing:0 set:0 unset:17610]
--- PASS: TestInterfaceUse (12.92s)
    --- PASS: TestInterfaceUse/all_conformance_traces_2025-06-16a.json (7.11s)
        --- PASS: TestInterfaceUse/all_conformance_traces_2025-06-16a.json/interface_bypass (0.01s)
        --- PASS: TestInterfaceUse/all_conformance_traces_2025-06-16a.json/contract_methods_set_rev_when_needed (0.03s)
            --- PASS: TestInterfaceUse/all_conformance_traces_2025-06-16a.json/contract_methods_set_rev_when_needed/Get_kubernetesEtcdContract (0.01s)
            --- PASS: TestInterfaceUse/all_conformance_traces_2025-06-16a.json/contract_methods_set_rev_when_needed/OptimisticPut_kubernetesEtcdContract (0.01s)
            --- PASS: TestInterfaceUse/all_conformance_traces_2025-06-16a.json/contract_methods_set_rev_when_needed/OptimisticDelete_kubernetesEtcdContract (0.01s)
            --- PASS: TestInterfaceUse/all_conformance_traces_2025-06-16a.json/contract_methods_set_rev_when_needed/List_kubernetesEtcdContract (0.01s)
    --- PASS: TestInterfaceUse/all_e2e_traces_2025-06-17a.json (5.81s)
        --- PASS: TestInterfaceUse/all_e2e_traces_2025-06-17a.json/interface_bypass (0.01s)
        --- PASS: TestInterfaceUse/all_e2e_traces_2025-06-17a.json/contract_methods_set_rev_when_needed (0.03s)
            --- PASS: TestInterfaceUse/all_e2e_traces_2025-06-17a.json/contract_methods_set_rev_when_needed/OptimisticPut_kubernetesEtcdContract (0.01s)
            --- PASS: TestInterfaceUse/all_e2e_traces_2025-06-17a.json/contract_methods_set_rev_when_needed/OptimisticDelete_kubernetesEtcdContract (0.00s)
            --- PASS: TestInterfaceUse/all_e2e_traces_2025-06-17a.json/contract_methods_set_rev_when_needed/List_kubernetesEtcdContract (0.01s)
            --- PASS: TestInterfaceUse/all_e2e_traces_2025-06-17a.json/contract_methods_set_rev_when_needed/Get_kubernetesEtcdContract (0.01s)
PASS
ok      go.etcd.io/etcd/tests/v3/robustness/coverage    12.950s

There are some TODO comments left in the code. More information in the related issue.

It pulls a file from testdata with traces collected by Jaeger and then
iterates over them in order to group them into those which use the
contract interface, are known (and usually valid) uses outside of it.
Later it can further analyze the parameters used over the run of
Kubernetes conformance tests.

It requires patched Kubernetes
with:
AwesomePatrol/kubernetes@5a18a66
or at:
https://github.com/AwesomePatrol/kubernetes/tree/add-kubernetes-etcd-contract-tracker
@k8s-ci-robot
Copy link

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: AwesomePatrol
Once this PR has been reviewed and has the lgtm label, please assign wenjiaswe for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot
Copy link

Hi @AwesomePatrol. Thanks for your PR.

I'm waiting for a etcd-io member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

Successfully merging this pull request may close these issues.

2 participants