[apiserver] Make local-e2e-test hermetic

[troychiu]

Why are these changes needed?

Make local-e2e-test hermetic by cleaning cluster first.

Related issue number

Closes #3368

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • This PR is not tested :(

[kevin85421]

cc @rueian

[rueian]

LGTM.

[dentiny]

I had some discussion with @troychiu offline, haven't fully decided the plan

[troychiu]

tldr @dentiny suggest keep e2e-test only run go test, so maybe we should put clean-kuberay-resources to local-e2e-test. However, I have found that local-e2e-test won't work on my side if the cluster already exists.

[rueian]

If we are going to move the cleanup to local-e2e-test, can we just reuse clean-cluster?

[troychiu]

I added it to e2e-test because I would like to run end-to-end test on a existing cluster so that I don't have to create a cluster every time. Wdyt?

[rueian]

I think both are ok.

[dentiny]

I discussed a little with @troychiu yesterday, here's my personal opinion and thoughts:

• There're two types of testing targets for now: end-to-end testing (local-e2e) and pure testing (e2e)
• The former does env setup + cleanup + test, so hermeticity is a necessity, otherwise the test would fail
• The later is only for test, as long as test itself succeeds and the test itself has no side effect, we could run any tests for any number of times
• So my preference is to add the cleanup logic in local-e2e target

[dentiny]

Thank you!

[dentiny]

nit: do you think it's better to rename uninstall to something more clear, for example, uninstall-api-server?
My concern is, we have too many things to uninstall, for example, apiserver, crd, etc.

[rueian]

Can we just use make clean-cluster?

[troychiu]

Yeah that's even simpler.

[kevin85421]

@troychiu can you fix the conflicts? Thanks!