🌱 TestReconcileMachinePhases: use apireader to directly talk to apiserver

[ivelichkovich]

What this PR does / why we need it: reduce flake in envtest test

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes # github.com//issues/12785

use APIReader with helper Get function. My comment in the issue explains what I'm thinking is going. For some of the flake you can see the logs reconcile so the status must be set to i.e. Pending. Instead the phase is empty and reads as Unknown. Completed reconciliations would set the status and the logs show those completing so the cache must be stale on the check where we Get the object and validate the status. This adds helper function and uses it for validation where it queries directly against apiserver.

Note: comments on APIReader say to use it sparingly. I would not suggest disabling cache in controller clients during testing however for validation of state after a test run is complete it should be safe to use and reduce flake.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign joelspeed for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-ci-robot]

This PR is currently missing an area label, which is used to identify the modified component when generating release notes.

Area labels can be added by org members by writing /area ${COMPONENT} in a comment

Please see the labels list for possible areas.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[k8s-ci-robot]

Hi @ivelichkovich. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.