daily checks to check k8s token's validity

[junminahn]

Q. How to test tokens validity
Q. How to get the tokens for Airflow tasks X

We have Airflow schedular tasks:
- Option 1: get tokens from Vault using annotations -> k9s api endpoint to test the tokens
- Option 2: get tokens from Vault using annotations -> oc login command to test the tokens
- Option 3: copy tokens from vault into airflow secret -> k9s api endpoint to test the tokens
- not secure, not automatically synced;
- Option 4: hit application endpoints -> metrics endpoint, deletion check endpoint
- Pros: no need to create new endpoints
- Cons: metrics, deletion check requires privielges checks because those information is somewhat sensitive; in airflow task, we will need to prepare the mechanism to pass the privielge checks.
efficiency: no need to know specific project's information
- Option 5: hit application new endpoint -> that checks the validity of the tokens > true/false