-
Notifications
You must be signed in to change notification settings - Fork 53
Refactoring apiclient apply
's get logic and apiclient apply
support for S3, SSM, and Secrets Manager URIs
#554
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: develop
Are you sure you want to change the base?
Conversation
use reqwest::Url; | ||
use crate::apply::{Error, Result}; | ||
use crate::apply::error::{ | ||
FileReadSnafu, FileUriSnafu, ReqwestSnafu, StdinReadSnafu, S3UriMissingBucketSnafu, InvalidFileUriSnafu, InvalidHTTPUriSnafu, S3UriSchemeSnafu |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why do we have both FileUriSnafu and InvalidFileUriSnafu
which is a bit confusing? Can we differentiate these error variants?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed - Got rid of InvalidFileUriSnafu
because it did the same thing as FileUriSnafu
and to maintain consistency with other resolvers.
.await | ||
.context(ReqwestSnafu { | ||
uri: self.url.to_string(), | ||
method: "GET".to_string(), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You don't need .to_string() here
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed - Addressing this in another comment: #554 (comment)
#[async_trait] | ||
impl UriResolver for S3Uri { | ||
async fn resolve(&self) -> Result<String> { | ||
// still unimplemented |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We generally don't want to merge a PR with unimplemented methods like this, but since this is a draft I guess its alright for now
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agree with Jarrett, but as another tip, in rust you can use the unimplemented!()
macro in functions to avoid having to "make stuff up" Like this:
async fn resolve(&self) -> Result<String> {
unimplemented!()
}
Instead of having to make up an Err
to return, unimplemented!()
just does what it takes to make the compiler happy (it panics, lol)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed - I've now implemented the resolve() method completely, so I have removed the unimplemented comment.
@@ -0,0 +1,159 @@ | |||
// src/uri_resolver.rs |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This module is lacking alot of documentation, especially a module level devinition, maybe addint something like:
//! URI resolution for different protocols.
//!
//! This module provides a trait-based approach to resolving different types of URIs
//! including stdin (`-`), file paths (`file://`), HTTP(S) URLs, and S3 URIs.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Try to imagine what someone reading this file for the first time would want.
The module-level docstring is your opportunity to explain how the abstractions/types in here relate to eachother.
If you prime programmers with the right information here, they will be able to read and comprehend your code much faster.
With more sparse information, you basically make the programmer guess at how the implementation works. Then they have to read the code to be absolutely certain (or what often times happen is that they don't do this, and then are surprised when their mental model doesn't totally match the actual code)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed - Added a module-level docstring to the top of uri_resolver.rs
sources/api/apiclient/src/apply.rs
Outdated
.context(error::StdinReadSnafu) | ||
.await?; | ||
return Ok(output); | ||
pub async fn get(input: &str) -> Result<String> { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there a reason the visibility on this method needs to be bumped to pub?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed - the function is no longer pub. It was temporarily bumped to pub
in the PR because one of the variables didn't have the correct scope.
sources/api/apiclient/src/apply.rs
Outdated
let url = Url::parse(input).context(error::UriSnafu { input_source: input.to_string() })?; | ||
|
||
// 3) file:// | ||
if let Ok(r) = FileUri::try_from(url.clone()) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wonder if it might be better to use a prefix strip and then match instead of trying to directly form the Uri type directly and failing out, but its not a real blocker here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed - fixed this issue in another comment: #554 (comment)
Pushed new Resolver for |
Reversed the changes I just made for |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks like something went awry with the rebase packages/kubernetes-1.26 files are showing up here.
sources/api/apiclient/src/apply.rs
Outdated
@@ -83,11 +84,16 @@ fn select_resolver(input: &str) -> Result<Box<dyn UriResolver>> { | |||
return Ok(Box::new(r)); | |||
} | |||
|
|||
// 6) secretsmanager:// |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These numberings look out of order, I used to do numbering too but now i've learned better to not do numbering as it prevents edits from creating inconsistencies.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed - Removed the numberings and fixed the order.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice work!
sources/api/apiclient/src/apply.rs
Outdated
} | ||
|
||
/// Choose which UriResolver applies to `input` (stdin, file://, http(s):// or s3://). | ||
fn select_resolver(input: &str) -> Result<Box<dyn UriResolver>> { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This function is fine, but I somewhat prefer an implementation where we have a static list of resolvers, and the resolver selection iterates through the list, calling an interface that each resolver exposes to decide if that resolver is capable of handling this input.
This makes it a little easier to reason about how to add more resolvers, rather than having to keep the stateful "if/else" ladder in your mind.
trait ResolverSelector {
fn try_create_resolver(input: &str) -> Option<Box<dyn UriResolver>>;
}
const RESOLVERS: &[&dyn ResolverSelector] = &[
&StdioResolverSelector,
&FileResolverSelector,
// etc
];
struct FileResolverSelector;
impl ResolverSelector for FileResolverSelector {
fn try_create_resolver(input: &str) -> Option<Box<dyn UriResolver>> {
let input_uri = Url::parse(&input).ok()?; // This maps the "Result" response into an "Option"
// try to make a FileResolver or return None if we can't
}
}
|
||
use reqwest::Url; | ||
use crate::apply::{Error, Result}; | ||
use crate::apply::error::{ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We shouldn't import Snafu's like this.
The "preferred" way is, in each function that needs these Snafu context selectors, do something like this:
fn something_that_can_fail() -> Result<(), ErrorType> {
use crate::apply::error::*;
// now the selectors are in scope just for this function
}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One reason is that this import list basically becomes a chore to maintain.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed - Used the preferred format for crate::uri_resolver
as well as crate::apply
.
method: "GET".to_string(), | ||
})? | ||
.error_for_status() | ||
.context(ReqwestSnafu { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These shouldn't all be ReqwestSnafu
. We want a different error variant to explain the failure mode more succinctly (e.g. an error variant for failing to get text()
from a response body should be different than the error variant for failing to make a web request at all)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed - replaced ReqwestSnafu
s with HttpRequest
, HttpStatus
, and HttpBody
.
} | ||
} | ||
|
||
/// s3://bucket/key URLs (stub; add aws-sdk-s3 later) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is useful information to include in a docstring, but you should start with a human-readable description first. Something like:
/// Uri Resolver that fetches content from S3
///
/// This resolver accepts input of the form s3://bucket/key and translates this into
/// authenticated AWS S3 get requests using the standard AWS credential resolution mechanism
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed - added a description for each resolver.
#[async_trait] | ||
impl UriResolver for S3Uri { | ||
async fn resolve(&self) -> Result<String> { | ||
// still unimplemented |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agree with Jarrett, but as another tip, in rust you can use the unimplemented!()
macro in functions to avoid having to "make stuff up" Like this:
async fn resolve(&self) -> Result<String> {
unimplemented!()
}
Instead of having to make up an Err
to return, unimplemented!()
just does what it takes to make the compiler happy (it panics, lol)
@@ -0,0 +1,159 @@ | |||
// src/uri_resolver.rs |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Try to imagine what someone reading this file for the first time would want.
The module-level docstring is your opportunity to explain how the abstractions/types in here relate to eachother.
If you prime programmers with the right information here, they will be able to read and comprehend your code much faster.
With more sparse information, you basically make the programmer guess at how the implementation works. Then they have to read the code to be absolutely certain (or what often times happen is that they don't do this, and then are surprised when their mental model doesn't totally match the actual code)
use aws_config::{self, BehaviorVersion, Region}; | ||
use aws_sdk_secretsmanager; | ||
|
||
// 1) load AWS config (region/account via env) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In general, comments which exactly say what the code below does aren't a great idea. The newline separation you have in this function is good, and you can basically skim each newline-separated section to figure out the flow of the function.
I think a better approach would be to hoist a lot of this information into the docstring:
/// Resolves incoming source artifacts by fetching from AWS Secrets Manager
///
/// This function uses the typical AWS credentials resolution mechanism to authenticate.
Docstrings like this are useful at two levels:
- People calling this function don't need to read the code to understand what's going to happen
- People who need to change this code can just read the docstring and be primed to understand what they're about to read, much like how I mentioned with module-level docstrings.
So when is it more useful to have inner comments?
- If the discrete actions of the function are hard to figure out by skimming each newline-separated section (this can also be a sign that you should refactor things, but like all things it's a tradeoff decision)
- If the what of the code is clear, but not the why (e.g. Imagine a function much like this one, but it reads AWS configurations from a separate file. You would probably need to add a comment there explaining why you are doing that)
apiclient apply
's get logic and apiclient apply
support for S3 URIsapiclient apply
's get logic and apiclient apply
support for S3, SSM, and Secrets Manager URIs
Twoliter.toml
Outdated
release-version = "9.1.0" | ||
release-version = "8.2.0-1" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We'll need to make sure we pull these changes out prior to merging.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed - Rebased my code on top of the bottlerocket-os:develop
branch and stopped tracking changes to Twoliter.toml.
sources/api/apiclient/Cargo.toml
Outdated
@@ -31,7 +37,7 @@ log.workspace = true | |||
models.workspace = true | |||
nix.workspace = true | |||
rand = { workspace = true, features = ["default"] } | |||
reqwest.workspace = true | |||
reqwest.workspace = true |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit:
reqwest.workspace = true | |
reqwest.workspace = true |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed - Removed the extra space
sources/api/apiclient/Cargo.toml
Outdated
@@ -44,6 +50,7 @@ tokio-tungstenite = { workspace = true, features = ["connect"] } | |||
toml.workspace = true | |||
unindent.workspace = true | |||
url.workspace = true | |||
aws-smithy-runtime-api.workspace = true |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think we actually want to directly take a dependency here. Most SDK crates re-export the error types they create.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed - This dependency is no longer being used
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It appears to still be present in Cargo.toml
sources/api/apiclient/src/apply.rs
Outdated
} | ||
|
||
|
||
/// Choose which UriResolver applies to `input` (stdin, file://, http(s):// or s3://). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This appears to be out-of-date.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed - Updated the comment to reflect the new resolvers that were added.
sources/api/apiclient/src/apply.rs
Outdated
#[snafu(display("Failed to read S3 object body '{bucket}/{key}': {}", source))] | ||
S3Body { | ||
bucket: String, | ||
key: String, | ||
source: aws_sdk_s3::primitives::ByteStreamError, | ||
}, | ||
|
||
#[snafu(display("Failed to fetch S3 object '{bucket}/{key}': {}", source))] | ||
S3Get { | ||
bucket: String, | ||
key: String, | ||
source: SdkError<GetObjectError, SdkHttpResponse>, | ||
}, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Instead of expanding the top-level error type with all of these error variants, I think it would be better to instead make a ResolverError type in the uri_resolver
module.
We could then add Resolver
as a specific type here. This would make it easier to determine at what point in the program each error variant can occur.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed - Moved all error variants associated with the resolvers into uri_resolver.rs
into a new ResolverError
enum. Additionally, the functions now return a new type ResolverResult
, so it is easier to associate errors with their sources.
sources/api/apiclient/src/apply.rs
Outdated
use crate::apply::SdkHttpResponse; | ||
use aws_smithy_runtime_api::client::result::SdkError; | ||
use aws_sdk_s3::operation::get_object::GetObjectError; | ||
//use aws_smithy_http::result::SdkHttpResponse; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please remove unused code.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed - I have removed the unused code
sources/api/apiclient/src/apply.rs
Outdated
let url = Url::parse(input).context(error::UriSnafu { input_source: input.to_string() })?; | ||
|
||
// file:// | ||
if let Ok(r) = crate::uri_resolver::FileUri::try_from(&url) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I might suggest adding use crate::uri_resolver
so we don't have to fully specify crate::
each time here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed - Imported the uri_resolver
crate, so that I no longer have to specify crate::
each time
sources/api/apiclient/src/apply.rs
Outdated
|
||
|
||
/// Choose which UriResolver applies to `input` (stdin, file://, http(s):// or s3://). | ||
fn select_resolver(input: &str) -> Result<Box<dyn crate::uri_resolver::UriResolver>> { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we add unit tests to assert that the correct resolver is selected?
I also think we really need tests to understand what happens when characters in the S3 keyspace are used which aren't supported by URL standard. We should add tests for these cases, as it seems like it's entirely up to our parser implementation to decide what to do given this current implementation.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed - I have added unit tests to assert that the correct resolver was selected.
Fixed - After trying out 4 different try_from() implementations for the S3Uri, I have found one that works on all sorts of URIs that have special characters, and ones that don't follow the URL standard. I have also added unit tests for testing different S3 URIs that have special characters and don't follow the URL standard.
parameter_name: String, | ||
} | ||
|
||
impl TryFrom<&str> for SsmUri { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we're still supporting TryFrom
on multiple types. Does it make sense to unify on each resolver accepting &str
types and allowing them each to parse?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is technically less efficient than the current setup, but it would make it easier for us to test our Resolver selector.
Another option could be to do something like this:
struct SettingsInput {
input: String,
parsed_url: Option<Url>,
}
impl SettingsInput {
pub(crate) fn new(input: impl Into<String>) -> Self {
let input = input.into();
let parsed_url = Url::parse(&input).ok();
Self { input, parsed_url }
}
}
Then you'd change select_resolver
to be like this:
fn select_resolver(input: &SettingsInput) -> Result<Box<dyn crate::uri_resolver::UriResolver>> {
// ...
}
And you could make it so that each resolver had a uniform TryFrom
impl on &SettingsInput
.
Doing this would let you parse the Url exactly once and standardize the TryFrom
. Resolvers that require a Url could basically return an error if the parsed_url
is None
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed - TryFrom
now takes &SettingsInput
as its input, and determines what input type is required for each resolver. Also, only one type (&SettingsInput
) is passed into all of the resolvers.
I think the build failed because the You can fix this locally by going into the |
SecretsManagerUriSnafu { input_source: input.input.as_str().to_string() } | ||
); | ||
// strip the prefix and ensure there's actually an ID | ||
let id = &input.input.as_str()["secretsmanager://".len()..]; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: you could use .strip_prefix("secretsmanager://")
here, which would be easier to read
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed - replaced ["secretsmanager://".len()..]
with .strip_prefix("secretsmanager://")
. Also rewrote some of the code in the try_from()
method to account for changing the type of id
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
let's update this in the TryFrom implementation for SsmUri as well. I think the S3 resolver's TryFrom implementation is the cleanest of the AWS resolvers as well, can you have all three of them align with that implementation?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed - All TryFrom
implementations are significantly cleaner and parsing logic for SSM and Secrets Manager
align similarly with the parsing logic in S3Uri's TryFrom implementation.
impl UriResolver for HttpUri { | ||
async fn resolve(&self) -> ResolverResult<String> { | ||
use resolver_error::*; | ||
const MAX_SIZE_BYTES: u64 = 100 * 1024 * 1024; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You should share this value between the two resolvers that use it, so we can configure it in one place
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed - MAX_SIZE_BYTE
is now defined at the top of the uri_resolver
crate, and has a comment explaining its purpose.
sources/api/apiclient/Cargo.toml
Outdated
@@ -39,11 +45,14 @@ serde_json.workspace = true | |||
signal-hook.workspace = true | |||
simplelog.workspace = true | |||
snafu = { workspace = true, features = ["futures"] } | |||
test-case.workspace = true |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should be listed under [dev-dependencies]
instead of [dependencies]
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed - moved test-case.workspace
to [dev-dependencies]
.
sources/api/apiclient/Cargo.toml
Outdated
@@ -44,6 +50,7 @@ tokio-tungstenite = { workspace = true, features = ["connect"] } | |||
toml.workspace = true | |||
unindent.workspace = true | |||
url.workspace = true | |||
aws-smithy-runtime-api.workspace = true |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It appears to still be present in Cargo.toml
sources/api/apiclient/src/apply.rs
Outdated
let parsed_url = Url::parse(&input).ok(); | ||
SettingsInput { input, parsed_url } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wouldn't be a bad idea to emit a debug!()
message if the URL parse fails that includes the error message from that failure. That would help us debug cases where the URL parser fails when we would expect it to succeed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed - a log::debug!
message is now logged whenever the File or HTTP URI resolvers fails to parse a URL correctly.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd rather see the debug message logged here where we still have access to the error message that would have been returned by Url::parse()
.
The error will only be meaningful in the File/HTTP URI cases, but the idea is that the message will be logged at debug
, allowing it to be ignored in cases where it isn't relevant.
sources/api/apiclient/src/apply.rs
Outdated
/// Choose which UriResolver applies to `input` (stdin, file://, http(s)://, s3://, secretsmanager://, and ssm://). | ||
fn select_resolver(input: &SettingsInput) -> Result<Box<dyn crate::uri_resolver::UriResolver>> { | ||
use crate::uri_resolver; | ||
|
||
// stdin ("-") | ||
if let Ok(r) = uri_resolver::StdinUri::try_from(input) { | ||
return Ok(Box::new(r)); | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The repetition in this function is still hard for me to thumbs up. I have a simple suggestion to improve this via macros:
/// Macro to try multiple settings resolver types in sequence, returning the first one that succeeds.
macro_rules! try_resolvers {
($input:expr, $($resolver_type:ty),+ $(,)?) => {
$(
if let Ok(r) = <$resolver_type>::try_from($input) {
return Ok(Box::new(r));
}
)+
};
}
/// Choose which UriResolver applies to `input` (stdin, file://, http(s)://, s3://, secretsmanager://, and ssm://).
fn select_resolver(input: &SettingsInput) -> Result<Box<dyn crate::uri_resolver::UriResolver>> {
use crate::uri_resolver::*;
try_resolvers!(
input,
StdinUri,
FileUri,
HttpUri,
S3Uri,
SecretsManagerArn,
SecretsManagerUri,
SsmArn,
SsmUri,
);
error::NoResolverSnafu {
input_source: input.input.clone(),
}
.fail()
}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should also probably emit a debug!
-level log message when we select a resolver saying which one we are using.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed - Made both changes.
sources/api/apiclient/src/apply.rs
Outdated
#[test_case("-", TypeId::of::<crate::uri_resolver::StdinUri>(); "stdin")] | ||
#[test_case("file:///tmp/folder", TypeId::of::<crate::uri_resolver::FileUri>(); "file")] | ||
#[test_case("http://amazon.com", TypeId::of::<crate::uri_resolver::HttpUri>(); "http")] | ||
#[test_case("https://amazon.com", TypeId::of::<crate::uri_resolver::HttpUri>(); "https")] | ||
#[test_case("s3://mybucket/path", TypeId::of::<crate::uri_resolver::S3Uri>(); "s3")] | ||
#[test_case("secretsmanager://sec", TypeId::of::<crate::uri_resolver::SecretsManagerUri>(); "secrets")] | ||
#[test_case("ssm://param", TypeId::of::<crate::uri_resolver::SsmUri>(); "ssmUri")] | ||
#[test_case("arn:aws:ssm:<region>:<account_id>:parameter/<name>", TypeId::of::<crate::uri_resolver::SsmArn>(); "ssmArn")] | ||
#[test_case("arn:aws:secretsmanager:<region>:<account-id>:secret:<secret-id>", TypeId::of::<crate::uri_resolver::SecretsManagerArn>(); "secretsmanagerArn")] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The spacing on these is very odd. You should either consistently use one space after each comma, or modify these to look like this:
#[test_case(
"-",
TypeId::of::<crate::uri_resolver::StdinUri>()
; "stdin")]
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed - The spacing is now consistent for all test cases.
// 3) Construct | ||
Ok(SsmArn { | ||
region: arn.region, | ||
parameter_name: input.input.to_string(), // still use the full `/parameter/...` segment |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The comment and parameter_name
are both incorrect. This contains the full ARN.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed - Fixed both variable names for SecretsManagerArn and SsmArn to be more descriptive of what they actually contain (full_arn
).
impl UriResolver for SsmArn { | ||
async fn resolve(&self) -> ResolverResult<String> { | ||
use resolver_error::*; | ||
// 1) Load default config, then override region & credentials |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Comment is not correct, we aren't changing credentials.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed - Forgot to update the comment. The original comment mentioned overwriting the credentials because that was my first approach, but then decided to use IAM permissions to access parameters and secrets instead (doesn't require overriding credentials).
#[derive(Debug, Snafu)] | ||
#[snafu(module)] | ||
pub enum ResolverError { | ||
#[snafu(display("Invalid ARN '{}': {}", input_source, reason))] | ||
InvalidArnFormat { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd prefer having this be composed of smaller Resolver-specific types like FileUriResolverError
, S3ResolverError
, SsmResolverError
, etc would improve maintainability.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Very nice work! I have a few comments about logging, but otherwise this LGTM.
sources/api/apiclient/src/apply.rs
Outdated
let parsed_url = Url::parse(&input).ok(); | ||
SettingsInput { input, parsed_url } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd rather see the debug message logged here where we still have access to the error message that would have been returned by Url::parse()
.
The error will only be meaningful in the File/HTTP URI cases, but the idea is that the message will be logged at debug
, allowing it to be ignored in cases where it isn't relevant.
let url = { | ||
if let None = &input.parsed_url { | ||
log::debug!("Failed to parse HTTP URI '{}'", input.input); | ||
} | ||
input.parsed_url.clone() | ||
}.context(FileUriSnafu { | ||
input_source: input.input.as_str(), | ||
})?; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
debug!
is the wrong log level for a failure here. We expect the URI to have parsed properly, so an error here would be error!
and not debug!
.
I'd be OK with removing the logging in this case in favor of just always debug logging at the top level as mentioned in the other comment.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed - updated the debug!()
statement to be thrown whenever there is an error parsing the Url for all parsed Urls, not just in the File/HTTP URI cases.
sources/api/apiclient/Cargo.toml
Outdated
@@ -44,6 +50,10 @@ tokio-tungstenite = { workspace = true, features = ["connect"] } | |||
toml.workspace = true | |||
unindent.workspace = true | |||
url.workspace = true | |||
tempfile.workspace = true |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should this also be a dev dependency?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed - It should be under [dev-dependencies]
since its only used to test the File parser.
05ba52f
to
0faab71
Compare
^ Force pushed to fix commit messages and remove unnecessary changes. |
Issue number: 4208
Closes #4208
Description of changes: Refactored the get logic for
apiclient apply
, implemented Structs for each of the resolver types, and added support for S3, SSM, Secrets Manager URIs as well as SSM and Secrets Manager ARNs toapiclient apply
Testing done: Manual Integration Testing for fetching and modifying configuration data from all sources (Stdin, File, HTTP/HTTPS, S3, SSM, Secrets Manager). Added Unit Tests to verify that the correct resolver is selected, positive and negative test cases to verify that URIs/ARNs are parsed correctly/throw an error, test cases to verify that S3 URIs with special characters and special URIs will be parsed correctly, and added a Unit Test to verify that files are parsed properly using Tempfile.
Terms of contribution:
By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.