Bug: CloudWatch logs do not use same logging level as PowerTools

[nCubed]

Expected Behaviour

When the logs are written to CloudWatch, the logging level is only applied at the structured logging. The CloudWatch logs have a different log level.

For example, when log level is set to "Error" with PowerTools, the CloudWatch log remains at "info". This is the same for all logging levels.

The issue here is that when our logs are ingested by company-wide logging aggregator (DataDog), the logs now are tagged with both "Error" and "info".

Is there a way to coerce the CloudWatch logging level to align with the PowerTools logging level?

Current Behaviour

2025-02-24T13:36:06.483Z	f6e6096f-5d28-57bd-99a3-6959252efdd6	info
{
    "cold_start": false,
    "xray_trace_id": "1-67bc<snip>",
    "function_name": "<snip>",
    "function_version": "$LATEST",
    "function_memory_size": 350,
    "function_arn": "arn:aws:lambda:us-east-2:<snip>",
    "function_request_id": "f6e6096f-<snip>",
    "timestamp": "2025-02-24T13:36:06.4837199Z",
    "level": "Error",
    "service": "<snip>",
    "name": "AWS.Lambda.Powertools.Logging.Logger",
    "message": "<snip>"
}

Code snippet

Use any standard logging message.

Possible Solution

No response

Steps to Reproduce

Log an event as an "Error". Json output is correctly set at "Error". CloudWatch is set at "info".

Powertools for AWS Lambda (.NET) version

latest

AWS Lambda function runtime

dotnet8

Debugging logs

[hjgraca]

@nCubed, are you using Advanced Logging Controls (https://docs.aws.amazon.com/lambda/latest/dg/monitoring-cloudwatchlogs-advanced.html) feature?

[nCubed]

@hjgraca Looks like we were using the OOB "text" format. I updated to "json" format and the structured logging is useless because the message is now an escaped string.

{
    "timestamp": "2025-02-24T13:36:06.483Z",
    "requestId": "f6e6096f-5d28-57bd-99a3-6959252efdd6",
    "level": "info",
    "traceId": "Root=1-67bc<snip>"
    "message": "{\"cold_start\":false,\"xray_trace_id\":\"1-67bc<snip>\",\"function_name\":\"<snip>\",\"function_version\":\"$LATEST\",\"function_memory_size\":350,\"function_arn\":\"arn:aws:lambda:us-east-2:<snip>\",\"function_request_id\":\"f6e6096f-<snip>\",\"timestamp\":\"2025-02-24T13:36:06.4837199Z\",\"level\":\"Error\",\"service\":\"<snip>\",\"name\":\"AWS.Lambda.Powertools.Logging.Logger\",\"message\":\"<snip>\"}"
}

Am I missing a configuration setting?

[hjgraca]

I was just trying to understand if you were using ALC because only ALC can change the log level.
Also in ALC mode Logging should be outputting LogLevel and not Level as you are showing and the json should not be inside a string.

Give me some time I am going to try reproduce your scenario, top level statement lambda correct? can you share a snippet of code that could help debug?

Thanks.

[nCubed]

Yes. Top level statement.

I'm going to need to run some more tests. In our actual lambdas, changing the ALC from text to json causes the json message to be an escaped string. In the standard sample I've shared with you in the past, the output is correctly structured.

This is a lower priority for us, so it may be a bit. However, I can confirm that in the message is a fully escaped string instead of structured json in our actual lambdas. Both this sandbox lambda and our actual lambdas all use the exact same build/deploy process, as well as same function definitions using Terraform.

[hjgraca]

Thanks, you should not need to enable ALC, was just wondering if you had enabled, but will try and get to the bottom of that.

[hjgraca]

Hi @nCubed I was not able to reproduce your issue. On our side from Lambda to CloudWatch everything seems fine.
But I want to understand your issue better, I might be missing something.
Please send me an email with your availability so we can have a chat to henrigra@amazon.com

Thank you.

[nCubed]

Hi @hjgraca - we're working through a release this week. I'm going to set aside another timeslot in the next day or two and re-validate the issue I encountered on my end. The log files still exist, so we at least have that as some evidence. Once I'm able to repro the issue, I'll ping you here and shoot you an email. Thanks for the follow up.

[hjgraca]

Hi @nCubed.
We had a meeting regarding this issue and we might have found the culprit.
We are calling Console.WriteLine and not the appropriate Console.Error, this will indeed affect the output to CloudWatch since the default log level is Info.

Regarding this line
2025-02-24T13:36:06.483Z f6e6096f-5d28-57bd-99a3-6959252efdd6 info
Are you doing any other logging without Powertools maybe using context.Logger? Where do you see it?

I am working on Logging v2 at the moment, but will do a fix for this and hopefully can get it out this week or next week.
Thanks for reporting the issue

[nCubed]

Hey @hjgraca - sorry for the delay on my side. We have been heads down; didn't quite expect to take this long. I'll still try to reproduce on my end when time permits.

Heads up... I will be submitting another bug for Tracing where it throws an exception which masks the underlying exception from the app. I saw a similar bug in the issues and it was closed. So this may be another avenue where the Trace tools can mask the system exception.