Skip to content

Credentials expiration using InstanceProvider and SQS #1634

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
pgrimaud opened this issue Dec 27, 2023 · 6 comments
Open

Credentials expiration using InstanceProvider and SQS #1634

pgrimaud opened this issue Dec 27, 2023 · 6 comments

Comments

@pgrimaud
Copy link
Contributor

Hello,

Thank you for your great packages! 😄

I'm using Symfony 5.4 along with the Messenger component, which includes the async-aws/core and async-aws/sqs packages. Our integration is configured to connect using the AWS Identity and Access Management (IAM) instance profile, which is valid for up to six hours.

My workers are managed by supervisord with a 3600-second time limit, after which the workers are automatically restarted.

Credentials are loaded when the worker starts, but if the credentials expire before the end of the worker's time limit, I encounter this error:

HTTP 403 returned for "https://sqs.eu-west-3.amazonaws.com/

Message: Signature expired: 20231227T074304Z is now earlier than 20231227T074415Z (20231227T075915Z - 15 min.)

I checked the AWS documentation and found this:

These security credentials are temporary and we rotate them automatically.
We make new credentials available at least five minutes before the expiration of the old credentials.

Is there a way to automatically rerun the authentication if the Expiration time plus 5 minutes is greater than the current datetime, without having to restart my workers?
@pgrimaud
Copy link
Contributor Author

Hi,

Just checking in on this. Any updates?

Thanks!

Pierre

@pgrimaud
Copy link
Contributor Author

Up

@pgrimaud
Copy link
Contributor Author

I wanted to kindly follow up on this issue. Is there any update or further information needed to move forward?

Thank you for your time and support!

@jderusse
Copy link
Member

I can't reproduce the issue, from the source code, we are using the expiration date provided by AWS to evict the cached (30 seconds before the expiration).

Could you add logs in your application to assert the Credentials::expireDate property is defined and credentials are re-regenerated when expired?

@C0piIot
Copy link

C0piIot commented Jan 10, 2025 •
edited
Loading

Hello,
I'm having this issue too.
In my case happens every now and then when reading some items from a dynamodb table:

{
  "message": "HTTP 400 returned for \"https://dynamodb.us-east-1.amazonaws.com/\".",
  "context": {
    "exception": {
      "class": "AsyncAws\\Core\\Exception\\Http\\ClientException",
      "message": "HTTP 400 returned for \"https://dynamodb.us-east-1.amazonaws.com/\".\n\nCode:    InvalidSignatureException\nMessage: Signature expired: 20250110T032928Z is now earlier than 20250110T033544Z (20250110T035044Z - 15 min.)\nType:    \nDetail:  \n",
      "code": 400,
      "file": "/app/vendor/async-aws/core/src/Response.php:432",
      "trace": [
        "/app/vendor/async-aws/core/src/Response.php:448",
        "/app/vendor/async-aws/core/src/Response.php:179",
        "/app/vendor/async-aws/core/src/Result.php:69",
        "/app/vendor/async-aws/core/src/Result.php:137",
        "/app/vendor/async-aws/dynamo-db/src/Result/QueryOutput.php:89",
        "/app/src/Redacted.php:59",
        "/app/vendor/symfony/http-kernel/HttpKernel.php:183",
        "/app/vendor/symfony/http-kernel/HttpKernel.php:76",
        "/app/vendor/symfony/http-kernel/Kernel.php:182",
        "/app/vendor/symfony/runtime/Runner/Symfony/HttpKernelRunner.php:35",
        "/app/vendor/autoload_runtime.php:29",
        "/app/public/index.php:5"
      ]
    },
    "aws_code": "InvalidSignatureException",
    "aws_message": "Signature expired: 20250110T032928Z is now earlier than 20250110T033544Z (20250110T035044Z - 15 min.)",
    "aws_type": null,
    "aws_detail": null
  },
  "level": 400,
  "level_name": "ERROR",
  "channel": "app",
  "datetime": "2025-01-10T03:29:28.855454+00:00",
  "extra": {
    "file": "/app/vendor/async-aws/core/src/Response.php",
    "line": 467,
    "class": "AsyncAws\\Core\\Response",
    "callType": "->",
    "function": "getResolveStatus"
  }
}

@Ardenexal
Copy link

Ardenexal commented May 8, 2025 •
edited
Loading

I've also been experiencing this issue. It doesn't happen often, only 12 times in the last 90 days, but the service that's pushing to that queue can't miss events, or it might cause downstream issues

AsyncAws\Core\Exception\Http\ClientException: HTTP 403 returned for "https://sqs.ap-southeast-2.amazonaws.com/".

Code:    InvalidSignatureException
Message: Signature expired: 20250505T143857Z is now earlier than 20250505T193050Z (20250505T194550Z - 15 min.)
Type:    
Detail:  

#44 /vendor/async-aws/core/src/Response.php(432): AsyncAws\Core\Response::AsyncAws\Core\{closure}
#43 /vendor/async-aws/core/src/Response.php(448): AsyncAws\Core\Response::getResolveStatus
#42 /vendor/async-aws/core/src/Response.php(179): AsyncAws\Core\Response::resolve
#41 /vendor/async-aws/core/src/Response.php(124): AsyncAws\Core\Response::__destruct
#40 /vendor/symfony/amazon-sqs-messenger/Transport/Connection.php(370): Symfony\Component\Messenger\Bridge\AmazonSqs\Transport\Connection::send
#39 /vendor/symfony/amazon-sqs-messenger/Transport/AmazonSqsSender.php(59): Symfony\Component\Messenger\Bridge\AmazonSqs\Transport\AmazonSqsSender::send
#38 /vendor/symfony/amazon-sqs-messenger/Transport/AmazonSqsTransport.php(69): Symfony\Component\Messenger\Bridge\AmazonSqs\Transport\AmazonSqsTransport::send
#37 /vendor/symfony/messenger/Middleware/SendMessageMiddleware.php(62): Symfony\Component\Messenger\Middleware\SendMessageMiddleware::handle
#36 /vendor/symfony/doctrine-bridge/Messenger/DoctrineCloseConnectionMiddleware.php(31): Symfony\Bridge\Doctrine\Messenger\DoctrineCloseConnectionMiddleware::handleForManager
#35 /vendor/symfony/doctrine-bridge/Messenger/AbstractDoctrineMiddleware.php(45): Symfony\Bridge\Doctrine\Messenger\AbstractDoctrineMiddleware::handle
#34 /vendor/symfony/messenger/Middleware/FailedMessageProcessingMiddleware.php(34): Symfony\Component\Messenger\Middleware\FailedMessageProcessingMiddleware::handle
#33 /vendor/symfony/messenger/Middleware/DispatchAfterCurrentBusMiddleware.php(61): Symfony\Component\Messenger\Middleware\DispatchAfterCurrentBusMiddleware::handle
#32 /vendor/symfony/messenger/Middleware/RejectRedeliveredMessageMiddleware.php(41): Symfony\Component\Messenger\Middleware\RejectRedeliveredMessageMiddleware::handle
#31 /vendor/symfony/messenger/Middleware/AddBusNameStampMiddleware.php(37): Symfony\Component\Messenger\Middleware\AddBusNameStampMiddleware::handle
#30 /vendor/symfony/messenger/Middleware/TraceableMiddleware.php(40): Symfony\Component\Messenger\Middleware\TraceableMiddleware::handle
#29 /vendor/symfony/messenger/MessageBus.php(70): Symfony\Component\Messenger\MessageBus::dispatch
#28 /vendor/symfony/messenger/TraceableMessageBus.php(38): Symfony\Component\Messenger\TraceableMessageBus::dispatch
#27 /src/MessageQueue/DispatchQueueEvent.php(76): App\MessageQueue\DispatchQueueEvent::toTransport
#26 /src/MessageQueue/DispatchQueueEvent.php(28): App\MessageQueue\DispatchQueueEvent::asAsyncNormal
#25 /src/EventScheduler/EventHandler/Redacted.php(75): App\EventScheduler\EventHandler\Redacted::__invoke
#24 /vendor/symfony/messenger/Middleware/HandleMessageMiddleware.php(152): Symfony\Component\Messenger\Middleware\HandleMessageMiddleware::callHandler
#23 /vendor/symfony/messenger/Middleware/HandleMessageMiddleware.php(91): Symfony\Component\Messenger\Middleware\HandleMessageMiddleware::handle
#22 /vendor/symfony/messenger/Middleware/SendMessageMiddleware.php(71): Symfony\Component\Messenger\Middleware\SendMessageMiddleware::handle
#21 /vendor/symfony/doctrine-bridge/Messenger/DoctrineCloseConnectionMiddleware.php(31): Symfony\Bridge\Doctrine\Messenger\DoctrineCloseConnectionMiddleware::handleForManager
#20 /vendor/symfony/doctrine-bridge/Messenger/AbstractDoctrineMiddleware.php(45): Symfony\Bridge\Doctrine\Messenger\AbstractDoctrineMiddleware::handle
#19 /vendor/symfony/messenger/Middleware/FailedMessageProcessingMiddleware.php(34): Symfony\Component\Messenger\Middleware\FailedMessageProcessingMiddleware::handle
#18 /vendor/symfony/messenger/Middleware/DispatchAfterCurrentBusMiddleware.php(68): Symfony\Component\Messenger\Middleware\DispatchAfterCurrentBusMiddleware::handle
#17 /vendor/symfony/messenger/Middleware/RejectRedeliveredMessageMiddleware.php(41): Symfony\Component\Messenger\Middleware\RejectRedeliveredMessageMiddleware::handle
#16 /vendor/symfony/messenger/Middleware/AddBusNameStampMiddleware.php(37): Symfony\Component\Messenger\Middleware\AddBusNameStampMiddleware::handle
#15 /vendor/symfony/messenger/Middleware/TraceableMiddleware.php(40): Symfony\Component\Messenger\Middleware\TraceableMiddleware::handle
#14 /vendor/symfony/messenger/MessageBus.php(70): Symfony\Component\Messenger\MessageBus::dispatch
#13 /vendor/symfony/messenger/TraceableMessageBus.php(38): Symfony\Component\Messenger\TraceableMessageBus::dispatch
#12 /vendor/symfony/messenger/RoutableMessageBus.php(51): Symfony\Component\Messenger\RoutableMessageBus::dispatch
#11 /vendor/symfony/messenger/Worker.php(161): Symfony\Component\Messenger\Worker::handleMessage
#10 /vendor/symfony/messenger/Worker.php(108): Symfony\Component\Messenger\Worker::run
#9 /vendor/symfony/messenger/Command/ConsumeMessagesCommand.php(244): Symfony\Component\Messenger\Command\ConsumeMessagesCommand::execute
#8 /vendor/symfony/console/Command/Command.php(326): Symfony\Component\Console\Command\Command::run
#7 /vendor/symfony/console/Application.php(1096): Symfony\Component\Console\Application::doRunCommand
#6 /vendor/symfony/framework-bundle/Console/Application.php(126): Symfony\Bundle\FrameworkBundle\Console\Application::doRunCommand
#5 /vendor/symfony/console/Application.php(324): Symfony\Component\Console\Application::doRun
#4 /vendor/symfony/framework-bundle/Console/Application.php(80): Symfony\Bundle\FrameworkBundle\Console\Application::doRun
#3 /vendor/symfony/console/Application.php(175): Symfony\Component\Console\Application::run
#2 /vendor/symfony/runtime/Runner/Symfony/ConsoleApplicationRunner.php(49): Symfony\Component\Runtime\Runner\Symfony\ConsoleApplicationRunner::run
#1 /vendor/autoload_runtime.php(29): require_once
#0 /bin/console(11): null
Symfony\Component\Messenger\Exception\TransportException: HTTP 403 returned for "https://sqs.ap-southeast-2.amazonaws.com/".

Code:    InvalidSignatureException
Message: Signature expired: 20250505T143857Z is now earlier than 20250505T193050Z (20250505T194550Z - 15 min.)
Type:    
Detail:  

#39 /vendor/symfony/amazon-sqs-messenger/Transport/AmazonSqsSender.php(68): Symfony\Component\Messenger\Bridge\AmazonSqs\Transport\AmazonSqsSender::send
#38 /vendor/symfony/amazon-sqs-messenger/Transport/AmazonSqsTransport.php(69): Symfony\Component\Messenger\Bridge\AmazonSqs\Transport\AmazonSqsTransport::send
#37 /vendor/symfony/messenger/Middleware/SendMessageMiddleware.php(62): Symfony\Component\Messenger\Middleware\SendMessageMiddleware::handle
#36 /vendor/symfony/doctrine-bridge/Messenger/DoctrineCloseConnectionMiddleware.php(31): Symfony\Bridge\Doctrine\Messenger\DoctrineCloseConnectionMiddleware::handleForManager
#35 /vendor/symfony/doctrine-bridge/Messenger/AbstractDoctrineMiddleware.php(45): Symfony\Bridge\Doctrine\Messenger\AbstractDoctrineMiddleware::handle
#34 /vendor/symfony/messenger/Middleware/FailedMessageProcessingMiddleware.php(34): Symfony\Component\Messenger\Middleware\FailedMessageProcessingMiddleware::handle
#33 /vendor/symfony/messenger/Middleware/DispatchAfterCurrentBusMiddleware.php(61): Symfony\Component\Messenger\Middleware\DispatchAfterCurrentBusMiddleware::handle
#32 /vendor/symfony/messenger/Middleware/RejectRedeliveredMessageMiddleware.php(41): Symfony\Component\Messenger\Middleware\RejectRedeliveredMessageMiddleware::handle
#31 /vendor/symfony/messenger/Middleware/AddBusNameStampMiddleware.php(37): Symfony\Component\Messenger\Middleware\AddBusNameStampMiddleware::handle
#30 /vendor/symfony/messenger/Middleware/TraceableMiddleware.php(40): Symfony\Component\Messenger\Middleware\TraceableMiddleware::handle
#29 /vendor/symfony/messenger/MessageBus.php(70): Symfony\Component\Messenger\MessageBus::dispatch
#28 /vendor/symfony/messenger/TraceableMessageBus.php(38): Symfony\Component\Messenger\TraceableMessageBus::dispatch
#27 /src/MessageQueue/DispatchQueueEvent.php(76): App\MessageQueue\DispatchQueueEvent::toTransport
#26 /src/MessageQueue/DispatchQueueEvent.php(28): App\MessageQueue\DispatchQueueEvent::asAsyncNormal
#25 /src/EventScheduler/EventHandler/Redacted.php(75): App\EventScheduler\EventHandler\Redacted::__invoke
#24 /vendor/symfony/messenger/Middleware/HandleMessageMiddleware.php(152): Symfony\Component\Messenger\Middleware\HandleMessageMiddleware::callHandler
#23 /vendor/symfony/messenger/Middleware/HandleMessageMiddleware.php(91): Symfony\Component\Messenger\Middleware\HandleMessageMiddleware::handle
#22 /vendor/symfony/messenger/Middleware/SendMessageMiddleware.php(71): Symfony\Component\Messenger\Middleware\SendMessageMiddleware::handle
#21 /vendor/symfony/doctrine-bridge/Messenger/DoctrineCloseConnectionMiddleware.php(31): Symfony\Bridge\Doctrine\Messenger\DoctrineCloseConnectionMiddleware::handleForManager
#20 /vendor/symfony/doctrine-bridge/Messenger/AbstractDoctrineMiddleware.php(45): Symfony\Bridge\Doctrine\Messenger\AbstractDoctrineMiddleware::handle
#19 /vendor/symfony/messenger/Middleware/FailedMessageProcessingMiddleware.php(34): Symfony\Component\Messenger\Middleware\FailedMessageProcessingMiddleware::handle
#18 /vendor/symfony/messenger/Middleware/DispatchAfterCurrentBusMiddleware.php(68): Symfony\Component\Messenger\Middleware\DispatchAfterCurrentBusMiddleware::handle
#17 /vendor/symfony/messenger/Middleware/RejectRedeliveredMessageMiddleware.php(41): Symfony\Component\Messenger\Middleware\RejectRedeliveredMessageMiddleware::handle
#16 /vendor/symfony/messenger/Middleware/AddBusNameStampMiddleware.php(37): Symfony\Component\Messenger\Middleware\AddBusNameStampMiddleware::handle
#15 /vendor/symfony/messenger/Middleware/TraceableMiddleware.php(40): Symfony\Component\Messenger\Middleware\TraceableMiddleware::handle
#14 /vendor/symfony/messenger/MessageBus.php(70): Symfony\Component\Messenger\MessageBus::dispatch
#13 /vendor/symfony/messenger/TraceableMessageBus.php(38): Symfony\Component\Messenger\TraceableMessageBus::dispatch
#12 /vendor/symfony/messenger/RoutableMessageBus.php(51): Symfony\Component\Messenger\RoutableMessageBus::dispatch
#11 /vendor/symfony/messenger/Worker.php(161): Symfony\Component\Messenger\Worker::handleMessage
#10 /vendor/symfony/messenger/Worker.php(108): Symfony\Component\Messenger\Worker::run
#9 /vendor/symfony/messenger/Command/ConsumeMessagesCommand.php(244): Symfony\Component\Messenger\Command\ConsumeMessagesCommand::execute
#8 /vendor/symfony/console/Command/Command.php(326): Symfony\Component\Console\Command\Command::run
#7 /vendor/symfony/console/Application.php(1096): Symfony\Component\Console\Application::doRunCommand
#6 /vendor/symfony/framework-bundle/Console/Application.php(126): Symfony\Bundle\FrameworkBundle\Console\Application::doRunCommand
#5 /vendor/symfony/console/Application.php(324): Symfony\Component\Console\Application::doRun
#4 /vendor/symfony/framework-bundle/Console/Application.php(80): Symfony\Bundle\FrameworkBundle\Console\Application::doRun
#3 /vendor/symfony/console/Application.php(175): Symfony\Component\Console\Application::run
#2 /vendor/symfony/runtime/Runner/Symfony/ConsoleApplicationRunner.php(49): Symfony\Component\Runtime\Runner\Symfony\ConsoleApplicationRunner::run
#1 /vendor/autoload_runtime.php(29): require_once
#0 /bin/console(11): null

The service is using AWS ECS Fargate instances using supervisor to run the queue. Each messenger queue consumes 10 messages before restarting, so this is variable in how often it restarts.

I will keep investigating to see if I can find more information.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@jderusse @C0piIot @pgrimaud @Ardenexal