-
Notifications
You must be signed in to change notification settings - Fork 2.8k
Description
Describe the Bug
I have an upcoming task uploading/downloading large files and it seems Payload will go OOM. Would it be possible to stream data to/from S3 instead of buffering?
Downloads are as easy as passing object.Body.transformToWebStream()
straight to new Response()
, not sure what the lift is on uploads but I assume skipping buffering for non image files should be feasible?
Alternatively would it be possible to expose storageClient
: https://github.com/payloadcms/payload/blob/main/packages/storage-s3/src/index.ts#L108 so that userland code can make a custom call without the need to install the same dependencies and create another instance.
Link to the code that reproduces this issue
https://github.com/payloadcms/payload/blob/main/packages/storage-s3/src/staticHandler.ts#L53
Reproduction Steps
Use s3Adapter. Try to upload/download 1GB file on a system with 512MB RAM
Which area(s) are affected? (Select all that apply)
plugin: cloud-storage
Environment Info
v3.12