Skip to content

OOM downloading large files #10286

@max-degterev

Description

@max-degterev

Describe the Bug

I have an upcoming task uploading/downloading large files and it seems Payload will go OOM. Would it be possible to stream data to/from S3 instead of buffering?

Downloads are as easy as passing object.Body.transformToWebStream() straight to new Response(), not sure what the lift is on uploads but I assume skipping buffering for non image files should be feasible?

Alternatively would it be possible to expose storageClient: https://github.com/payloadcms/payload/blob/main/packages/storage-s3/src/index.ts#L108 so that userland code can make a custom call without the need to install the same dependencies and create another instance.

Link to the code that reproduces this issue

https://github.com/payloadcms/payload/blob/main/packages/storage-s3/src/staticHandler.ts#L53

Reproduction Steps

Use s3Adapter. Try to upload/download 1GB file on a system with 512MB RAM

Which area(s) are affected? (Select all that apply)

plugin: cloud-storage

Environment Info

v3.12

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions