Skip to content

Use streaming semantics for DownloadFile through io.Reader/Writer interfaces instead of byte slices for memory efficiency #689

@mikn

Description

@mikn

I am using this library to download an OS image of roughly 500MB through the Updater abstraction. Currently this consumes roughly 3GB of RAM, as the abstractions in this library passes around the full downloaded contents as a slice of bytes.

You could make the implementation more efficient by using io.Reader and io.Writer interfaces, which could also allow for accumulating the hash instead of doing it after the download, as well as streaming the write to the destination file (or at least a temporary file next to the destination file using a tempfile, then doing an atomic rename once the file is done) instead of accumulating it in RAM before flushing to disk.

This could also open up for a new interface, called DownloadToWriter() which could accept an io.Writer which would not do the atomic rename.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Status

    No status

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions