Skip to content

PERF: read list of parquet files at once #5723

@mvashishtha

Description

@mvashishtha

You can provide a list of local files (but not a list of directories or s3 files) to read_parquet, as in #5698. For #5698 I will make a fix to read the files separately and concat the results. I attempted a more general solution but it required too much surgery on modin's read_parquet code, which assumes in many places that there's just one file or directory to read. One difficulty I had is that ParquetDirectory can take a list of files but not a list of directories, so we can't convert to a list of directories at the beginning.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Minor bugs or low-priority feature requestsPerformance 🚀Performance related issues and pull requests.new feature/request 💬Requests and pull requests for new features

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions