Skip to content

ENH: Add Polars engine to read_csv (#61813) #61983

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 3 commits into from

Conversation

abujabarmubarak
Copy link

What does this PR do?

This PR adds support for engine="polars" in the pandas.read_csv() function. It enables users to leverage the performance of the Polars DataFrame engine when reading CSV files in pandas.


Why is this needed?

This enhancement addresses issue #61813. Since Polars is a high-performance DataFrame library with fast CSV parsing, adding it as an engine allows pandas users to benefit from its speed while staying within the pandas API.


What changes were made?

✅ Added Polars Support in _read() Function

  • Included a conditional block inside the _read() function in pandas/io/parsers/readers.py to handle engine="polars"
  • This helps pandas use polars.read_csv() under the hood and convert the result to a pandas DataFrame using .to_pandas()

✅ Updated Engine Validation

  • Modified _refine_defaults_read() to accept "polars" as a valid engine
  • This ensures pandas doesn’t raise a ValueError when engine="polars" is passed

✅ Created a New Test File

  • Created test_read_csv_polars.py inside pandas/tests/io/parser/
  • The test verifies that using engine="polars" in read_csv() loads a simple CSV correctly
  • Ensures code coverage and prevents future regressions

How to use it?

import pandas as pd

# Requires Polars to be installed
# pip install polars

df = pd.read_csv("example.csv", engine="polars")
print(df.head())

This allows pandas users to benefit from Polars' speed and memory efficiency while still using the familiar pandas API.


Dependencies

Requires the user to have Polars installed:

pip install polars

If polars is not installed, the engine will raise an ImportError with instructions.


Related Issues

Closes #61813


Let me know if any additional tests or validations are needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

ENH: Add Polars engine to read_csv
1 participant