Skip to content

Use of LazyFrames for Polars implementation #52

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
adamamer20 opened this issue Aug 3, 2024 · 2 comments · May be fixed by #144
Open

Use of LazyFrames for Polars implementation #52

adamamer20 opened this issue Aug 3, 2024 · 2 comments · May be fixed by #144
Labels
enhancement Improvements to existing features or performance.
Milestone

Comments

@adamamer20
Copy link
Member

adamamer20 commented Aug 3, 2024

Polars has a lazy API, which allows for query optimization. Currently, AgentSetPolars and PolarsMixins are implemented with the eager API, which may lead to a significant performance loss. Collection could be done at each step.

References:

@adamamer20 adamamer20 added the enhancement Improvements to existing features or performance. label Aug 3, 2024
@adamamer20
Copy link
Member Author

Closed in favour of Ibis (#89).

@adamamer20 adamamer20 closed this as not planned Won't fix, can't repro, duplicate, stale Oct 1, 2024
@adamamer20
Copy link
Member Author

I am reopening the issue as converting everything in ibis takes a significant effort and:

  1. the performance of duckdb isn't far from lazy polars
  2. polars can now run on GPU
  3. one can use narwhals to achieve ibis functionality

@adamamer20 adamamer20 reopened this Mar 20, 2025
@adamamer20 adamamer20 added this to the 0.1.0-stable milestone Mar 20, 2025
@adamamer20 adamamer20 changed the title Use/Addition of LazyFrames for Polars implementation Use of LazyFrames for Polars implementation Apr 3, 2025
@adamamer20 adamamer20 linked a pull request Apr 4, 2025 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Improvements to existing features or performance.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant