Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add GPU support to the Polars LazyFrame profiler #17723

Draft
wants to merge 1 commit into
base: branch-25.02
Choose a base branch
from

Conversation

Matt711
Copy link
Contributor

@Matt711 Matt711 commented Jan 13, 2025

Description

Closes #16224. Depends on pola-rs/polars#20693

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@Matt711 Matt711 added feature request New feature or request non-breaking Non-breaking change labels Jan 13, 2025
Copy link

copy-pr-bot bot commented Jan 13, 2025

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@github-actions github-actions bot added Python Affects Python cuDF API. cudf.polars Issues specific to cudf.polars labels Jan 13, 2025
@Matt711
Copy link
Contributor Author

Matt711 commented Jan 13, 2025

This is what the profile would looks like.
TODO

  • report the time in us
  • time each node separately
In [1]: import polars as pl

In [2]: lf = pl.LazyFrame(
   ...:     {
   ...:         "a": ["a", "b", "a", "b", "b", "c"],
   ...:         "b": [1, 2, 3, 4, 5, 6],
   ...:         "c": [6, 5, 4, 3, 2, 1],
   ...:     }
   ...: )
   ...: lf.group_by("a", maintain_order=True).agg(pl.all().sum()).sort(
   ...:     "a"
   ...: ).profile(engine="gpu")
Out[2]: 
(shape: (3, 3)
 ┌─────┬─────┬─────┐
 │ abc   │
 │ --------- │
 │ stri64i64 │
 ╞═════╪═════╪═════╡
 │ a410  │
 │ b1110  │
 │ c61   │
 └─────┴─────┴─────┘,
 shape: (2, 3)
 ┌──────────────┬───────────────┬───────────────┐
 │ nodestartend           │
 │ ---------           │
 │ stru64u64           │
 ╞══════════════╪═══════════════╪═══════════════╡
 │ optimization05700362213737 │
 │ GPU callback57003622137375700362795311 │
 └──────────────┴───────────────┴───────────────┘)

@@ -84,7 +84,7 @@ def translate_ir(self, *, n: int | None = None) -> ir.IR:
# IR is versioned with major.minor, minor is bumped for backwards
# compatible changes (e.g. adding new nodes), major is bumped for
# incompatible changes (e.g. renaming nodes).
if (version := self.visitor.version()) >= (4, 3):
if (version := self.visitor.version()) >= (5, 1):
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ignore, will delete

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cudf.polars Issues specific to cudf.polars feature request New feature or request non-breaking Non-breaking change Python Affects Python cuDF API.
Projects
Status: In Progress
Development

Successfully merging this pull request may close these issues.

[FEA] Support GPU execution engine in LazyFrame profiler
1 participant