[mypyc] feat: cache len for iterating over immutable types and expressions with length known at compile time #19503
+1,098
−149
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This comment was marked as outdated.
This comment was marked as outdated.
BobTheBuidler
changed the title
2 tasks
|
@ilevkivskyi this one is also ready for review if and when you get a chance, though its definitely a bit more involved than #19497 |
BobTheBuidler
force-pushed
the
for-loop-len-cache
branch
from
07f40fc to
622f38f
Compare
BobTheBuidler
commented
Aug 3, 2025
| @@ -1147,3 +1187,33 @@ def gen_step(self) -> None: | |||
| def gen_cleanup(self) -> None: | |||
| for gen in self.gens: | |||
| gen.gen_cleanup() | |||
|
|
|||
|
|
|||
| def get_expr_length(expr: Expression) -> int | None: | |||
There was a problem hiding this comment.
These 2 helper functions can be extended to cover more cases and used for other length-based optimizations I have in mind
BobTheBuidler
changed the title
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
BobTheBuidler
force-pushed
the
for-loop-len-cache
branch
from
904945c to
537c8af
Compare
|
This PR has mixed results with self-check, but I'm not sure if self-check is a reliable benchmark for this particular set of changes. It's a pretty niche-case optimization_ |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently, if a user uses an immutable type as the sequence input for a for loop, the length is checked once at each iteration which, while necessary for some container types such as list and dictionaries, is not necessary for iterating over immutable types tuple, str, and bytes.
This PR modifies the codebase such that the length is only checked at the first iteration, and reused from there.
Also, in cases where a simple genexp is the input argument for a tuple, the length is currently checked one additional time before entering the iteration (this is done to determine how to size the new tuple). In those cases, we don't even need a length check at the first iteration step, and can reuse the result of that first
lencall (or compile-time determined constant) instead.Lastly, in cases where a tuple is created from a genexp and the length of the genexp is knowable at compile time, this PR replaces PyList_AsTuple with the tuple constructor fast-path.