[SPARK-53696][PYTHON][CONNECT][SQL]Default to bytes for BinaryType in PySpark #52467
+654
−85
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| safecheck, | ||
| input_types, | ||
| int_to_decimal_coercion_enabled=False, | ||
| int_to_decimal_coercion_enabled, |
There was a problem hiding this comment.
the default value for int_to_decimal_coercion_enabled is not used at all
xianzhe-databricks
changed the title
HyukjinKwon
reviewed
Sep 29, 2025
python/pyspark/sql/tests/arrow/test_arrow_binary_as_bytes_udf.py
Outdated
Show resolved
Hide resolved
ueshin
reviewed
Sep 29, 2025
xianzhe-databricks
changed the title
where shall I add it? I found the pyspark migration guide is archived https://github.com/apache/spark/blob/master/docs/pyspark-migration-guide.md |
xianzhe-databricks
requested review from
allisonwang-db,
HyukjinKwon and
ueshin
ueshin
reviewed
Oct 1, 2025
| def _get_binary_as_bytes(self) -> bool: | ||
| """Get the binary_as_bytes configuration value from Spark session.""" | ||
| conf_value = self._session.conf.get("spark.sql.execution.pyspark.binaryAsBytes", "true") | ||
| return (conf_value or "true").lower() == "true" |
There was a problem hiding this comment.
Which case is (conf_value or "true") for?
There was a problem hiding this comment.
The method self._session.conf.get returns only an Optional[str]. I think the conf is retrieved from Spark Connect Server. (code pointer)
This way, mypy does not complain about type incompatibility.
sql/core/src/main/scala/org/apache/spark/sql/execution/python/EvaluatePython.scala
Outdated
Show resolved
Hide resolved
xianzhe-databricks
changed the title
xianzhe-databricks
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Currently,
BinaryTypeis mapped inconsistently in PySpark:Cases when it is mapped to
bytearray:Cases when it is mapped to
bytes:This complicates the data mapping model. With this PR,
BinaryTypewill be consistently mapped tobytesin all aforementioned cases.We gate the change with a SQL Conf, and enable the conversion to bytes by default.
This PR is based on #52370
Why are the changes needed?
bytesis more efficient as it is immutable and requires zero copy.Does this PR introduce any user-facing change?
Yes. For the aforementioned cases where
BinaryTypeis mapped tobytearray, we changed the mapping tobytes.How was this patch tested?
Many tests.
Was this patch authored or co-authored using generative AI tooling?
Yes, with the help of claude code.