Skip to content

Switch default string storage from python to pyarrow (if installed) also for NA-variant of the StringDtype #62118

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

Switch default string storage from python to pyarrow (if installed) also for NA-variant of the StringDtype #62118

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
f88f0c5
Switch default string storage from python to pyarrow (if installed) a…
jorisvandenbossche Aug 15, 2025
d01326f
update tests
jorisvandenbossche Aug 15, 2025
64ea6a8
keep string instead of large_string for ArrowDtype
jorisvandenbossche Aug 15, 2025
123c777
update docstrings
jorisvandenbossche Aug 15, 2025
89f8a66
update more tests
jorisvandenbossche Aug 15, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 11 additions & 3 deletions pandas/_libs/parsers.pyx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,10 @@ import warnings

from pandas.util._exceptions import find_stack_level

from pandas import StringDtype
from pandas import (
ArrowDtype,
StringDtype,
)
from pandas.core.arrays import (
ArrowExtensionArray,
BooleanArray,
Expand Down Expand Up @@ -43,7 +46,6 @@ from libc.string cimport (
strncpy,
)


import numpy as np

cimport numpy as cnp
Expand Down Expand Up @@ -1452,7 +1454,13 @@ def _maybe_upcast(

elif arr.dtype == np.object_:
if use_dtype_backend:
dtype = StringDtype()
if dtype_backend == "pyarrow":
# using the StringDtype below would use large_string by default
# keep here to pyarrow's default of string
import pyarrow as pa
dtype = ArrowDtype(pa.string())
else:
dtype = StringDtype()
Comment on lines +1457 to +1463
Copy link
Member Author

@jorisvandenbossche jorisvandenbossche Aug 15, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is one behaviour question we should decide on. Currently, pd.read_csv(..., dtype_backend="pyarrrow") gives you the ArrowDtype using pa.string() (string[pyarrow]) in general, but without the change above, with this PR this now gives you pa.large_string() (large_string[pyarrow]).

The reason for this is because this implementation first goes through creating a column with StringDtype, and then in a next step below converts those to the equivalent ArrowExtensionArray. Before, with "python" storage string dtype, pyarrow converts that to pa.string(), but now with using "pyarrow" storage by default for the StringDtype (which uses pa.large_string()), the pyarrow conversion gives large_string.

But we could also not special case this here in the code, and just update the tests to expect large_string for the c parser

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would support this returning pa.large_string instead of pa.string to align with Patrick's changes in 2.2 #56220.

And generally ArrowExtensionArray._from_sequence should probably return pa.large_string instead of pa.string for a sequence of strings (if it doesn't already)

Copy link
Member

@WillAyd WillAyd Aug 15, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a blocker but I would prefer returning whatever pyarrow provides and not forcing this to a particular type with dtype_backend="pyarrow".

Sure, many users of pandas could probably care less about the differences between string / large_string, and there likely isn't too much over head upcasting the format to the latter in pandas. However, it gets murky when you start thinking about string_view and the larger I/O system. I don't think the pyarrow backend should force string_view to large_string, because that can have non-trivial performance impacts, and if we follow that train of thought it would be inconsistent to cast string to large_string

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm totally ignorant on the differences between string/large_string/string_view/whatever. I lean toward "always give a pd.StringDtype"

Copy link
Member Author

@jorisvandenbossche jorisvandenbossche Aug 16, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I lean toward "always give a pd.StringDtype"

To be clear, this is not about StringDtype .. but about ArrowDtype. If the user asks for dtype_backend="pyarrow", we currently give you a dataframe with all ArrowDtype columns. Unless that is what you would change? (specifically for strings, use StringDtype backed by pyarrow instead of ArrowDtype(string))

EDIT: I see the link to #62129 now, so yes that is what you meant ;)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And generally ArrowExtensionArray._from_sequence should probably return pa.large_string instead of pa.string for a sequence of strings (if it doesn't already)

It does not, because there it just relies on the type inference of pyarrow, and pyarrow will always prefer pa.string() over pa.large_string() when inferring (unless the data source has specific data type information).

And I think I agree with Will's comment about preferring to rely on what pyarrow gives, and not forcing specific types, specifically for ArrowDtype (but no strong opinion here)

The reason we get large_string here is because for StringDtype, we do make a very specific choice on our side to use large_string instead of string. And at that point of course pyarrow preserves that choice when converting to a pyarrow array.

cls = dtype.construct_array_type()
arr = cls._from_sequence(arr, dtype=dtype)

Expand Down
27 changes: 11 additions & 16 deletions pandas/core/arrays/string_.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -126,10 +126,10 @@ class StringDtype(StorageExtensionDtype):
Examples
--------
>>> pd.StringDtype()
<StringDtype(storage='python', na_value=<NA>)>

>>> pd.StringDtype(storage="pyarrow")
<StringDtype(na_value=<NA>)>

>>> pd.StringDtype(storage="python")
<StringDtype(storage='python', na_value=<NA>)>
"""

@property
Expand All @@ -154,16 +154,11 @@ def __init__(
) -> None:
# infer defaults
if storage is None:
if na_value is not libmissing.NA:
storage = get_option("mode.string_storage")
if storage == "auto":
if HAS_PYARROW:
storage = "pyarrow"
else:
storage = "python"
else:
storage = get_option("mode.string_storage")
if storage == "auto":
storage = get_option("mode.string_storage")
if storage == "auto":
if HAS_PYARROW:
storage = "pyarrow"
else:
storage = "python"

if storage == "pyarrow_numpy":
Expand Down Expand Up @@ -617,7 +612,7 @@ class StringArray(BaseStringArray, NumpyExtensionArray): # type: ignore[misc]
Examples
--------
>>> pd.array(["This is", "some text", None, "data."], dtype="string")
<StringArray>
<ArrowStringArray>
['This is', 'some text', <NA>, 'data.']
Length: 4, dtype: string

Expand All @@ -629,15 +624,15 @@ class StringArray(BaseStringArray, NumpyExtensionArray): # type: ignore[misc]
['1', 1]
Length: 2, dtype: object
>>> pd.array(["1", 1], dtype="string")
<StringArray>
<ArrowStringArray>
['1', '1']
Length: 2, dtype: string

However, instantiating StringArrays directly with non-strings will raise an error.

For comparison methods, `StringArray` returns a :class:`pandas.BooleanArray`:

>>> pd.array(["a", None, "c"], dtype="string") == "a"
>>> pd.array(["a", None, "c"], dtype="string[python]") == "a"
<BooleanArray>
[True, <NA>, False]
Length: 3, dtype: boolean
Expand Down
6 changes: 3 additions & 3 deletions pandas/core/construction.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -230,14 +230,14 @@ def array(
Length: 2, dtype: Float64

>>> pd.array(["a", None, "c"])
<StringArray>
<ArrowStringArray>
['a', <NA>, 'c']
Length: 3, dtype: string

>>> with pd.option_context("string_storage", "pyarrow"):
>>> with pd.option_context("string_storage", "python"):
... arr = pd.array(["a", None, "c"])
>>> arr
<ArrowStringArray>
<StringArray>
['a', <NA>, 'c']
Length: 3, dtype: string

Expand Down
4 changes: 1 addition & 3 deletions pandas/tests/arrays/categorical/test_constructors.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -736,9 +736,7 @@ def test_interval(self):

def test_categorical_extension_array_nullable(self, nulls_fixture):
# GH:
arr = pd.arrays.StringArray._from_sequence(
[nulls_fixture] * 2, dtype=pd.StringDtype()
)
arr = pd.array([nulls_fixture] * 2, dtype=pd.StringDtype())
result = Categorical(arr)
assert arr.dtype == result.categories.dtype
expected = Categorical(Series([pd.NA, pd.NA], dtype=arr.dtype))
Expand Down
5 changes: 3 additions & 2 deletions pandas/tests/arrays/string_/test_string_arrow.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
import numpy as np
import pytest

from pandas.compat import HAS_PYARROW
import pandas.util._test_decorators as td

import pandas as pd
Expand All @@ -26,10 +27,10 @@ def test_eq_all_na():
tm.assert_extension_array_equal(result, expected)


def test_config(string_storage, using_infer_string):
def test_config(string_storage):
# with the default string_storage setting
# always "python" at the moment
assert StringDtype().storage == "python"
assert StringDtype().storage == "pyarrow" if HAS_PYARROW else "python"

with pd.option_context("string_storage", string_storage):
assert StringDtype().storage == string_storage
Expand Down
2 changes: 1 addition & 1 deletion pandas/tests/copy_view/test_array.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -128,7 +128,7 @@ def test_dataframe_array_ea_dtypes():


def test_dataframe_array_string_dtype():
df = DataFrame({"a": ["a", "b"]}, dtype="string")
df = DataFrame({"a": ["a", "b"]}, dtype="string[python]")
arr = np.asarray(df)
assert np.shares_memory(arr, get_array(df, "a"))
assert arr.flags.writeable is False
Expand Down
8 changes: 3 additions & 5 deletions pandas/tests/copy_view/test_astype.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,7 @@ def test_astype_numpy_to_ea():


@pytest.mark.parametrize(
"dtype, new_dtype", [("object", "string"), ("string", "object")]
"dtype, new_dtype", [("object", "string[python]"), ("string[python]", "object")]
)
def test_astype_string_and_object(dtype, new_dtype):
df = DataFrame({"a": ["a", "b", "c"]}, dtype=dtype)
Expand All @@ -96,7 +96,7 @@ def test_astype_string_and_object(dtype, new_dtype):


@pytest.mark.parametrize(
"dtype, new_dtype", [("object", "string"), ("string", "object")]
"dtype, new_dtype", [("object", "string[python]"), ("string[python]", "object")]
)
def test_astype_string_and_object_update_original(dtype, new_dtype):
df = DataFrame({"a": ["a", "b", "c"]}, dtype=dtype)
Expand Down Expand Up @@ -224,9 +224,7 @@ def test_convert_dtypes(using_infer_string):
df_orig = df.copy()
df2 = df.convert_dtypes()

if using_infer_string and HAS_PYARROW:
# TODO the default nullable string dtype still uses python storage
# this should be changed to pyarrow if installed
if HAS_PYARROW:
assert not tm.shares_memory(get_array(df2, "a"), get_array(df, "a"))
else:
assert tm.shares_memory(get_array(df2, "a"), get_array(df, "a"))
Expand Down
2 changes: 1 addition & 1 deletion pandas/tests/dtypes/test_common.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -117,7 +117,7 @@ def test_period_dtype(self, dtype):
"float": np.dtype(np.float64),
"object": np.dtype(object),
"category": com.pandas_dtype("category"),
"string": pd.StringDtype(),
"string": pd.StringDtype("python"),
}


Expand Down
4 changes: 2 additions & 2 deletions pandas/tests/frame/methods/test_convert_dtypes.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -199,7 +199,7 @@ def test_convert_dtypes_avoid_block_splitting(self):
{
"a": [1, 2, 3],
"b": [4, 5, 6],
"c": pd.Series(["a"] * 3, dtype="string[python]"),
"c": pd.Series(["a"] * 3, dtype="string"),
}
)
tm.assert_frame_equal(result, expected)
Expand All @@ -209,7 +209,7 @@ def test_convert_dtypes_from_arrow(self):
# GH#56581
df = pd.DataFrame([["a", datetime.time(18, 12)]], columns=["a", "b"])
result = df.convert_dtypes()
expected = df.astype({"a": "string[python]"})
expected = df.astype({"a": "string"})
tm.assert_frame_equal(result, expected)

def test_convert_dtype_pyarrow_timezone_preserve(self):
Expand Down
4 changes: 4 additions & 0 deletions pandas/tests/io/excel/test_readers.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -657,6 +657,10 @@ def test_dtype_backend(self, read_ext, dtype_backend, engine, tmp_excel):
for col in df.columns
}
)

# pandas uses large_string by default, but pyarrow infers string
expected["d"] = expected["d"].astype(pd.ArrowDtype(pa.string()))
expected["h"] = expected["h"].astype(pd.ArrowDtype(pa.string()))
# pyarrow by default infers timestamp resolution as us, not ns
expected["i"] = ArrowExtensionArray(
expected["i"].array._pa_array.cast(pa.timestamp(unit="us"))
Expand Down
6 changes: 1 addition & 5 deletions pandas/tests/io/test_clipboard.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -350,11 +350,7 @@ def test_read_clipboard_dtype_backend(
# GH#50502
if dtype_backend == "pyarrow":
pa = pytest.importorskip("pyarrow")
if engine == "c" and string_storage == "pyarrow":
# TODO avoid this exception?
string_dtype = pd.ArrowDtype(pa.large_string())
else:
string_dtype = pd.ArrowDtype(pa.string())
string_dtype = pd.ArrowDtype(pa.string())
else:
string_dtype = pd.StringDtype(string_storage)

Expand Down
11 changes: 3 additions & 8 deletions pandas/tests/io/test_orc.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@
import pandas as pd
from pandas import read_orc
import pandas._testing as tm
from pandas.core.arrays import StringArray

pytest.importorskip("pyarrow.orc")

Expand Down Expand Up @@ -368,13 +367,9 @@ def test_orc_dtype_backend_numpy_nullable():

expected = pd.DataFrame(
{
"string": StringArray(np.array(["a", "b", "c"], dtype=np.object_)),
"string_with_nan": StringArray(
np.array(["a", pd.NA, "c"], dtype=np.object_)
),
"string_with_none": StringArray(
np.array(["a", pd.NA, "c"], dtype=np.object_)
),
"string": pd.array(np.array(["a", "b", "c"], dtype=np.object_)),
"string_with_nan": pd.array(np.array(["a", pd.NA, "c"], dtype=np.object_)),
"string_with_none": pd.array(np.array(["a", pd.NA, "c"], dtype=np.object_)),
"int": pd.Series([1, 2, 3], dtype="Int64"),
"int_with_nan": pd.Series([1, pd.NA, 3], dtype="Int64"),
"na_only": pd.Series([pd.NA, pd.NA, pd.NA], dtype="Int64"),
Expand Down
4 changes: 3 additions & 1 deletion pandas/tests/series/test_constructors.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2137,7 +2137,9 @@ def test_series_string_inference_storage_definition(self):
# but after PDEP-14 (string dtype), it was decided to keep dtype="string"
# returning the NA string dtype, so expected is changed from
# "string[pyarrow_numpy]" to "string[python]"
expected = Series(["a", "b"], dtype="string[python]")
expected = Series(
["a", "b"], dtype="string[pyarrow]" if HAS_PYARROW else "string[python]"
)
with pd.option_context("future.infer_string", True):
result = Series(["a", "b"], dtype="string")
tm.assert_series_equal(result, expected)
Expand Down
Loading