[WIP][TESTS] Upgrade pypy to 3.11

[zhengruifeng]

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

[zhengruifeng]

The daily job keeps getting stuck in streaming tests, and after upgrading to pypy 3.11, they no longer hang, but started to fail.

given that:
1, 10 out of 18 pyspark testing modules are already skipped with excluded_python_implementations in modules.py;
2, there are extra skips with "pypy" in platform.python_implementation().lower() in tests like udtf, dstream, etc.

The test coverage for pypy is low

I am wondering shall we just drop the support of pypy? @HyukjinKwon @dongjoon-hyun @LuciferYang

[dongjoon-hyun]

That's too bad, @zhengruifeng .

• I'm wondering PySpark on CPython performance is enough to cover the loss of PyPy.
• In addition, did we file the JIRA issues about the failures to give more visibility to the community? Without JIRA issue ID, we cannot discuss. Before dropping, it would be great to inform the issues properly in the community.

[zhengruifeng]

@dongjoon-hyun Ok, I will just focus on fixing the workflow for now.
And will raise the discussion on dropping pypy later in the mail list.

[dongjoon-hyun]

Thank you!

[zhengruifeng]

@dongjoon-hyun @HyukjinKwon it seems almost all remaining tests are broken

• with pypy3.10 in [SPARK-53254][PYTHON][TESTS] Skip hanging tests in PyPy daily workflow #51983, they get stuck;
• with pypy3.11 in this PR, they fail

one failure seems related to pickle that the protocol between JVM and Python is broken

Caused by: org.apache.spark.api.python.PythonException: Traceback (most recent call last):
  File "/__w/spark/spark/python/lib/pyspark.zip/pyspark/worker.py", line 2868, in main
    func, profiler, deserializer, serializer = read_command(pickleSer, infile)
                                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/__w/spark/spark/python/lib/pyspark.zip/pyspark/worker_util.py", line 64, in read_command
    command = serializer._read_with_length(file)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/__w/spark/spark/python/lib/pyspark.zip/pyspark/serializers.py", line 173, in _read_with_length
    return self.loads(obj)
           ^^^^^^^^^^^^^^^
  File "/__w/spark/spark/python/lib/pyspark.zip/pyspark/serializers.py", line 473, in loads
    return cloudpickle.loads(obj, encoding=encoding)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/pypy/pypy3.11/lib/pypy3.11/pickle.py", line 1785, in _loads
    return _Unpickler(file, fix_imports=fix_imports, buffers=buffers,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/pypy/pypy3.11/lib/pypy3.11/pickle.py", line 1226, in load
    dispatch[key[0]](self)
  File "/usr/local/pypy/pypy3.11/lib/pypy3.11/pickle.py", line 1603, in load_reduce
    stack[-1] = func(*args)
                ^^^^^^^^^^^
TypeError: expected str, got int object

[zhengruifeng]

I checked the last successful run at Jun 25 and the first failure at Jun 27, the package versions are the same.

And the commits between Jun 25 and Jun 27, haven't find a suspicious one.

[dongjoon-hyun]

I merged the following to help you.

• [SPARK-53271][PYTHON][INFRA] Show Python Versions in PySpark Jobs #52013