request: Pyodide support for `polars`; Use `polars` in shinylive examples

[schloerke]

Related:

• Polars built in pyodide ? pola-rs/polars#3672
• Package request: polars pyodide/pyodide#4611

Goal: Display an example app using polars on shinylive (wasm / pyodide).

Failing reprex: link

Error:

>>> Installing polars ...
Traceback (most recent call last):
  File "<exec>", line 369, in _start_app
  File "<exec>", line 302, in _install_requirements_from_dir
  File "/lib/python3.12/site-packages/micropip/_commands/install.py", line 142, in install
    await transaction.gather_requirements(requirements)
  File "/lib/python3.12/site-packages/micropip/transaction.py", line 55, in gather_requirements
    await asyncio.gather(*requirement_promises)
  File "/lib/python3.12/site-packages/micropip/transaction.py", line 62, in add_requirement
    return await self.add_requirement_inner(Requirement(req))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lib/python3.12/site-packages/micropip/transaction.py", line 151, in add_requirement_inner
    await self._add_requirement_from_package_index(req)
  File "/lib/python3.12/site-packages/micropip/transaction.py", line 190, in _add_requirement_from_package_index
    wheel = find_wheel(metadata, req)
            ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lib/python3.12/site-packages/micropip/transaction.py", line 281, in find_wheel
    raise ValueError(
ValueError: Can't find a pure Python 3 wheel for 'polars'.
See: https://pyodide.org/en/stable/usage/faq.html#why-can-t-micropip-find-a-pure-python-wheel-for-a-package
You can use `await micropip.install(..., keep_going=True)` to get a list of all packages with missing wheels.

---

Shiny uses shinylive to display its examples in the browser (via pyodide) without a central compute server. It would be nice to have polars in these examples. However, polars currently does not compile for pyodide due to threading.

Talking with @georgestagg , he believes we can follow pattern of huggingface/tokenizers's flag where they turn off threads given an envvar value (or if we could determine if the target is wasm).

If a fix is found, this could be a possible technical blog post?

---

Made issue for tracking. Will pick back up after posit::conf(2024). The fix will most likely need to be in the pola-rs/polars repo.

cc @gadenbuie @wch

[georgestagg]

Sneak preview:

This currently requires patches to LLVM's wasm-ld & Rust's rustc, so that the inventory crate can be made to work with Wasm. With those patches, plus a bunch of stubbing out of currently problematic features (e.g. cloud, aws, a bunch more...), the core of polars can be built for Pyodide.

I will attempt to re-enable as many features as I can, then write things up in a document/blog post.

Later, we can start thinking about merging PRs to LLVM and inventory. If the required patches turn out to be no good, we could still create a Docker image with a custom compiler suite to build the polars wheel, and distribute the resulting binary with shinylive (I do this for webR system libraries and it works mostly OK).

[georgestagg]

Pyodide wheels are now being built for Polars and v1.18.0 is available at https://github.com/pola-rs/polars/releases/tag/py-1.18.0.

[kv9898]

Seems like they removed polars from the latest version (0.28.0) due to build issues: https://pyodide.org/en/stable/project/changelog.html