feat(QA-200): Automated pythonindex package tests

.gitignore

@@ -18,3 +18,4 @@ __pycache__
 /pkgs/*.build
 /pkgs/*.prepared
 /pkgs/*.sysroot
+wasix-site-packages

.python-version

@@ -0,0 +1 @@
+3.13

README.md

@@ -84,10 +84,10 @@ Buildscripts to build numpy and other wheels for wasix. For convenience, this pa
 
 The build script is controlled by the following environment variables:
 
-* `CC`, `CXX`, `AR`, `LD`, `RANLIB`, etc... : The cross-compiler tools. These should all be normal clang tools, but target wasm32-wasix by default and use the wasix sysroot.
-* `WASIX_SYSROOT`: The path to the wasix sysroot that is used by the toolchain. Libraries will get installed here when you run `make install` or when they are required to build a package.
-* `INSTALL_DIR`: The path to the python library path. Wheels will get installed here when you run `make install`.
-* `WASMER`: The path to the wasmer binary. You must have it registered to handle wasm files as binfmt_misc. You can do this with `sudo $WASMER binfmt reregister`.
+- `CC`, `CXX`, `AR`, `LD`, `RANLIB`, etc... : The cross-compiler tools. These should all be normal clang tools, but target wasm32-wasix by default and use the wasix sysroot.
+- `WASIX_SYSROOT`: The path to the wasix sysroot that is used by the toolchain. Libraries will get installed here when you run `make install` or when they are required to build a package.
+- `INSTALL_DIR`: The path to the python library path. Wheels will get installed here when you run `make install`.
+- `WASMER`: The path to the wasmer binary. You must have it registered to handle wasm files as binfmt_misc. You can do this with `sudo $WASMER binfmt reregister`.
 
 The easiest way to setup all the environment variables is to activate the wasix-clang environment using `source wasix-clang/activate`.
 
@@ -154,111 +154,149 @@ Here is a list of the versions of the wheels and libraries that are included in
 
 #### Wheels
 
-* numpy: numpy/numpy main
-* markupsafe: 3.0.2
-* pandas: 2.3.2
-* pytz: 2025.2
-* dateutil: 2.9.0
-* tzdata: 2025.2
-* six: 1.17.0
-* msgpack: 1.1.0
-* pycryptodome: 3.23.0
-* pycryptodomex: 3.23.0
-* pyzbar: 0.1.9
-* cpython: 3.1.2
-* pypandoc: 1.15
-* pypandoc_binary: 1.15
-* psycopg: 3.2.9
-* psycopg-binary: 3.2.9
-* psycopg-pool: pool-3.2.6
-* brotlicffi: 1.1.0.0
-* cffi: 1.17.1
-* pillow: 11.3.0
-* matplotlib: 3.10.6
-* uvloop: 0.21.0
-* mysqlclient: 2.2.7
-* python-qrcode: 8.2
-* pycparser: 2.22
-* pydantic: 2.11.7
-* typing_extensions: 4.14.1
-* typing-inspection: 0.4.1
-* annotated-types: 0.7.0
-* shapely: 2.1.1
-* mrab-regex: 2025.7.31
-* lxml: 6.0.0
-* protobuf: 31.1
-* grpc: 1.74.1
-* numpy: 1.26.5
-* numpy: 2.0.2
-* numpy: 2.3.2
-* python-crc32c: 1.7.1
-* requests: 2.32.4
-* urllib3: 2.5.0
-* idna: 3.10
-* certifi: 2025.08.03
-* charset-normalizer: 3.4.3
-* pypng: 0.20250521.0
-* pyarrow: 19.0.1
-* pyarrow: 21.0.0
-* packaging: 25.0
-* pyparsing: 3.2.3
-* cycler: 0.12.1
-* kiwisolver: 1.4.9
-* contourpy: 1.3.3
-* pyopenssl: 25.1.0
-* aspw: 3.50.4.0
+- numpy: numpy/numpy main
+- markupsafe: 3.0.2
+- pandas: 2.3.2
+- pytz: 2025.2
+- dateutil: 2.9.0
+- tzdata: 2025.2
+- six: 1.17.0
+- msgpack: 1.1.0
+- pycryptodome: 3.23.0
+- pycryptodomex: 3.23.0
+- pyzbar: 0.1.9
+- cpython: 3.1.2
+- pypandoc: 1.15
+- pypandoc_binary: 1.15
+- psycopg: 3.2.9
+- psycopg-binary: 3.2.9
+- psycopg-pool: pool-3.2.6
+- brotlicffi: 1.1.0.0
+- cffi: 1.17.1
+- pillow: 11.3.0
+- matplotlib: 3.10.6
+- uvloop: 0.21.0
+- mysqlclient: 2.2.7
+- python-qrcode: 8.2
+- pycparser: 2.22
+- pydantic: 2.11.7
+- typing_extensions: 4.14.1
+- typing-inspection: 0.4.1
+- annotated-types: 0.7.0
+- shapely: 2.1.1
+- mrab-regex: 2025.7.31
+- lxml: 6.0.0
+- protobuf: 31.1
+- grpc: 1.74.1
+- numpy: 1.26.5
+- numpy: 2.0.2
+- numpy: 2.3.2
+- python-crc32c: 1.7.1
+- requests: 2.32.4
+- urllib3: 2.5.0
+- idna: 3.10
+- certifi: 2025.08.03
+- charset-normalizer: 3.4.3
+- pypng: 0.20250521.0
+- pyarrow: 19.0.1
+- pyarrow: 21.0.0
+- packaging: 25.0
+- pyparsing: 3.2.3
+- cycler: 0.12.1
+- kiwisolver: 1.4.9
+- contourpy: 1.3.3
+- pyopenssl: 25.1.0
+- aspw: 3.50.4.0
 <!-- WHEEL_VERSIONS_END -->
 
-psycopg3-c is just the sdist of psycopg3-binary
+psycopg3-c is just the sdist of psycopg3-binary## Running Tests
 
 #### Libraries
 
-* libzbar: 0.23.93
-* libffi: wasix-org/libffi main
-* pandoc: haskell-wasm/pandoc wasm
-* postgresql: 17.5
-* brotli: 1.1.0
-* zlib: develop
-  * 1.3.1 does not have proper cmake support, so we are using develop for now
-* libjpeg-turbo: 3.1.1
-* xz: 5.8.1
-* libtiff: 4.7.0
-* libwebp: 1.5.0
-* giflib: 5.2.2
-* libpng: 1.6.50
-* SDL: 3.2.16
-  * SDL has all subsystems disabled
-* openjpeg: 2.5.3
-* libuv: 1.51.0
-* mariadb-connector-c: 3.4.6
-* openssl: 3.5.1
-* bzip2: 1.0.8
-* util-linux: 2.41.1
-  * We only build libuuid from util-linux
-* openssh: 10.0p2
-* dropbear: 2025.88
-* tinyxml2: 11.0.0
-* geos: 3.13.1
-* libxslt: 1.1.43
-* libxml2: 2.14.5
-* google-crc32c: 1.1.2
-* arrow: 19.0.1
-* arrow: 21.0.0
-* rapidjson: 1.1.0
-* icu: 77.1
-* readline: 8.2
-* ncurses: 6.4.20230225
-* curl: curl/curl ab18c04218ff316cd67b1e928c5cee579b2f66a0
-  * This was the current commit in the wasix fork. We can probably update to the next release
-* pycurl: 7.45.6
-* sqlite: 3.50.4
-* wasix-libc: wasix-org/wasix-libc e4e5cfcf1009e690207821e9b3a63cdce2ee9e86
-  * The commit before we removed fork. A few packages don't build if fork is not defined.
-* libcxx: wasix-org/llvm-project 6bb93a243f6d15855f485f5aec3810d9e2de150d
-  * This is the commit that is currently used by wasix-libc
-* compiler-rt: wasix-org/llvm-project 6bb93a243f6d15855f485f5aec3810d9e2de150d
-  * This is the commit that is currently used by wasix-libc
-<!-- LIB_VERSIONS_END -->
+- libzbar: 0.23.93
+- libffi: wasix-org/libffi main
+- pandoc: haskell-wasm/pandoc wasm
+- postgresql: 17.5
+- brotli: 1.1.0
+- zlib: develop
+  - 1.3.1 does not have proper cmake support, so we are using develop for now
+- libjpeg-turbo: 3.1.1
+- xz: 5.8.1
+- libtiff: 4.7.0
+- libwebp: 1.5.0
+- giflib: 5.2.2
+- libpng: 1.6.50
+- SDL: 3.2.16
+  - SDL has all subsystems disabled
+- openjpeg: 2.5.3
+- libuv: 1.51.0
+- mariadb-connector-c: 3.4.6
+- openssl: 3.5.1
+- bzip2: 1.0.8
+- util-linux: 2.41.1
+  - We only build libuuid from util-linux
+- openssh: 10.0p2
+- dropbear: 2025.88
+- tinyxml2: 11.0.0
+- geos: 3.13.1
+- libxslt: 1.1.43
+- libxml2: 2.14.5
+- google-crc32c: 1.1.2
+- arrow: 19.0.1
+- arrow: 21.0.0
+- rapidjson: 1.1.0
+- icu: 77.1
+- readline: 8.2
+- ncurses: 6.4.20230225
+- curl: curl/curl ab18c04218ff316cd67b1e928c5cee579b2f66a0
+  - This was the current commit in the wasix fork. We can probably update to the next release
+- pycurl: 7.45.6
+- sqlite: 3.50.4
+- wasix-libc: wasix-org/wasix-libc e4e5cfcf1009e690207821e9b3a63cdce2ee9e86
+  - The commit before we removed fork. A few packages don't build if fork is not defined.
+- libcxx: wasix-org/llvm-project 6bb93a243f6d15855f485f5aec3810d9e2de150d
+  - This is the commit that is currently used by wasix-libc
+- compiler-rt: wasix-org/llvm-project 6bb93a243f6d15855f485f5aec3810d9e2de150d
+  - This is the commit that is currently used by wasix-libc
+  <!-- LIB_VERSIONS_END -->
+
+## Tests
+
+The whole point of these tests is to ensure that the packages works within the wasmer runtime.
+So there are two ways of running the tests, natively and with wasmer:
+
+### Native
+
+Requirements:
+
+- Python 3.10+
+- `python -m venv env`
+- `source venv/bin/activate`
+- `pip install .`
+- `pip install pytest`
+
+I'm sure it's possible to do via uv, or poetry, or many other ways. But this works also.
+
+Run tests:
+
+- `python -m pytest`
+- Discovers files matching `*-test.py`, `*_test.py`, and `test_*.py` under `tests/`.
+- Files ending with `.skip.py` are ignored.
+- Files ending with `-broken.py` are marked as expected failures (strict). If they pass, the run reports XPASS and fails.
+
+### Wasmer
+
+These are the instructions as of 2025-10, but there's a shipit looming, which may optimize the flow.
+If this doesn't work though, we have a bug:
+
+Requirements:
+
+- wasmer 6.1.0-rc.3+
+
+Run tests:
+
+- `uv pip compile pyproject.toml --python-version=3.13 --universal --extra-index-url https://pythonindex.wasix.org/simple --index-url=https://pypi.org/simple --emit-index-url --only-binary :all: -o wasmer-requirements.txt`
+- `uvx pip install -r wasmer-requirements.txt --target wasix-site-packages --platform wasix_wasm32 --only-binary=:all: --python-version=3.13 --compile`
 
 ### Notes
 
@@ -286,7 +324,7 @@ to check which python libraries depend on shared libs. We try to keep that to a
 
 ### Structure
 
-<!-- 
+<!--
 There is the pkgs folder that contains most stuff
 
 For each project that can be built there are multiple files depending on the type.
@@ -312,52 +350,52 @@ TODO: Make this more understandable
 
 Inside the pkgs/ folder there can be the following directories:
 
-* `*.source`: clean submodule checkout
-* `*.prepared`: patched worktree of source
-* `*.build`: temporary build directory
-* `*.tar.gz`: python sdist
-* `*.sdist`: unpacked python sdist
-* `*.whl`: compiled python wheel
-* `*.wheel`: unpacked python wheel
-* `*.lib`: unpacked library/application
-* `*.tar.xz`: packed library/application
+- `*.source`: clean submodule checkout
+- `*.prepared`: patched worktree of source
+- `*.build`: temporary build directory
+- `*.tar.gz`: python sdist
+- `*.sdist`: unpacked python sdist
+- `*.whl`: compiled python wheel
+- `*.wheel`: unpacked python wheel
+- `*.lib`: unpacked library/application
+- `*.tar.xz`: packed library/application
 
 #### Base structure
 
 Each project follows a consistent flow through the first three main directories.
 
-* `*.source`
-  * This is a clean checkout of the project's upstream source code, tracked as a git submodule.
-  * We avoid modifying this directly, since changes here would slow down git operations in the build-scripts repo.
-* `*.prepared`
-  * A git worktree created from the `*.source` repository.
-  * If patches are needed, they're applied here.
-  * If no patches are needed, it's just a clean mirror of the source.
-  * This directory is persistent and only refreshed if the source changes so new patches can be developed in this directory
-* `*.build`
-  * A copy of the `*.prepared` directory, used for the actual build step.
-  * Contains all intermediate build artifacts.
-  * This directory is temporary and may be deleted between builds. Never make manual changes here.
+- `*.source`
+  - This is a clean checkout of the project's upstream source code, tracked as a git submodule.
+  - We avoid modifying this directly, since changes here would slow down git operations in the build-scripts repo.
+- `*.prepared`
+  - A git worktree created from the `*.source` repository.
+  - If patches are needed, they're applied here.
+  - If no patches are needed, it's just a clean mirror of the source.
+  - This directory is persistent and only refreshed if the source changes so new patches can be developed in this directory
+- `*.build`
+  - A copy of the `*.prepared` directory, used for the actual build step.
+  - Contains all intermediate build artifacts.
+  - This directory is temporary and may be deleted between builds. Never make manual changes here.
 
 The remaining steps are different depending on the type of project.
 
 #### Python modules
 
-* The build step creates a `*.tar.gz` sdist from the `*.build` directory.
-* The sdist is then extracted into a `*.sdist` folder.
-* Finally, a wheel (`*.whl`) is built from the `*.sdist`.
-* If you want to you can make a `*.wheel` directory to view the unpacked wheel
+- The build step creates a `*.tar.gz` sdist from the `*.build` directory.
+- The sdist is then extracted into a `*.sdist` folder.
+- Finally, a wheel (`*.whl`) is built from the `*.sdist`.
+- If you want to you can make a `*.wheel` directory to view the unpacked wheel
 
 #### WASIX libraries and applications
 
-* The build step builds the library and installs it into a `*.lib` folder, following the correct directory structure.
-* That folder is then compressed into a final distributable *.tar.xz.
+- The build step builds the library and installs it into a `*.lib` folder, following the correct directory structure.
+- That folder is then compressed into a final distributable \*.tar.xz.
 
 #### Interdependencies
 
 If a project depends on other project they can either be direct dependencies of that project or you can define a `*.sysroot` target with the dependencies as prerequisites.
 
-* `*.sysroot`
-  * Contains the merged builds of multiple other projects
-  * Useful when a project is using pkg-config to find its dependencies
-  * Automatically builds a sysroot from its list of prerequisites
+- `*.sysroot`
+  - Contains the merged builds of multiple other projects
+  - Useful when a project is using pkg-config to find its dependencies
+  - Automatically builds a sysroot from its list of prerequisites