new versions have no source distributions or arm64 wheels #601

ViviMine · 2025-01-24T15:59:59Z

Hey,
I'm sorry if this is not really precise but I'm helped by someone else there
I tried to install pyxel on an arm64 but a lot of things goes wrong. First thing is that it only find the 1.8 version, I was told that I thought first it was because of missing thing like Cargo for Rust language, or VS 2022, but after everything it still doesn't work. It still saying that it can't find CMake.
I tried also to clone the git repository, but it keeps saying that I didn't installed CMake, but I did installed it and Virtual Studio 2022 with Desktop C++ tools and Windows SDK
Here is the git error I get for 1.8 which does have a source distribution:

  Using cached pyxel-1.8.5.tar.gz (1.1 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
info: please complete authentication in your browser...
  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [18 lines of output]
      ðŸ’¥ maturin failed
        Caused by: Cargo metadata failed. Does your crate compile with `cargo build`?
        Caused by: `cargo metadata` exited with an error:     Updating git repository `https://github.com/messense/cmake-rs.git`
      remote: Repository not found.
      fatal: repository 'https://github.com/messense/cmake-rs.git/' not found
      error: failed to load source for dependency `cmake`

      Caused by:
        Unable to update https://github.com/messense/cmake-rs.git?branch=cross-compile

      Caused by:
        failed to fetch into: C:\Users\minec\.cargo\git\db\cmake-rs-1a1c3c7fd02f9517

      Caused by:
        process didn't exit successfully: `git fetch --no-tags --force --update-head-ok https://github.com/messense/cmake-rs.git +refs/heads/cross-compile:refs/remotes/origin/cross-compile` (exit code: 128)
      Error running maturin: Command '['maturin', 'pep517', 'write-dist-info', '--metadata-directory', 'C:\\Users\\minec\\AppData\\Local\\Temp\\pip-modern-metadata-_r1888m7', '--interpreter', 'G:\\Programing Folder\Python\\.venv\\Scripts\\python.exe']' returned non-zero exit status 1.
      Checking for Rust toolchain....
      Running `maturin pep517 write-dist-info --metadata-directory C:\Users\minec\AppData\Local\Temp\pip-modern-metadata-_r1888m7 --interpreter G:\Programing Folder\Python\.venv\Scripts\python.exe`
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

And here is the msbuild error I get for 2.3 when I try to build from source:

Running `maturin pep517 build-wheel -i G:\Mon Drive\Programmation\Python\.venv\Scripts\python.exe --compatibility off --editable`
      ðŸ\x8d¹ Building a mixed python/rust project
      ðŸ”— Found pyo3 bindings with abi3 support for Python â‰¥ 3.8
         Compiling pyxel-platform v2.3.0 (G:\Mon Drive\Programmation\Python\pyxel\rust\pyxel-platform)
      error: failed to run custom build command for `pyxel-platform v2.3.0 (G:\Mon Drive\Programmation\Python\pyxel\rust\pyxel-platform)`

      Caused by:
        process didn't exit successfully: `G:\Mon Drive\Programmation\Python\pyxel\rust\target\release\build\pyxel-platform-3829ce30b08c5b63\build-script-build` (exit code: 101)
        --- stdout
        CMAKE_TOOLCHAIN_FILE_aarch64-pc-windows-msvc = None
        CMAKE_TOOLCHAIN_FILE_aarch64_pc_windows_msvc = None
        HOST_CMAKE_TOOLCHAIN_FILE = None
        CMAKE_TOOLCHAIN_FILE = None
        CMAKE_GENERATOR_aarch64-pc-windows-msvc = None
        CMAKE_GENERATOR_aarch64_pc_windows_msvc = None
        HOST_CMAKE_GENERATOR = None
        CMAKE_GENERATOR = None
        CMAKE_PREFIX_PATH_aarch64-pc-windows-msvc = None
        CMAKE_PREFIX_PATH_aarch64_pc_windows_msvc = None
        HOST_CMAKE_PREFIX_PATH = None
        CMAKE_PREFIX_PATH = None
        CMAKE_aarch64-pc-windows-msvc = None
        CMAKE_aarch64_pc_windows_msvc = None
        HOST_CMAKE = None
        CMAKE = None
        running: "cmake" "G:\\Mon Drive\\Programmation\\Python\\pyxel\\rust\\target\\release\\build\\pyxel-platform-000c95749b2cecf3\\out/SDL2-2.28.4" "-G" "Visu
al Studio 17 2022" "-Thost=x64" "-AARM64" "-DSDL_SHARED=OFF" "-DSDL_STATIC=ON" "-DSDL_MAIN_HANDLED=ON" "-DCMAKE_INSTALL_PREFIX=G:\\Mon Drive\\Programmation\\Pyth
on\\pyxel\\rust\\target\\release\\build\\pyxel-platform-000c95749b2cecf3\\out" "-DCMAKE_C_FLAGS= -D__FLTUSED__ -nologo -MD -Brepro" "-DCMAKE_C_FLAGS_RELEASE= -D_
_FLTUSED__ -nologo -MD -Brepro" "-DCMAKE_CXX_FLAGS= -nologo -MD -Brepro" "-DCMAKE_CXX_FLAGS_RELEASE= -nologo -MD -Brepro" "-DCMAKE_ASM_FLAGS= -nologo -MD -Brepro" "-DCMAKE_ASM_FLAGS_RELEASE= -nologo -MD -Brepro" "-DCMAKE_BUILD_TYPE=release"
        -- Configuring incomplete, errors occurred!

        --- stderr
        CMake Deprecation Warning at CMakeLists.txt:8 (cmake_minimum_required):
          Compatibility with CMake < 3.10 will be removed from a future version of
          CMake.

          Update the VERSION argument <min> value.  Or, use the <min>...<max> syntax
          to tell CMake that the project requires at least <min> but has been updated
          to work with policies introduced by <max> or earlier.


        CMake Error at CMakeLists.txt:9 (project):
          Failed to run MSBuild command:

            C:/Program Files (x86)/Microsoft Visual Studio/2022/BuildTools/MSBuild/Current/Bin/arm64/MSBuild.exe

          to get the value of VCTargetsPath:

            Version MSBuild 17.12.12+1cce77968 pour .NET Framework
            Generation started at 23/01/2025 22:44:13.

            Project "G:\Mon Drive\Programmation\Python\pyxel\rust\target\release\build\pyxel-platform-000c95749b2cecf3\out\build\CMakeFiles\3.31.4\VCTargetsPath.vcxproj" on node 1 (default target).
            C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\MSBuild\Current\Bin\arm64\Microsoft.Common.CurrentVersion.targets(844,5): error : The 
BaseOutputPath/OutputPath property is not set for project 'VCTargetsPath.vcxproj'.  Please check to make sure that you have specified a valid combination of Conf
iguration and Platform for this project.  Configuration='Debug'  Platform='ARM64'.  You may be seeing this message because you are trying to build a project with
out a solution file, and have specified a non-default Configuration or Platform that doesn't exist for this project. [G:\Mon Drive\Programmation\Python\pyxel\rust\target\release\build\pyxel-platform-000c95749b2cecf3\out\build\CMakeFiles\3.31.4\VCTargetsPath.vcxproj]
            Generation of the project "G:\Mon Drive\Programmation\Python\pyxel\rust\target\release\build\pyxel-platform-000c95749b2cecf3\out\build\CMakeFiles\3.31.4\VCTargetsPath.vcxproj" ended (default target) -- ERROR.

            ERROR of the build.

            "G:\Mon Drive\Programmation\Python\pyxel\rust\target\release\build\pyxel-platform-000c95749b2cecf3\out\build\CMakeFiles\3.31.4\VCTargetsPath.vcxproj" (default target) (1) ->
            (_CheckForInvalidConfigurationAndPlatform cible) ->
              C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\MSBuild\Current\Bin\arm64\Microsoft.Common.CurrentVersion.targets(844,5): error : Th
e BaseOutputPath/OutputPath property is not set for project 'VCTargetsPath.vcxproj'.  Please check to make sure that you have specified a valid combination of Co
nfiguration and Platform for this project.  Configuration='Debug'  Platform='ARM64'.  You may be seeing this message because you are trying to build a project wi
thout a solution file, and have specified a non-default Configuration or Platform that doesn't exist for this project. [G:\Mon Drive\Programmation\Python\pyxel\rust\target\release\build\pyxel-platform-000c95749b2cecf3\out\build\CMakeFiles\3.31.4\VCTargetsPath.vcxproj]

                0 Warning(s)
                1 Error(s)

            Time estimated 00:00:00.86


          Exit code: 1



        thread 'main' panicked at C:\Users\minec\.cargo\registry\src\index.crates.io-6f17d22bba15001f\cmake-0.1.52\src\lib.rs:1115:5:

        command did not execute successfully, got: exit code: 1

        build script failed, must exit now
        note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
      ðŸ’¥ maturin failed
        Caused by: Failed to build a native library through cargo
        Caused by: Cargo build finished with "exit code: 101": `"cargo" "rustc" "--message-format" "json-render-diagnostics" "--manifest-path" "G:\\Mon Drive\\Programmation\\Python\\pyxel\\rust\\pyxel-wrapper\\Cargo.toml" "--release" "--lib"`
      Error: command ['maturin', 'pep517', 'build-wheel', '-i', 'G:\\Mon Drive\\Programmation\\Python\\.venv\\Scripts\\python.exe', '--compatibility', 'off', '--editable'] returned non-zero exit status 1

The text was updated successfully, but these errors were encountered:

kitao · 2025-01-25T02:35:46Z

For Linux, Pyxel wheel includes the following targets:

x86_64-unknown-linux-gnu
i686-unknown-linux-gnu
aarch64-unknown-linux-gnu
armv7-unknown-linux-gnueabihf

Some users run Pyxel on ARM64 Chromebook by using Pyxel wheel, I think.

ViviMine · 2025-01-25T11:54:42Z

I'm on windows :(

kitao · 2025-01-25T12:08:46Z

In that case, just download the source code from GitHub and follow this instruction:
https://github.com/kitao/pyxel/blob/main/Makefile

ViviMine · 2025-01-25T15:09:13Z

I can't execute the make clean install command, I guess I'm doing it wrong, everything is set up

ViviMine · 2025-01-25T23:43:06Z

I've tried to execute the Makefile by hand and even that it keep failing saying that cmake doesn't exists. I had this error before and I fixed it by going on the cmd. But now even in the cmd it doesn't work

kitao · 2025-02-11T09:09:28Z

Try executing Pyxel's self-build process by following the steps written in the Makefile.
Since Pyxel's Makefile simply calls a series of commands, an error like "cmake not found" indicates that the necessary commands are not properly installed in your build environment.
On Windows, the build process assumes a Git Bash environment. Is everything set up correctly on your end?

ViviMine · 2025-02-11T16:58:48Z

Isn't what I did when I manually done the "make clean install" command? I've done by step all things in clean and install, and it crashed the same à a certain command, I'll redo the process on another project to get the precise command that crashes
EDIT: my bad I've done "make clean build", is there any differences?

ViviMine · 2025-02-11T17:02:57Z

Also yes I have git bash

ViviMine · 2025-02-11T19:01:06Z

It crashed at the maturin command in the build thing:

error: failed to run custom build command for `pyxel-platform v2.3.6 (G:\Mon Drive\Programmation\TempPython\pyxel\rust\pyxel-platform)`

Caused by:
process didn't exit successfully: `G:\Mon Drive\Programmation\TempPython\pyxel\rust\target\release\build\pyxel-platform-d2c6d62507dd53d7\build-script-build` (exit code: 101)
--- stdout
CMAKE_TOOLCHAIN_FILE_aarch64-pc-windows-msvc = None
CMAKE_TOOLCHAIN_FILE_aarch64_pc_windows_msvc = None
HOST_CMAKE_TOOLCHAIN_FILE = None
CMAKE_TOOLCHAIN_FILE = None
CMAKE_GENERATOR_aarch64-pc-windows-msvc = None
CMAKE_GENERATOR_aarch64_pc_windows_msvc = None
HOST_CMAKE_GENERATOR = None
CMAKE_GENERATOR = None
CMAKE_PREFIX_PATH_aarch64-pc-windows-msvc = None
CMAKE_PREFIX_PATH_aarch64_pc_windows_msvc = None
HOST_CMAKE_PREFIX_PATH = None
CMAKE_PREFIX_PATH = None
CMAKE_aarch64-pc-windows-msvc = None
CMAKE_aarch64_pc_windows_msvc = None
HOST_CMAKE = None
CMAKE = None
running: "cmake" "G:\\Mon Drive\\Programmation\\TempPython\\pyxel\\rust\\target\\release\\build\\pyxel-platform-bb3be4484fa70548\\out/SDL2-2.28.4" "-G" "Visual
Studio 17 2022" "-Thost=x64" "-AARM64" "-DSDL_SHARED=OFF" "-DSDL_STATIC=ON" "-DSDL_MAIN_HANDLED=ON" "-DCMAKE_INSTALL_PREFIX=G:\\Mon Drive\\Programmation\\TempPy
thon\\pyxel\\rust\\target\\release\\build\\pyxel-platform-bb3be4484fa70548\\out" "-DCMAKE_C_FLAGS= -D__FLTUSED__ -nologo -MD -Brepro" "-DCMAKE_C_FLAGS_RELEASE= -
D__FLTUSED__ -nologo -MD -Brepro" "-DCMAKE_CXX_FLAGS= -nologo -MD -Brepro" "-DCMAKE_CXX_FLAGS_RELEASE= -nologo -MD -Brepro" "-DCMAKE_ASM_FLAGS= -nologo -MD -Brepro" "-DCMAKE_ASM_FLAGS_RELEASE= -nologo -MD -Brepro" "-DCMAKE_BUILD_TYPE=release"

--- stderr
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 7172k 100 7172k 0 0 3329k 0 0:00:02 0:00:02 --:--:-- 3332k
thread 'main' panicked at C:\Users\minec\.cargo\registry\src\index.crates.io-6f17d22bba15001f\cmake-0.1.54\src\lib.rs:1119:5:

failed to execute command: program not found
is `cmake` not installed?

build script failed, must exit now
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
💥 maturin failed
Caused by: Failed to build a native library through cargo
Caused by: Cargo build finished with "exit code: 101": `"cargo" "rustc" "--message-format" "json-render-diagnostics" "--manifest-path" "G:\\Mon Drive\\Programmation\\TempPython\\pyxel\\rust\\pyxel-wrapper\\Cargo.toml" "--release" "--lib"`

with maturin build -o ../dist --release --manylinux 2014 --auditwheel skip

EDIT : I've tried with a venv made with amd64 and arm64 python, both have the same error

agriyakhetarpal · 2025-02-15T15:54:10Z

Hi there from the Pyodide project,

Slightly related to the conversations but not directly to the build issues noted above, we would greatly appreciate it if pyxel could provide an sdist on PyPI with its releases. We currently use the GitHub archives for packaging pyxel downstream for the Pyodide distribution, which we have noted might not contain the correct files in some cases for specific projects, as GitHub release archives usually don't include Git submodules or nested Git submodules (none for pyxel's case, fortunately, besides the examples/ submodule).

Another benefit of the presence of source distributions would be that they would allow updating pyxel in Pyodide's recipes quickly, as we access PyPI's JSON API for packages to bump their versions (along with their checksums) through the pyodide skeleton pypi --update pyxel command. There was a checksum mismatch as well in pyodide/pyodide#5427 with the GitHub release for version 1.9.10 for some reason that we haven't explored, which would not happen with immutable storage(s) such as PyPI.

Also, please feel free to join in as recipe maintainers (as I notice that there are out-of-tree WASM builds in this repository already) for pyxel's recipe in Pyodide!

kitao · 2025-03-16T10:07:26Z

I haven’t used sdist before, so I’m not very familiar with it. However, in Pyxel, several preprocessing steps are executed via scripts before running maturin with a specific Rust version and build options to generate the binary wheel. Because of this, these steps are defined in the Makefile. In such a case, if an sdist is created, would users be able to reproduce the same build process using it?

agriyakhetarpal · 2025-03-16T13:38:41Z

Thanks, @kitao. Looking at https://github.com/kitao/pyxel/blob/3fead432aaaf881e25da0c85da17c874bf7a19b9/Makefile as you suggested, I think the primary step is to build SDL2 using embuilder. The other options, such as setting a nightly Rust toolchain, etc., will be possible (https://pyodide.org/en/stable/development/meta-yaml.html#build-script), which will run before the compilation from the sdist is attempted.

Whether users can reproduce the same build process with the sdist depends on whether these prerequisites remain documented well enough. I understand that there are troubles with sdists, especially for those on niche platforms. Most users will enjoy the availability of the wheels, anyway. :)

kitao · 2025-03-21T13:11:03Z

If the sdist for Pyxel is uploaded to PyPI, the following would likely happen:
• A user runs pip install pyxel on a platform that Pyxel doesn’t officially support.
• The sdist is downloaded and a build is attempted, which fails and results in an error.
• The user reports it as a bug, and I end up replying each time saying, “That platform is not supported or tested — please read the documentation carefully.”

I’d like to avoid getting into this kind of cycle.
Is there a good way to handle this?

agriyakhetarpal · 2025-03-21T16:50:21Z

There isn't, unfortunately. pip has historically selected newer sdists over older wheels and will comtinue to do so; I would even go on to say that this is one of the hardest problems that is still unsolved in Python packaging. :)

Of course, there are ways to mitigate it – for example, supporting as many platforms/architectures as one can, but that prospect would have an exorbitantly high cost of maintainership.

Is there a list of platforms that will not receive support? The quartet of Windows, GNU/Linux, macOS, and WebAssembly has support already. Let's say:

Windows arm64 is still coming up and CI providers aren't ready yet
Linux s390x and ppc64le
RISC-V?
WASI
BSD derivatives (FreeBSD, NetBSD, DragonflyBSD)
someone might want musl wheels someday for Alpine Linux
armv6l (Raspberry Pi, through the piwheels project)
PyPy (which I notice that you dropped due to maintenance costs)

are the only unsupported platforms at this time (unless they are more niche ones I don't recall). Some of these rely on Python packaging tooling support, and some might need support from the Rust toolchain. I guess the best way to go ahead depends on how much of pyxel's user base is and can be on such platforms. However, I'm cognizant of the fact that this initiative isn't something that can be handled solitarily by a maintainer or even with a small team of maintainers. Downstream packagers would need to step up for this. We can help from the Pyodide side, for sure.

On this note, one thing that an sdist does help with is that it provides a minimal set of source files that is vetted by the package developer for correctness and further usage for downstream packaging. It can be a starting point to handle these troubles – the fact that you can allow compilation to go through and pass if all the prerequisites for building from source are met is pretty useful, and there is a place (PyPI) to put those files which is trustworthy enough for relying on it for software supply-chain security and a host of other reasons. If there is/can be some data on the percentage of prospective/attempted pyxel users on niche platforms for which wheels have not been available on PyPI so far, it would be possible to make an informed decision.

kitao · 2025-03-22T00:07:58Z

The platforms currently supported by Pyxel are as follows:
• x86_64-pc-windows-msvc
• i686-pc-windows-msvc
• x86_64-apple-darwin
• aarch64-apple-darwin
• wasm32-unknown-emscripten
• x86_64-unknown-linux-gnu
• i686-unknown-linux-gnu
• aarch64-unknown-linux-gnu
• armv7-unknown-linux-gnueabihf

As a general policy, Pyxel officially supports only platforms where I can personally verify and fix issues as they arise. For other platforms, users are expected to build Pyxel themselves and investigate any issues on their own.

There are a couple of exceptions to this. While I cannot check all browsers across all platforms, I still support the browser version because it provides significant benefits to users. For Linux, even though I can only do build verification for the targets that can be built automatically via maturin, I still include them in official releases.

I’ve been developing Pyxel for eight years now, and during that time, I’ve rarely received requests to support platforms beyond those listed above. I believe this is because such platforms have a very small user base, and beyond just being able to build Pyxel, users on those platforms often need to take care of many other things themselves — such as ensuring proper drawing performance, or installing dependencies like SDL or OpenGL ES — so self-building has become the norm for them.

Also, at this point, a considerable amount of my development time is already taken up by maintenance, support, and platform verification. Therefore, it’s difficult to expand the list of supported platforms any further.

If we’re talking about a minimal and verified set of source files, the GitHub release packages serve that purpose exactly. The contents under the doc folder are also used as the README for pip, so while there are a few extra files, most of them are actually required to build with pip.

For that reason, the easiest and most reliable way to obtain Pyxel’s source code is either via git or by downloading the release package from GitHub.

ViviMine · 2025-03-22T09:32:25Z

Would you like that I do another issue so that this one is dedicated to Pyodide?

ViviMine changed the title ~~ARM64 incompatibility + no source distribution~~ new versions have no source distributions or arm64 wheels Jan 24, 2025

This was referenced Feb 15, 2025

Update sha256 checksum for pyxel recipe pyodide/pyodide#5427

Merged

Update Pyxel to version 2.3.6 pyodide/pyodide#5435

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

new versions have no source distributions or arm64 wheels #601

new versions have no source distributions or arm64 wheels #601

ViviMine commented Jan 24, 2025 •

edited

Loading

kitao commented Jan 25, 2025

Uh oh!

ViviMine commented Jan 25, 2025

Uh oh!

kitao commented Jan 25, 2025

Uh oh!

ViviMine commented Jan 25, 2025

Uh oh!

ViviMine commented Jan 25, 2025

Uh oh!

kitao commented Feb 11, 2025

Uh oh!

ViviMine commented Feb 11, 2025 •

edited

Loading

Uh oh!

ViviMine commented Feb 11, 2025

Uh oh!

ViviMine commented Feb 11, 2025 •

edited

Loading

Uh oh!

agriyakhetarpal commented Feb 15, 2025

Uh oh!

kitao commented Mar 16, 2025

Uh oh!

agriyakhetarpal commented Mar 16, 2025

Uh oh!

kitao commented Mar 21, 2025

Uh oh!

agriyakhetarpal commented Mar 21, 2025

Uh oh!

kitao commented Mar 22, 2025

Uh oh!

ViviMine commented Mar 22, 2025

Uh oh!

Uh oh!

new versions have no source distributions or arm64 wheels #601

new versions have no source distributions or arm64 wheels #601

Comments

ViviMine commented Jan 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

kitao commented Jan 25, 2025

Uh oh!

ViviMine commented Jan 25, 2025

Uh oh!

kitao commented Jan 25, 2025

Uh oh!

ViviMine commented Jan 25, 2025

Uh oh!

ViviMine commented Jan 25, 2025

Uh oh!

kitao commented Feb 11, 2025

Uh oh!

ViviMine commented Feb 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ViviMine commented Feb 11, 2025

Uh oh!

ViviMine commented Feb 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

agriyakhetarpal commented Feb 15, 2025

Uh oh!

kitao commented Mar 16, 2025

Uh oh!

agriyakhetarpal commented Mar 16, 2025

Uh oh!

kitao commented Mar 21, 2025

Uh oh!

agriyakhetarpal commented Mar 21, 2025

Uh oh!

kitao commented Mar 22, 2025

Uh oh!

ViviMine commented Mar 22, 2025

Uh oh!

ViviMine commented Jan 24, 2025 •

edited

Loading

ViviMine commented Feb 11, 2025 •

edited

Loading

ViviMine commented Feb 11, 2025 •

edited

Loading