Skip to content

Emscripten's webgpu.h is behind the standard and Dawn implementations #23432

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
orhun-sketchup opened this issue Jan 16, 2025 · 59 comments · May be fixed by #24220
Open

Emscripten's webgpu.h is behind the standard and Dawn implementations #23432

orhun-sketchup opened this issue Jan 16, 2025 · 59 comments · May be fixed by #24220
Assignees
Labels

Comments

@orhun-sketchup
Copy link

Is it possible to update to a recent version of webgpu_cpp.h?

I am specifically interested in missing templated async functions.

like in the Buffer class.

template <typename F, typename T,
              typename Cb = BufferMapCallback<T>,
              typename CbChar = void (MapAsyncStatus status, const char* message, T userdata),
              typename = std::enable_if_t<std::is_convertible_v<F, Cb*> || std::is_convertible_v<F, CbChar*>>>
Future MapAsync(MapMode mode, size_t offset, size_t size, CallbackMode callbackMode, F callback, T userdata) const;
template <typename L,
              typename Cb = BufferMapCallback<>,
              typename CbChar = std::function<void(MapAsyncStatus status, const char* message)>,
              typename = std::enable_if_t<std::is_convertible_v<L, Cb> || std::is_convertible_v<L, CbChar>>>
Future MapAsync(MapMode mode, size_t offset, size_t size, CallbackMode callbackMode, L callback) const;
@kainino0x
Copy link
Collaborator

For the moment we won't be bringing any new changes into Emscripten's WebGPU bindings because we don't want to continuously make breaking changes on every Emscripten release. However we are currently working on an updated fork which lives in Dawn. You can use this fork along with regular Emscripten - it's a big pain right now but it's possible. See:
https://source.chromium.org/chromium/chromium/src/+/main:third_party/dawn/src/emdawnwebgpu/README.md
This sample repo is set up to do this:
https://github.com/kainino0x/webgpu-cross-platform-demo/tree/dawnwasm
You can follow here for updates on the un-forking: https://crbug.com/371024051 (+1 button in the top right)

@ypujante
Copy link
Contributor

ypujante commented Jan 18, 2025

@kainino0x Now that you can write contrib ports in Emscripten, wouldn't it make sense to have a contrib.webgpu (or contrib.dawn_webgpu) to do this? That way you are not breaking Emscripten WebGPU binding. You just upgrade the port version (which might contain breaking changes).

You can also have the port not be a contrib port at all meaning it is a port that anybody can use (simply) without being part of Emscripten. Like I did for ImGui. This port file could live in the Dawn project.

I don't know the internals of Dawn or why it was chosen to have an Emscripten fork which lives in Dawn. I am just pointing out that there is a new mechanism (Emscripten ports) that is available...

I am very familiar with Emscripten contrib/external ports, since I am the one who implemented the feature, so don't hesitate to reach out if you need help/guidance...

@kainino0x
Copy link
Collaborator

I wasn't aware of emscripten-ports! Regardless of that, we actually just started talking to sbc100 about this topic, we might end up maintaining the bindings as a library outside of emscripten instead. The reason we did it in Emscripten to start was that it relied on a lot of internal things. We need a stableish interface between emscripten and js libraries, and a stableish interface for gen_struct_info.py. But it sounds like we may be able to achieve that!

TBA, and thank you for your offer!

@kainino0x
Copy link
Collaborator

FYI: I'm working on updating the code and documentation in Dawn for how to use emdawnwebgpu (our fork). It's much easier with Emscripten 4.0.3+!

https://dawn-review.googlesource.com/c/dawn/+/227335/4/src/emdawnwebgpu/README.md

@kainino0x kainino0x changed the title WebGPU missing functions from webgpu_cpp.h Emscripten's webgpu.h is behind the standard and Dawn implementations Feb 27, 2025
@sbc100
Copy link
Collaborator

sbc100 commented Feb 27, 2025

At the end of that process should we delete the webgpu support from emscripten? (we can still test it via some kind of vendoring, or add it as a "port")

@ypujante
Copy link
Contributor

I am not sure what is the plan (it is hard to follow this thread and a link to the diff of an upcoming README file change is not really helping) but I hope that no matter the solution, there continues to be an easy way to use WebGPU from Emscripten.

I really don't want to have to clone an external project and use CMake to build WebGPU especially that now Emscripten has support for integrating external projects, not maintained in Emscripten (contrib ports).

I am hoping that Dawn/WebGPU can at the very minimum be offered as a contrib port (and I reiterate what I stated in a previous message, that I am more than happy to provide help and guidance...)

@kainino0x
Copy link
Collaborator

I've been working on producing a "packageable" version of emdawnwebgpu that can be easily used by just linking into an emscripten build. Once that's done I should be able to point to updated docs that actually make sense :)

I'll give an update here when that's ready - so I can get help figuring out how this package should be provided to people (I don't know how "ports" work).

@kainino0x
Copy link
Collaborator

kainino0x commented Mar 12, 2025

Those readme changes I linked to have been updated and landed, so there are at least (partial) instructions on how to use emdawnwebgpu:
https://source.chromium.org/chromium/chromium/src/+/main:third_party/dawn/src/emdawnwebgpu/README.md

I'm still working on making pre-packaged releases of emdawnwebgpu. Basically what that's going to look like is something like:

├── CMakeLists.txt
├── LICENSE
├── include
│   ├── dawn
│   │   └── webgpu_cpp_print.h
│   └── webgpu
│       ├── webgpu.h
│       ├── webgpu_cpp.h
│       ├── webgpu_cpp_chained_struct.h
│       └── webgpu_enum_class_bitmasks.h
├── library_webgpu.js
├── library_webgpu_enum_tables.js
├── library_webgpu_generated_sig_info.js
├── webgpu-externs.js
├── webgpu.cpp
├── webgpu_generated_struct_info32.json
└── webgpu_generated_struct_info64.json

where CMakeLists.txt includes all of the flags needed to make that work.

For maintenance reasons it seems easier for this to be totally separate from Emscripten, rather than a port. A bunch of these files are generated (not checked into Dawn) so it's not possible to point directly at Dawn's source.

But I'm curious if you think it would be better to add it to the ports. It will be easier to install, but it seems you will only be able to use the version pinned in Emscripten? At least, the contrib ports that exist now seem to pin a version.

@ypujante
Copy link
Contributor

ypujante commented Mar 12, 2025

@kainino0x I still think that having it as a port would still be very beneficial: the README that you published is quite involved with lots of steps, including complicated instructions to download and build Dawn. Using the port would be as simple as emcc --use-port=contrib.dawn and that's it for the instructions... hard to be simpler...

The fact that the Dawn version is pinned to Emscripten version is not really a problem. This is how SDL, GLFW, (etc...) ports work. Upgrading Emscripten for the new version of Dawn now becomes trivial. If you want to use a different version you can always copy the port file and use it directly since now Emscripten allow non built-in ports.

You mention that there are files that are generated and not checked in into Dawn. I do not know how Dawn gets released (when there is a new version), but if you could automatically publish a zip file containing all the necessary files to build Dawn for Emscripten (both generated and checked in) that would be the best approach. Maybe this is what you are talking about when you say: I'm still working on making pre-packaged releases of emdawnwebgp

I am doing exactly this for the GLFW port:

Image

and then in the port, the downloaded artifact is simply:

TAG = '3.4.0.20250305'
HASH = 'd5d1496cf1e05a640c182584a09ed0b43c8d6c42386513e5c2b1e05a2289854a0a0abe085e0a8b64c3a484836a8e63dc5ddb683ce4d2899fccbbf89536d58ee1'
ZIP_URL = f'https://github.com/pongasoft/emscripten-glfw/releases/download/v{TAG}/emscripten-glfw3-{TAG}.zip'

I will spend some time looking at a first pass at the Emscripten Dawn port if I can follow your instructions.

@sbc100
Copy link
Collaborator

sbc100 commented Mar 12, 2025

We certainly want emscripten users to be able to use WebGPU APIs with as little friction as possible.

There are various ways we could achieve that without having to maintain emdawn directly in the emscirpten tree. I think a "port" is likely the best option. We might also want to consider using a git submodule, but I'm not sure that would be any better. I think having emscripten default to particular pinned version is fine (good even) as long as we document some way to experiment to non-pinned version too.

@kainino0x
Copy link
Collaborator

Ports do sound extremely convenient. As long as there is some easy way to use a non-pinned version, I'm OK with having a pinned default. The bindings are definitely still buggy so for the near-term future, people will commonly need the latest version, ideally without having to upgrade their whole toolchain.

@kainino0x
Copy link
Collaborator

kainino0x commented Mar 13, 2025

I do not know how Dawn gets released (when there is a new version), but if you could automatically publish a zip file containing all the necessary files to build Dawn for Emscripten (both generated and checked in) that would be the best approach. Maybe this is what you are talking about when you say: I'm still working on making pre-packaged releases of emdawnwebgpu

Dawn doesn't have any releases (pre-built or otherwise), but since we need to provide pre-built files for emdawnwebgpu, yes, that's my plan - upload packages like the one described above, either as zips on the github releases page, or in some git repository.

@ypujante
Copy link
Contributor

Dawn doesn't have any releases (pre-built or otherwise), but since we need to provide pre-built files for emdawnwebgpu, yes, that's my plan - upload packages like the one described above, either as zips on the github releases page, or in some git repository.

Awesome. When you do that let me know and I can take a pass at writing the (contrib?) port for Emscripten or help @sbc100 if he would prefer to do it himself. I can think of a couple of ways on how to allow changing the pinned version.

@sbc100
Copy link
Collaborator

sbc100 commented Mar 13, 2025

@kainino0x and I had a quick VC meeting today and it sounds like a port is going to be the way to go. They might even just go with an external port (e.g: --use-port=/path/to/emdawn_files/emscripten_port.py).

Ryu204 added a commit to Ryu204/stay3 that referenced this issue Mar 16, 2025
@krauthaufen
Copy link

Any updates on that?
I‘m building a dotnet wrapper around dawn and would love to see it run in the browser too. I did the same with WebGL a few years ago and have a running pipeline for auto-generating all my stuff. Sadly the dotnet environment pins me to emscripten 3.0.56 for now, so all cutting edge things won‘t work for me.

Any hints how to make this work?

@sbc100
Copy link
Collaborator

sbc100 commented Mar 19, 2025

The plan is for emdawn to maintain an out-of-tree emscripten port. The goal is to make this trivial/easy to use from emscripten directly, but it almost certainly will not work with old emscripten version like 3.0.56.

I would advice you to petition the dotnet folks to update their emscripten version.

@krauthaufen
Copy link

Thanks for the reply. I just found that dotnet is actually using 3.1.56 where ports should be working. I‘d be happy to help with testing things when you have the port ready.

@sbc100
Copy link
Collaborator

sbc100 commented Mar 20, 2025

3.1.56 is still a year old. I would imagine that emdawn will require a current (e.g. 4.0.0+) version.. but that still remains to be seen.

@albertrodriguezclo
Copy link

Hey @sbc100, any advances with the emdawn port?

@sbc100
Copy link
Collaborator

sbc100 commented Apr 7, 2025

I believe @kainino0x is working on it right now. I'll make sure we update this issue if / when stuff lands here.

@kainino0x
Copy link
Collaborator

Apologies, I had other important work the past few weeks. I will be working on it this week.

@kainino0x
Copy link
Collaborator

You may be aware of this, but there are some symbols that are defined with an extension number in library_webgpu.js but not in the webgpu.h that is provided next to it. Thinking of wgpuCommandEncoderInsertDebugMarker for instance (they have the same signature, beside that '2' at the end of the name in library_webgpu.js).

@eliemichel thanks for the report, totally didn't realize that. Filed https://crbug.com/412395835

@kainino0x
Copy link
Collaborator

kainino0x commented Apr 28, 2025

Update!

@eliemichel
Copy link

Nice! What's the minimum version of emscripten that is needed to use these? (in particular to automatically get the --closure-args added). BTW, do emscripten port standardize a way to know what minimum version of emscripten is supported by the port?

@ypujante
Copy link
Contributor

@eliemichel This will be available when 4.0.8 is released. I do not know if there is a way from within the port to be able to know the version of Emscripten and as a result handle different behavior. In its latest iteration, @kainino0x implemented it like this (detecting if the CLOSURE_ARGS is available).

As far as I am concerned, as I mentioned (and posted) previously, I have a version of the port that behaves more like a port, which means you only need the port file which downloads the necessary artifact as oppose to having to download the artifact manually and have the port file refers to local files (which is backward and not the real intention of ports). I will release this version of the port once 4.0.8 is released and will update this thread when I do.

I wished the deprecation of WebGPU in Emscripten would lead to a contrib port so that people could simply install the latest version of Emscripten and still be able to use WebGPU (if the issue is the tag version that changes every day, the tag could be made a parameter of the port FWIW: --use-port=contrib.dawn:tag=v20250428.160623). But it doesn't sound that it is the direction it is going to go. That is why I am planning to release a version of the port that behaves like an actual port.

@sbc100
Copy link
Collaborator

sbc100 commented Apr 29, 2025

I wished the deprecation of WebGPU in Emscripten would lead to a contrib port so that people could simply install the latest version of Emscripten and still be able to use WebGPU (if the issue is the tag version that changes every day, the tag could be made a parameter of the port FWIW: --use-port=contrib.dawn:tag=v20250428.160623). But it doesn't sound that it is the direction it is going to go. That is why I am planning to release a version of the port that behaves like an actual port.

That think there there are still plans to perhaps do that in the future once the port seems stable. The nice thing about starting off with the port being external is that it can all be maintained in a single repo without kai (or someone working on dawn) having to constantly update emscripten. We will see how the cost / benefit plays out over time.

@kainino0x
Copy link
Collaborator

+1, we definitely might still do it in the future once the new bindings are more polished. I was thinking of maybe doing it around the same time we fully remove the old implementation from Emscripten.

I also had a thought that maybe a real port could handle ONLY the downloading, then pull all the flags and stuff from the package's emdawnwebgpu.port.py, if it's possible to get the download before computing those. Then hopefully we wouldn't need to change it, which would be helpful - there is at least one planned optimization that will require additional compile/link flags (https://crbug.com/370856875), that may not happen for a while.

@ypujante
Copy link
Contributor

ypujante commented Apr 30, 2025

@kainino0x a port is just a python script. The zip file that gets downloaded by the port has full access to the content of the download (that is how the file webgpu.cpp gets compiled). So yes you could have a file called port.config or flags.txt (or whatever else is suitable) inside the zip file and read it/parse it from the port and do whatever you want with it. Even the list of files that get added to settings.JS_LIBRARIES could be outside the port file itself. In other words, by extracting the hard coded pieces inside the port, you could build something pretty generic where most of the details are in 1 (or more) files inside the downloaded artifact.

@ypujante
Copy link
Contributor

ypujante commented May 1, 2025

As promised, since Emscripten 4.0.8 has been released, I have now released my version of the WebGPU port that behaves more like a traditional port.

You can find it at emscripten-ports.

A few points:

  • the port file exposes tag (and optional hash) as port options since the version changes quite frequently (at the moment)
  • there is an example (derived from @kainino0x code) that uses the port

@kainino0x
Copy link
Collaborator

@kainino0x a port is just a python script. The zip file that gets downloaded by the port has full access to the content of the download (that is how the file webgpu.cpp gets compiled).

I know, the thing I wasn't sure about was whether we could trigger the download early enough for all of the hooks. IIRC something, I think it was process_args, runs before get, but get is what (typically) downloads the zip.

@ypujante
Copy link
Contributor

ypujante commented May 2, 2025

As far as I can tell, process_args is called after get: see build with print statement

> emcc -sASYNCIFY=1 --use-port=../../ports/Dawn/dawn.py main.cpp -o /tmp/dawn/index.html
get
process_args
get

@sbc100
Copy link
Collaborator

sbc100 commented May 5, 2025

I've been thinking about how we can make out-of-tree ports work better.

How about something like this:

$ cat tools/ports/contrib/emdawn.py
EXTERNAL_PORT=http://foo.bar/releases/release.zip
SHA1=8407a6393c858ad5e7c7f23f7579d524c403be97
PORT_FILE=myport.py

That would then be shorthand the current manual steps of the downloading and extracting the zip file and then using then embedded the .py file from the zip file file.

We could even just maybe just call it emdawn.yaml since it will be pure download&redirect metadata.

@sbc100
Copy link
Collaborator

sbc100 commented May 5, 2025

Folks who use --use-port=contrib/emdawn would then get a particular blessed version of emdawn from the emscripten release they are using.

Folks who want more control can do the manual download and --use-port=/path/to/emdawn.py

@ypujante
Copy link
Contributor

ypujante commented May 5, 2025

@sbc100 I personally like the idea very much. I assume you meant --use-port=contrib.emdaw (you wrote --use-port=contrib/emdawn)

Since I worked on the port system, let me know if you want me to look at a possible way of implementing the changes. Be more than happy to.

@sbc100
Copy link
Collaborator

sbc100 commented May 5, 2025

Yes, I think this would help for your glfw port maybe? Since it would allow you to keep the glfw3.py logic alongside the upstream code.

If you would to take a crack an implementing what I described above that would be great. I would basically mean two different types of port files:
A) Ones that describe the build actions
B) Ones that describe how to download files of type (A)

(I think using yaml perhaps doesn't actually work since we don't want to take a dependency on pyyaml).

@sbc100
Copy link
Collaborator

sbc100 commented May 5, 2025

It looks like we could possibly use configparser: https://docs.python.org/3/library/configparser.html

And maybe once we depend on python 3.11 we could even use toml!

@ypujante
Copy link
Contributor

ypujante commented May 5, 2025

I will take a look at it. I created a new ticket (#24260) for it.

@Honya2000
Copy link

Hello,

Unfortunately i coudn't make emdawn port working.
A soon as i add this line in cmake:
--use-port=c:/Projects/emsdk/emdawn/emdawnwebgpu.port.py
for both compiler and linker - cmake generator completely stops working, throwing errors on any cpp test step, like:

-- Looking for sys/types.h
-- Looking for sys/types.h - not found
-- Looking for stdint.h
-- Looking for stdint.h - not found
-- Looking for stddef.h
-- Looking for stddef.h - not found
-- Check size of off64_t
-- Check size of off64_t - failed
-- Looking for fseeko
-- Looking for fseeko - not found
-- Looking for unistd.h
-- Looking for unistd.h - not found
-- Performing Test ABSL_INTERNAL_AT_LEAST_CXX17
-- Performing Test ABSL_INTERNAL_AT_LEAST_CXX17 - Failed
-- Performing Test ABSL_INTERNAL_AT_LEAST_CXX20
-- Performing Test ABSL_INTERNAL_AT_LEAST_CXX20 - Failed

pycache was created in the port's folder. So it definitely compiled the py file during the build process...

I'm using emsdk 4.0.8.
Dawn release v20250505.211659

@kainino0x
Copy link
Collaborator

kainino0x commented May 6, 2025

It doesn't work on Windows because I totally forgot to test on Windows. Bug is https://crbug.com/415850603, please star it.

@kainino0x
Copy link
Collaborator

Never mind, it works fine on Windows. These are two separate issues. Continuing discussion of this one in #24253 for now even though it's closed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

9 participants