Initial commit

Author: mrexodia

.gitignore

@@ -0,0 +1,3 @@
+__pycache__/
+venv*/
+/rosetta

README.md

@@ -0,0 +1,183 @@
+# rosetta-multipass
+
+Would you also like to run `amd64` binaries in your [Multipass](https://multipass.run/) VMs on an M1 Mac? This is a guide/tool to enable [Rosetta](https://developer.apple.com/documentation/virtualization/running_intel_binaries_in_linux_vms_with_rosetta) without official support from Multipass. 
+
+## Rosetta Installation (macOS host)
+
+First you need to [install Rosetta](https://osxdaily.com/2020/12/04/how-install-rosetta-2-apple-silicon-mac/) on your host system:
+
+```sh
+softwareupdate --install-rosetta --agree-to-license
+```
+
+You should now have the `rosetta` translator binary:
+
+```sh
+% tree /Library/Apple/usr/libexec/oah
+/Library/Apple/usr/libexec/oah
+├── RosettaLinux
+│   └── rosetta # <- this one
+├── debugserver -> /usr/libexec/rosetta/debugserver
+├── libRosettaRuntime
+├── runtime -> /usr/libexec/rosetta/runtime
+└── translate_tool -> /usr/libexec/rosetta/translate_tool
+```
+
+**Note**: If you do not get the `RosettaLinux/rosetta` binary, try following the [UTM Rosetta Guide](https://docs.getutm.app/advanced/rosetta/) first, which should install Rosetta for you.
+
+## Rosetta Installation (Ubuntu guest)
+
+Copy the `RosettaLinux/rosetta` binary to your guest VM. In this example the binary has been copied to `/bin/rosetta`, but any path should work. Make sure that you can run the binary from inside the VM:
+
+```
+$ /bin/rosetta
+rosetta error: Rosetta is only intended to run on Apple Silicon with a macOS host using Virtualization.framework with Rosetta mode enabled
+Trace/breakpoint trap (core dumped)
+```
+
+The error is expected, you can read more about it in a [Quick look at Rosetta on Linux](https://threedots.ovh/blog/2022/06/quick-look-at-rosetta-on-linux/). 
+
+### Using `mount-rosetta.py`
+
+To fix the error you can create a FUSE mount using `mount-rosetta.py`, which implements the `ioctl` required for the Rosetta handshake:
+
+```sh
+sudo apt install libfuse2
+sudo python -m pip install fusepy
+```
+
+**Note**: To mount without root, edit `/etc/fuse.conf` and uncomment `user_allow_other`. This shouldn't be necessary unless you are developing the script.
+
+Now you can run the script:
+
+```sh
+sudo python mount-rosetta.py /bin/rosetta
+```
+
+This will shadow the `/bin/rosetta` binary and handle the necessary `ioctl` calls. To confirm you you can run:
+
+```sh
+$ /bin/rosetta
+Usage: rosetta <x86_64 ELF to run>
+
+Optional environment variables:
+ROSETTA_DEBUGSERVER_PORT    wait for a debugger connection on given port
+
+version: Rosetta-289.7
+```
+
+**Note**: You will have to run `mount-rosetta.py` in the background.
+
+### Installing the binfmt handler (guest)
+
+To register `rosetta` for `amd64` binaries:
+
+```
+sudo apt install binfmt-support
+sudo update-binfmts --install rosetta /bin/rosetta \
+    --magic "\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x3e\x00" \
+    --mask "\xff\xff\xff\xff\xff\xfe\xfe\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff" \
+    --credentials yes --preserve no --fix-binary no
+```
+
+**Important**: If you want to use `--fix-binary yes` you will have to run `mount-rosetta.py` before the `update-binfmts` command.
+
+### Running an `amd64` executable
+
+You should now be able to execute a statically linked executable:
+
+```sh
+$ ./tests/main-static
+Hello from Rosetta!
+```
+
+### Creating a service
+
+You can create `/etc/systemd/system/rosetta.service`:
+
+```
+[Unit]
+Description=Rosetta Mount Service
+
+[Service]
+ExecStart=/usr/bin/python3 /home/ubuntu/.local/bin/mount-rosetta.py /bin/rosetta
+Environment=PYTHONUNBUFFERED=1
+Restart=on-failure
+
+[Install]
+WantedBy=default.target
+```
+
+See [here](https://github.com/torfsen/python-systemd-tutorial) for more information.
+
+```
+sudo systemctl list-unit-files | grep rosetta
+sudo systemctl enable rosetta.service
+sudo systemctl start rosetta.service
+```
+
+## Installing shared libraries (experimental)
+
+**Warning**: This section is super experimental and unlikely to yield great results. That being said, it is possible to run clang compiled for `amd64` and build something.
+
+Running a dynamically linked binary will not work properly at this point:
+
+```
+$ ./tests/main-dynamic
+./tests/main-dynamic: error while loading shared libraries: libc.so.6: cannot open shared object file: No such file or directory
+```
+
+To fix this (only tested on Ubuntu 22.04) you can run the following commands:
+
+```
+sudo apt install g++-multilib-x86-64-linux-gnu gcc-multilib-x86-64-linux-gnu
+sudo ln -s /usr/x86_64-linux-gnu/lib64 /lib64
+export LD_LIBRARY_PATH=/usr/x86_64-linux-gnu/lib
+```
+
+At this point things are working:
+
+```
+./tests/main-dynamic
+Hello from Rosetta!
+```
+
+**Note**: For other distributions you can search for `multilib` or `multiarch` packages or look for guides on getting qemu usermode emulation to work.
+
+### Fixing some dynamic linker errors
+
+Since `/usr/x86_64-linux-gnu` is meant for compiling things and not running you might have to replace some `.so` files that are actually linker scripts with the binaries:
+
+```
+$ sudo -i
+$ cd /usr/x86_64-linux-gnu/lib
+$ rg GROUP # lists fake files
+libc.so
+5:GROUP ( /usr/x86_64-linux-gnu/lib/libc.so.6 /usr/x86_64-linux-gnu/lib/libc_nonshared.a  AS_NEEDED ( /usr/x86_64-linux-gnu/lib64/ld-linux-x86-64.so.2 ) )
+libm.so
+4:GROUP ( /usr/x86_64-linux-gnu/lib/libm.so.6  AS_NEEDED ( /usr/x86_64-linux-gnu/lib/libmvec.so.1 ) )
+
+$ mv libc.so libc.so.script
+$ cp libc.so.6 libc.so
+$ mv libm.so libm.so.script
+$ cp libm.so.6 libm.so
+```
+
+### Cross-compiling libraries
+
+If you need to install something like [zlib](https://zlib.net) for `amd64` you can use [Zig](https://zig.news/kristoff/cross-compile-a-c-c-project-with-zig-3599) for easy cross compiling:
+
+```
+sudo snap install zig --classic --edge
+git clone https://github.com/madler/zlib
+cd zlib
+CC="zig cc -target x86_64-linux-musl" CXX="zig c++ -target x86_64-linux-musl" AR="zig ar" RANLIB="zig ranlib" uname_S="Linux" uname_M="x86_64" C11_ATOMIC=yes USE_JEMALLOC=no USE_SYSTEMD=no ./configure --prefix /usr/x86_64-linux-gnu
+make
+sudo make install
+```
+
+For CMake project you can use [zig-cross](https://github.com/mrexodia/zig-cross) as a base.
+
+### Multithreading
+
+According to [Quick look at Rosetta on Linux](https://threedots.ovh/blog/2022/06/quick-look-at-rosetta-on-linux/) you might run into issues if the VM is not running in Total Store Ordering (TSO) mode. The solution should be to wrap `rosetta` in a `taskset` command during `update-binfmts`. This was not explored further.

mount-rosetta.py

@@ -0,0 +1,102 @@
+#!/usr/bin/env python
+
+import os
+import sys
+import errno
+import ctypes
+import argparse
+import logging
+import stat
+
+from fuse import FUSE, FuseOSError, Operations
+
+# References:
+# - https://github.com/libfuse/libfuse/blob/master/example/hello.c
+# - https://github.com/fusepy/fusepy/blob/master/examples/memory.py
+# - https://github.com/fusepy/fusepy/blob/master/examples/ioctl.py
+# - https://threedots.ovh/blog/2022/06/quick-look-at-rosetta-on-linux/
+
+class RosettaFS(Operations):
+    log = logging.getLogger('RosettaFS')
+
+    def __init__(self, rosetta_binary: str):
+        # Virtualization check constants (https://threedots.ovh/blog/2022/06/quick-look-at-rosetta-on-linux/)
+        self.ioctl_cmd = 0x80456122
+        self.handshake = b"Our hard work\nby these words guarded\nplease don\'t steal\n\xc2\xa9 Apple Inc\0"
+
+        # Load the contents of the original binary
+        with open(rosetta_binary, "rb") as f:
+            self.data = f.read()
+        if self.handshake not in self.data:
+            self.log.warning(f"Could not find handshake in binary. Either you are mounting the wrong binary or the virtualization check changed")
+
+        # Proxy file attributes
+        st = os.lstat(rosetta_binary)
+        self.attr = dict((key, getattr(st, key)) for key in ('st_atime', 'st_ctime',
+                     'st_gid', 'st_mode', 'st_mtime', 'st_nlink', 'st_size', 'st_uid'))
+        self.attr["st_mode"] = stat.S_IFREG | 0o755
+        self.attr["st_nlink"] = 1
+
+        self.log.info(f"Created mount, to test you can run: {rosetta_binary}. To register the binfmt handler:")
+        self.log.info(f"sudo update-binfmts --install rosetta {rosetta_binary} --magic \"\\x7fELF\\x02\\x01\\x01\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x02\\x00\\x3e\\x00\" --mask \"\\xff\\xff\\xff\\xff\\xff\\xfe\\xfe\\x00\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xfe\\xff\\xff\\xff\" --credentials yes --preserve no --fix-binary no")
+
+    def __call__(self, op, path, *args):
+        def truncate(s):
+            if len(s) > 256:
+                return s[:256] + "..."
+            return s
+
+        self.log.debug('-> %s %s %s', op, path, truncate(repr(args)))
+
+        ret = '[Unhandled Exception]'
+        try:
+            ret = getattr(self, op)(path, *args)
+            return ret
+        except OSError as e:
+            ret = str(e)
+            raise
+        finally:
+            self.log.debug('<- %s %s', op, truncate(repr(ret)))
+
+    def readdir(self, path, fh):
+        raise FuseOSError(errno.ENOTDIR)
+
+    def getattr(self, path, fh=None):
+        if path == "/":
+            return self.attr
+        raise FuseOSError(errno.ENOENT)
+
+    def open(self, path, flags):
+        if path == "/":
+            return 1
+        raise FuseOSError(errno.ENOENT)
+
+    def read(self, path, size, offset, fh):
+        if path == "/":
+            return self.data[offset:offset + size]
+        raise FuseOSError(errno.EIO)
+
+    def ioctl(self, path, cmd, arg, fip, flags, data):
+        if path == "/":
+            if cmd == self.ioctl_cmd:
+                self.log.debug(f"Handling rosetta handshake")
+                ctypes.memmove(data, self.handshake, len(self.handshake))
+                return 0
+            else:
+                self.log.error(f"Unsupported ioctl({hex(cmd)}) -> Probably the virtualization check changed")
+
+        raise FuseOSError(errno.ENOTTY)
+
+def main():
+    parser = argparse.ArgumentParser("mount-rosetta")
+    parser.add_argument("rosetta_binary", help="Path to the rosetta binary to mount")
+    parser.add_argument('--debug', help="Enable debug logging", action=argparse.BooleanOptionalAction)
+    args = parser.parse_args()
+    logging.basicConfig(level=logging.DEBUG if args.debug else logging.INFO)
+    binary = args.rosetta_binary
+    if not os.path.isfile(binary):
+        raise FileNotFoundError(f"Rosetta binary not found: {binary}")
+    FUSE(RosettaFS(binary), binary, nothreads=True, foreground=True, allow_other=True, nonempty=True)
+
+if __name__ == "__main__":
+    main()

requirements.txt

@@ -0,0 +1 @@
+fusepy

tests/main-dynamic

@@ -0,0 +1,11 @@
+#include <stdio.h>
+
+/* Compilation:
+zig cc -target x86_64-linux-musl -static main.c -o main-static
+zig cc -target x86_64-linux-gnu main.c -o main-dynamic
+*/
+
+int main()
+{
+    puts("Hello from Rosetta!");
+}