Skip to content

Commit 673b2b9

Browse files
per1234per1234
committed
Add build infrastructure for native Apple Silicon host
Introduction ------------ On every release tag and manual trigger when the "Build for Apple Silicon" checkbox is checked, use a self-hosted GitHub Actions runner on an Apple Silicon-based Mac AWS EC2 instance to make the Arduino IDE build for native Apple Silicon host. Previously, the build workflow only produced a build for x86-64 (AKA "Intel") macOS hosts. These can be used on Apple Silicon machines via the Rosetta 2 translation software, but there is a performance impact. Native Apple Silicon builds had to be produced and published manually. Runner Setup ------------ The process of AWS EC2 configuration, preparation, runner installation, and repository configuration is quite complex. Documentation was added to the repository for these procedures as well as the script used to install the runner on the instance. Hosting this content in the project repository allows it to be version controlled. In addition to benefiting the maintainers of Arduino's own build system, these resources will make it possible for contributors to run the Apple Silicon builds in their forks in order to validate proposed changes. Sequence of Operations for Build -------------------------------- The sequence of events necessary to run the build: 1. Check for an existing EC2 host appropriate for use with the runner's instance. 2. If there is no existing host, allocate a host. 3. Start the instance. 4. Run the build on the self-hosted runner installed on the instance. 5. Stop the instance. 6. Release the host. Host Release System ------------------- In order to comply with the macOS license, AWS requires the host to be allocated for a minimum of 24 hours. It can not be released before that time has passed. For this reason, the host release can not be done by the build workflow. A dedicated GitHub Actions workflow named "Release Self-Hosted Runner Host" was added for this purpose. This workflow runs on a 15 minute interval. It checks for hosts that were allocated for this project's build runs. If a host is found, it attempts to release it. If the release attempt is successful or fails due to the 24 hour minimum allocation not having passed yet, the workflow run passes. If the release attempt fails for any other reason, the workflow run fails. A workflow status badge was added to the project readme to provide a visible indication of failed runs, which should be investigated in order to avoid excess charges to Arduino's AWS account that would result from the host being unnecessarily allocated for longer than the 24 hour minimum duration.
1 parent f5621db commit 673b2b9

File tree

6 files changed

+800
-16
lines changed

6 files changed

+800
-16
lines changed

‎.github/assets/install-mac2-github-actions-runner.sh‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
#!/bin/bash
2+
3+
# Install self-hosted GitHub Actions runner on macOS Apple Silicon machine
4+
# See:
5+
# - https://docs.github.com/actions/hosting-your-own-runners/adding-self-hosted-runners#adding-a-self-hosted-runner-to-a-repository
6+
# - https://docs.github.com/en/actions/hosting-your-own-runners/configuring-the-self-hosted-runner-application-as-a-service
7+
8+
runner_folder="${HOME}/actions-runner"
9+
10+
mkdir "$runner_folder"
11+
cd "$runner_folder" || exit 1
12+
# Download the latest runner package
13+
brew install jq
14+
runner_tag="$(curl -s -X GET 'https://api.github.com/repos/actions/runner/releases/latest' | jq -r '.tag_name')"
15+
runner_version="${runner_tag:1}"
16+
runner_file="actions-runner-osx-arm64-${runner_version}.tar.gz"
17+
curl -O -L "https://github.com/actions/runner/releases/download/${runner_tag}/${runner_file}"
18+
# Extract the installer
19+
tar xzf "./$runner_file"
20+
21+
# Create the runner
22+
# See: https://docs.github.com/en/actions/hosting-your-own-runners/using-labels-with-self-hosted-runners#programmatically-assign-labels
23+
./config.sh \
24+
--labels ec2,mac2,arduino_arduino-ide \
25+
--token "$RUNNER_REGISTRATION_TOKEN" \
26+
--unattended \
27+
--url "https://github.com/$REPO_SLUG"
28+
29+
# Install the runner service
30+
./svc.sh install

‎.github/workflows/build.yml‎

Lines changed: 246 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,11 @@ on:
1515
tags:
1616
- '[0-9]+.[0-9]+.[0-9]+*'
1717
workflow_dispatch:
18+
inputs:
19+
build-for-apple-silicon:
20+
description: Build for Apple Silicon
21+
type: boolean
22+
default: false
1823
pull_request:
1924
paths-ignore:
2025
- '.github/**'
@@ -32,40 +37,239 @@ env:
3237
GO_VERSION: "1.17"
3338
JOB_TRANSFER_ARTIFACT: build-artifacts
3439
CHANGELOG_ARTIFACTS: changelog
40+
SELF_HOSTED_RUNNER_AWS_PURPOSE_TAG_KEY: purpose
41+
SELF_HOSTED_RUNNER_AWS_PURPOSE_TAG_VALUE: GitHub Actions runner
42+
SELF_HOSTED_RUNNER_HOST_AVAILABILITY_ZONE: us-east-2b
43+
SELF_HOSTED_RUNNER_INSTANCE_REGION: us-east-2
44+
# certificate-secret: Name of the secret that contains the certificate.
45+
# certificate-password-secret: Name of the secret that contains the certificate password.
46+
# certificate-extension: File extension for the certificate.
47+
# APPLE_SIGNING_CERTIFICATE_P12 secret was produced by following the procedure from:
48+
# https://www.kencochrane.com/2020/08/01/build-and-sign-golang-binaries-for-macos-with-github-actions/#exporting-the-developer-certificate
49+
BASE_BUILD_MATRIX: |
50+
[
51+
{
52+
"name": "Windows",
53+
"runs-on": "windows-2019",
54+
"certificate-secret": "WINDOWS_SIGNING_CERTIFICATE_PFX",
55+
"certificate-password-secret": "WINDOWS_SIGNING_CERTIFICATE_PASSWORD",
56+
"certificate-extension": "pfx"
57+
},
58+
{
59+
"name": "Linux",
60+
"runs-on": "ubuntu-18.04"
61+
},
62+
{
63+
"name": "macOS x86",
64+
"runs-on": "macos-latest",
65+
"certificate-secret": "APPLE_SIGNING_CERTIFICATE_P12",
66+
"certificate-password-secret": "KEYCHAIN_PASSWORD",
67+
"certificate-extension": "p12"
68+
}
69+
]
70+
SELF_HOSTED_MATRIX: |
71+
[
72+
{
73+
"name": "macOS ARM",
74+
"runs-on": [
75+
"self-hosted",
76+
"ec2",
77+
"mac2",
78+
"arduino_arduino-ide"
79+
],
80+
"certificate-secret": "APPLE_SIGNING_CERTIFICATE_P12",
81+
"certificate-password-secret": "KEYCHAIN_PASSWORD",
82+
"certificate-extension": "p12"
83+
}
84+
]
3585
3686
jobs:
87+
select-targets:
88+
runs-on: ubuntu-latest
89+
outputs:
90+
build-matrix: ${{ steps.generate-build-matrix.outputs.matrix }}
91+
use-self-hosted-runner: ${{ steps.self-hosted-runner-determination.outputs.use }}
92+
steps:
93+
- name: Determine whether to run build on self-hosted runner
94+
id: self-hosted-runner-determination
95+
run: |
96+
# Only run the build on self-hosted runner on release or select manually triggered runs.
97+
if [[
98+
"${{ secrets.AWS_SECRET_ACCESS_KEY }}" != "" && (
99+
"${{ startsWith(github.ref, 'refs/tags/') }}" == "true" ||
100+
(
101+
"${{ github.event_name }}" == "workflow_dispatch" &&
102+
"${{ github.event.inputs.build-for-apple-silicon }}" == "true"
103+
)
104+
)
105+
]]; then
106+
echo "use=true" >> $GITHUB_OUTPUT
107+
else
108+
echo "use=false" >> $GITHUB_OUTPUT
109+
fi
110+
111+
- name: Generate build matrix
112+
id: generate-build-matrix
113+
run: |
114+
if [[ "${{ steps.self-hosted-runner-determination.outputs.use }}" == "true" ]]; then
115+
# Use -c to avoid the need to deal with multi-line content in workflow step output
116+
matrix="$(echo '{"base": ${{ env.BASE_BUILD_MATRIX }}, "self_hosted": ${{ env.SELF_HOSTED_MATRIX }}}' | jq -c '.base + .self_hosted')"
117+
else
118+
matrix="$(echo '${{ env.BASE_BUILD_MATRIX }}' | jq -c '.')"
119+
fi
120+
121+
echo "matrix=$matrix" >> $GITHUB_OUTPUT
122+
123+
allocate-self-hosted-runner-host:
124+
needs: select-targets
125+
runs-on: ubuntu-latest
126+
outputs:
127+
host-id: ${{ steps.allocate-host.outputs.id }}
128+
env:
129+
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
130+
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
131+
steps:
132+
- name: Check for existing host
133+
if: needs.select-targets.outputs.use-self-hosted-runner == 'true'
134+
id: host-available
135+
run: |
136+
# See: https://awscli.amazonaws.com/v2/documentation/api/latest/reference/ec2/describe-hosts.html
137+
hosts="$(
138+
aws ec2 describe-hosts \
139+
--filter \
140+
Name="instance-type",Values="mac2.metal" \
141+
Name="state",Values="available" \
142+
Name="tag:${{ env.SELF_HOSTED_RUNNER_AWS_PURPOSE_TAG_KEY }}",Values="${{ env.SELF_HOSTED_RUNNER_AWS_PURPOSE_TAG_VALUE }}" \
143+
--output "text" \
144+
--query 'Hosts[?AvailableCapacity.AvailableInstanceCapacity[0].AvailableCapacity<`0`].HostId' \
145+
--region "${{ env.SELF_HOSTED_RUNNER_INSTANCE_REGION }}"
146+
)"
147+
148+
if [[ "$hosts" == "" ]]; then
149+
echo "available=false" >> $GITHUB_OUTPUT
150+
else
151+
echo "available=true" >> $GITHUB_OUTPUT
152+
fi
153+
154+
- name: Allocate host
155+
if: >
156+
needs.select-targets.outputs.use-self-hosted-runner == 'true' &&
157+
steps.host-available.outputs.available == 'false'
158+
run: |
159+
# See: https://awscli.amazonaws.com/v2/documentation/api/latest/reference/ec2/allocate-hosts.html
160+
aws ec2 allocate-hosts \
161+
--auto-placement "on" \
162+
--availability-zone "${{ env.SELF_HOSTED_RUNNER_HOST_AVAILABILITY_ZONE }}" \
163+
--instance-type "mac2.metal" \
164+
--output "text" \
165+
--quantity 1 \
166+
--region "${{ env.SELF_HOSTED_RUNNER_INSTANCE_REGION }}" \
167+
--tag-specifications \
168+
'ResourceType="dedicated-host",Tags=[{Key=Name,Value=mac2-github-actions-runner-arduino_arduino-ide},{Key=${{ env.SELF_HOSTED_RUNNER_AWS_PURPOSE_TAG_KEY }},Value=${{ env.SELF_HOSTED_RUNNER_AWS_PURPOSE_TAG_VALUE }}}]'
169+
170+
start-self-hosted-runner:
171+
needs:
172+
- select-targets
173+
- allocate-self-hosted-runner-host
174+
runs-on: ubuntu-latest
175+
outputs:
176+
instance-id: ${{ steps.get-instance-id.outputs.id }}
177+
env:
178+
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
179+
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
180+
steps:
181+
- name: Get instance ID
182+
if: needs.select-targets.outputs.use-self-hosted-runner == 'true'
183+
id: get-instance-id
184+
env:
185+
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
186+
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
187+
run: |
188+
# See: https://awscli.amazonaws.com/v2/documentation/api/latest/reference/ec2/describe-instances.html
189+
id="$(
190+
aws ec2 describe-instances \
191+
--filters \
192+
Name="instance-type",Values="mac2.metal" \
193+
Name="tag:${{ env.SELF_HOSTED_RUNNER_AWS_PURPOSE_TAG_KEY }}",Values="${{ env.SELF_HOSTED_RUNNER_AWS_PURPOSE_TAG_VALUE }}" \
194+
Name="tag:project",Values="arduino/arduino-ide" \
195+
--output "text" \
196+
--query "Reservations[0].Instances[0].InstanceId" \
197+
--region "${{ env.SELF_HOSTED_RUNNER_INSTANCE_REGION }}"
198+
)"
199+
200+
if [[ "$id" == "None" ]]; then
201+
echo "::error::Instance not found."
202+
exit 1
203+
fi
204+
205+
echo "id=$id" >> $GITHUB_OUTPUT
206+
207+
- name: Start instance
208+
if: needs.select-targets.outputs.use-self-hosted-runner == 'true'
209+
id: start-instance
210+
env:
211+
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
212+
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
213+
run: |
214+
# See: https://awscli.amazonaws.com/v2/documentation/api/latest/reference/ec2/start-instances.html
215+
aws ec2 start-instances \
216+
--instance-ids "${{ steps.get-instance-id.outputs.id }}" \
217+
--output "text" \
218+
--region "${{ env.SELF_HOSTED_RUNNER_INSTANCE_REGION }}"
219+
220+
echo "Instance started."
221+
echo "It may take up to 40 minutes for the runner to be ready."
222+
echo "During that time, the build job will remain in a \"Waiting for a runner to pick up this job...\" state."
223+
37224
build:
38-
name: build (${{ matrix.config.os }})
225+
name: build (${{ matrix.config.name }})
226+
needs:
227+
- select-targets
228+
- start-self-hosted-runner
39229
strategy:
230+
fail-fast: false
40231
matrix:
41-
config:
42-
- os: windows-2019
43-
certificate-secret: WINDOWS_SIGNING_CERTIFICATE_PFX # Name of the secret that contains the certificate.
44-
certificate-password-secret: WINDOWS_SIGNING_CERTIFICATE_PASSWORD # Name of the secret that contains the certificate password.
45-
certificate-extension: pfx # File extension for the certificate.
46-
- os: ubuntu-18.04 # https://github.com/arduino/arduino-ide/issues/259
47-
- os: macos-latest
48-
# APPLE_SIGNING_CERTIFICATE_P12 secret was produced by following the procedure from:
49-
# https://www.kencochrane.com/2020/08/01/build-and-sign-golang-binaries-for-macos-with-github-actions/#exporting-the-developer-certificate
50-
certificate-secret: APPLE_SIGNING_CERTIFICATE_P12
51-
certificate-password-secret: KEYCHAIN_PASSWORD
52-
certificate-extension: p12
53-
runs-on: ${{ matrix.config.os }}
232+
config: ${{ fromJson(needs.select-targets.outputs.build-matrix) }}
233+
runs-on: ${{ matrix.config.runs-on }}
54234
timeout-minutes: 90
55235

56236
steps:
57237
- name: Checkout
58238
uses: actions/checkout@v3
59239

240+
- name: Set up Apple Silicon macOS self-hosted runner
241+
if: >
242+
runner.os == 'macOS' &&
243+
runner.arch == 'ARM64' &&
244+
contains(matrix.config.runs-on, 'self-hosted')
245+
run: |
246+
# Rosetta 2 is required to run tools used during the build process that were built for x86 architecture.
247+
softwareupdate --agree-to-license --install-rosetta
248+
249+
# See: https://github.com/actions/setup-python/blob/main/docs/advanced-usage.md#macos
250+
python_installation_folder="/Users/runner/hostedtoolcache"
251+
sudo mkdir -p "$python_installation_folder"
252+
sudo chown "$USER" "$python_installation_folder"
253+
60254
- name: Install Node.js 16.x
61255
uses: actions/setup-node@v3
62256
with:
63257
node-version: '16.x'
64258
registry-url: 'https://registry.npmjs.org'
65259

260+
- name: Install Yarn
261+
shell: bash
262+
run: |
263+
npm install --global "yarn@<2.x"
264+
66265
- name: Install Python 3.x
67266
uses: actions/setup-python@v4
68267
with:
268+
# This is required for the Apple Silicon runner because the action does not provide Python for arm64
269+
# architecture. Python >=3.11 for macOS is universal2 binary so the x64 architecture Python works for Apple
270+
# Silicon runner as well:
271+
# https://github.com/actions/python-versions#user-content-building-installation-packages
272+
architecture: x64
69273
python-version: '3.x'
70274

71275
- name: Install Go
@@ -114,6 +318,28 @@ jobs:
114318
name: ${{ env.JOB_TRANSFER_ARTIFACT }}
115319
path: electron/build/dist/build-artifacts/
116320

321+
stop-self-hosted-runner:
322+
needs:
323+
- select-targets
324+
- start-self-hosted-runner
325+
- build
326+
# The job must run any time the self-hosted runner instance was started.
327+
if: >
328+
always() &&
329+
needs.select-targets.outputs.use-self-hosted-runner == 'true' &&
330+
needs.start-self-hosted-runner.result == 'success'
331+
runs-on: ubuntu-latest
332+
steps:
333+
- name: Stop self-hosted runner instance
334+
env:
335+
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
336+
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
337+
run: |
338+
# See: https://awscli.amazonaws.com/v2/documentation/api/latest/reference/ec2/stop-instances.html
339+
aws ec2 stop-instances \
340+
--instance-ids "${{ needs.start-self-hosted-runner.outputs.instance-id }}" \
341+
--region "${{ env.SELF_HOSTED_RUNNER_INSTANCE_REGION }}"
342+
117343
artifacts:
118344
name: ${{ matrix.artifact.name }} artifact
119345
needs: build
@@ -128,9 +354,13 @@ jobs:
128354
- path: '*Linux_64bit.AppImage'
129355
name: Linux_X86-64_app_image
130356
- path: '*macOS_64bit.dmg'
131-
name: macOS_dmg
357+
name: macOS_X86-64_dmg
132358
- path: '*macOS_64bit.zip'
133-
name: macOS_zip
359+
name: macOS_X86-64.zip
360+
- path: '*macOS_ARM64.dmg'
361+
name: macOS_ARM64_dmg
362+
- path: '*macOS_ARM64.zip'
363+
name: macOS_ARM64_zip
134364
- path: '*Windows_64bit.exe'
135365
name: Windows_X86-64_interactive_installer
136366
- path: '*Windows_64bit.msi'

0 commit comments

Comments
 (0)