Improve CI builds

[jt-nti]

The builds have been getting more unreliable recently, which seems like it could be related to this discussion:
containers/podman#19258

There's also a muddle of build and publish steps without a clear separation, requiring more permissions than ideal in a lot of build jobs.

Perhaps something along these lines would be better?

• Cross compile build commands and upload the resulting artifacts
• Use those pre-build artifacts in the docker image instead of recompiling
• Separate publish jobs