Executing the custom container implementation failed. Please contact your self hosted runner administrator.

Process completed with exit code 1.

Error: failed to run script step: command terminated with non-zero exit code: error executing command [sh -e /__w/_temp/f641c890-4636-11f0-8283-b19cda9ee69a.sh], exit code 1

buildx failed with: ERROR: failed to receive status: rpc error: code = Unavailable desc = error reading from server: read tcp 10.0.163.28:40776->10.4.180.29:1234: read: connection reset by peer

Executing the custom container implementation failed. Please contact your self hosted runner administrator.

Process completed with exit code 1.

Error: failed to run script step: command terminated with non-zero exit code: error executing command [sh -e /__w/_temp/0505d7e0-4637-11f0-9c8f-574d62cfe945.sh], exit code 1

buildx failed with: ERROR: failed to solve: process "/bin/bash -eo pipefail -c export MAX_JOBS=\"${BUILD_MAX_JOBS:-$(./scale.sh \"$(./effective_cpu_count.sh)\" 3 32)}\" && echo \"MAX_JOBS: ${MAX_JOBS}\" && export NVCC_APPEND_FLAGS=\"$(cat /build/nvcc.conf)\" && echo \"NVCC_APPEND_FLAGS: ${NVCC_APPEND_FLAGS}\" && if [ -n \"${BUILD_CXX11_ABI}\" ]; then export _GLIBCXX_USE_CXX11_ABI=\"${BUILD_CXX11_ABI}\"; fi && ./storage-info.sh . && cd pytorch && ../storage-info.sh . && mkdir build && ln -s /usr/bin/cc build/cc && ln -s /usr/bin/c++ build/c++ && if [ \"$(uname -m)\" = 'aarch64' ]; then export USE_PRIORITIZED_TEXT_FOR_LD=1; fi && { if [ -d /opt/nccl-tests ]; then export USE_DISTRIBUTED=1 USE_NCCL=1 USE_SYSTEM_NCCL=1 UCC_HOME=${HPCX_UCC_DIR} UCX_HOME=${HPCX_UCX_DIR} USE_NCCL_WITH_UCC=1 USE_UCC=1 USE_SYSTEM_UCC=1; fi; } && USE_CUDNN=1 BUILD_TORCH=ON BUILD_TEST=0 CUDA_HOST_COMPILER=cc USE_CUDA=1 USE_NNPACK=1 CC=cc CXX=c++ USE_BLAS=1 USE_LAPACK=1 WITH_BLAS=FLAME PYTORCH_BUILD_VERSION=\"$(../version-string.sh \"$TORCH_VERSION\")\" PYTORCH_BUILD_NUMBER=0 TORCH_NVCC_FLAGS=\"-Xfatbin -compress-all\" python3 setup.py bdist_wheel --dist-dir ../dist 2>&1 | grep -Ev --line-buffered '^(ptxas /tmp/|copying .+/|creating build/)'" did not complete successfully: exit code: 1

Executing the custom container implementation failed. Please contact your self hosted runner administrator.

Process completed with exit code 1.

Error: failed to run script step: command terminated with non-zero exit code: error executing command [sh -e /__w/_temp/06d9fba0-4637-11f0-9a39-e1b47e11fe3b.sh], exit code 1

buildx failed with: ERROR: failed to solve: process "/bin/sh -c export CC=$(realpath -e ./compiler) MAX_JOBS=\"${BUILD_FLASH_ATTN_MAX_JOBS:-$(./scale.sh \"$(./effective_cpu_count.sh)\" 8 12)}\" && echo \"MAX_JOBS: ${MAX_JOBS}\" && export NVCC_APPEND_FLAGS=\"$(cat /build/nvcc.conf)\" && echo \"NVCC_APPEND_FLAGS: ${NVCC_APPEND_FLAGS}\" && cd flash-attention && for EXT_DIR in $(realpath -s -e . csrc/ft_attention csrc/fused_dense_lib csrc/fused_softmax csrc/layer_norm csrc/rotary csrc/xentropy); do /build/fa-build.sh \"$EXT_DIR\" || exit 1; done" did not complete successfully: exit code: 1

Executing the custom container implementation failed. Please contact your self hosted runner administrator.

Process completed with exit code 1.

Error: failed to run script step: command terminated with non-zero exit code: error executing command [sh -e /__w/_temp/fd5638f0-4636-11f0-b966-5f9153265abe.sh], exit code 1

buildx failed with: ERROR: failed to receive status: rpc error: code = Unavailable desc = closing transport due to: connection error: desc = "error reading from server: EOF", received prior goaway: code: NO_ERROR, debug data: "graceful_stop"

Executing the custom container implementation failed. Please contact your self hosted runner administrator.

Process completed with exit code 1.

Error: failed to run script step: command terminated with non-zero exit code: error executing command [sh -e /__w/_temp/fa5c0bc0-4636-11f0-a262-437e4bfcea47.sh], exit code 1

buildx failed with: ERROR: failed to solve: DeadlineExceeded: failed to push ghcr.io/coreweave/ml-containers/nightly-torch: no active session for iyqzl2x6jlukm59zwoun0q6v1: context deadline exceeded

Executing the custom container implementation failed. Please contact your self hosted runner administrator.

Process completed with exit code 1.

Error: failed to run script step: command terminated with non-zero exit code: error executing command [sh -e /__w/_temp/fc2a6320-4636-11f0-b19e-530d51bbbb15.sh], exit code 1

buildx failed with: ERROR: failed to solve: process "/bin/sh -c export CC=$(realpath -e ./compiler) MAX_JOBS=\"${BUILD_FLASH_ATTN_MAX_JOBS:-$(./scale.sh \"$(./effective_cpu_count.sh)\" 8 12)}\" && echo \"MAX_JOBS: ${MAX_JOBS}\" && export NVCC_APPEND_FLAGS=\"$(cat /build/nvcc.conf)\" && echo \"NVCC_APPEND_FLAGS: ${NVCC_APPEND_FLAGS}\" && cd flash-attention && for EXT_DIR in $(realpath -s -e . csrc/ft_attention csrc/fused_dense_lib csrc/fused_softmax csrc/layer_norm csrc/rotary csrc/xentropy); do /build/fa-build.sh \"$EXT_DIR\" || exit 1; done" did not complete successfully: exit code: 1

Failed to save: reserveCache failed: Cache service responded with 503

Docker is required to export a build record

Failed to restore: getCacheEntry failed: Cache service responded with 503

Failed to save: reserveCache failed: Cache service responded with 503

Docker is required to export a build record

Failed to restore: getCacheEntry failed: Cache service responded with 503

Failed to save: reserveCache failed: Cache service responded with 503

Docker is required to export a build record

Failed to restore: getCacheEntry failed: Cache service responded with 503

Failed to save: reserveCache failed: Cache service responded with 503

Docker is required to export a build record

Failed to restore: getCacheEntry failed: Cache service responded with 503

Failed to save: reserveCache failed: Cache service responded with 503

Docker is required to export a build record

Failed to restore: getCacheEntry failed: Cache service responded with 503

Failed to save: reserveCache failed: Cache service responded with 503

Docker is required to export a build record