FreeBSD on ARM64 fails to acquire an IP

[yorickpeterse]

An example is found in https://github.com/inko-lang/inko/actions/runs/6518403904/job/17703959993: the build fails to acquire an IP for the MAC address:

  Initializing VM
  Booting VM
  Getting IP address for MAC address: 4e:70:ef:c2:f2:ed
  Waiting for IP to become available...
  Error return from kevent change: Invalid argument
  Consoles: userboot  
  
  FreeBSD/amd64 User boot, Revision 1.1
  (root@fbsd, Sat Jun 20 15:45:53 PDT 2015)
  |�/�-�\�|�/�-�\�|�/�-�\�|�/�-�\�|�/�-�\�|�/�-�\�|�/�-�\�|�/�-�\�|�/�-�\�|�/�-�\�|�/�-�\�|�/�-�\�|�/�-�\�|�/�-�\�|�/�-�\�|�/�-�\�Loading /boot/defaults/loader.conf
  |�/�-�\�|�/�-�\�|�/�-�\�|�/�-�\�|�/�-�\�|�/�-�\�|�/�-�\�|�/�-�\�|�/�-�\�|�/�-�\�|�/�-�\�|�/�-�\�|�/�-�\�|�/�-�\�|�/�-�\�|�/�-�\�|�/�-�\�|�/�-�\�|�/�-�\�|�/�-�\�|�/�don't know how to load module '/boot/kernel/kernel'
  -�\�|�/�-�\�|�/�-�\�
  |�/�-�\�|�/�-�\�|�/�-�\�|�/�-�\�|�/�-�\�|�/�-�\�|�/�-�\�|�/�-�\�|�/�-�\�|�/�-�\�|�/�can't load 'kernel'
  
  Type '?' for a list of commands, 'help' for more detailed help.
  OK Waiting for IP to become available...
  Waiting for IP to become available...
  Waiting for IP to become available...
  Waiting for IP to become available...
...

The configuration for the job is as follows:

jobs:
  compiler:
    strategy:
      fail-fast: false
      matrix:
        version:
          - '1.68'
          - stable
        arch:
          - x86-64
          - arm64
    runs-on: macos-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/cache@v3
        with:
          path: |
            ${{ env.CARGO_HOME }}
            target
          key: freebsd-${{ matrix.arch }}-${{ matrix.version }}
      - name: Run tests
        uses: cross-platform-actions/action@v0.19.1
        with:
          operating_system: freebsd
          version: '13.2'
          architecture: '${{ matrix.arch }}'
          environment_variables: 'CARGO_HOME LIBRARY_PATH CARGO_REGISTRIES_CRATES_IO_PROTOCOL'
          hypervisor: qemu
          image_url: 'https://github.com/inko-lang/freebsd-builder/releases/download/v0.6.2/freebsd-13.2-${{ matrix.arch }}.qcow2'
          run: |
            ./scripts/rust.sh ${{ matrix.version }}
            setenv PATH "${CARGO_HOME}/bin:${PATH}"
            echo "::group::Running tests"
            cargo test
            echo "::endgroup::"

Did I perhaps misconfigure something?

[jacob-carlborg]

Looks like it's using Xhyve as the hypervisor, for some reason. The output contains: FreeBSD/amd64 User boot, which is not correct for ARM64. Could you please try to enable debug info by setting these two variables: ACTIONS_RUNNER_DEBUG and ACTIONS_STEP_DEBUG to true? You can set them in project settings -> Secrets and variables -> Actions -> Variables.

[jacob-carlborg]

Could you also please try without the custom image?

[yorickpeterse]

Enabling the debug flags and not using the custom image resulted in build https://github.com/inko-lang/inko/actions/runs/6520442318/job/17708002043. This build made it to the actual test stage, then failed (as intended) due to the default image not including all the necessary dependencies.

I'm guessing that if a custom image URL is specified, somehow the wrong hypervisor is picked? I'll run a new job with the custom image and the debug output to see what happens then.

[yorickpeterse]

Of course with the debug flags enabled, it seems to work just fine: https://github.com/inko-lang/inko/actions/runs/6520506031/job/17708119514

I'm wondering if it's the matrix configuration: to speed up this debugging, I changed my matrix to only run using 1 instead of 2 Rust versions. I'll change it back and see what happens then.

[yorickpeterse]

Ah shit I'm an idiot: I had stashed the changes to run FreeBSD on ARM, so these jobs are all running on x86-64 🤦 I'll change the config to actually use ARM and try again.

[yorickpeterse]

With ARM, the first job failed due to #64: https://github.com/inko-lang/inko/actions/runs/6520549770/job/17708200134

https://github.com/inko-lang/inko/actions/runs/6520549770/job/17708585192 is a retry of that job, and it's currently stuck waiting for an IP, and will fail as a result in the next few minutes. This is without a custom image.

[jacob-carlborg]

inko-lang/inko/actions/runs/6520549770/job/17708585192 is a retry of that job, and it's currently stuck waiting for an IP, and will fail as a result in the next few minutes. This is without a custom image.

Yeah, it definitely selects Xhyve as the hypervisor, which is wrong for ARM64. My FreeBSD ARM64 tests are only running on Linux runners [1]. Seems to be some condition that is wrong which selects the hypervisor and prefers Xhyve on macOS runners. Try using Linux runners instead.

[1]

action/.github/workflows/ci.yml

Lines 47 to 52 in 6785ab1

	exclude:
	- version: '12.2'
	architecture: { name: arm64 }
	- architecture: { name: arm64 }
	host: { name: macos-latest }

[yorickpeterse]

@jacob-carlborg Using ubuntu-latest as the runner, it seems the job makes it past the setup stage: https://github.com/inko-lang/inko/actions/runs/6523844533/job/17714887627. The failure is because there's apparently no Rust toolchain for ARM64 FreeBSD, but that's unrelated to this issue.

So it seems the combination of using ARM64 on the macOS runner causes the action to pick the wrong hypervisor.

[jacob-carlborg]

Looks like I never tested ARM64 on a macOS runner. There's really no point (yet) to run ARM64 on macOS. The only point to use a macOS runner is to get hardware accelerated nested virtualization, but since that doesn't apply to ARM64 there's no point. But one can argue it should work anyway. It starts to become quite a few jobs in the CPA matrix 😃.

[yorickpeterse]

@jacob-carlborg I'm fine with it not being supported, but in that case a mention of this in the README may be useful (unless I'm blind and overlooked such a mention already being present) 😃

[jacob-carlborg]

I added it to the readme for now [1]. It should get fix because GitHub recently added support for macOS Apple Silicon runners.

[1] https://github.com/cross-platform-actions/action/#runners

[yorickpeterse]

@jacob-carlborg Thanks!