Support for native macOS agents with local backend on provisioned VMs

[gsaslis]

I would like to add ephemeral macOS builders to our woodpecker setup (in order to build / sign macOS binaries) and as far as I can see, there is currently no support across the various providers for macOS VMs to be provisioned - then connected to Woodpecker as agents. Woodpecker Autoscaler sounds ideal to cater for this.

At the same time, I do see that at least one of the autoscaler's currently supported providers (Scaleway) does have an Apple Silicon offering, so I was wondering if this is something the maintainers would be open to.

I do see that the Scaleway API does not support "user data" (cloud-config), which seems to be the canonical method to provision VMs with woodpecker agent here, so I guess there is an open question with regards to how to install/configure woodpecker-agent on those VMs, post instance-creation.

Before further investigating, I wanted to check whether there's something I've missed and this is somehow an already-supported setup ?

As far as I understand, as long as the agent itself is running in a container, the local backend doesn't make much sense for this use case. It would only make sense if the agent was running natively on macOS itself, right?

Thanks in advance for your help!

[xoxys]

I'm not sure if the autoscaler is the best place to be honest. Right now, the autoscaler uses container setups for agents on the cloud provider VMs. As far as I know, you can't create and sign macOS apps in Docker, even if Docker is running on the Mac, right?

If that's the case, I would suggest doing the deployment yourself, e.g. with Terraform and Ansible but keep in mind spinning up macOS systems "on-request" is super slow and the headless management capabilities are real crap.

---

I've had to deploy macOS build capabilities to Drone CI (the origin of Woodpecker) in the past, and we've worked with several hosted mac solutions including mac stadium and scaleway. In my experience, this is all less stable than, Linux VMs or even Windows. We ended up with a handful of hosted mac minis from Hetzner. They were running all the time, and we only deployed the Drone Runner binary via Ansible. This was the most stable solution we could find over the years.

[gsaslis]

I'm not sure if the autoscaler is the best place to be honest. Right now, the autoscaler uses container setups for agents on the cloud provider VMs. As far as I know, you can't create and sign macOS apps in Docker, even if Docker is running on the Mac, right?

Yes, and this is exactly why I was asking. We'd need a non-container setup.

I've had to deploy macOS build capabilities to Drone CI (the origin of Woodpecker) in the past, and we've worked with several hosted mac solutions including mac stadium and scaleway. In my experience, this is all less stable than, Linux VMs or even Windows. We ended up with a handful of hosted mac minis from Hetzner. They were running all the time, and we only deployed the Drone Runner binary via Ansible. This was the most stable solution we could find over the years.

Thanks so much for sharing - this really helps! I'll probably go down a similar path with scaleway for now (hetzner doesn't seem to provide those any more).

Are there any plans at all to extend to support non-container setups, so that it could also provision VMs and run the agent natively? As far as I understand it's just the VMs user-data that would need to change to make that happen, right?

[xoxys]

As far as I understand it's just the VMs user-data that would need to change to make that happen, right?

Yes, that could work. If we can support this out of the box IDK, but custom user-data is supported already. The main problem is testing, especially in such cases, as we don't have access to all cloud providers. If the user experience (startup-times, reliability, features etc) would be the same for macOS and Linux this would be straight forward, but I fear that's not the case.

[gsaslis]

Great, thanks again for the quick reply!

I'll see if I can spare a few cycles to make that happen and report back here with progress if/when I do.

Do I have the correct understanding that the agent would need to start with WOODPECKER_BACKEND=local in that case? (And then jobs that want to use that agent would use e.g. image: bash instead of a container image name? )

I know this probably isn't the right place to ask about that but I couldn't find a lot of info on this setup.

[anbraten]

Do I have the correct understanding that the agent would need to start with WOODPECKER_BACKEND=local in that case? (And then jobs that want to use that agent would use e.g. image: bash instead of a container image name? )

That's correct.

In general you would adjust user-data to curl / wget the agent binary and run it as a service. So in case a new agent would be required it would spin up a macos system somewhere and start an agent. As you probably wont have any docker related stuff installed on the agent's system it would automatically fallback to the local backend, which executes commands as if you would enter them on the agent's shell directly.

The image is a kinda hacky way we use to specify the shell used by the local backend.

[gsaslis]

Perfect!

Thanks so much for confirming @anbraten !

I'll report back here as soon as I have something I can share.