illumos support

[nshalman]

Is your feature request related to a problem? Please describe.

The main branch of tailscale/tailscale doesn't compile on/for illumos.

Describe the solution you'd like

I have hosts where GOOS=illumos and I want to run Tailscale on them.
Ideally I'd like to be able to just build from main.

Describe alternatives you've considered

I have configured Wireguard manually, and I've used Algo, but Tailscale is pretty sweet.

Additional context

I have managed to hack together working binaries based on a fork of wireguard-go built on top of the work of someone else who does not have the bandwidth to get the changes upstream, but has no objection to them being upstreamed. My branch is here.

Attempting to merge main fully this evening fails with errors I haven't had time to debug yet:

# tailscale.com/wgengine/tsdns
wgengine/tsdns/forwarder.go:211:23: undefined: os.ErrDeadlineExceeded
wgengine/tsdns/forwarder.go:228:23: undefined: os.ErrDeadlineExceeded
# tailscale.com/derp/derphttp
derp/derphttp/derphttp_client.go:518:9: undefined: tls.Dialer

Some questions on my mind:

• Is it reasonable to open up a PR against tailscale/wireguard-go with a cleaned up commit to bring in illumos support?
• Is there documentation on how to run the tests (I'm still relatively new to Go and am not finding it obvious...)
• Can I help provide documentation/tooling for how to get a usable test environment for illumos CI builds?

[danderson]

Thanks for doing the work! I used Illumos briefly, and would love to have Tailscale working on it :)

To answer your questions:

• It would be better to open PRs against wireguard/wireguard-go, which is a copy of the authoritative upstream for that codebase. Assign crawshaw and myself as reviewers, and we can get it merged in the official WireGuard upstream. That would benefit all WireGuard users, rather than just Tailscale.
• To run tests in the OSS repo, run go test ./... from the root of the git checkout. That'll run all tests in all subdirectories. You can also go test ./some/specific/thing.
• The main thing we'd need for an illumos CI is instructions on how to run an Illumos box in a cheap cloud somewhere (e.g. DigitalOcean VPS), and getting the Github Actions runner working on it so we can plug it into our existing CI.
• As an aside, we would also need some operational stuff in general: how do we package stuff for Illumos? How do we run stuff under SMF? Can this be automated in a similar vein to Debian packages?

[jclulow]

* The main thing we'd need for an illumos CI is instructions on how to run an Illumos box in a cheap cloud somewhere (e.g. DigitalOcean VPS), and getting the Github Actions runner working on it so we can plug it into our existing CI.

If DO is an option, this will be pretty easy to get going. I have a pipeline for emitting images of OpenIndiana and OmniOS that will run in a droplet -- you can use the "import custom image" function and provide it a URL and it all pretty much just works.

* As an aside, we would also need some operational stuff in general: how do we package stuff for Illumos? How do we run stuff under SMF? Can this be automated in a similar vein to Debian packages?

Packaging is generally distribution-specific. If it builds and works out of the box, it will be pretty easy for OmniOS and OpenIndiana to include it in their build recipe repository that they use to build the binary packages they ship.

For SMF you need a particular XML file that is analogous to a systemd unit file. The contents will, like a unit file, depend on where the binaries go in the file system, but can often be otherwise relatively similar or the same between the common distributions.

[danderson]

DO is an option, as is Vultr. I should also add that for OSes we don't officially support (e.g. currently FreeBSD and OpenBSD, which work thanks to third-party contribs), our CI usually just cross-compiles the Go code to make sure things still build, but the unit tests don't run. Illumos would be a novelty for us in that department, in that we'd end up running a CI box for a platform we're not officially supporting. Maybe the simplest to get started would be to do cross-compile only in CI, and we can build out more over time.

Operational stuff SGTM. We'd happily take patches for an SMF config, and whatever extra packaging niceties the downstream distros might want to keep in upstream (I don't know if they usually operate more like BSD Ports, where the distro packaging is disconnected from upstream).

As I mentioned on Twitter, some additional challenges for illumos support are going to be in the bowels of the network engine, to get from "mostly works" to "every feature works" :

• Routing loop avoidance. tailscaled's own traffic must not use Tailscale routes, otherwise some setups can end up with routing loops for WireGuard traffic, or loss of connectivity to the Tailscale control plane.
  • Linux does this with policy routing and SO_MARK, so that all packets originated from the tailscale binary ignore Tailscale routes. MacOS/iOS/Android have things Just Work because those platforms's VPN frameworks take care of that for us. Windows and the BSDs currently don't avoid routing loops. I'm poised to fix Windows in a poor way (SO_BINDTODEVICE on sockets, which will only really work right on non-routers). I'm hoping Illumos's networking facilities have something we can use to get a similar outcome to linux/iDevices/android?
• We need a bit of firewall configuration to work correctly. In particular, optional SNAT of traffic coming from Tailscale and going to a local network, and filtering of 100.64.0.0/10 packets coming from somewhere other than tailscale0 (to prevent spoofing-based attacks). Tailscale configures this in the wgengine/router package on startup. I'm hoping Illumos has similar facilities, so we can get a setup on par with linux in terms of security and functionality.

If those two pieces are missing, Tailscale will still mostly work, but some topologies will result in routing loops that break Tailscale, and illumos will only be usable as an end-host, not a subnet router.

[nshalman]

* It would be better to open PRs against `wireguard/wireguard-go`, which is a copy of the authoritative upstream for that codebase. Assign crawshaw and myself as reviewers, and we can get it merged in the official WireGuard upstream. That would benefit all WireGuard users, rather than just Tailscale.

Opened WireGuard/wireguard-go#39 I will note that nshalman/wireguard-go@e06f6c5 seems to be necessary when trying to merge into the tailscale fork.

* To run tests in the OSS repo, run `go test ./...` from the root of the git checkout. That'll run all tests in all subdirectories. You can also `go test ./some/specific/thing`.

Thanks! That shows me failures to hunt down. :)

[admin@tailscale2 ~/tailscale]$ GOOS=illumos go test ./...
?       tailscale.com/atomicfile        [no test files]
?       tailscale.com/cmd/cloner        [no test files]
# github.com/kr/pty
../go/pkg/mod/github.com/kr/pty@v1.1.1/ioctl.go:8:29: undefined: syscall.SYS_IOCTL
../go/pkg/mod/github.com/kr/pty@v1.1.1/util.go:55:3: undefined: syscall.SYS_IOCTL
ok      tailscale.com/cmd/derper        0.037s
?       tailscale.com/cmd/microproxy    [no test files]
?       tailscale.com/cmd/mkpkg [no test files]
?       tailscale.com/cmd/tailscale     [no test files]
?       tailscale.com/cmd/tailscale/cli [no test files]
?       tailscale.com/cmd/tailscaled    [no test files]
<remainder "ok">

* As an aside, we would also need some operational stuff in general: how do we package stuff for Illumos? How do we run stuff under SMF? Can this be automated in a similar vein to Debian packages?

I have a rudimentary SMF manifest built with manifold that mostly works (I have a bug somewhere that means things don't get fully cleaned up on exit so restarts fail without manual intervention.) Once I solve that I'll make sure to include an example SMF manifest in my PR. I'll give some thought to the packaging question.

[jelmer]

Attempting to merge main fully this evening fails with errors I haven't had time to debug yet:

# tailscale.com/wgengine/tsdns
wgengine/tsdns/forwarder.go:211:23: undefined: os.ErrDeadlineExceeded
wgengine/tsdns/forwarder.go:228:23: undefined: os.ErrDeadlineExceeded
# tailscale.com/derp/derphttp
derp/derphttp/derphttp_client.go:518:9: undefined: tls.Dialer

This is probably because you're using a version of golang < 1.15

[fserb]

Can confirm that upgrading to 1.15 fixes the problem.

[bradfitz]

AFAICT, we have a number of Illumos active users on Tailscale. So I think this bug can be closed. Feel free to keep sending improvements, though.

[nshalman]

I don't think the code has actually landed upstream in your repo yet. Still, from an issue management perspective on your end, I'm fine with this being marked closed, as it can still be referenced in any future pull requests or I can open a new one when I (or someone else??) have the free time to work on it.

[bradfitz]

Oh, I'll reopen. I just assumed it must've been in our tree if people were using it.

So what tree are the Illumos users using? Or is it just 1 person with N nodes? (I didn't look)

[nshalman]

In terms of tree(s):

I have managed to hack together working binaries based on a fork of wireguard-go built on top of the work of someone else who does not have the bandwidth to get the changes upstream, but has no objection to them being upstreamed. My branch is here.

In terms of users, you'd have to look on your end. I account for 2 nodes. I haven't checked logs to see how many people have downloaded my binaries linked from https://blog.shalman.org/tailscale-for-illumos/ and even less insight into people who may have compiled it on their own.

[nshalman]

With help from @danderson, WireGuard/wireguard-go#39 is getting closer to being ready to land (waiting behind golang/sys#100) but there is still work do to on the tailscale side. I'm hacking away in a branch, but I think I'm going to have to borrow some stuff from the BSD side, and strangely enough, some of the quirks are similar to the Windows side (e.g. needs an IP address when adding a route.) Also, ipv6 doesn't seem to work on the illumos wireguard tun device yet. I'm hopeful it's a relatively straightforward change, but I don't know yet.)