Running Buildbot on IPv6

2022-11-26 — mosu

As I’ve struggled a bit here I thought I’d make this short write-up for others to find. The following writeup was done against Buildbot 3.6.1 on Ubuntu 22.04 & Arch Linux.

There are two well-known components to Buildbot: one master process (purposefully ignoring multi-master setup) & one or more workers. The master is the one providing the web interface, scheduling the work, keeping track of results & sending notifications. The workers do the work (well, duh!).

This means that we’re basically dealing with two different scenarios here:

The master process must listen on IPv6 addresses for connections, both for the web interface & the workers which connect on a dedicated port.
The workers must connect via IPv6 to the master.

Using the default configuration or the examples from the website neither of the two components use IPv6, for different reasons.

Both use the twistd Python framework for connection handling.

Making the master listen on IPv6

The configuration for the master is done solely via master.cfg, not in buildbot.tac.

Basically for every connection the master listens on a ConnectionString is used which contains both the port number & the address to listen on, e.g. "tcp:8010:interface=127.0.0.1" One might attempt to simply replace the IPv4 address 127.0.0.1 with an IPv6 address, but that doesn’t work as colons are used to separate key/value pairs within the ConnectionString. Therefore the colons must be escaped. Note that the syntax often used in other programs such as web browser to escape IPv6 addresses, placing them in square brackets such as [::], doesn’t work here either.

How are those colons escaped, though? With backslashes. However, these are Python strings, and Python uses backslashes for escaping in strings as well. Therefore in the source code we have to use two backslashes per colon. The resulting connection string might look like this: "tcp:8010:interface=\\:\\:" (which means “listen on all IPv6 & IPv4 addresses, regardless of the interface, on port 8010”).

Web interface handler

For the web interface listener, which is usually referred to as c["www"] (with c being the an instance of BuildmasterConfig), a dict is used. The port key can be a port number, or a full-blown connection string. Example:

``` c['www'] = { 'port': "tcp:8010:interface=\\:\\:", 'plugins': { 'waterfall_view': {}, 'console_view': {}, 'grid_view': {} }, 'auth': util.UserPasswordAuth({'user': 'supersecret'}), 'authz': authz, 'change_hook_auth': [strcred.makeChecker("file:changehook.passwd")], 'change_hook_dialects': { 'gitlab': { 'secret': 'reallysecret' } } } ```

As hinted in the example the web interface handles both requests from us humans as well as change hooks from version control systems.

Worker handler

The worker handler is configured via c["protocols"]["pb"], and again a dict with a key called port is used which can be set to a connection string. For example:

``` c['protocols'] = {'pb': {'port': "tcp:interface=\\:\\::port=9989"}} ```

Making the workers connect via IPv6

The configuration is done in buildbot.tac.

The worker creates an instance of the Worker class. The first two parameters are the host address & the port number of the worker handler on the master’s side. Unlike the master, the parameters cannot seem to be ConnectionStrings.

Furthermore, the twistd client framework used here cannot resolve DNS AAAA records at all. If you provide a host name instead of an IPv6 or IPv4 address as the first argument, the library will only try to resolve it to an A record. For AAAA-only DNS records this means that name resolution fails, and the worker will not start. For DNS entries with both AAAA and A records, only the A record will be used, and the connection will be attempted via IPv4.

Luckily you can use an IPv6 address directly as the first argument. As there are no ConnectionStrings involved, colons can be left as-is. For example (all following arguments left out as they aren’t important):

``` s = Worker('2001:db8:1234::2', 9989, …) ```

Making the master connect via IPv6

We have to go back to the master once more as it can also make certain outbound connections that you might want to run via IPv6. These include but aren’t limited to:

email notifications
version control system status updates (Gitlab, Github etc.)

Email notifications

The email notifier also uses a pair of parameters specifying the SMTP server’s hostname & its port. Unfortunately it uses the same twistd client components that the worker uses, too, with the same DNS resolution issues. Sure, you can specify an IPv6 address directly instead of a host name, but that’ll be problematic if you want to enable TLS as certificate validation will now fail. I don’t have a good solution for IPv6-only setups in this case.

An example without authentication & without TLS:

``` mailer = reporters.MailNotifier( fromaddr="buildbot-master@example.com", sendToInterestedUsers=False, extraRecipients=["buildbot-failure-recipient@example.com"], relayhost='2001:db8:fefe::9876', generators=[generator_failing]) c["services"].append(mailer) ```

Version control status handlers

Handlers that signal the build status to version control systems such as Gitlab & Github don’t use the twistd framework for communication, it seems. They probably use something like urllib2. For their configuration they actually expect a base URL such as https://gitlab.com, and luckily for us, if that address resolves to an AAAA record then the status is indeed sent via IPv6. Nothing for us to do here.

Other considerations

Last but not least: the build instructions you create must also take IPv6 connectivity into account. If you configure git repositories on IPv4-only websites such as Github (shame!), you won’t be able to use them in an IPv6-only environment. Dual-stack setups will be fine, though.

On the other hand: external tools such as git that do not use the same Python libraries will resolve DNS entries to AAAA & A records & prefer AAAA = IPv6 over A = IPv4 — meaning configuring a git repo on sites such as Gitlab will indeed be retrieved via IPv6.

mosu's Matroska stuff