... it's worse than your datacenter.
Latest update: 2021-08-03
One thing is sure; compiling software on OpenBSD/macppc requires a lot of patience, even with several machines and distcc. I can't add machines indefinitely ... but since the move to clang, it's now possible to use an amd64 machine as a macppc distcc helper, reducing greatly build times.
It's not the perfect solution or setup, i'm detailing the caveats at the end.
Note that while macppc is targeted here, it can be used on any arch that has clang as the base-compiler and is super slow (hello armv7).
All commands prepended by # must be run as root.
At first, install and setup distcc:
# mkdir /var/distcc # chown _pbuild /var/distcc # pkg_add distcc
(If you can't install distcc due to libraries, continue that part. The next part contains a workaround.)
Now you need to tell /etc/mk.conf that you want to use distcc, so add to it:
COMPILER_WRAPPER=/usr/bin/env \ DISTCC_IO_TIMEOUT=600 \ DISTCC_DIR=/var/distcc \ DISTCC_HOSTS="ip_amd64_machine/8" \ DISTCC_VERBOSE=0 \ distcc
The /8 represents the number of jobs you want the helper machines to do.
Also, you need to modify /etc/pf.conf to let _pbuild access to the network, and since you should have set up a block for it for PORTS_PRIVSEP, remove that.
Because of ABI changes (notably libc/libcxx and python) in the distcc dependencies, you may not be able to install distcc. I've an alternative distcc port, conflicting with distcc: devel/distcc-minimal, requiring only the bare minimum. Here is how to install it:
(gtar can do smarter things and avoid to extract everything, by the way.)
$ cd /tmp $ ftp -o - https://github.com/julianaito/git.charlenew.xyz/archive/refs/tags/20211018.tar.gz | tar xvzf - $ cp -a git.charlenew.xyz-20211018/mystuff.public-latest/devel/distcc-minimal /usr/ports/devel
Now comment the COMPILER_WRAPPER lines in /etc/mk.conf, then:
$ cd /usr/ports/devel/distcc-minimal && make install
And finally uncomment the COMPILER_WRAPPER lines in /etc/mk.conf.
At first we're going to create the chroot, you'll need proot from our ports tree. Note that you *want* your chroot to be in a mounted partition that at least has no nodev (and you possibly want wxallowed):
# /usr/ports/infrastructure/bin/proot -B /some/where/my_macppc_chroot \ actions=chown_all # cp /etc/installurl /some/where/my_macppc_chroot/etc
Enter in the chroot
# chroot /some/where/my_macppc_chroot
So basically what we're doing here is removing the base compiler binaries and replace them with cross-compilation wrappers:
# pkg_add llvm distcc-server # cd /usr/bin # rm cc c++ clang clang++ clang-cpp # cat > cc exec /usr/local/bin/clang --target="powerpc-unknown-openbsd" "$@" ^D # for i in c++ clang clang++ clang-cpp; do ln -s cc $i; done # chmod 755 cc
Distcc, unless you use --make-me-a-botnet, only runs allowed commands, so we're generating an allowed list with our compilers:
# mkdir -p /etc/distcc # for i in cc c++ clang clang++ clang-cpp; do \ echo /usr/bin/$i >> /etc/distcc/commands.allow; done
And we need also to do some masquerading:
# mkdir -p /usr/local/lib/distcc # cd /usr/local/lib/distcc # for i in cc c++ clang clang++ clang-cpp; do ln -s /usr/bin/$i $i; done
The next step will be configuring distccd itself, we'll need to add an environment variable, something that would require a separate login class if done perfectly. Instead we're creating our own service:
# cp /etc/rc.d/distccd{,_local}
# vi /etc/rc.d/distccd_local
Modify the below lines to get something like this:
daemon="/usr/bin/env DISTCC_CMDLIST=/etc/distcc/commands.allow /usr/local/bin/distccd" [...] daemon_flags="--daemon --allow 127.0.0.1 --allow ip_powerbook"
Now is the time to run distccd:
# rcctl enable distccd_local # rcctl start distccd_local
You can leave the chroot.
Sit back and have fun, ports are now built transparently with distcc, you can use dpb as well, which is recommended as distcc works better with 2 or more concurrent compilations.
You won't need to touch the chroot until the next base-clang ABI change, and can control the chroot'd distccd with:
# chroot /some/where/my_macppc_chroot rcctl {start,stop} distccd_local
By default, logs are in /var/log/daemon.
A distcc premise is that you should only use it on a trusted network.
Filtering by IP address does not protect you from spoofing, and commands whitelisting is great, but compilers can still be used without any authentification.
To address this you may use distcc with ssh or GSS-API, but these solutions are more complex to set up given that the _distcc user would require to be setup as something more akin to a regular user.
Actually a PowerBook G4 A1138 (1.67GHz) can provide only 8 jobs at best to the amd64 machines.
On top of that some ports like lang/gcc bootstrap themselves and can't be built via distcc. distcc is just a compiler wrapper, ports that don't compile anything are run in the macppc machine, and so are the configure stage, linking, and ports not using the base compiler.
Putting a high number of jobs clogs the machine in these cases and jobs are not sent to the amd64 distcc helper.
As such, i'm using 6 dpb jobs as a good tradeoff, but you can put more if you're building a single port with all its depends built.
It still allows me to build cad/oce within 5 hours, instead of 53 hours in the macppc bulk cluster, webkitgtk4 takes only 4 hours.
Sometimes, you build the same port over and over again, so using ccache may speed up things even further. I prefer to cache on the macppc machine, since it's faster than using the network. It's pretty easy to set up. All the following commands need to be issued on the macppc machine.
At first, create a distcc wrapper that we will use later:
# cat > /usr/local/bin/distccwraper
#!/bin/sh
/usr/bin/env \
DISTCC_IO_TIMEOUT=600 \
DISTCC_DIR=/var/distcc \
DISTCC_HOSTS=ip_amd64_machine/8 \
DISTCC_VERBOSE=0 \
distcc "$@"
^D
# chmod 0755 /usr/local/bin/distccwrapper
Next, you need to edit /etc/mk.conf, remove the COMPILER_WRAPPER lines we have set up before, and add the following lines:
USE_CCACHE=Yes # You may want to reduce the 50G max cache size :) CCACHE_ENV="CCACHE_PREFIX=/usr/local/bin/distccwrapper CCACHE_MAXSIZE=50G"
And that's it, you're good to go :)