Why we don't run freebsd-update inside freebsd.org

We don't run freebsd-update or portsnap in the freebsd.org cluster, but not for reasons you might think. It's more for philosophical and practical reasons. Having recently mentioned this in a firestorm on twitter, I will attempt to shed some light on why.

The first reason is because I'm an OS developer as well as doing cluster admin work. I am a firm believer in frequent updates and exercising the code under fire to try and provoke problems before things get too far out of hand. If I can cause a crash and get it fixed before you even see it then I'm happy. Cluster machines are updated typically every 1-4 weeks.

To that end, we run internal mini-releases over an internal secure update mechanism that is strictly based on what's in svn.freebsd.org. We do the same for base system binaries and for cluster-specific ports binary packages.

These internal mini-releases enable us to update 300+ hosts and jails as often as we like. It only takes a few hours now. About 90-95% of our systems run 12-current, including the master subversion repository server itself.

Our quick-and-dirty mini-releases work for us because I was able to define our operational environment for this to work. Most permutations were eliminated. I can make huge short cuts that won't work in the real world. As a result it takes longer for DNS to update than it takes to bring up a new host or new jail.

I also designed it so that everything can be tracked to corresponding svn revisions. We change every single file during a refresh. We use etcupdate(8) to manage a dynamic /etc environment. Everything is source repository derived. We churn the osreldate header in the ELF PT_NOTE section relentlessly.

This is all done on same-network servers. Pulling a 100MB update because of a 1-character typo is of no concern. At no point does any of this cross the naked scary internet.

On the flip side there is freebsd-update. Instead of being source repository based, freebsd-update makes a point of divorcing itself from the repository. It explicitly uses ftp://ftp.freebsd.org to fetch release tarballs and applies a series of patch(1) files that correspond to security updates and erratas. It replicates a release build and generates efficient binary diffs. It goes out of its way to avoid touching any file it doesn't absolutely have to. It does not have any relationship with the source repository at all.

This is the exact opposite of what our goal was for dogfooding and exercising the top-of-source-tree. freebsd-update is a mechanism optimized to avoid change rather than facilitate it.

While my mechanism would replace hundreds of megabytes of binaries to fix one file, freebsd-update would efficiently make that 1-character patch. Update size is of no concern in the cluster's local ethernet environment.

The other big reason for not using freebsd-update is philosophical. As we are providing source code and binaries to people, it is a reasonable expectation that we make extra effort to safeguard the process.

Right now, freebsd-update has an attack surface. It uses third party mirrors and depends on end-to-end integrity checks as it crosses the naked scary internet. Imagine a hypothetical data update path: freebsd-update-builder.freebsd.org -> scary internet -> mirrors -> scary internet -> releasebuilder.freebsd.org. Or think of that for svn.freebsd.org. If anything were to go wrong with the end-to-end integrity checking.... Ouch.

We do not do this because we don't have to.

portsnap has an additional practical concern - In my common use test cases, modern svn operating from https://svn.freebsd.org/ports/ is on par with or faster than portsnap. svn also supports local tweaks while portsnap erases your work. Like portsnap, svn is in the base system.

In summary; we have no use cases for either freebsd-update or portsnap in the freebsd.org cluster.

This is not to say I can't trust freebsd-update - I run it on my EC2 VM instances to do binary updates - I have no interest in (or cpu credits for) compiling things there. It also run stock freebsd.org packages. I have no use for freebsd-update at home. I do not run portsnap anywhere because of the speed and usability issues.

Update: The mechanism that we use for update distribution inside the cluster is actually portsnap / freebsd-update derived. The main difference is that the signature is on the entire blob set. Things like libarchive/tar/etc never touch anything until after the signature is verified.

Contact Discussion