Saturday, April 11, 2015

Ports are more than just a makefile

If you are coming from a GNU/Linux background, used to the usual choice of apt, yum, pacman, portage & the others. You might wonder what's the fuss about the ports system in OpenBSD. I myself came from those communities and incorrectly categorised the BSD ports system as something much more bare-bones than it actually is. Obviously some things cross over between BSD systems but this post is mainly focused on the one I have on-hand experience with. Want a glimpse of the BSD ecosystem? Read on.


Some of you might have noticed Gnome 3.16.0 on the photo. Yes, my new amd64 laptop is now working perfectly on OpenBSD thanks to a patch I received today that fixed the re0 issues I had before. Since this is a bit stronger box compared to my ageing i386 MSI netbook I decided to give Gnome a shot. Don't worry, this post is not a 3 page instruction on how to set up the latest gnome on OpenbSD - that takes one command and 3 lines of configuration.

pkg_add gnome # gnome-extras for the whole suite
$ cat /etc/rc.conf.local                                                                     
ntpd_flags=
# xdm_flags=
multicast_host=YES
pkg_scripts="messagebus avahi_daemon gdm"
Looks familiar? Yes folks, the ports system is just as powerful as the package managers you are used to. Let's take a look at all of this from the plain makefile and what sits on top of that. You will then understand why I decided to write up about this topic.

The OpenBSD ports system originated from FreeBSD but underwent tremendous changes during the years since it was introduced. The makefile is the most basic level of the ports ecosystem - everything starts here. Port makefiles tell the infrastructure:
  • where & how to obtain source code/binaries
  • what are the software dependencies
  • how to patch the software for the platform
  • how to configure & build it
  • how to test it
  • how to install/uninstall it
  • and much more (who is the maintainer, what platforms are supported, etc)
This is a basic recipe that tells the ports infrastructure on how to make a package that can be installed in the system.  Ports themselves are stored in /usr/ports (you have to obtain that folder from a mirror or cvs), categorized into folders like www, editors, security, x11 etc. Most users never need to download the ports tree. You will most probably just set a PKG_PATH which tells the system where to look for binary packages and use commands like pkg_add, pkg_info, pkg_delete to install, find & investigate ports and finally to remove them.

You can of course build ports from source, which allows you to customize the way they are built. The important part is that the result will be a normal package that all the typical system tools know how to work with. The Arch Build System works in a similar way - in fact it was inspired by the ports infrastructure - what really stood out for me is everything around ports in the system. If you want to learn more about porting software or how the internals of the ports system work go to the official documentation. I'm going to tackle things that are a bit less apparent. The whole ecosystem itself.

People porting software for OpenBSD engage in a special relationship. You will be encouraged to maintain contact with your upstream (the software you are porting) and people will most probably report problems to you before going up the chain.

The baby steps every porter makes are joining the relevant ports@ & ports-cvs@ mailing lists. The first list is where each port change, whether completely new or just an update goes up for a review. No port is committed, compiled and distributed to users without being reviewed & tested by the community and confirmed by developers. The second list emails you each time a commit is made to the ports tree. People will take your port, compile it locally & test how the software works. This is not a 'compilation' test. The whole port is reviewed - starting with the makefile layout & ending on application specific functionality (otter-browser crashes when I open a new tab). The porter is expected to address those issues - fix your makefile, write platform specific patches or upstream the issues if they are outside of your control.

During my first port I encountered several Qt5 related issues. The MAINTAINER of the port spent his personal time on debugging the crashes I reported, upstreamed them & back-ported patches to the OpenBSD port. He was also my 'upstream' and was extremely helpful with packaging the browser itself as he knew a lot more about Qt5 than I did. That's a great level of responsibility for ones work that I exceptionally admire.

This setup is great both for port developers & users. You can immediately see whom to ping if something is wrong with a port. If otter browser crashes for you on OpenBSD for whatever reason - you can email me directly and I will investigate it to my fullest potential.

Porters run current, the packages here are extremely fresh. If you're coming from Arch you won't be missing your bleeding edge software but you can for sure expect fewer unexpected issues between packages. If you are a user and don't want to jump on the bleeding edge - you don't have to. OpenBSD regularly builds binary packages for all supported platforms (on real hardware - not on virtual machines) - though you will be bound to the ports set that was current during the release.

What? Outdated Firefox! Well yes. Before a release happens at some arbitrary point a ports lock will be announced. During that time the system is undergoing stabilization and increased testing. No new ports are accepted and only updates deemed as critical are committed. There are third party providers like mtier which allow you to stay on a release & have up to date packages without running -CURRENT or building your own from ports (which you can also do of course!). Personally I think that you should just roll on with snapshots.

Patches. They range from altering makefiles to adhere to hier(7) through  exchanging unsafe API usage to platform best practices & up to back-porting fixes or developing new ones specifically for the platform. In all cases as the porter you are expected to engage with upstream to incorporate those patches. This benefits you as the porter (less things to maintain) & the general users of that software - especially if those patches increase portability/security of the upstream software. Good upstream relations will also result in better software, when porting otter on two occasions we were asked to ship a different version of the software than the one tagged by upstream as a specific 'release'. That's the only reason you can now pkg_add otter-browser on OpenBSD to obtain the beta 5 version of the browser & be free from issues like 'crash when opening a new background tab', 'crash when duplicating a tab' and a few more.


The Arch Linux Wiki is amazing. I think everyone will agree that they found it invaluable while working with the system & setting up Gnome, KDE or in fact any bit of a little more complicated software. I'm surprised that I don't miss it in OpenBSD at all. If you are an outsider like I was - you might think that the system misses critical documentation. How the hell did I knew what to put in the /etc/rc.conf.local file & why I did that? Well you don't need a wiki for it on OpenBSD. Porters provide a package. You are supposed to install it and have everything that is necessary to run software without problem on the platform. If a specific piece of software requires unusual configuration the porter will prepare a pkg-readme file for you. Those files are conveniently stored in your system under /usr/local/share/doc/pkg-readmes. Those files will contain every bit of information you need to run the package. By reading /usr/local/share/doc/pkg-readmes/gnome-3.16.0p3 I learned everything I needed to have a working Gnome setup on my system.

By now you should understand that ports are far more than a simple makefile dumped into source control. The testing you will receive on ports@, the dedication of the porter to ship a quality package to you are impressive to say the least. Those mailing lists are also used by developers so people who provide the platform you are porting to are looking and able to act if the base OS stands in the way of porting a package. Porters also make informed decisions - they run this stuff on their boxes, sometimes a package may be outdated specifically to avoid a critical issue that the porter & upstream are aware of. OpenBSD has a nice system called portroach that you can track to see outdated ports that need some love, before you try to update any one of them - contact the current MAINTAINER. He might have a good reason not to update it. You can also count on a lot of help from the ports team, while I was on a trip to the UK, otter changed their github account & repository location. People from the ports team acted & fixed my port while I was away while still keeping me informed on what's going on. Not to mention Stuart fixing Otter beta 4 release while I was working on packaging up Otter beta 5. Thank you if you happen to be reading this!

Hats off, to the maintainer of Gnome. I was able to obtain a large software package with a single command and set it up with 3 lines of configuration without ever needing to search for a HOWTO online. Heck, they even shipped a nice demo while I was writing this blog post so here is the relevant tweet - go watch the demo now!



I packaged software for Arch Linux & Gentoo in the past. The process was extremely pleasant in both cases but felt really lonely in comparison to OpenBSD. I'm sure that porters for various Linux distributions put in the same amount of dedication & work into their ports. I personally just found the OpenBSD community much nicer to approach & engaged in the process.

Ports are important. Up to date software incorporates security fixes, often comes with performance improvements (& unfortunately new bugs). It's really important for you to take some time and test a port once in a while. That way all of the community will benefit from a more current system. If you don't test a package before the ports freeze - don't complain about old software on a release CD.