Debugging a strange permission issue on Unbound

I recently upgraded a VM from Debian 9 to Debian 10 and found myself unable to start Unbound.

This is not uncommon when “dist-upgrading” and my experience told me it would probably not be a big deal, but the error message was quite a surprise:

unbound[429:0] error: Could not open /etc/unbound/unbound.conf: Permission denied
unbound[429:0] fatal error: Could not read config file: /etc/unbound/unbound.conf. Maybe try unbound -dd, it stays on the commandline to see more errors, or unbound-checkconf

Let’s see what the permissions are on that file:

# ls -ld /etc /etc/unbound /etc/unbound/unbound.conf
drwxr-xr-x 93 root    root    4096 Jun  4 14:22 /etc
drwxr-x---  3 unbound unbound 4096 May 29 06:03 /etc/unbound
-rw-r-----  1 unbound unbound   94 May 17 02:10 /etc/unbound/unbound.conf

Ok, that’s strange!

A bit of context

Before I dig into how I resolved this issue, I must signal that I use Puppet to manage my infrastructure and especially I use a module that I developed myself to manage Unbound, meaning everything is pretty much automated, from specific settings to access rights on configuration files.

I also already knew something was wrong with this module, as, months before I faced this issue myself, a kind contributor submitted a patch aiming at fixing a permission issue on Debian Buster.

The birth of a daemon

So what surprised me here has to do with the way daemons start on UNIX systems¹.

Most daemons run as an unprivileged user, but they often start by running as root for a short period of time before forking and switching to another user. This allows the process to make actions otherwise restricted, like reading its configuration file, creating sockets or listening on privileged ports. Unbound is no exception and follows that exact same process, there’s no reason it would fail at reading its configuration file with EACCES (permission denied).

I couldn’t figure what was going on, and couldn’t reproduce this behavior in an interactive shell running as root:

# cat /etc/unbound/unbound.conf
# This file is managed by Puppet. DO NOT EDIT.

include: "/etc/unbound/unbound.conf.d/*.conf"

# touch /etc/unbound/unbound.conf; echo $?
0

The merge request I received on my Puppet module months ago quickly came to my mind: the contributor told me they had to set root as the owner of the configuration files, otherwise Unbound wouldn’t start.

Let’s give it a try:

# chown root: -R /etc/unbound /etc/unbound/unbound.conf
# unbound
[1591299980] unbound[10638:0] error: error for private key file: /etc/letsencrypt/live/rdns01.br0.fr/privkey.pem
[1591299980] unbound[10638:0] error: Error in SSL_CTX use_PrivateKey_file crypto error:0200100D:system library:fopen:Permission denied
[1591299980] unbound[10638:0] error: and additionally crypto error:20074002:BIO routines:file_ctrl:system lib
[1591299980] unbound[10638:0] error: and additionally crypto error:140B0002:SSL routines:SSL_CTX_use_PrivateKey_file:system lib
[1591299980] unbound[10638:0] fatal error: could not set up listen SSL_CTX

There was another issue here that needed attention, but at least Unbound was now able to read its configuration file and go further in its startup process.

Still, I couldn’t understand why it wouldn’t work in the first place and it seemed super strange that root would need explicit read access on these files. This was something I’d never encountered before.

I decided to search for similar issues on the Web and came across a post on Server Fault with an answer that could explain the certificates issue: I needed to tweak AppArmor in order to allow Unbound to read its certificates in /etc/letsencrypt.

AppArmor

AppArmor is a security mechanism that act kind of like a “sandbox” by restricting what a software can or can’t do on the system using per-process profiles. For example, it can limit what file or directory a software can read or write to, what POSIX capabilities are assigned to a process or what kind of network socket it can create. Profiles shipped in Linux distributions are pretty much transparent when running daemon close to their default configuration.

In this case, I fixed the certificates issue by adding the following two lines in /etc/apparmor.d/local/usr.sbin.unbound:

# Allows Unbound to recursively read files in those two directory
/etc/letsencrypt/archive/** r,
/etc/letsencrypt/live/** r,

After running apparmor_parser --replace /etc/apparmor.d/usr.sbin.unbound to reload the profile, Unbound was able to read the certificates used for DNS over TLS and it was finally able to start.

I could’ve stopped here, but I’m not the kind of person to be satisfied with workarounds, and I really wanted to understand what was going on in the first place. I started looking at Unbound’s AppArmor profile (in /etc/apparmor.d/usr.sbin.unbound) and I finally figured that my issue was related to POSIX capabilities.

POSIX capabilities

Besides limiting access to files and directory, AppArmor can assign or remove POSIX capabilities to limit what a process is allowed to do on the system.

In the case of Unbound, it is only granted the ones it really needs (for instance the NET_BIND_SERVICE capability allows it to listen ports lower than 1024). This means that even when Unbound is running as root, during the early phase of its startup process, it is limited to the capabilities that was explicitly granted.

Actually, this is the key difference between Unbound and my interactive shell, the later is basically granted with the full list of capabilities. There’s one specific capabilities that explains this difference: DAC_OVERRIDE allows a process to bypass access right checks on files.

That explains it all: Unbound can’t read a file that is owned by the unbound user without read permission for other users while running as root, because it doesn’t have that capability assigned.

Conclusion

To summarize, Unbound was unable to read its configuration file because AppArmor limited its capabilities such that even while running as root the kernel acted as if it was a unprivileged user. The fix was simply to set root the owner of Unbound’s configuration files.

In hindsight, I believe this could be a good practice: most daemons don’t need to have write access on their configuration file and preventing that can help mitigate the impact of some security issues.

It took a long time to debug this issue, and this is mainly due to my lack of experience with AppArmor to the point that I didn’t even think of it until I saw the post on Server Fault. I knew about AppArmor for a while but I only started experimenting with it recently (mainly by installing the apparmor-profiles package which ships with a lot of profiles).