I recently upgraded a VM from Debian 9 to Debian 10 and found myself unable to start Unbound.
This is not uncommon when “dist-upgrading” and my experience told me it would probably not be a big deal, but the error message was quite a surprise:
unbound[429:0] error: Could not open /etc/unbound/unbound.conf: Permission denied unbound[429:0] fatal error: Could not read config file: /etc/unbound/unbound.conf. Maybe try unbound -dd, it stays on the commandline to see more errors, or unbound-checkconf
Let’s see what the permissions are on that file:
# ls -ld /etc /etc/unbound /etc/unbound/unbound.conf drwxr-xr-x 93 root root 4096 Jun 4 14:22 /etc drwxr-x--- 3 unbound unbound 4096 May 29 06:03 /etc/unbound -rw-r----- 1 unbound unbound 94 May 17 02:10 /etc/unbound/unbound.conf
Ok, that’s strange!
Before I dig into how I resolved this issue, I must signal that I use Puppet to manage my infrastructure and especially I use a module that I developed myself to manage Unbound, meaning everything is pretty much automated, from specific settings to access rights on configuration files.
I also already knew something was wrong with this module, as, months before I faced this issue myself, a kind contributor submitted a patch aiming at fixing a permission issue on Debian Buster.
So what surprised me here has to do with the way daemons start on UNIX systems1.
Most daemons run as an unprivileged user, but they often start by running as
root for a short period of time before forking and switching to another user.
This allows the process to make actions otherwise restricted, like reading its
configuration file, creating sockets or listening on privileged ports. Unbound
is no exception and follows that exact same process, there’s no reason it would
fail at reading its configuration file with
I couldn’t figure what was going on, and couldn’t reproduce this behavior in an
interactive shell running as
# cat /etc/unbound/unbound.conf # This file is managed by Puppet. DO NOT EDIT. include: "/etc/unbound/unbound.conf.d/*.conf"
# touch /etc/unbound/unbound.conf; echo $? 0
The merge request I received on my Puppet module months ago quickly came to my
mind: the contributor told me they had to set
root as the owner of the
configuration files, otherwise Unbound wouldn’t start.
Let’s give it a try:
# chown root: -R /etc/unbound /etc/unbound/unbound.conf # unbound  unbound[10638:0] error: error for private key file: /etc/letsencrypt/live/rdns01.br0.fr/privkey.pem  unbound[10638:0] error: Error in SSL_CTX use_PrivateKey_file crypto error:0200100D:system library:fopen:Permission denied  unbound[10638:0] error: and additionally crypto error:20074002:BIO routines:file_ctrl:system lib  unbound[10638:0] error: and additionally crypto error:140B0002:SSL routines:SSL_CTX_use_PrivateKey_file:system lib  unbound[10638:0] fatal error: could not set up listen SSL_CTX
There was another issue here that needed attention, but at least Unbound was now able to read its configuration file and go further in its startup process.
Still, I couldn’t understand why it wouldn’t work in the first place and it
seemed super strange that
root would need explicit read access on these
files. This was something I’d never encountered before.
I decided to search for similar issues on the Web and came across a post on
Server Fault with an answer that could explain the certificates
issue: I needed to tweak AppArmor in order
to allow Unbound to read its certificates in
AppArmor is a security mechanism that act kind of like a “sandbox” by restricting what a software can or can’t do on the system using per-process profiles. For example, it can limit what file or directory a software can read or write to, what POSIX capabilities are assigned to a process or what kind of network socket it can create. Profiles shipped in Linux distributions are pretty much transparent when running daemon close to their default configuration.
In this case, I fixed the certificates issue by adding the following two lines
# Allows Unbound to recursively read files in those two directory /etc/letsencrypt/archive/** r, /etc/letsencrypt/live/** r,
apparmor_parser --replace /etc/apparmor.d/usr.sbin.unbound to
reload the profile, Unbound was able to read the certificates used for DNS
over TLS and it was finally able
I could’ve stopped here, but I’m not the kind of person to be satisfied with
workarounds, and I really wanted to understand what was going on in the first
place. I started looking at Unbound’s AppArmor profile (in
/etc/apparmor.d/usr.sbin.unbound) and I finally figured that my issue was
related to POSIX
Besides limiting access to files and directory, AppArmor can assign or remove POSIX capabilities to limit what a process is allowed to do on the system.
In the case of Unbound, it is only granted the ones it really needs (for
NET_BIND_SERVICE capability allows it to listen ports lower than
1024). This means that even when Unbound is running as root, during the early
phase of its startup process, it is limited to the capabilities that was
Actually, this is the key difference between Unbound and my interactive shell,
the later is basically granted with the full list of capabilities. There’s one
specific capabilities that explains this difference:
DAC_OVERRIDE allows a
process to bypass access right checks on files.
That explains it all: Unbound can’t read a file that is owned by the
user without read permission for other users while running as root, because it
doesn’t have that capability assigned.
To summarize, Unbound was unable to read its configuration file because
AppArmor limited its capabilities such that even while running as
kernel acted as if it was a unprivileged user. The fix was simply to set
the owner of Unbound’s configuration files.
In hindsight, I believe this could be a good practice: most daemons don’t need to have write access on their configuration file and preventing that can help mitigate the impact of some security issues.
It took a long time to debug this issue, and this is mainly due to my lack of experience with AppArmor to the point that I didn’t even think of it until I saw the post on Server Fault. I knew about AppArmor for a while but I only started experimenting with it recently (mainly by installing the apparmor-profiles package which ships with a lot of profiles).
I use UNIX as a generic term here. ↩︎