Docker DNS deadlock (with ipvlan l2 and addguard dns container)

Gert-JanFebruary 11, 2024Tech

I stumbled on an interesting issue, that I think is worth mentioning.

Behaviour

I had problems with traefik getting the certificates from let’s Encrypt. I knew it worked before, but I didn’t have a clue why it suddenly didn’t work.

Investigation

The first investigation focused of course on traefik. After examining the logs I saw the following message:

time="2024-02-11T00:25:23+01:00" level=error msg="Unable to obtain ACME certificate for domains \"XXX.aalderink.nl\": cannot get ACME client get directory at 'https://acme-v02.api.letsencrypt.org/directory': Get \"https://acme-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-v02.api.letsencrypt.org on 127.0.0.11:53: server misbehaving" routerName=aalderink@docker rule="Host(`XXX.aalderink.nl`)" ACME CA="https://acme-v02.api.letsencrypt.org/directory" providerName=script.acme

After some further investigation it turned out that the traefik container wasn’t able to resolve hostnames. So the next step was to figure out why. I have docker installed on an ubuntu system. The docker system uses a split dns, which results in the /etc/resolv.conf pointing to localhost (127.0.0.11) as the nameserver. Depending on the dns request (docker internal or external) it’s being answered by nameservers mentioned in either of the following files in /var/run/systemd/resolve:

  • resolv.conf
  • stub-resolv.conf

The stub-resolv.conf points to the local docker dns server and the resolv.conf file points to the nameserver which is configured by DHCP or statically.

Slowly it dawned on my that I recently created an Adguard container within this docker host and have my dhcp server serve that as the resolving DNS server.

I generally configure my servers’ static IP adres by configuring a static lease in my DHCP server. so my docker host is also pointing to the Adguard DNS server.

Here is where the deadlock arose as they are expecting both (docker host and adguard container) to provide the resolving of external hosts.

Resolution

The way I solved this is by using configuring the static IP on the host itself and not use DHCP, in order for me to override the DNS server provided by the DHCP server.

In my case I use netplan to configure the static IP

network:
  ethernets:
    enp2s0f0:
#      dhcp4: true
      dhcp4: no
      addresses: [172.20.20.129/24]
#      gateway4: 172.19.19.1
      routes:
        - to: default
          via: 172.19.19.1
      nameservers:
        addresses: [8.8.8.8,8.8.4.4]
  version: 2

Based on what I found on the Internet I first used “gateway4” as the option to define the default gateway, but when I issued “netplan apply”, I received the message that gateway4 was deprecated, so after consulting the Interwebs a bit more, I updated the config, resulting in the config mentioned above.

Conclusion

After I implemented this change, nameresolving started to work, resulting in the certificates being requested. And again there was peace in the Docker realm.