Symptoms:
Once you deployed the docker environment inside a simple CT (centos, vzlinux, rocky, almalinux), without tunning it, it'll display the docker service as failed as the following example:
CT-102 /# journalctl -xe
{"nftables": [{"metainfo": {"json_schema_version": 1}}, {"insert": {"rule": {"family": "inet", "table": "firewalld", "chain": "filter_INPUT_ZONES", "expr": [>
Dec 21 11:02:57 102.example.net firewalld[76]: ERROR: COMMAND_FAILED: 'python-nftables' failed: internal:0:0-0: Error: Could not process rule: No such file or directory
JSON blob:
{"nftables": [{"metainfo": {"json_schema_version": 1}}, {"insert": {"rule": {"family": "inet", "table": "firewalld", "chain": "filter_INPUT_ZONES", "expr": [>
Dec 21 11:02:57 102.example.net dockerd[28684]: failed to start daemon: Error initializing network controller: Error creating default "bridge" network: Failed to program NAT chain: COMMAND_FAILED: 'python>
Dec 21 11:02:57 102.example.net dockerd[28684]: JSON blob:
Dec 21 11:02:57 102.example.net dockerd[28684]: {"nftables": [{"metainfo": {"json_schema_version": 1}}, {"insert": {"rule": {"family": "inet", "table": "firewalld", "chain": "filter_INPUT_ZONES", "expr": >
Dec 21 11:02:57 102.example.net systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Dec 21 11:02:57 102.example.net systemd[1]: docker.service: Failed with result 'exit-code'.
Cause:
Not enough features are loaded inside the CT's namespace like (netfilter or tun options).
Resolution:
The following procedure describes how to install and configure properly a CT to deploy a docker environment inside it.
Pre-requisites:
- Install template and create its cache (for this example we will deploy on vzlinux8):
# vzpkg install template vzlinux-8-x86_64
# vzpkg create cache vzlinux-8-x86_64
# vzpkg list -O | grep vzlinux-8-x
vzlinux-8-x86_64 2022-12-22 13:03:23
Creation of the CT and its tunning:
1) Create CT:
# vzctl create 102 --ostemplate vzlinux-8-x86_64 --diskspace 20G
# vzlist -a | grep 102
102 - stopped - -
2) With CT turned off, turn on bridge feature to allow docker creating brdiged network:
# vzctl set 102 --features bridge:on --save
Saved parameters for Container 102
3) Setup ct veth-based network:
# vzctl set 102 --netif_add eth0 --save
Saved parameters for Container 102
4) Allow all iptables modules to be used in CT:
# vzctl set 101 --netfilter full --save
Saved parameters for Container 101
5) Enable tun device access for CT:
# vzctl set 102 --devnodes net/tun:rw --save
Saved parameters for Container 102
6) Configure custom cgroups in systemd:
# vzctl mount 102
Mount image: /vz/private/102/root.hdd
Container is mounted
Backing up the config, then add cgroups options:
# cp /vz/root/102/etc/systemd/system.conf /vz/root/102/etc/systemd/system.conf.bck
# echo "JoinControllers=cpu,cpuacct,cpuset freezer,devices" >> /vz/root/102/etc/systemd/system.conf
Note: systemd reads /proc/cgroups and mounts all cgroups enabled there, though it doesn't know there's a restriction that only freezer, devices and cpuacct, cpu, cpuset can be mounted in container, but not freezer, cpu, etc. separately
7) Start CT and network configuration:
# vzctl start 102
# prlctl set 102 --device-set net0 --ipadd 172.16.30.70 --gw 172.16.30.1 --nameserver 8.8.8.8
Installing & Preparing docker environment:
8) Update, install dnf-utils, adding docker repo, and verifying that it is properly added:
CT-102 /# dnf install -y dnf-utils
CT-102 /# dnf config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
CT-102 /# dnf repolist -v | grep docker
Last metadata expiration check: 0:00:24 ago on Thu Dec 22 14:44:01 2022.
Repo-id : docker-ce-stable
Repo-baseurl : https://download.docker.com/linux/centos/8/x86_64/stable
Repo-filename : /etc/yum.repos.d/docker-ce.repo
9) Install the docker community edition environment:
CT-102 /# dnf install -y docker-ce
CT-102 /# docker --version
Docker version 20.10.22, build 3a2c30b
10) Finally, start/enable the docker service as well as verifying its correct working status:
CT-102 /# systemctl start docker
CT-102 /# systemctl enable docker
CT-102 /# systemctl status docker
● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2022-12-22 14:47:35 CST; 39s ago
Docs: https://docs.docker.com
Main PID: 26983 (dockerd)
CGroup: /system.slice/docker.service
└─26983 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
Dec 22 14:47:28 localhost.localdomain dockerd[26983]: time="2022-12-22T14:47:28.480676198-06:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Dec 22 14:47:28 localhost.localdomain dockerd[26983]: time="2022-12-22T14:47:28.488725746-06:00" level=error msg="Failed to built-in GetDriver graph btrfs /var/lib/docker"
Dec 22 14:47:28 localhost.localdomain dockerd[26983]: time="2022-12-22T14:47:28.548753479-06:00" level=info msg="Loading containers: start."
Dec 22 14:47:33 localhost.localdomain dockerd[26983]: time="2022-12-22T14:47:33.576617103-06:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Dae>
Dec 22 14:47:35 localhost.localdomain dockerd[26983]: time="2022-12-22T14:47:35.210156207-06:00" level=info msg="Firewalld: interface docker0 already part of docker zone, returning"
Dec 22 14:47:35 localhost.localdomain dockerd[26983]: time="2022-12-22T14:47:35.372906531-06:00" level=info msg="Loading containers: done."
Dec 22 14:47:35 localhost.localdomain dockerd[26983]: time="2022-12-22T14:47:35.513645192-06:00" level=info msg="Docker daemon" commit=42c8b31 graphdriver(s)=overlay2 version=20.10.22
Dec 22 14:47:35 localhost.localdomain dockerd[26983]: time="2022-12-22T14:47:35.513972713-06:00" level=info msg="Daemon has completed initialization"
Dec 22 14:47:35 localhost.localdomain systemd[1]: Started Docker Application Container Engine.
Dec 22 14:47:35 localhost.localdomain dockerd[26983]: time="2022-12-22T14:47:35.619745394-06:00" level=info msg="API listen on /var/run/docker.sock"
Diagnostic steps:
When you install docker inside a CT and the service is falied due to nftables, policies, rules or firewall issues, it is so probably you are missing enable certain features described in the procedure of this article.
Related Ticket(s):
#398886