Docker resource management via Cgroups and systemd

Level: Advanced-intermediate

In this post, I will focus on resource management in docker using cgroups. If you are new in the container world and especially Docker that will use for demos, please read either my linked article about underlying technologies or other corresponded resources out there.

In Docker, the resources are managed by Control Groups (cgroups) which a Linux kernel feature allows you to limit, modify, or allocate resources as needed. Docker allows limiting container resources per flag for each individual container. You can see a bunch of flags here. Let’s see a sample:

docker run -it --cpuset-cpus="1-3" --memory=50MiB ubuntu

We let the docker use CPU cores from 0 to 3 and maximum 50MiB memory. The container will not consume more than the assigned resources. To see this simply hit the docker stats command. As I said there are many more flags, please see the linked docker documentation.

Now let’s move deeper use cgroups and systemd to manage resources at the daemon level.

Without any resource limitation: Stressing CPUs via docker container

docker run -it — rm jess/stress — cpu 100

At the and you will see with the same CPU stressing it will look different.

Use case: The docker daemon will not consume more resources than allowed and no matter how many containers you spin up.

Keep in Mind: “Everything is a file” in Linux.

Systemd allows us to create custom Units for services, where we define custom properties for services. This Unit configuration can be achieved with systemd.slices. Through this systemd feature, we can create a node on cgroups hierarchy. Steps:

1. Create a custom slice file and define resources
2. Add the created slice file to docker.service file
3. Change cgroup driver
4. Add cgroups-parent to the docker daemon
5. Reload daemon and restart the docker daemon

1 Step:

The custom slice file should be created under /etc/systemd/system

$ sudo vim /etc/systemd/system/docker-engine.slice 

I called docker-engine.slice` you can call it whatever you wish. Note that lowercase and — (dash) is allowed to use.

As you see the content of the file is self-explanatory. My comments start with a # sign

$ cat /etc/systemd/system/docker-engine.slice
[Unit]
Description=Slice that limits docker resources 
Before=slices.target
[Slice]
CPUAccounting=true
CPUQuota=50%    
MemoryAccounting=true
MemoryHigh=2G  
MemoryMax=3G 
MemoryMaxSwap=10G

here are explanations

[Unit]
Description=Slice that limits docker resources #any custom desc.
Before=slices.target #is importand because over layer should be running 
[Slice]
CPUAccounting=true
CPUQuota=50%    #docker daemond can use max 50% of cpus
#Memory Management
MemoryAccounting=true
MemoryHigh=2G  #allowed amount, might be go over then process will slow down
MemoryMax=3G #maximum usage of memory, the process cannot use more
MemoryMaxSwap=10G #maximum swap capacity

2. Step:

Now we have to tell the docker service how many resources assigned. The service file is under /etc/systemd/system/multi-user.target.wants/docker.service`. It may be located on a different path in your machine. Please find it. Add following line to Service section

sudo vim /etc/systemd/system/multi-user.target.wants/docker.service`

[Service] 
Slice=docker-engine.slice

3. Step:

Change cgroup driver to systemd . This step is also will be done in docker.service file in [Service] part

default Execution

ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

Docker installed default look like:

As you see Cgroup is using /system.slice which is for the entire host resources

Default state

To change systemd cgroup

ExecStart=/usr/bin/dockerd --exec-opt native.cgroupdriver=systemd

PS: After all step done it will look as below

docker-engine.slice

4. Step

So tell the daemon to use cgroup parents Set parent cgroup for all containers.

Therefore we have to edit /etc/docker/daemon.json the file. If it is not created then create it before.

$ sudo vim /etc/docker/daemon.json
{
 "cgroup-parent": "docker-engine.slice"
}

5. Step:

Reload daemon and restart docker-engine

$ sudo systemctl daemon-reload
$ sudo systemctl restart docker.service

Let’s stress CPU again with 100% loads as we did at the beginning of the article

docker run -it --rm jess/stress --cpu 100`

As you see docker daemon is not allowed to use entire host resources. You can check it under cgroups as below.

celcin@celcin:/sys/fs/cgroup/memory$ tree docker.slice/
docker.slice/
├── cgroup.clone_children
├── cgroup.event_control
├── cgroup.procs
├── docker-engine.slice
│   ├── cgroup.clone_children
│   ├── cgroup.event_control
│   ├── cgroup.procs
│   ├── docker.service
│   │   ├── cgroup.clone_children
│   │   ├── cgroup.event_control
│   │   ├── cgroup.procs
│   └── tasks
├── memory.failcnt
├── memory.force_empty
├── memory.limit_in_bytes
├── notify_on_release
└── tasks

NOTE: Using Cgroups you can limit/manage resources for any services and process. Docker is just an example here.

Resources:
* https://www.freedesktop.org/software/systemd/man/systemd.resource-control.html
*https://docs.docker.com/engine/reference/commandline/dockerd/
*https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/resource_management_guide/ch01
*https://man7.org/linux/man-pages/man5/systemd.slice.5.html
*https://man7.org/linux/man-pages/man7/cgroups.7.html
*https://unix.stackexchange.com/questions/537645/how-to-limit-docker-total-resources