Using cgroups v2 with OpenLiteSpeed

HomeUsing cgroups v2 with OpenLiteSpeed
May 21, 2021
How Can We Help?
< Back
You are here:
Print
Posted
Last Updated On
ByRobert

If you are a web server administrator and your users are creating CGI services that are chewing up your server’s CPU, memory, or tasks, you have a powerful tool to deal with it in OpenLiteSpeed: cgroups.

“cgroup” is short for “control group”. cgroups is a Linux kernel feature that, according to Wikipedia, “limits, accounts for, and isolates the resource usage (CPU, memory, disk I/O, network, etc.) of a collection of processes.”

The feature is part of the Linux Operating System and can be activated and utilized with version 1.7.9 (v1.7.x)/1.6.20 (v1.6.x) or higher of OpenLiteSpeed.  Previous versions used an updated v1 cgroups which will no longer be supported in the future. cgroups v2 requires a kernel 5.2 or higher and some additional configuration discussed below.

cgroups v2 has a much richer set of features supported for control than cgroups v1 as discussed here.
The features that have been certified to work with cgroups are:

  • CPU utilization
  • real memory
  • maximum tasks

There are other features of cgroups, and if you enable them, you do so at your own risk. OpenLiteSpeed will not attempt to stop you.

Things to Know About cgroups

cgroups is only available for true CGI applications at this time.  Below we discuss the details for enabling it, but it works on the operating system resource definitions for the owner of the CGI script file.

Containers (like Docker containers) are operating system facilities that run within the base OS.  Containers may use cgroups to avoid completely dominating the machine, and they can only do this if using cgroups. This is because cgroups is one of the few methods that can be used to address overuse of resources without killing the processes (unlike ulimit and other limiting schemes).  cgroups helps you better balance the use of your system, keeping it active and functional for all of your users.

cgroups affects more than just OpenLiteSpeed, since it is a kernel feature. It also affects tasks running as specific users that start their own user sessions, like SSH or SFTP sessions.  Thus, if you set a CPUQuota of 10% for a user, all accesses by the user must not exceed 10%. This includes those in OpenLiteSpeed and user login sessions. All will be shared fairly by the operating system.  If you start more processes, the total percentage for the user does not change, but each process runs a bit slower so as to avoid monopolizing the CPU.

Note that this is managed by systemd, which means that all services within your operating system can be controlled similarly, but only at the service level.  To control it at the user level, OpenLiteSpeed’s CGI functions utilize specific user-management code.

The reason why this method is not more widespread is that users who are logged in with su or a screen window manager (KDE, Gnome, etc.) can’t use cgroups, as their sessions are managed by the service that initialized the slice rather than the user slice.  OpenLiteSpeed creates its CGI tasks (when configured to) within the user’s sessions and thus can take advantage of them.

Supported cgroup Features

The following are cgroup facilities supported by the operating system, tested, and certified to work with OpenLiteSpeed.  The names specified below are the ones that you will use in systemctl to modify them:

  1. CPUQuota
  2. TasksMax
  3. MemoryMax

CPUQuota

This is specified as a percentage of a single processor to use.  By default, the CPUQuota of all tasks is infinite, and explains why a single user’s task can pretty much monopolize the CPU resources of your system.  This is by far the most useful of the supported features as CPU affects not only processor I/O but also access to buffers in buffered I/O.  Tools like top help you monitor CPU utilization for a task and display the user and process that are running.  Note that values above 100% are valid and indicate that multiple CPUs are to be allowed.

TasksMax

TasksMax refers to the number of tasks that can be started by a user. This ensures that the number of tasks accounted for the user stays below a specific limit. This takes an absolute number of tasks for the user on the system.  A sample default used by the system is 12000 and can result in a very large number of tasks, each of which is managed by the OS.  A reasonable value might be 2500.

MemoryMax

This is the maximum amount of real memory allocated to a process, specified in bytes.  It is not the amount of virtual memory that a process uses.  Specify the limit on maximum memory usage of the executed processes. The limit specifies how much process and kernel memory can be used by tasks in this unit. If the value is suffixed with K, M, G or T, the specified memory size is parsed as Kilobytes, Megabytes, Gigabytes, or Terabytes (with the base 1024), respectively.  Remember that this is not virtual memory (which is stored both on disk and in real memory), but real memory.  Note that real memory affects virtual memory and that tasks that take too much of it can weigh a system down.  A reasonable value might be 10M.

Using cgroups with OpenLiteSpeed

Step 1. System Configuration

To verify that you are running cgroups v2 check to see if you have the cgroups v2 controllers file:

ls /sys/fs/cgroup/cgroup.controllers

If this file is not there then you are not running v2 cgroups.
By default only a limited number of commands can be controlled through cgroups. You can see this list:

cat /sys/fs/cgroup/cgroup.subtree_control

If this list does not include the features you wish to control: cpuset and cpu for CPUQuota, memory for MemoryMax and tasks for MaxTasks, then you will need to add them. If these features are already included, you can skip this step and proceed to User Configuration below.

Note that all configuration will need to be done either running as root, or by using the sudo command as a prefix for your command. The notes below will show the entries as if you were running as root. Add the sudo prefix if you need to on your system.

Extend the set of commands by creating a drop-in file:

mkdir -p /etc/systemd/system/user@.service.d
cat > /etc/systemd/system/user@.service.d/delegate.conf << EOF
[Service]
Delegate=cpu cpuset io memory pids
EOF
systemctl daemon-reload

Step 2. User Configuration

User configuration is within the operating system.  What's nice about the systemd programs loginctl and systemctl is that configuration is applied immediately and preserved on reboot.

You will need to identify the users you want to limit and obtain their userids.  The /etc/passwd file contains a list of users and passwords, though there are many ways of identifying IDs for a user.  Note that all configuration will need to be done either running as root, or by using the sudo command as a prefix for your command.  The notes below will show the entries as if you were running as root, add the sudo prefix if you need to on your system.

You can only modify a service or user that is either active, logged-in or lingered. Lingering is requesting that a user slice stay active even if the user is not logged in. There is virtually no overhead in enabling linger for a user. Linger is one of the very few commands which is done with a different program, in this case loginctl.

To turn on linger for user2, enter the following:

loginctl enable-linger user2

You can see if a user is lingered by entering:

loginctl user-status user2

Using systemctl

systemctl is the primary tool for accessing services, slices and units on a machine.  It is accessed from a terminal session.  You can see its options by using:

systemctl --help

You may have noted that this is the same command used to start and stop services on your system.  This is not an accident; these facilities were initially intended for services and have been extended by kernel developers for users.

systemctl requires that you specify the user using the userid (uid) in the form user-UID.slice, where UID is replaced by the userid. For example, if user2 has a uid of 1001, then you'd specify the user as user-1001.slice

systemctl show user-1001.slice

The titles and values for the properties are mentioned above (CPUQuota=xx%, TasksMax=xx, and MemoryMax=xx).  To set a property like CPUQuota to 10% for user2 (uid=1001), enter:

systemctl set-property user-1001.slice CPUQuota=10%

Properties are set in the system and take effect immediately for future processes.  Current ones are not affected, as mentioned above, they are also saved and may work on reboot.  If you don't want them saved, you can avoid this by adding the modifier --runtime after systemctl and before the command.  For example:

systemctl --runtime set-property user-1001.slice CPUQuota=10%

To see its current value, enter (as you did above):

systemctl show user-1001.slice

Its current value will be buried within all of the other parameters, but it will be there.  Except CPUQuota, which is listed in a systemctl show command as CPUQuotaPerSecUSec (USec means microseconds, but it's displayed in milliseconds).  10% is displayed as 100ms.

You can see where the information you update gets stored by using systemctl with the status command and the user slice.  For example:

$ systemctl status user-1001.slice
    
● user-1001.slice 
  Loaded: loaded 
 Drop-In: /etc/systemd/system/user-1001.slice.d 
          └─50-CPUQuota.conf, 50-TasksMax.conf 
  Active: active since Wed 2019-01-23 15:56:52 EST; 1 day 22h ago 
   Tasks: 0 (limit: 1000)

Note that it shows its status and the names and locations of the drop-in files.

You will need to repeat the steps above for all of the users you wish to control.

Step 3. OpenLiteSpeed Setup.

In the WebAdmin Console, configure OpenLiteSpeed so that CGI apps use the user ID of the owner of the file.  Navigate to Configuration > Virtual Hosts and select the virtual host you wish to manage.  In the Basic tab, modify the Security entry and set the ExtApp Set UID Mode to CGI File UID.  Save the entry.

To enable cgroups at the server level, in OpenLiteSpeed WebAdmin, navigate to Server Configuration > Security and edit the CGI Settings group.  The parameter cgroup controls whether cgroups can be used and if so, sets the default value:

  • Off (the same as not set):  Allows cgroups to be used by OpenLiteSpeed, but does not enable cgroups by default. cgroups must be explicitly enabled at the virtual host level (see below).
  • On:  Allows cgroups to be used by OpenLiteSpeed and enables cgroups by default. cgroups can be disabled at the virtual host level.
  • Disabled:  No matter the setting at the virtual host level, the cgroups feature is disabled.

Save the group to proceed.  If you do not want to set cgroups at the virtual host level, do a graceful restart of OpenLiteSpeed to activate your configuration and proceed to Step 3.

Optional Virtual Host Configuration

If you have not disabled cgroups at the server level and want to further control them at the vhost level, navigate to Configuration > Virtual Hosts and select the virtual host you wish to manage.  In the General tab, modify the General entry and set cgroups:

  • Off: Disables cgroups for this vhost, regardless of the server setting.
  • On: Enables cgroups for this vhost if not disabled at the server level.

Save the group to proceed.  Do a graceful restart of OpenLiteSpeed to activate your configuration and proceed to Step 3.

Modifying the Configuration in Files

This is only for reference if you're using the admin panel.

In $SERVER_ROOT/conf/httpd.config.conf there is a single parameter in the CGIRLimit section with parameters like procHardLimit:

  • cgroups:  The possible values of 0, 1, and 2 correspond to Off, On, and Disabled, respectively, as described above.

You can specify specific virtual hosts to enable/disable at the virtual host level unless you have disabled it entirely.  That is done in the $SERVER_ROOT/conf/vhosts/<virtual host>/vhconf.conf file.  For example, for the Example virtual host you'd edit $SERVER_ROOT/conf/vhosts/Example/vhconf.conf and set in the main group under no specific section with parameters like docRoot and enableGzip:

  • cgroups:  Set to 1 to override the default and turn cgroups on for the virtual host, and 0 to override the default and turn cgroups off for the virtual host.

You will need to gracefully restart OpenLiteSpeed to activate the configuration.

Step 4. Validation

After configuring users and OpenLiteSpeed to use cgroups, you want to make sure that they are applied correctly.  Configure a user to use a small amount of CPU (like 10%).  Then create a CGI file in your virtual host directory which is owned by your configured user (with a command like chown).  Below is an example.

Warning: This is a dangerous example as it uses 100% of CPU so it should only be used on test systems. 

I've named it $SERVER_ROOT/<virtual host>/cgi-bin/pound and changed it's owner to be user2:

#!/bin/sh

date=date -u '+%a, %d %b %Y %H:%M:%S %Z'
results=/usr/bin/dd if=/dev/zero of=/dev/null 
cat << EOF
Content-type: text/plain
Expires: $date

$results

EOF

Then change its owner:
chown user2:users pound

Note that in Ubuntu, the dd command is in /bin/dd rather than /usr/bin/dd (on the results line).

When you specify the site in your browser: http://127.0.0.1/cgi-bin/pound the dd command should be owned by user2 and it should take only 10%.  You can view this with the top program and kill it there.  If there are problems you should check the $SERVER_ROOT/logs/error.log file to see if there are any cgroup messages.

Other Useful Tools

There are other useful tools which can help you understand how configuration is implemented in your system:

  • systemd-cgls -a:  Enter this command to display a tree-like structure of the processes and under which slice and scope they are running.  The dd command in the pound example above should be running under user-1001.slice.
  • The /proc/<process-id>/cgroup file:  In the 1:name line you should see the slice that the process is running in.

Problems With cgroups in OpenLiteSpeed

Since enabling and disabling cgroups is so easy in OpenLiteSpeed, almost all problems will be in the systemctl configuration. We recommend that you create a simple program like the pound program above and that you use top to verify the results. If they show 100% CPU after setup you should verify the following:

  • Use systemctl show user-$UID.slice where $UID is your user's UID. Look for the following properties:
    • If you are controlling CPU: CPUAccounting=yes and CPUQuotaPerSecUSec set to a non-infinite value.
    • If you are controlling Tasks: TasksAccounting=yes and TasksMax set to the value you specified. This can be a bit difficult to test as you need to create a program that generates a lot of tasks and we do not provide one.
    • If you are controlling Memory: MemoryAccounting=yes and MemoryMax set to the value you specified, but in bytes. Note that this is for real memory and can't be identified easily in testing.
    • ControlGroup should exist and be set to the user slice. If it does not you may need to validate that it's lingered correctly. This can also indicate that there are problems with the drop in files.
  • Use systemctl status user-$UID.slice where $UID is your user's UID. Look to verify that the status is loaded and that in the log under the status there aren't any errors that refer to the drop in files. Note that it may show an Active state of inactive and there may be red colored messages in the log entries below the status that are not drop in file errors and can generally be ignored.