If you are a web server administrator and your users are creating CGI services that are chewing up your server’s CPU, memory, or tasks, you have a powerful tool to deal with it in OpenLiteSpeed: cgroups.
“cgroup” is short for “control group”. cgroups is a Linux kernel feature that, according to Wikipedia, “limits, accounts for, and isolates the resource usage (CPU, memory, disk I/O, network, etc.) of a collection of processes.”
The feature is part of the Linux Operating System and can be activated and utilized with version 1.7.9 (v1.7.x)/1.6.20 (v1.6.x) or higher of OpenLiteSpeed. Previous versions used an updated v1 cgroups which will no longer be supported in the future. cgroups v2 requires a kernel 5.2 or higher and some additional configuration discussed below.
cgroups v2 has a much richer set of features supported for control than cgroups v1 as discussed here.
The features that have been certified to work with cgroups are:
- CPU utilization
- real memory
- maximum tasks
There are other features of cgroups, and if you enable them, you do so at your own risk. OpenLiteSpeed will not attempt to stop you.
Things to Know About cgroups
cgroups is only available for true CGI applications at this time. Below we discuss the details for enabling it, but it works on the operating system resource definitions for the owner of the CGI script file.
Containers (like Docker containers) are operating system facilities that run within the base OS. Containers may use cgroups to avoid completely dominating the machine, and they can only do this if using cgroups. This is because cgroups is one of the few methods that can be used to address overuse of resources without killing the processes (unlike ulimit and other limiting schemes). cgroups helps you better balance the use of your system, keeping it active and functional for all of your users.
cgroups affects more than just OpenLiteSpeed, since it is a kernel feature. It also affects tasks running as specific users that start their own user sessions, like SSH or SFTP sessions. Thus, if you set a CPUQuota of 10% for a user, all accesses by the user must not exceed 10%. This includes those in OpenLiteSpeed and user login sessions. All will be shared fairly by the operating system. If you start more processes, the total percentage for the user does not change, but each process runs a bit slower so as to avoid monopolizing the CPU.
Note that this is managed by systemd, which means that all services within your operating system can be controlled similarly, but only at the service level. To control it at the user level, OpenLiteSpeed’s CGI functions utilize specific user-management code.
The reason why this method is not more widespread is that users who are logged in with
su or a screen window manager (KDE, Gnome, etc.) can’t use cgroups, as their sessions are managed by the service that initialized the slice rather than the user slice. OpenLiteSpeed creates its CGI tasks (when configured to) within the user’s sessions and thus can take advantage of them.
Supported cgroup Features
The following are cgroup facilities supported by the operating system, tested, and certified to work with OpenLiteSpeed. The names specified below are the ones that you will use in
systemctl to modify them:
This is specified as a percentage of a single processor to use. By default, the CPUQuota of all tasks is infinite, and explains why a single user’s task can pretty much monopolize the CPU resources of your system. This is by far the most useful of the supported features as CPU affects not only processor I/O but also access to buffers in buffered I/O. Tools like
top help you monitor CPU utilization for a task and display the user and process that are running. Note that values above 100% are valid and indicate that multiple CPUs are to be allowed.
TasksMax refers to the number of tasks that can be started by a user. This ensures that the number of tasks accounted for the user stays below a specific limit. This takes an absolute number of tasks for the user on the system. A sample default used by the system is 12000 and can result in a very large number of tasks, each of which is managed by the OS. A reasonable value might be 2500.
This is the maximum amount of real memory allocated to a process, specified in bytes. It is not the amount of virtual memory that a process uses. Specify the limit on maximum memory usage of the executed processes. The limit specifies how much process and kernel memory can be used by tasks in this unit. If the value is suffixed with K, M, G or T, the specified memory size is parsed as Kilobytes, Megabytes, Gigabytes, or Terabytes (with the base 1024), respectively. Remember that this is not virtual memory (which is stored both on disk and in real memory), but real memory. Note that real memory affects virtual memory and that tasks that take too much of it can weigh a system down. A reasonable value might be 10M.
Using cgroups with OpenLiteSpeed
Step 1. System Configuration
To verify that you are running cgroups v2 check to see if you have the cgroups v2 controllers file:
If this file is not there then you are not running v2 cgroups.
By default only a limited number of commands can be controlled through cgroups. You can see this list:
If this list does not include the features you wish to control:
cpu for CPUQuota,
memory for MemoryMax and
tasks for MaxTasks, then you will need to add them. If these features are already included, you can skip this step and proceed to User Configuration below.
Note that all configuration will need to be done either running as root, or by using the
sudo command as a prefix for your command. The notes below will show the entries as if you were running as root. Add the
sudo prefix if you need to on your system.
Extend the set of commands by creating a drop-in file:
mkdir -p /etc/systemd/system/user@.service.d cat > /etc/systemd/system/user@.service.d/delegate.conf << EOF [Service] Delegate=cpu cpuset io memory pids EOF systemctl daemon-reload
Step 2. User Configuration
User configuration is within the operating system. What's nice about the systemd programs
systemctl is that configuration is applied immediately and preserved on reboot.
You will need to identify the users you want to limit and obtain their userids. The
/etc/passwd file contains a list of users and passwords, though there are many ways of identifying IDs for a user. Note that all configuration will need to be done either running as root, or by using the
sudo command as a prefix for your command. The notes below will show the entries as if you were running as root, add the
sudo prefix if you need to on your system.
You can only modify a service or user that is either active, logged-in or lingered. Lingering is requesting that a user slice stay active even if the user is not logged in. There is virtually no overhead in enabling linger for a user. Linger is one of the very few commands which is done with a different program, in this case
To turn on linger for user2, enter the following:
loginctl enable-linger user2
You can see if a user is lingered by entering:
loginctl user-status user2
systemctl is the primary tool for accessing services, slices and units on a machine. It is accessed from a terminal session. You can see its options by using:
You may have noted that this is the same command used to start and stop services on your system. This is not an accident; these facilities were initially intended for services and have been extended by kernel developers for users.
systemctl requires that you specify the user using the userid (uid) in the form
user-UID.slice, where UID is replaced by the userid. For example, if user2 has a uid of 1001, then you'd specify the user as
systemctl show user-1001.slice
The titles and values for the properties are mentioned above (
MemoryMax=xx). To set a property like CPUQuota to 10% for user2 (uid=1001), enter:
systemctl set-property user-1001.slice CPUQuota=10%
Properties are set in the system and take effect immediately for future processes. Current ones are not affected, as mentioned above, they are also saved and may work on reboot. If you don't want them saved, you can avoid this by adding the modifier
--runtime after systemctl and before the command. For example:
systemctl --runtime set-property user-1001.slice CPUQuota=10%
To see its current value, enter (as you did above):
systemctl show user-1001.slice
Its current value will be buried within all of the other parameters, but it will be there. Except CPUQuota, which is listed in a
systemctl show command as
CPUQuotaPerSecUSec (USec means microseconds, but it's displayed in milliseconds). 10% is displayed as 100ms.
You can see where the information you update gets stored by using systemctl with the status command and the user slice. For example:
$ systemctl status user-1001.slice
● user-1001.slice Loaded: loaded Drop-In: /etc/systemd/system/user-1001.slice.d └─50-CPUQuota.conf, 50-TasksMax.conf Active: active since Wed 2019-01-23 15:56:52 EST; 1 day 22h ago Tasks: 0 (limit: 1000)
Note that it shows its status and the names and locations of the drop-in files.
You will need to repeat the steps above for all of the users you wish to control.
Step 3. OpenLiteSpeed Setup.
In the WebAdmin Console, configure OpenLiteSpeed so that CGI apps use the user ID of the owner of the file. Navigate to Configuration > Virtual Hosts and select the virtual host you wish to manage. In the Basic tab, modify the Security entry and set the ExtApp Set UID Mode to
CGI File UID. Save the entry.
To enable cgroups at the server level, in OpenLiteSpeed WebAdmin, navigate to Server Configuration > Security and edit the CGI Settings group. The parameter cgroup controls whether cgroups can be used and if so, sets the default value:
Off(the same as
not set): Allows cgroups to be used by OpenLiteSpeed, but does not enable cgroups by default. cgroups must be explicitly enabled at the virtual host level (see below).
On: Allows cgroups to be used by OpenLiteSpeed and enables cgroups by default. cgroups can be disabled at the virtual host level.
Disabled: No matter the setting at the virtual host level, the cgroups feature is disabled.
Save the group to proceed. If you do not want to set cgroups at the virtual host level, do a graceful restart of OpenLiteSpeed to activate your configuration and proceed to Step 3.
Optional Virtual Host Configuration
If you have not disabled cgroups at the server level and want to further control them at the vhost level, navigate to Configuration > Virtual Hosts and select the virtual host you wish to manage. In the General tab, modify the General entry and set cgroups:
Off: Disables cgroups for this vhost, regardless of the server setting.
On: Enables cgroups for this vhost if not disabled at the server level.
Save the group to proceed. Do a graceful restart of OpenLiteSpeed to activate your configuration and proceed to Step 3.
Modifying the Configuration in Files
This is only for reference if you're using the admin panel.
$SERVER_ROOT/conf/httpd.config.conf there is a single parameter in the CGIRLimit section with parameters like procHardLimit:
- cgroups: The possible values of
Disabled, respectively, as described above.
You can specify specific virtual hosts to enable/disable at the virtual host level unless you have disabled it entirely. That is done in the
$SERVER_ROOT/conf/vhosts/<virtual host>/vhconf.conf file. For example, for the Example virtual host you'd edit
$SERVER_ROOT/conf/vhosts/Example/vhconf.conf and set in the main group under no specific section with parameters like docRoot and enableGzip:
- cgroups: Set to
1to override the default and turn cgroups on for the virtual host, and
0to override the default and turn cgroups off for the virtual host.
You will need to gracefully restart OpenLiteSpeed to activate the configuration.
Step 4. Validation
After configuring users and OpenLiteSpeed to use cgroups, you want to make sure that they are applied correctly. Configure a user to use a small amount of CPU (like 10%). Then create a CGI file in your virtual host directory which is owned by your configured user (with a command like
chown). Below is an example.
Warning: This is a dangerous example as it uses 100% of CPU so it should only be used on test systems.
I've named it
$SERVER_ROOT/<virtual host>/cgi-bin/pound and changed it's owner to be
date -u '+%a, %d %b %Y %H:%M:%S %Z'results=
/usr/bin/dd if=/dev/zero of=/dev/nullcat << EOF Content-type: text/plain Expires: $date $results EOF
Then change its owner:
chown user2:users pound
Note that in Ubuntu, the dd command is in
/bin/dd rather than
/usr/bin/dd (on the results line).
When you specify the site in your browser:
dd command should be owned by
user2 and it should take only 10%. You can view this with the top program and kill it there. If there are problems you should check the
$SERVER_ROOT/logs/error.log file to see if there are any cgroup messages.
Other Useful Tools
There are other useful tools which can help you understand how configuration is implemented in your system:
systemd-cgls -a: Enter this command to display a tree-like structure of the processes and under which slice and scope they are running. The
ddcommand in the
poundexample above should be running under user-1001.slice.
/proc/<process-id>/cgroupfile: In the
1:nameline you should see the slice that the process is running in.
Problems With cgroups in OpenLiteSpeed
Since enabling and disabling cgroups is so easy in OpenLiteSpeed, almost all problems will be in the
systemctl configuration. We recommend that you create a simple program like the
pound program above and that you use
top to verify the results. If they show 100% CPU after setup you should verify the following:
systemctl show user-$UID.slicewhere $UID is your user's UID. Look for the following properties:
- If you are controlling CPU:
CPUQuotaPerSecUSecset to a non-infinite value.
- If you are controlling Tasks:
TasksMaxset to the value you specified. This can be a bit difficult to test as you need to create a program that generates a lot of tasks and we do not provide one.
- If you are controlling Memory:
MemoryMaxset to the value you specified, but in bytes. Note that this is for real memory and can't be identified easily in testing.
ControlGroupshould exist and be set to the user slice. If it does not you may need to validate that it's lingered correctly. This can also indicate that there are problems with the drop in files.
- If you are controlling CPU:
systemctl status user-$UID.slicewhere $UID is your user's UID. Look to verify that the status is
loadedand that in the log under the status there aren't any errors that refer to the drop in files. Note that it may show an Active state of
inactiveand there may be red colored messages in the log entries below the status that are not drop in file errors and can generally be ignored.