How to Maintain Updated Rocky Linux 9 Servers

2 years ago

10 minutes

In the beginning

In this tutorial, you will learn about effective methods to maintain your Rocky Linux 9 server stack up to date. Like network security hardening, there are numerous measures you can implement to ensure your servers remain secure and functional without further intervention.

If you are managing your own Rocky Linux servers, there are several tools and configurations available that can be applied additionally to the default settings. Patching all your environments manually can be quite disruptive and prone to errors.

In this tutorial, we will discuss the following topics:

Testing graceful reboots following best practices for application management, to minimize any complications from maintenance updates
Configuring automatic updates for most of the packages and libraries running on your machine
Live kernel patching, and other best practices around kernel updates

Requirements

A Rocky Linux 9 server and a non-root user with sudo privileges. You can learn more about how to set up a user with these privileges in our Initial Server Setup with Rocky Linux 9 guide.

Firstly, adhere to the recommended methods for application management.

One important aspect of setting up your server for automated updates is to make sure that all the applications running on the server can restart properly in the event of unforeseen downtime or a reboot. Linux package managers are specifically designed to operate smoothly in the background, causing minimal disruption during necessary maintenance activities. However, despite this advantage, fear of potential server behavior following a restart remains a major obstacle for many individuals who fail to establish a robust update strategy.

As much as possible, it is recommended that your server’s init system manages the applications in your stack. On most modern Linux distributions like Rocky Linux, this init system is systemd. Systemd offers the systemctl command for conveniently managing and automatically restarting your running services when necessary. Following best practices, almost all software installed through your package manager and intended to run in the background should come with a systemd service and a configuration unit file.

It is advisable to create your own unit files for systemd when you are using your own software or software from Git repositories. Alternatively, you can utilize a tool such as supervisor as a lighter option. Another approach is to employ your system’s cron scheduler using the @reboot syntax.

Once your setup is complete, it is important to verify its functionality by testing it during system restarts. To reboot, you can execute the command “sudo shutdown now -r” which will properly terminate ongoing processes and instantly reboot the system. Alternatively, you can specify a specific time in the format hh:mm or indicate the number of minutes from now to schedule a reboot for a future time. In normal circumstances, production deployments should not demand your involvement following unforeseen disruptions as all essential services and access points should automatically be restored.

After ensuring the stability of your environment during maintenance reboots, the next step will involve acquiring the knowledge to schedule automated upgrades.

Step 2 involves the setup of automatic upgrades.

Rocky’s package manager, dnf, offers two primary methods for completing a comprehensive system upgrade. One approach is executing dnf upgrade command without indicating a specific package to upgrade, resulting in the upgrade of all packages on your system. Another option is using dnf upgrade-minimal command which upgrades every package solely to the most recent bugfix or security patch release, ensuring essential maintenance without risking any disruptive modifications from upstream sources. More information regarding dnf commands can be found in the dnf documentation.

Rocky also offers a distinctive utility named dnf-automatic, enabling the automatic retrieval and installation of crucial upgrades, including security patches, for your server. Installation can be done through dnf.

sudo dnf install dnf-automatic -y

After installation, dnf-automatic does not get enabled automatically. Instead, it offers various services that can be registered with Systemd to personalize its functioning. These services include:

dnf-automatic – this service follows the dnf-automatic configuration file options in /etc/dnf/automatic.
dnf-automatic-notifyonly – this overrides the configuration file by only notifying you of available updates and not installing them.
dnf-automatic-download – this overrides the configuration file by only downloading packages and not installing them.
dnf-automatic-install – this overrides the configuration file by always installing downloaded packages.

To begin this tutorial, you will activate the dnf-automatic-install service. However, before that, you need to modify one aspect in the dnf-automatic configuration file. Use vi or any preferred text editor to open the file.

sudo vi /etc/dnf/automatic

One possible paraphrase could be:

“The automatic configuration file for DNF can be found at the location /etc/dnf/automatic.conf.”

[commands]
#  What kind of upgrade to perform:
# default                            = all available upgrades
# security                           = only the security upgrades
upgrade_type = security
random_sleep = 0

# Maximum time in seconds to wait until the system is on-line and able to
# connect to remote repositories.
network_online_timeout = 60

# To just receive updates use dnf-automatic-notifyonly.timer

# Whether updates should be downloaded when they are available, by
# dnf-automatic.timer. notifyonly.timer, download.timer and
# install.timer override this setting.
download_updates = yes

# Whether updates should be applied when they are available, by
# dnf-automatic.timer. notifyonly.timer, download.timer and
# install.timer override this setting.
apply_updates = no
…

The majority of choices in this document correspond to the different overrides offered by the various Systemd services. However, there is one exception – the upgrade_type option, which is initially set to default. If you plan on implementing automatic upgrades (which is the most proactive setup), it is advisable to only install security upgrades as the default setting. This will help prevent any unexpected alterations in functionality. To make this adjustment, modify the upgrade_type value to security, as shown above, and then save and close the file. In case you are utilizing vi, you can save and exit by typing :x.

You can now activate the service by utilizing systemctl.

sudo systemctl enable dnf-automatic-install.timer

To verify if the dnf-automatic-install service is functioning correctly, you can utilize systemctl.

sudo systemctl status dnf-automatic-install

Output

● dnf-automatic-install.service – dnf automatic install updates Loaded: loaded (/usr/lib/systemd/system/dnf-automatic-install.service; static; vendor preset: disabled) Active: inactive (dead) Jul 14 21:01:03 droplet-name dnf-automatic[40103]: No security updates needed, but 154 updates available Jul 14 21:01:03 droplet-name systemd[1]: dnf-automatic-install.service: Succeeded. Jul 14 21:01:03 droplet-name systemd[1]: Started dnf automatic install updates.

dnf-automatic is designed as a timer instead of a service that continuously runs in the background, unlike some Systemd services. Therefore, it is normal for the service’s status to be inactive (dead) as long as the service is loaded and the logs show successful executions.

To inquire about the information of that timer, you may utilize the “systemctl cat” command.

sudo systemctl cat dnf-automatic-install.timer

Output

[Unit] Description=dnf-automatic-install timer # See comment in dnf-makecache.service ConditionPathExists=!/run/ostree-booted Wants=network-online.target [Timer] OnCalendar=*-*-* 6:00 RandomizedDelaySec=60m Persistent=true [Install] WantedBy=timers.target

The service is set to automatically run at approximately 6:00 every day as the default setting. There should be no need for you to alter this value.

By now, it is important to have implemented measures that guarantee all the packages on your server are updated with necessary security updates automatically, without any additional effort. In the final phase, you will understand how to maintain your kernel up to date and efficiently manage server reboots whenever they are required.

In the third step, we will proceed with the process of updating and livepatching the kernel.

Updating your system’s kernel is less frequent compared to other packages. The Linux kernel encompasses almost all hardware drivers in operation and handles most low-level system interactions. Kernel updates are typically required when addressing significant vulnerabilities, utilizing new kernel features that have been publicized, or when the kernel becomes outdated and accumulates various bugs and vulnerabilities that might go unnoticed.

A single option for native paraphrasing of the given statement could be:
“Automatically scheduling Linux kernel updates is not universally possible due to the historical need for a complete system reboot. Scheduling reboots becomes unfeasible without making assumptions about the specific environment. Given that numerous servers are expected to maintain maximum availability, reboots can consume uncertain time or demand manual intervention.”

Many production deployments necessitate increased complexity when it comes to rebooting. This is done to ensure that the services remain available. As an illustration, you could employ a load balancer to automatically redirect traffic to alternative servers in a horizontally scaled deployment. This way, each server can be rebooted one after another, ensuring identical functionality and preventing any noticeable downtime.

To ensure uninterrupted server operations during kernel updates, Live Patching can be enabled.

To prevent any interruptions while upgrading the kernel, you have the option to utilize the live patching feature of the Linux kernel. This feature enables you to implement kernel updates without the need to reboot your system. In the Rocky Linux ecosystem, there are two primary maintainers for kernel live patches: Red Hat’s kpatch, which offers live patching for Red Hat Enterprise Linux, and KernelCare, which supports Rocky Linux along with other popular Linux distributions. To access these features, registration is required for both options.

Installing the kpatch-dnf package is the first step to enabling kernel live patching on Rocky Linux, even though it operates similarly to an unlicensed version of Red Hat. Bear in mind that without running a fully licensed Red Hat instance, you won’t be able to receive live kernel patches via kpatch, as per the licensing model of Red Hat Enterprise Linux. Nevertheless, the process to enable it remains the same for both Rocky Linux and RHEL.

sudo dnf install kpatch-dnf

Afterwards, execute the command “dnf kpatch auto” to conveniently subscribe to live patching services automatically.

sudo dnf kpatch auto

Output

Last metadata expiration check: 0:00:06 ago on Thu 14 Jul 2022 09:12:07 PM UTC. Dependencies resolved. Nothing to do. Complete!

If no configured patching services are found, kpatch will gracefully exit. However, on Red Hat, it will conduct a license check and register with the Red Hat kpatch servers. On Rocky, it would be advisable to explore KernelCare as a commercial support alternative.

In conclusion,

During this tutorial, you have discovered various approaches to automatically update your Rocky Linux servers. Additionally, you gained insights into package repositories, kernel updates, and managing server reboots. These concepts are vital in the field of DevOps and server administration, forming the foundation for most production setups.

Afterwards, it might be beneficial for you to acquire the skill of utilizing Watchtower, which can automatically refresh Docker container images.