Skip to content

GPFS GPL Update Runbook⚓︎

This document describes a repeatable process for rebuilding and distributing the GPFS Portability Layer (GPL) after a Linux kernel update.

Purpose⚓︎

When the kernel changes on IBM Storage Scale / GPFS nodes, the GPL module may need to be rebuilt so GPFS can start correctly on the new kernel.

This runbook covers:

  • checking whether a GPL rebuild is needed
  • rebuilding the GPL package on a designated build node
  • distributing the resulting package to other nodes
  • handling common reboot and rollout issues

Example Update Flow⚓︎

One practical workflow looks like this:

  1. A designated build node detects that a new kernel is available.
  2. The node reboots into the new kernel.
  3. A new GPL package is built on that node.
  4. The package is published to a shared location.
  5. Other nodes reboot and install the updated GPL package on startup.

Before You Start⚓︎

Confirm the following first:

  • the new kernel is supported by your Storage Scale version
  • kernel headers and development packages are available
  • you know which node will be used as the GPL build node

Useful references:

Quick Checks⚓︎

Check GPFS state⚓︎

Bash
mmgetstate

Check installed and running kernels⚓︎

On RHEL-compatible systems:

Bash
yum list kernel
uname -r

On Ubuntu:

Bash
ls /boot/vmlinuz*
uname -r

Step 1: Prepare the Build Node⚓︎

Prepare the selected build node before updating its kernel.

Optional checks:

Bash
hostnamectl
yum versionlock list

If kernel version locks are in place and must be removed:

Bash
yum versionlock clear

Step 2: Update the Kernel⚓︎

Install the new kernel on the build node:

Bash
yum update kernel

Useful checks:

Bash
yum list kernel
uname -r

If you need to confirm which older kernel was previously used, record it before rebooting.

Step 3: Reboot the Build Node⚓︎

Reboot the node so it starts with the new kernel:

Bash
reboot

After the system returns, verify the running kernel:

Bash
uname -r

Step 4: Build the New GPFS GPL Package⚓︎

Run the GPL build command on the build node:

Bash
/usr/lpp/mmfs/bin/mmbuildgpl --build-package

Typical success indicators include:

  • kernel headers are found
  • compiler toolchain is present
  • make rpm completes successfully
  • a gpfs.gplbin-...rpm package is produced

The output package is commonly written under a path similar to:

Text Only
/root/rpmbuild/RPMS/x86_64/gpfs.gplbin-<kernel-version>-<scale-version>.x86_64.rpm

If the build fails due to a Kbuild-related issue, see:

Step 5: Activate the New GPL on the Build Node⚓︎

After a successful build, start GPFS with the new GPL:

Bash
mmstartup

Then confirm cluster state:

Bash
mmgetstate

Step 6: Publish the Built Package⚓︎

One simple approach is to publish the build artifacts over NFS from the build node.

Server-side example⚓︎

Install and verify NFS services:

Bash
sudo yum install -y nfs-utils
sudo systemctl status nfs-server
sudo systemctl status rpcbind
sudo systemctl status nfs-mountd
sudo systemctl status nfs-idmapd

Allow firewall services:

Bash
sudo firewall-cmd --permanent --add-service=nfs
sudo firewall-cmd --permanent --add-service=mountd
sudo firewall-cmd --permanent --add-service=rpc-bind
sudo firewall-cmd --reload

Create the export directory:

Bash
mkdir -p /buildGPL

Example /etc/exports entry:

Text Only
/buildGPL <cluster-subnet>(ro,sync,no_root_squash)

Reload exports:

Bash
exportfs -ra
showmount -e localhost

Client-side example⚓︎

Check the export:

Bash
showmount -e <nfs-server-ip>

Mount it locally:

Bash
mkdir -p /mnt/buildGPL
mount -t nfs <nfs-server-ip>:/buildGPL /mnt/buildGPL
df -h /mnt/buildGPL

Step 7: Roll Out to Other Nodes⚓︎

If you use helper scripts from an admin or base node, keep them generic and parameterized.

A common pattern is:

  • a pre-reboot script checks whether an update is needed
  • a scheduled reboot is triggered
  • a startup script installs the latest GPL package from /mnt/buildGPL

Rocky Linux Notes⚓︎

If the target nodes use Rocky Linux or another RHEL-compatible distribution, ensure the matching kernel packages are installed:

Bash
sudo yum install kernel-devel
sudo dnf install kernel-headers-$(uname -r)

Reference:

Known Operational Issues⚓︎

Some nodes may not return after reboot⚓︎

In some environments, a node may come back from reboot in an unhealthy state. If that happens:

  1. confirm the node is not returning to service
  2. use your standard recovery procedure for the affected hardware
  3. verify it boots the intended kernel

After the rollout:

  • confirm all target nodes are on the expected kernel
  • confirm GPFS starts cleanly on each node
  • update any version locks if your policy requires them
  • archive the built GPL package with a clear version label

References⚓︎