Troubleshooting⚓︎

I will outline the steps I use to identify the issue and provide direct links to relevant documentation where the solution is described.

First we need to check GPFS state on the node.

#mmgetstate

The second step involves running mmhealth command to gain an overview of the issue.

Cluster Level

#mmhealth cluster show --verbose

Node Level

#mmhealth node show -N all

#mmhealth node show --verbose

Event Analysis

#mmhealth node eventlog --verbose

#mmhealth event show quorum_down

Logs⚓︎

The GPFS log can be found in the /var/adm/ras directory on each node. The GPFS log file is named mmfs.log.date.nodeName, where date is the time stamp when the instance of GPFS started on the node and nodeName is the name of the node. The latest GPFS log file can be found by using the symbolic file name /var/adm/ras/mmfs.log.latest.

#/var/adm/ras/mmfs.log.latest

#/var/adm/ras/mmsysmonitor.localhost.log

#grep "mmfs:" /var/log/messages

#/var/mmf/ccr/*

#egrep '([0-9]{1,3}\.){3}[0-9]{1,3}$' /var/adm/ras/log.wb-<domain>

Monitoring Events⚓︎

The recorded events are stored in the local database on each node. The user can get a list of recorded events by using the mmhealth node eventlog command. Users can use the mmhealth node show or mmhealth cluster show commands to display the active events in the node and cluster respectively.

when using mmhealth node eventlog you will be presented with Event Name for each event type we can get more details and solution recommendations from the RAS events list. This can be very helpful for locating the issue.

Log Dump⚓︎

In case needed for incident report or sending to IBM support for diagonics.

#dssg.snap

#gpfs.snap --gather-logs -d /tmp/logs -N all

Also you can look into the Trace facility.

Extra Commands⚓︎

CCR⚓︎

#mmccr check -e -Y

#mmccr lsnodes