Home > Storage > PowerScale (Isilon) > Product Documentation > Storage (general) > Dell PowerScale: Considerations and Best Practices for Large Clusters > Group changes
Group changes may be caused by drive removals or replacements, node additions, node removals, node reboots or shutdowns, backend (internal) network events, and the transition of a node into read-only mode. For debugging purposes, group change messages can be reviewed to determine whether any devices are currently in a failure state.
When a group change occurs, a cluster-wide process writes a message describing the new group membership to /var/log/messages on every node. Similarly, if a cluster ‘splits’, the newly formed clusters behave in the same way: Each node records its group membership to /var/log/messages. When a cluster splits, it breaks into multiple clusters (multiple groups). Rarely, if ever, is this a desirable event.
The terms ‘cluster’ and ‘group’ are synonymous: A cluster is defined by its group members. Group members which lose sight of other group members no longer belong to the same group and thus no longer belong to the same cluster.
To view group changes from an individual node’s perspective, ‘grep’ for the expression ‘new group’ to extract the group change statements from that node’s ‘messages’ logfile. For example:
# grep -i 'new group' /var/log/messages | tail –n 10
May 30 08:07:50 (id1) /boot/kernel/kernel: [gmp_info.c:530](pid 1814="kt: gmpdrive-upda") new group: : { 1:0-4, down: 1:5-11, 2-3 }
May 30 08:07:50 (id1) /boot/kernel/kernel: [gmp_info.c:530](pid 1814="kt: gmpdrive-upda") new group: : { 1:0-5, down: 1:6-11, 2-3 }
May 30 08:07:50 (id1) /boot/kernel/kernel: [gmp_info.c:530](pid 1814="kt: gmpdrive-upda") new group: : { 1:0-6, down: 1:7-11, 2-3 }
May 30 08:07:50 (id1) /boot/kernel/kernel: [gmp_info.c:530](pid 1814="kt: gmpdrive-upda") new group: : { 1:0-7, down: 1:8-11, 2-3 }
May 30 08:07:50 (id1) /boot/kernel/kernel: [gmp_info.c:530](pid 1814="kt: gmpdrive-upda") new group: : { 1:0-8, down: 1:9-11, 2-3 }
May 30 08:07:50 (id1) /boot/kernel/kernel: [gmp_info.c:530](pid 1814="kt: gmpdrive-upda") new group: : { 1:0-9, down: 1:10-11, 2-3 }
May 30 08:07:50 (id1) /boot/kernel/kernel: [gmp_info.c:530](pid 1814="kt: gmpdrive-upda") new group: : { 1:0-10, down: 1:11, 2-3 }
May 30 08:07:50 (id1) /boot/kernel/kernel: [gmp_info.c:530](pid 1814="kt: gmpdrive-upda") new group: : { 1:0-11, down: 2-3 }
May 30 08:07:51 (id1) /boot/kernel/kernel: [gmp_info.c:530](pid 1814="kt: gmpmerge") new group: : { 1:0-11, 3:0-7,9-12, down: 2 }
May 30 08:07:52 (id1) /boot/kernel/kernel: [gmp_info.c:530](pid 1814="kt: gmpmerge") new group: : {
In this case, the tail -10 command has been used to limit the returned group change output to the last ten reported in the file. All of these occur within two seconds, so you would want to go further back, to before whatever incident was under investigation.