Hi all,
I'm preparing for EX436 exam on RH labs and testing many scenarios. One of these scenarios is prepare cluster with assigned two IP addresses to each node and then use iscsi target with dlm, multipath and LVM's.
Everything goes somoothly till... I starting with LVM configuration. I getting error "Global lock failed: check that global lockspace is started".
Below steps which I using for reproduce that error:
I could reproduce the reported issue, as follows:
[root@nodea ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 10G 0 disk
└─diska 253:0 0 10G 0 mpath
sdb 8:16 0 10G 0 disk
└─diska 253:0 0 10G 0 mpath
vda 252:0 0 10G 0 disk
├─vda1 252:1 0 1M 0 part
├─vda2 252:2 0 100M 0 part /boot/efi
└─vda3 252:3 0 9.9G 0 part /
[root@nodea ~]# pcs resource create dlm ocf:pacemaker:controld op monitor interval=30s on-fail=fence --group=locking
[root@nodea ~]# pcs resource create lvmlockd ocf:heartbeat:lvmlockd op monitor interval=30s on-fail=fence --group=locking
[root@nodea ~]# pcs resource clone locking interleave=true
[root@nodea ~]# pvs
Skipping global lock: lockspace not found or started
[root@nodea ~]# pvs
Global lock failed: error -210
[root@nodea ~]# pvs
Skipping global lock: lockspace not found or started
[root@nodea ~]# pvcreate /dev/mapper/diska
Global lock failed: check that global lockspace is started
As you can see, I'm running my cluster with two links for each node. Apparently, that's the issue, according to the following KB:
https://access.redhat.com/solutions/5099971
So, to solve it, remove the extra ring, like the following example:
[root@nodea ~]# pcs cluster corosync | grep nodelist -A21
nodelist {
node {
ring0_addr: 192.168.0.10
ring1_addr: 192.168.2.10
name: nodea.private.example.com
nodeid: 1
}
node {
ring0_addr: 192.168.0.11
ring1_addr: 192.168.2.11
name: nodeb.private.example.com
nodeid: 2
}
node {
ring0_addr: 192.168.0.12
ring1_addr: 192.168.2.12
name: nodec.private.example.com
nodeid: 3
}
}
[root@nodea ~]# pcs cluster link remove 1
Sending updated corosync.conf to nodes...
nodea.private.example.com: Succeeded
nodeb.private.example.com: Succeeded
nodec.private.example.com: Succeeded
nodea.private.example.com: Corosync configuration reloaded
[root@nodea ~]# pcs cluster corosync | grep nodelist -A18
nodelist {
node {
ring0_addr: 192.168.0.10
name: nodea.private.example.com
nodeid: 1
}
node {
ring0_addr: 192.168.0.11
name: nodeb.private.example.com
nodeid: 2
}
node {
ring0_addr: 192.168.0.12
name: nodec.private.example.com
nodeid: 3
}
}
[root@nodea ~]# pcs cluster stop --all
nodea.private.example.com: Stopping Cluster (pacemaker)...
nodec.private.example.com: Stopping Cluster (pacemaker)...
nodeb.private.example.com: Stopping Cluster (pacemaker)...
nodea.private.example.com: Stopping Cluster (corosync)...
nodeb.private.example.com: Stopping Cluster (corosync)...
nodec.private.example.com: Stopping Cluster (corosync)...
[root@nodea ~]# reboot
[root@nodeb ~]# reboot
[root@nodec ~]# reboot
[root@nodea ~]# pcs status --full
Cluster name: cluster1
Cluster Summary:
* Stack: corosync
* Current DC: nodec.private.example.com (3) (version 2.0.4-6.el8-2deceaa3ae) - partition with quorum
* Last updated: Wed May 29 21:38:34 2024
* Last change: Wed May 29 21:31:46 2024 by root via crm_resource on nodea.private.example.com
* 3 nodes configured
* 9 resource instances configured
Node List:
* Online: [ nodea.private.example.com (1) nodeb.private.example.com (2) nodec.private.example.com (3) ]
Full List of Resources:
* fence_nodea (stonith:fence_ipmilan): Started nodea.private.example.com
* fence_nodeb (stonith:fence_ipmilan): Started nodeb.private.example.com
* fence_nodec (stonith:fence_ipmilan): Started nodec.private.example.com
* Clone Set: locking-clone [locking]:
* Resource Group: locking:0:
* dlm (ocf::pacemaker:controld): Started nodec.private.example.com
* lvmlockd (ocf::heartbeat:lvmlockd): Started nodec.private.example.com
* Resource Group: locking:1:
* dlm (ocf::pacemaker:controld): Started nodea.private.example.com
* lvmlockd (ocf::heartbeat:lvmlockd): Started nodea.private.example.com
* Resource Group: locking:2:
* dlm (ocf::pacemaker:controld): Started nodeb.private.example.com
* lvmlockd (ocf::heartbeat:lvmlockd): Started nodeb.private.example.com
Migration Summary:
Tickets:
PCSD Status:
nodea.private.example.com: Online
nodeb.private.example.com: Online
nodec.private.example.com: Online
Daemon Status:
corosync: active/enabled
pacemaker: active/enabled
pcsd: active/enabled
[root@nodea ~]# pvs
[root@nodea ~]# pvcreate /dev/mapper/diska
Physical volume "/dev/mapper/diska" successfully created.
[root@nodea ~]# vgcreate --shared vg1 /dev/mapper/diska
Volume group "vg1" successfully created
VG vg1 starting dlm lockspace
Starting locking. Waiting until locks are ready...
[root@nodea ~]# ssh nodeb vgchange --lock-start vg1
VG vg1 starting dlm lockspace
Starting locking. Waiting until locks are ready...
[root@nodea ~]# ssh nodec vgchange --lock-start vg1
VG vg1 starting dlm lockspace
Starting locking. Waiting until locks are ready...
[root@nodea ~]# lvcreate --activate sy -L4G -n lv1 vg1
Logical volume "lv1" created.
[root@nodea ~]# pvs
PV VG Fmt Attr PSize PFree
/dev/mapper/diska vg1 lvm2 a-- <10.00g <6.00g
[root@nodea ~]# vgs
VG #PV #LV #SN Attr VSize VFree
vg1 1 1 0 wz--ns <10.00g <6.00g
[root@nodea ~]# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
lv1 vg1 -wi-a----- 4.00g
I can see this KB article which explains that DLM does not support configuring more than one ring. Please refer this KB article for details :
https://access.redhat.com/solutions/5099971
Here is another policy document with more details : https://access.redhat.com/login?redirectTo=https%3A%2F%2Faccess.redhat.com%2Fsolutions%2F5099971
Yes. Without any problem.
Hello @Bartas1979 !
Thanks for reaching out !
Could you please quickly let me know the guided exercise or the chapter -section which you are practising for this issue in RH436 course ?
Hello,
Well... there is no section exactly like this. I'm basing it on "compreview-gfs2" with a few small changes (as described above).
Ok @Bartas1979 ! What about use_lvmlockd = 1 in the global section of /etc/lvm/lvm.conf ? if it is currently set to 1 then change it to 0 and then recheck.
or Ensure that only one heartbeat address is configured per cluster node in /etc/corosync/corosync.conf
( check /var/log/messages for the dlm error )
Thank you for your suggestions. I tried some time ago change use_lvmlockd parameter on 0. Result is pvcreate /dev/mapper/diska command works but next step is failed again:
vgcreate --shared vg1 /dev/mapper/diska
Using a shared lock type requires lvmlockd.
About second part of your suggestion - "Ensure that only one heartbeat address is..." I guess you mean to remove one address from section:
ok @Bartas1979 , check this if this doc is of any help
Thank you @Chetan_Tiwary_ for links with documentation. Really appreciate that but... it still not cover my issue.
are you able to fix this issue ? because i am facing the same senario also and I don't understand where is the issue really, the shared volume group will require "use_lvmlockd = 1 " which will activate automatic i shouldn't touch the file /etc/lvm/lvm.conf
Red Hat
Learning Community
A collaborative learning environment, enabling open source skill development.