mount.ocfs2: Invalid argument while mounting /dev/sdc1 on /data2. Check ‘dmesg’ for more information on this error.
如题所示的问题发生在RHEL5U10 + Oracle Database 10g RAC + OCFS2的环境里。
具体的报错场景如下：
|
1 2 3 |
[root@wsscm04 ~]# mount -t ocfs2 /dev/sdc1 /data2 mount.ocfs2: Invalid argument while mounting /dev/sdc1 on /data2. Check 'dmesg' for more information on this error. [root@wsscm04 ~]# |
在RAC的其他节点上挂载OCFS2分区成功，但是在其中一个节点上挂载OCFS2分区失败，报错如上所示。
该错误发生的时候，Linux后台日志的报错如下：
/var/log/message
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
Jun 25 23:22:19 wsscm04 modprobe: FATAL: Module ocfs2_stackglue not found. Jun 25 23:25:39 wsscm04 kernel: o2net: connected to node wsscm01 (num 0) at 188.0.1.71:7777 Jun 25 23:25:39 wsscm04 kernel: o2net: connected to node wsscm02 (num 1) at 188.0.1.72:7777 Jun 25 23:25:39 wsscm04 kernel: o2net: connected to node wsscm03 (num 2) at 188.0.1.73:7777 Jun 25 23:25:41 wsscm04 kernel: o2net: accepted connection from node wsscm05 (num 4) at 188.0.1.75:7777 Jun 25 23:25:43 wsscm04 kernel: OCFS2 1.4.10 Tue Oct 1 04:50:12 PDT 2013 (build b703e5e0906b370c876b657dabe8d4c8) Jun 25 23:25:43 wsscm04 kernel: o2dlm: Nodes in domain B2B93781687E4E0DB6CF58DCF4F14BFA: 0 1 2 3 4 Jun 25 23:25:43 wsscm04 kernel: (mount.ocfs2,31659,2):ocfs2_find_slot:243 ERROR: no free slots available! Jun 25 23:25:43 wsscm04 kernel: (mount.ocfs2,31659,2):ocfs2_mount_volume:1461 ERROR: status = -22 Jun 25 23:25:47 wsscm04 kernel: ocfs2: Unmounting device (8,33) on (node 3) Jun 25 23:25:49 wsscm04 kernel: o2net: no longer connected to node wsscm01 (num 0) at 188.0.1.71:7777 Jun 25 23:25:49 wsscm04 kernel: o2net: no longer connected to node wsscm02 (num 1) at 188.0.1.72:7777 Jun 25 23:25:49 wsscm04 kernel: o2net: no longer connected to node wsscm03 (num 2) at 188.0.1.73:7777 Jun 25 23:25:49 wsscm04 kernel: o2net: no longer connected to node wsscm05 (num 4) at 188.0.1.75:7777 |
通过上述日志，可以看到，挂载失败的原因是：ocfs2_find_slot:243 ERROR: no free slots available!
查看下当前要挂在的OCFS存储的Slot数，及其当前挂载情况：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
[root@wsscm02 ~]# fsck.ocfs2 -n /dev/sdc1 fsck.ocfs2 1.6.3 WARNING!!! Running fsck.ocfs2 (read-only) on a mounted filesystem may detect invalid errors. Do you really want to continue (y/N): y Checking OCFS2 filesystem in /dev/sdc1: Label: oracle UUID: B2B93781687E4E0DB6CF58DCF4F14BFA Number of blocks: 478150619 Block size: 4096 Number of clusters: 478150619 Cluster size: 4096 Number of slots: 4 ** Skipping slot recovery because -n was given. ** /dev/sdc1 is clean. It will be checked after 20 additional mounts. [root@wsscm02 ~]# [root@wsscm02 ~]# mounted.ocfs2 -f Device FS Nodes /dev/sda1 ocfs2 wsscm05, wsscm01 /dev/sdb1 ocfs2 wsscm05, wsscm01 /dev/sdc1 ocfs2 wsscm01, wsscm05, wsscm02, wsscm03 [root@wsscm02 ~]# |
可以看到，确实达到了磁盘SLOT的上限：4.
需要为OCFS2的磁盘增加SLOT数量。
1.从所有节点卸载/dev/sdc1所对应的共享存储（可能在各个节点上它对应的卷标不同）。
虽然OCFS2在各个节点上的卷标可能不同，但是它的UUID是不变的：
|
1 2 3 4 5 6 |
[root@wsscm02 ~]# mounted.ocfs2 -d Device FS Stack UUID Label /dev/sda1 ocfs2 o2cb 7794FA513A2547C7AA69C6D0055E58A7 oradatafiles /dev/sdb1 ocfs2 o2cb B68172211A124E82B3DEF12280907B38 oradatafiles /dev/sdc1 ocfs2 o2cb B2B93781687E4E0DB6CF58DCF4F14BFA oracle [root@wsscm02 ~]# |
你可以通过以上的命令，在各个节点上通过UUID，确认真正的盘符表现。
卸载：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 |
[root@wsscm01 ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/VolGroup00-LV07 39G 13G 25G 34% / /dev/mapper/VolGroup00-LV01 7.8G 147M 7.3G 2% /home /dev/mapper/VolGroup00-LV05 39G 27G 11G 72% /oracle /dev/mapper/VolGroup00-LV02 7.8G 147M 7.3G 2% /tmp /dev/mapper/VolGroup00-LV03 12G 4.0G 7.1G 36% /usr /dev/mapper/VolGroup00-LV04 7.8G 3.2G 4.3G 43% /var /dev/sda1 190M 37M 144M 21% /boot tmpfs 5.9G 0 5.9G 0% /dev/shm /dev/sdb1 2.0T 2.0T 8.4G 100% /Data /dev/sdc1 200G 23G 178G 12% /Arch /dev/sdd1 1.8T 2.9G 1.8T 1% /data2 [root@wsscm01 ~]# [root@wsscm01 ~]# mounted.ocfs2 -f Device FS Nodes /dev/sdb1 ocfs2 wsscm05, wsscm01 /dev/sdc1 ocfs2 wsscm05, wsscm01 /dev/sdd1 ocfs2 wsscm01, wsscm05, wsscm02, wsscm03 [root@wsscm01 ~]# [root@wsscm01 ~]# umount /dev/sdd1 [root@wsscm01 ~]# mounted.ocfs2 -f Device FS Nodes /dev/sdb1 ocfs2 wsscm05, wsscm01 /dev/sdc1 ocfs2 wsscm05, wsscm01 /dev/sdd1 ocfs2 wsscm05, wsscm02, wsscm03 [root@wsscm01 ~]# [root@wsscm01 ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/VolGroup00-LV07 39G 13G 25G 34% / /dev/mapper/VolGroup00-LV01 7.8G 147M 7.3G 2% /home /dev/mapper/VolGroup00-LV05 39G 27G 11G 72% /oracle /dev/mapper/VolGroup00-LV02 7.8G 147M 7.3G 2% /tmp /dev/mapper/VolGroup00-LV03 12G 4.0G 7.1G 36% /usr /dev/mapper/VolGroup00-LV04 7.8G 3.2G 4.3G 43% /var /dev/sda1 190M 37M 144M 21% /boot tmpfs 5.9G 0 5.9G 0% /dev/shm /dev/sdb1 2.0T 2.0T 8.4G 100% /Data /dev/sdc1 200G 23G 178G 12% /Arch [root@wsscm01 ~]# |
其他节点如法炮制。
全部卸载后：
|
1 2 3 4 5 6 7 8 9 10 11 |
[root@wsscm01 ~]# mounted.ocfs2 -d Device FS Stack UUID Label /dev/sdb1 ocfs2 o2cb 7794FA513A2547C7AA69C6D0055E58A7 oradatafiles /dev/sdc1 ocfs2 o2cb B68172211A124E82B3DEF12280907B38 oradatafiles /dev/sdd1 ocfs2 o2cb B2B93781687E4E0DB6CF58DCF4F14BFA oracle [root@wsscm01 ~]# mounted.ocfs2 -f Device FS Nodes /dev/sdb1 ocfs2 wsscm05, wsscm01 /dev/sdc1 ocfs2 wsscm05, wsscm01 /dev/sdd1 ocfs2 Not mounted [root@wsscm01 ~]# |
增加SLOT数：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 |
[root@wsscm01 ~]# fsck.ocfs2 -n /dev/sdd1 fsck.ocfs2 1.6.3 Checking OCFS2 filesystem in /dev/sdd1: Label: oracle UUID: B2B93781687E4E0DB6CF58DCF4F14BFA Number of blocks: 478150619 Block size: 4096 Number of clusters: 478150619 Cluster size: 4096 Number of slots: 4 ** Skipping slot recovery because -n was given. ** /dev/sdd1 is clean. It will be checked after 20 additional mounts. [root@wsscm01 ~]# [root@wsscm01 ~]# tunefs.ocfs2 -N 5 /dev/sdd1 [root@wsscm01 ~]# [root@wsscm01 ~]# fsck.ocfs2 -n /dev/sdd1 fsck.ocfs2 1.6.3 Checking OCFS2 filesystem in /dev/sdd1: Label: oracle UUID: B2B93781687E4E0DB6CF58DCF4F14BFA Number of blocks: 478150619 Block size: 4096 Number of clusters: 478150619 Cluster size: 4096 Number of slots: 5 ** Skipping slot recovery because -n was given. ** /dev/sdd1 is clean. It will be checked after 20 additional mounts. [root@wsscm01 ~]# |
再次在所有节点挂载：
|
1 2 |
[root@wsscm04 ~]# mount.ocfs2 /dev/sdc1 /data2 [root@wsscm04 ~]# |
挂载成功时，后台日志的情况：
/var/log/message
|
1 2 3 4 5 6 7 8 |
Jun 25 23:45:54 wsscm04 kernel: o2net: connected to node wsscm01 (num 0) at 188.0.1.71:7777 Jun 25 23:45:54 wsscm04 kernel: o2net: connected to node wsscm03 (num 2) at 188.0.1.73:7777 Jun 25 23:45:54 wsscm04 kernel: o2net: connected to node wsscm02 (num 1) at 188.0.1.72:7777 Jun 25 23:45:56 wsscm04 kernel: o2net: accepted connection from node wsscm05 (num 4) at 188.0.1.75:7777 Jun 25 23:45:58 wsscm04 kernel: o2dlm: Nodes in domain B2B93781687E4E0DB6CF58DCF4F14BFA: 0 1 2 3 4 Jun 25 23:45:58 wsscm04 kernel: JBD: Ignoring recovery information on journal Jun 25 23:45:58 wsscm04 kernel: kjournald starting. Commit interval 5 seconds Jun 25 23:45:58 wsscm04 kernel: ocfs2: Mounting device (8,33) on (node 3, slot 4) with ordered data mode. |
挂载成功后：
|
1 2 3 4 5 6 7 8 9 10 11 |
[root@wsscm04 ~]# mounted.ocfs2 -d Device FS Stack UUID Label /dev/sda1 ocfs2 o2cb 7794FA513A2547C7AA69C6D0055E58A7 oradatafiles /dev/sdb1 ocfs2 o2cb B68172211A124E82B3DEF12280907B38 oradatafiles /dev/sdc1 ocfs2 o2cb B2B93781687E4E0DB6CF58DCF4F14BFA oracle [root@wsscm04 ~]# mounted.ocfs2 -f Device FS Nodes /dev/sda1 ocfs2 wsscm05, wsscm01 /dev/sdb1 ocfs2 wsscm05, wsscm01 /dev/sdc1 ocfs2 wsscm01, wsscm05, wsscm02, wsscm03, wsscm04 [root@wsscm04 ~]# |
——————————————————
Done。
