The most important thing about WWN numbers are they may not be static. In general, whichever LUN first connected to the OS during the reboot, the sd* names are assigned respectively. If the order changes for any reason (this can be a confiuration change on storage or on SAN switches), then the very same WWN number can point to different sd* SCSI disk. To overcome this problem, Oracle released ASMLib for tagging the SCSI devices.
ASMLib is a package runs on Linux which assigns a label to the disk header to identify the correct disk itself, independently from the name of the SCSI device. This prevents the problems of the changes in the device names and/or in WWN numbers.
So, after installing ASMLib the disks should be defined (labeled) in order as follows.
[root@tslnx01 disks]# oracleasm create disk ASMDISK01 /dev/mapper/mpathv
[root@tslnx01 disks]# oracleasm create disk ASMDISK02 /dev/mapper/mpathu
[root@tslnx01 disks]# oracleasm create disk ASMDISK03 /dev/mapper/mpatht
[root@tslnx01 disks]# oracleasm create disk ASMDISK04 /dev/mapper/mpaths
[root@tslnx01 disks]# ls -l /dev/oracleasm/disks
brw-rw---- 1 oracle dba 252, 5 Aug 14 16:55 ASMDISK01
brw-rw---- 1 oracle dba 252, 4 Aug 14 16:56 ASMDISK02
brw-rw---- 1 oracle dba 252, 6 Aug 14 16:56 ASMDISK03
brw-rw---- 1 oracle dba 252, 7 Aug 14 16:56 ASMDISK04After configuring ASM disks, configured disks can be found under the /dev/oracleasm/disks directory. Consider the device numbers of the defined disks in this directory. For instance ASMDISK01 is using the 252,5 device number. To identify which multipath devices are connected to the ASMDisks /proc/partitions can be examined as follows.
[root@tslnx01 disks]# cat /proc/partitions
major minor #blocks name
7 0 3763200 loop0
8 0 285671424 sda
8 1 65536000 sda1
8 2 220134400 sda2
8 208 10485760 sdn
8 64 10485760 sde
8 144 10485760 sdj
8 32 10485760 sdc
8 96 10485760 sdg
8 80 10485760 sdf
8 240 10485760 sdp
8 192 10485760 sdm
8 160 10485760 sdk
8 16 10485760 sdb
8 112 10485760 sdh
8 48 10485760 sdd
8 128 31457280 sdi
8 224 10485760 sdo
8 176 10485760 sdl
65 0 31457280 sdq
11 0 1048575 sr0
252 0 10485760 dm-0
252 1 10485760 dm-1
252 2 10485760 dm-2
252 3 10485760 dm-3
252 4 10485760 dm-4
252 5 10485760 dm-5
252 6 10485760 dm-6
252 7 31457280 dm-7As can be seen from the output, ASMDisks are linked to dm-4, dm-5, dm-6 and dm-7 respectively.
Another (probably one of the most important) issue about the ASMLib comes after rebooting the system. While configuring ASMLib on multipath devices, it is important to know the order of the device creation on the OS level. dm-* devices are always created after the sd* devices. This behaviour is natural as dm-* devices are dependent to the sd* devices.
During the reboot ASMLib devices are linked to the first available devices which are sd* devices. This not the expected configuration. Because, sd* devices are not high available but dm-* devices are the ones which are high available. Desired configuration is configuring ASMLib devices on the dm-* devices.
The issue can be investigates as follows after the system reboot;
[root@tslnx01 ~]# cd /dev/oracleasm/
[root@tslnx01 oracleasm]# ls
disks iid
[root@tslnx01 oracleasm]# cd disks
[root@tslnx01 disks]# ls -lrt
total 0
brw-rw---- 1 oracle dba 8, 112 Aug 14 17:01 ASMDISK03
brw-rw---- 1 oracle dba 8, 96 Aug 14 17:01 ASMDISK01
brw-rw---- 1 oracle dba 8, 208 Aug 14 17:01 ASMDISK02
brw-rw---- 1 oracle dba 65, 0 Aug 14 17:01 ASMDISK04
[root@tslnx01 disks]# cat /proc/partitions
major minor #blocks name
7 0 3763200 loop0
8 0 285671424 sda
8 1 65536000 sda1
8 2 220134400 sda2
8 112 10485760 sdh
8 48 10485760 sdd
8 240 10485760 sdp
8 176 10485760 sdl
8 64 10485760 sde
8 96 10485760 sdg
8 32 10485760 sdc
8 144 10485760 sdj
8 224 10485760 sdo
8 208 10485760 sdn
8 80 10485760 sdf
65 0 31457280 sdq
8 160 10485760 sdk
8 128 31457280 sdi
8 16 10485760 sdb
8 192 10485760 sdm
11 0 1048575 sr0
252 0 10485760 dm-0
252 1 10485760 dm-1
252 2 10485760 dm-2
252 3 10485760 dm-3
252 4 10485760 dm-4
252 5 10485760 dm-5
252 6 10485760 dm-6
252 7 31457280 dm-7
[root@tslnx01 disks]#
[root@tslnx01 disks]# multipath -ll
mpathv (360080e50004316c8000001d655b8432e) dm-7 NETAPP,INF-01-00
size=30G features='3 queue_if_no_path pg_init_retries 50' hwhandler='1 rdac' wp=rw
|-+- policy='round-robin 0' prio=14 status=active
| `- 1:0:0:8 sdi 8:128 active ready running
`-+- policy='round-robin 0' prio=9 status=enabled
`- 2:0:0:8 sdq 65:0 active ready running
mpathu (360080e500029b50c0000021155b84318) dm-6 NETAPP,INF-01-00
size=10G features='3 queue_if_no_path pg_init_retries 50' hwhandler='1 rdac' wp=rw
|-+- policy='round-robin 0' prio=14 status=active
| `- 2:0:0:6 sdp 8:240 active ready running
`-+- policy='round-robin 0' prio=9 status=enabled
`- 1:0:0:6 sdh 8:112 active ready running
mpatht (360080e500029b50c0000021055b842f0) dm-4 NETAPP,INF-01-00
size=10G features='3 queue_if_no_path pg_init_retries 50' hwhandler='1 rdac' wp=rw
|-+- policy='round-robin 0' prio=14 status=active
| `- 2:0:0:4 sdn 8:208 active ready running
`-+- policy='round-robin 0' prio=9 status=enabled
`- 1:0:0:4 sdf 8:80 active ready running
mpaths (360080e50004316c8000001d555b8430e) dm-5 NETAPP,INF-01-00
size=10G features='3 queue_if_no_path pg_init_retries 50' hwhandler='1 rdac' wp=rw
|-+- policy='round-robin 0' prio=14 status=active
| `- 1:0:0:5 sdg 8:96 active ready running
`-+- policy='round-robin 0' prio=9 status=enabled
`- 2:0:0:5 sdo 8:224 active ready running
[root@tslnx01 disks]#To overcome this issue, default setting of the ASMLib should be altered.
The configration should be done by updating the ORACLEASM_SCANORDER and ORACLEASM_SCANEXCLUDE parameters in the /etc/sysconfig/oracleasm configuration file.
ORACLEASM_SCANORDER defines the order of the matching multipath devices. This parameter should be altered as "mpath dm" to inform ASMLib to first match the mpath devices first and then dm devices afterwards.
[root@tslnx01 disks]# cd /etc/sysconfig
[root@tslnx01 sysconfig]# vi oracleasm
#
# This is a configuration file for automatic loading of the Oracle
# Automatic Storage Management library kernel driver. It is generated
# By running /etc/init.d/oracleasm configure. Please use that method
# to modify this file
#
# ORACLEASM_ENABLED: 'true' means to load the driver on boot.
ORACLEASM_ENABLED=true
# ORACLEASM_UID: Default user owning the /dev/oracleasm mount point.
ORACLEASM_UID=oracle
# ORACLEASM_GID: Default group owning the /dev/oracleasm mount point.
ORACLEASM_GID=dba
# ORACLEASM_SCANBOOT: 'true' means scan for ASM disks on boot.
ORACLEASM_SCANBOOT=true
# ORACLEASM_SCANORDER: Matching patterns to order disk scanning
ORACLEASM_SCANORDER="mpath dm"
# ORACLEASM_SCANEXCLUDE: Matching patterns to exclude disks from scan
ORACLEASM_SCANEXCLUDE="sd"
# ORACLEASM_USE_LOGICAL_BLOCK_SIZE: 'true' means use the logical block size
# reported by the underlying disk instead of the physical. The default
# is 'false'
ORACLEASM_USE_LOGICAL_BLOCK_SIZE=false
[root@tslnx01 sysconfig]# After the configuration update, system reboot shows us the correct multipath and high available devices are matched with ASMLib devices.
[root@tslnx01 ~]# cd /dev/oracleasm/disks/
[root@tslnx01 disks]# ls -lrt
total 0
brw-rw---- 1 oracle dba 252, 4 Aug 14 17:06 ASMDISK02
brw-rw---- 1 oracle dba 252, 5 Aug 14 17:06 ASMDISK01
brw-rw---- 1 oracle dba 252, 6 Aug 14 17:06 ASMDISK03
brw-rw---- 1 oracle dba 252, 7 Aug 14 17:06 ASMDISK04
[root@tslnx01 disks]# cat /proc/partitions
major minor #blocks name
7 0 3763200 loop0
8 0 285671424 sda
8 1 65536000 sda1
8 2 220134400 sda2
8 144 10485760 sdj
8 160 10485760 sdk
8 240 10485760 sdp
8 192 10485760 sdm
8 96 10485760 sdg
8 16 10485760 sdb
8 48 10485760 sdd
65 0 31457280 sdq
8 208 10485760 sdn
8 128 31457280 sdi
8 32 10485760 sdc
8 112 10485760 sdh
8 64 10485760 sde
8 176 10485760 sdl
8 224 10485760 sdo
8 80 10485760 sdf
11 0 1048575 sr0
252 0 10485760 dm-0
252 1 10485760 dm-1
252 2 10485760 dm-2
252 3 10485760 dm-3
252 4 10485760 dm-4
252 5 10485760 dm-5
252 6 10485760 dm-6
252 7 31457280 dm-7
[root@tslnx01 disks]# ls -l /dev/ | grep dm
crw-rw---- 1 root root 10, 62 Aug 14 17:05 cpu_dma_latency
brwxrwxrwx 1 oracle disk 252, 0 Aug 14 17:33 dm-0
brwxrwxrwx 1 oracle disk 252, 1 Aug 14 17:08 dm-1
brwxrwxrwx 1 oracle disk 252, 2 Aug 14 17:08 dm-2
brwxrwxrwx 1 oracle disk 252, 3 Aug 14 17:08 dm-3
brwxrwxrwx 1 oracle disk 252, 4 Aug 14 17:08 dm-4
brwxrwxrwx 1 oracle disk 252, 5 Aug 14 17:08 dm-5
brwxrwxrwx 1 oracle disk 252, 6 Aug 14 17:08 dm-6
brwxrwxrwx 1 oracle disk 252, 7 Aug 14 17:08 dm-7
crw-rw---- 1 root root 1, 12 Aug 14 17:05 oldmem
brw-rw---- 1 root disk 8, 192 Aug 14 17:05 sdm
[root@tslnx01 disks]# ^C
[root@tslnx01 disks]#
[root@tslnx01 disks]# multipath -ll
mpathv (360080e50004316c8000001d655b8432e) dm-7 NETAPP,INF-01-00
size=30G features='3 queue_if_no_path pg_init_retries 50' hwhandler='1 rdac' wp=rw
|-+- policy='round-robin 0' prio=14 status=active
| `- 1:0:0:8 sdi 8:128 active ready running
`-+- policy='round-robin 0' prio=9 status=enabled
`- 2:0:0:8 sdq 65:0 active ready running
mpathu (360080e500029b50c0000021155b84318) dm-6 NETAPP,INF-01-00
size=10G features='3 queue_if_no_path pg_init_retries 50' hwhandler='1 rdac' wp=rw
|-+- policy='round-robin 0' prio=14 status=active
| `- 2:0:0:6 sdp 8:240 active ready running
`-+- policy='round-robin 0' prio=9 status=enabled
`- 1:0:0:6 sdh 8:112 active ready running
mpatht (360080e500029b50c0000021055b842f0) dm-4 NETAPP,INF-01-00
size=10G features='3 queue_if_no_path pg_init_retries 50' hwhandler='1 rdac' wp=rw
|-+- policy='round-robin 0' prio=14 status=active
| `- 2:0:0:4 sdn 8:208 active ready running
`-+- policy='round-robin 0' prio=9 status=enabled
`- 1:0:0:4 sdf 8:80 active ready running
mpaths (360080e50004316c8000001d555b8430e) dm-5 NETAPP,INF-01-00
size=10G features='3 queue_if_no_path pg_init_retries 50' hwhandler='1 rdac' wp=rw
|-+- policy='round-robin 0' prio=14 status=active
| `- 1:0:0:5 sdg 8:96 active ready running
`-+- policy='round-robin 0' prio=9 status=enabled
`- 2:0:0:5 sdo 8:224 active ready running
[root@tslnx01 disks]#
Комментарии